Compare commits

..
Author SHA1 Message Date
Garry TanandClaude Fable 5 b46b349028 test: reset gateway in afterAll in cli-multimodal-integration — cross-file config leak
Review hardening on #3116: the file resets the gateway in beforeEach but
left it configured after the last test, leaking a multimodal gateway
config into whatever file runs next in its shard (same class as the
adaptive-embed-batch leak fixed on master).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 12:12:41 -07:00
73256e56a0 fix(config): route remaining hand-rolled configureGateway callsites through buildGatewayConfig
Takeover of #2430 (rebased onto master; the providers.ts hunk already
landed there). Five command callsites still rebuilt partial gateway
configs by hand, dropping file-plane API keys (config.json
openai/anthropic/zeroentropy/openrouter keys), env-provided provider
base URLs, and provider_chat_options:

- src/commands/eval-takes-quality.ts: spread `{...cfg, ...process.env}`
  at top level, so the gateway's `env` field was never populated at all —
  availability/diagnose checks dereferenced undefined and threw.
- src/commands/eval-cross-modal.ts: configureGatewayForCli hand-rolled
  both branches; now one buildGatewayConfig call.
- src/commands/init.ts: all three configureGateway sites (merged-
  precedence helper + PGLite + Postgres init) now overlay the resolved
  model fields on loadConfig() and route through the adapter.
- src/commands/migrations/in-process.ts: runMigrateOnlyCore same.

Tests: new test/eval-takes-quality-gateway.test.ts (fails with a throw
on the old code path); LLAMA_SERVER_RERANKER_BASE_URL added to the
adapter passthrough sweep; cli-multimodal-integration now imports the
real adapter instead of a hand-maintained mirror copy. KEY_FILES.md
entries updated to current state.

Co-authored-by: TheAngryPit <TheAngryPit@users.noreply.github.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 14:34:19 -07:00
50 changed files with 262 additions and 1615 deletions
+2 -2
View File
@@ -71,8 +71,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
+2 -2
View File
@@ -79,7 +79,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `docs/architecture/RETRIEVAL.md` + `docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md` — retrieval-pipeline architecture reference + the named-thing-miss incident write-up (root cause, the five-layer fix, the eval that pins it).
- `src/core/types.ts` extension + `src/core/operations.ts:search` + `src/core/import-file.ts` + `src/cli.ts` + `src/core/search/telemetry.ts` — the wiring layer for the retrieval cathedral. `SearchResult` gains `evidence`, `create_safety`, `title_match_boost`, `alias_hit` (all optional; evidence/create_safety reference the union types in `evidence.ts`). The `search` MCP op uses a cheap-hybrid path by default and accepts a per-call `mode` (conservative|balanced|tokenmax) honored ONLY for trusted/local callers (`resolvePerCallMode(ctx, ...)` — remote callers use the configured mode so a remote provider can't force tokenmax spend); every search path stamps evidence fail-soft. `importFromContent` projects frontmatter `aliases:` into `page_aliases` via `normalizeAliasList` + `engine.setPageAliases` so new + changed pages register aliases at ingest. `src/cli.ts` adds the `gbrain search diagnose` dispatch (lazy import) and reconciles the `search` CLI path with the cheap-hybrid op. `src/core/search/telemetry.ts` extends the rollup with the rank-1 base_score drift signal (sum/count + 3 coarse buckets, aggregate not per-query), surfaced via `gbrain search stats`, backed by migration v111's `search_telemetry` columns. Tests: `test/cli-search-dispatch.test.ts`, `test/search/per-call-mode.test.ts`, `test/search/telemetry-rank1.test.ts`, `test/search/title-boost-stage.test.ts`, `test/search/alias-hop.test.ts`, `test/search/evidence.test.ts`, `test/search/searchvector-maxpool.test.ts`, `test/search/pre-migration-failopen.test.ts`.
- `src/commands/eval.ts``gbrain eval` command: single-run table + A/B config comparison. Sub-subcommand dispatch on `args[0]` routes `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. `gbrain eval cross-modal` is in the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine).
- `src/commands/eval-cross-modal.ts` — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (partial cost guardrail) via the shared `resolveCycleDefault(explicit, isTty)` in `src/core/eval/cycle-default.ts`; the cost-estimate banner appends `cycleDefaultSuffix(...)` (`for 1 cycle(s) (non-interactive default; --cycles N for more)`) when the value is the silent non-TTY fallback, so the 1-vs-3 difference isn't hidden. Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` fans out cross-modal scoring across a LongMemEval-shape JSONL; mutually exclusive with `--task` (fail-fast usage error if both set); filters `kind: "by_type_summary"` rows; pre-flight cost estimate refuses if `> --max-usd` without `--yes` (default cap 5.00 USD). Semaphore-bounded fan-out via inline `runWithLimit<T>(items, limit, fn)` (exported for unit tests): max N questions in-flight × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run; the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})`; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by `test/eval-cross-modal-batch.test.ts`.
- `src/commands/eval-cross-modal.ts` — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway via `buildGatewayConfig(loadConfig() ?? {})` since the cli.ts dispatch bypasses `connectEngine()`, so file-plane keys, env base URLs, and provider_chat_options follow the same adapter path as runtime. Default cycles 3 in TTY, 1 in non-TTY (partial cost guardrail) via the shared `resolveCycleDefault(explicit, isTty)` in `src/core/eval/cycle-default.ts`; the cost-estimate banner appends `cycleDefaultSuffix(...)` (`for 1 cycle(s) (non-interactive default; --cycles N for more)`) when the value is the silent non-TTY fallback, so the 1-vs-3 difference isn't hidden. Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` fans out cross-modal scoring across a LongMemEval-shape JSONL; mutually exclusive with `--task` (fail-fast usage error if both set); filters `kind: "by_type_summary"` rows; pre-flight cost estimate refuses if `> --max-usd` without `--yes` (default cap 5.00 USD). Semaphore-bounded fan-out via inline `runWithLimit<T>(items, limit, fn)` (exported for unit tests): max N questions in-flight × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run; the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})`; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by `test/eval-cross-modal-batch.test.ts`.
- `src/core/eval/cycle-default.ts` — single source of truth for the eval cycle-count default. Exports `DEFAULT_CYCLES_TTY = 3`, `DEFAULT_CYCLES_NONTTY = 1`, `resolveCycleDefault(explicit, isTty): {cycles, usedNonTtyDefault}`, and `cycleDefaultSuffix(r)` (returns ` (non-interactive default; --cycles N for more)` only when the non-TTY default was applied, else `''`). Consumed by `eval-cross-modal.ts`, `eval-takes-quality.ts` (run + regress), and `takes-quality-eval/runner.ts` (core uses only the constant — library stays TTY-agnostic; the CLI owns the TTY=3 upgrade + banner annotation). `eval-suspected-contradictions.ts` applies the same transparency to its `$5`/`$1` budget default via a `budgetUsdExplicit` flag (the budget is overwritten in-place so explicitness can't be inferred post-hoc). Not shared with `resolveWorkersWithClamp` (different domain, no engine, no dedup). Pinned by `test/eval/cycle-default.test.ts`, `test/eval-suspected-contradictions-budget-default.test.ts`.
- `src/core/cross-modal-eval/json-repair.ts``parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes.
- `src/core/cross-modal-eval/aggregate.ts` — pure verdict logic. Pass criterion: `(successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5)`. Inconclusive when <2/3 models returned parseable scores (regression guard for the v1 `Object.values({}).every(...) === true` empty-array PASS bug).
@@ -138,7 +138,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/minions/queue.ts` extension — `MinionQueue.add()` rejects `subagent` jobs whose `data.model` resolves via `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (layers 2+3: `model-config.ts:enforceSubagentAnthropic` runtime fallback + `src/commands/doctor.ts` `subagent_provider` check). Pinned by `test/agent-cli.test.ts`.
- `src/commands/models.ts``gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain), every per-task override (11 `PER_TASK_KEYS`: `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map, and a source-of-truth column (`default` / `config: <key>` / `env: <VAR>`). `gbrain models doctor [--skip=<provider>] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}`. Wired into `cli.ts` dispatch + `CLI_ONLY` set. A zero-token `embedding_config` probe runs FIRST, before any chat/expansion probes spend money: `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. `ProbeStatus` variant `'config'` + optional `fix?: string` on `ProbeResult` surface a paste-ready `gbrain config set ...` line in human + JSON output; touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'`.
- `src/core/init-embed-check.ts` — embedding-key validation at `gbrain init`. `runInitEmbedCheck(opts)` runs a config-only `diagnoseEmbedding` (catches a missing key for ANY provider) plus a best-effort `liveTestEmbed` (1-token `gateway.embed(['probe'], {inputType:'query', abortSignal})`, 5s `AbortController` timeout, never throws — catches an invalid/expired key). Loud warning to stderr; init still exits 0 (`--no-embedding` is the deferred-setup escape; `--skip-embed-check` / `GBRAIN_INIT_SKIP_EMBED_CHECK=1` skip the check). Builds the effective env (`process.env` + file-plane `openai/anthropic/zeroentropy_api_key` from `loadConfigFileOnly()` + `opts.apiKey`) and configures the gateway via `buildGatewayConfig` before diagnose/probe, so the check sees the same keys AND provider base URLs runtime will (no false "missing key" for config.json-keyed users; the probe hits the right endpoint). Init-specific warning text names `--no-embedding` / `--skip-embed-check`, not the sync-flavored `--no-embed`. Wired into `initPGLite` + `initPostgres` in `src/commands/init.ts`, with the result added to the `--json` envelope as `embedding_check {ok, reason?, live_ok?}`. Pinned by `test/init-embed-check.test.ts` (hermetic via the gateway embed-transport seam + `withEnv`).
- `src/core/ai/build-gateway-config.ts``buildGatewayConfig(c: GBrainConfig): AIGatewayConfig`, extracted from `src/cli.ts` (which re-exports it for back-compat). Lets core modules (`init-embed-check.ts`) reuse it without importing the CLI entrypoint. Single owner of folding file-plane API keys (openai/anthropic/zeroentropy) into the gateway env and threading local-server `*_BASE_URL` env vars into base_urls. `process.env` wins EXCEPT empty-string / undefined values are dropped before the merge, so an injected empty `ANTHROPIC_API_KEY=''` (Claude Code neuters subprocess LLM calls this way) can't clobber a valid config-plane key; `'0'` / `'false'` are preserved. Pinned by `test/ai/build-gateway-config.test.ts`.
- `src/core/ai/build-gateway-config.ts``buildGatewayConfig(c: GBrainConfig): AIGatewayConfig`, extracted from `src/cli.ts` (which re-exports it for back-compat). Single owner of translating stored config into gateway config — consumed by CLI runtime, init (`gbrain init`'s three configureGateway sites), `init-embed-check.ts`, the eval commands (cross-modal, takes-quality), provider diagnostics, and the in-process migration path. Folds file-plane API keys (openai/anthropic/zeroentropy/openrouter) into the gateway env and threads local-server `*_BASE_URL` env vars into base_urls; caller-provided `provider_base_urls` config wins over env base URLs. `process.env` wins EXCEPT empty-string / undefined values are dropped before the merge, so an injected empty `ANTHROPIC_API_KEY=''` (Claude Code neuters subprocess LLM calls this way) can't clobber a valid config-plane key; `'0'` / `'false'` are preserved. Pinned by `test/ai/build-gateway-config.test.ts`.
- `src/commands/doctor.ts` extension — `subagent_provider` check (layer 3 of 3). Warns when `models.tier.subagent` is explicitly set non-Anthropic (message names the bad value + paste-ready fix `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK when subagent tier resolves to Anthropic. Tests in `test/doctor.test.ts`.
- `src/core/skill-trigger-index.ts` — Shared loader that unions per-skill SKILL.md frontmatter `triggers:` with curated RESOLVER.md / AGENTS.md rows from `skillsDir` AND the parent dir (preserves the OpenClaw workspace-root layout). UNION semantics: explicit RESOLVER.md rows ADD to frontmatter triggers (don't replace). Dedup keyed on `(skillPath, trigger.trim().toLowerCase())`. Three consumers fold through this primitive — `checkResolvable`, `runRoutingEvalCli`, `mounts-cache.composeResolvers` — so fixing frontmatter reaches all of them. Exports `loadSkillTriggerIndex(skillsDir): SkillTriggerEntry[]`, `entriesToResolverContent(entries): string` (synthesizes a markdown-table resolver string for `runRoutingEval`'s string-content API), `findPrimaryResolverPath(skillsDir): string | null`, the `FRONTMATTER_SECTION` constant, and `_resetWarnedSkillsForTests`. Skip rules: non-directory entries, `_*`/`.*` prefixes, `conventions/`+`migrations/` subdirs, skills with no `SKILL.md` (deprecated `install/` graceful-skipped), no `triggers:` array, or malformed YAML (warn-once + skip). Reuses `parseSkillFrontmatter` from `src/core/skill-frontmatter.ts` (regex-based, not full YAML). Pinned by `test/skill-trigger-index.test.ts` (18 hermetic cases). CI gate `bun run check:resolver` (= `bun src/cli.ts check-resolvable --strict --skills-dir skills/`) wired into `bun run verify`.
- `src/core/skill-catalog.ts` — host-repo skill catalog backing the MCP `list_skills` / `get_skill` ops. Lets a thin MCP client (Codex desktop, Claude Code, Claude Cowork, Perplexity) DISCOVER + FOLLOW the agent repo's fat-markdown skills over `gbrain serve` — a skill is prose, so "using" one = fetching its body then calling the gbrain MCP tools the server already exposes. Read-scope, NOT localOnly (defensible only via the full mitigation stack): (1) **publish gate**`assertPublishEnabled(ctx, publishSkills)`; remote callers require `mcp.publish_skills === true`, default-OFF so an upgrade never silently grants existing read tokens host-skill read; local callers (`ctx.remote === false`) always pass. (2) **path confinement**`assertSkillNameShape` rejects separators/`..`/null/space before any FS access; the client `name` is a manifest LOOKUP KEY (via `loadOrDeriveManifest`), never a raw path segment; `confineManifestPath` does realpath + relative-containment + `SKILL.md`-regular-file check on EVERY entry (defeats poisoned manifest.json `path`, symlink/`..` escape). (3) **frontmatter allowlist**`GetSkillResult.frontmatter` projects a safe subset; private `writes_to` + `sources` dropped. (4) **prose-only + 256KB cap** (`MAX_SKILL_MD_BYTES`, env `GBRAIN_MAX_SKILL_MD_BYTES`), size-checked twice (statSync + UTF-8 byte length). (5) **no install_path serve for remote** — remote callers use `autoDetectSkillsDir` (no install-path tier) so a hosted gbrain with no agent repo returns `storage_error`; local callers use `autoDetectSkillsDirReadOnly`. (6) MCP rate-limiter caps call rate. Config reads honor BOTH planes: `readMcpPublishSkills` / `readMcpSkillsDir` prefer the DB plane (`engine.getConfig`) over the file plane (`ctx.config.mcp`). Tool-honesty: `crossReferenceTools(declared, ctx)` splits a skill's declared `tools:` into `usable_tools` vs `unavailable_tools`; `buildSkillCatalog`'s `instructions` envelope (`SKILL_CATALOG_INSTRUCTIONS`) carries the "these are prose, follow-then-call-tools" protocol. Skills are host-filesystem repo-global — `sourceScopeOpts(ctx)` / `ctx.brainId` deliberately do NOT apply. `buildSkillCatalog` is resilient (one malformed/escaping skill is skipped, never throws). Config keys in `src/core/config.ts`: `GBrainConfig.mcp?: { publish_skills?, skills_dir? }` + `KNOWN_CONFIG_KEYS` entries `mcp.publish_skills`/`mcp.publish_skills_prompted`/`mcp.skills_dir` + `mcp.` prefix in `KNOWN_CONFIG_KEY_PREFIXES`. `src/commands/init.ts` writes `config.mcp = { publish_skills: true, ... }` for new installs (existing config wins on re-init). `src/commands/upgrade.ts:runPostUpgrade` adds a one-time consent prompt (gated by `mcp.publish_skills_prompted`; existing installs stay OFF until owner opts in). Two ops register in `src/core/operations.ts` (`list_skills` with optional `section` filter + `cliHints:{name:'skills'}`; `get_skill` taking `name` + `cliHints:{name:'skill', positional:['name']}`) and dynamically import this module to avoid the import cycle (skill-catalog statically imports the `operations` array). Descriptions in `src/core/operations-descriptions.ts` (`LIST_SKILLS_DESCRIPTION`, `GET_SKILL_DESCRIPTION`, `SKILL_CATALOG_INSTRUCTIONS`, `SKILL_CLIENT_GUIDANCE`), pinned by `test/operations-descriptions.test.ts`. CLI: `gbrain skills` / `gbrain skill <name>`. Pinned by `test/skill-catalog.test.ts`, `test/skill-catalog-security.test.ts` (path-confinement / poisoned-manifest / symlink-escape), `test/skill-catalog-transports.test.ts` (publish-gate + remote-vs-local) over `test/fixtures/skill-catalog/`.
+1 -3
View File
@@ -131,9 +131,7 @@ into gbrain so other clients can scaffold it. Default behavior:
`~/.gbrain/harvest-private-patterns.txt` plus built-in defaults
(canonical private fork name, common email regex, Slack channel pattern). Any
match → rollback (delete the harvested files) and exit non-zero.
- `openclaw.plugin.json` updated with the new slug, sorted. Harvest must preserve
the top-level OpenClaw-native plugin fields (`id`, `configSchema`, `contracts`)
because OpenClaw validates those before it can install the package.
- `openclaw.plugin.json` updated with the new slug, sorted.
- `--no-lint` bypasses the linter (after a manual editorial scrub).
Use the `skillpack-harvest` skill (its companion editorial workflow)
@@ -233,14 +233,13 @@ keep it or `git checkout` to throw it away. Nothing is committed for you.
**For a skill that ships with gbrain** (anything under the gbrain repo's own
`skills/`): SkillOpt refuses to overwrite it by default and writes the winner to
`skills/<name>/skillopt/proposed.md` instead (while keeping `best.md` as the
optimizer's current-best pointer), so an optimization pass can never silently
mutate a skill other people depend on. Two ways to handle that:
`skills/<name>/skillopt/best.md` instead, so an optimization pass can never
silently mutate a skill other people depend on. Two ways to handle that:
```bash
# See the proposed improvement without touching SKILL.md (works for ANY skill):
gbrain skillopt meeting-prep --split 1:1:1 --no-mutate
# → writes skills/meeting-prep/skillopt/proposed.md, updates best.md, and prints the proposal path.
# → writes skills/meeting-prep/skillopt/best.md (the proposed rewrite), prints its path. Copy what you want.
# Actually rewrite a bundled skill (explicit opt-in + an independent held-out set):
gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled \
+2 -2
View File
@@ -1565,8 +1565,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
-1
View File
@@ -1,5 +1,4 @@
{
"id": "gbrain-context-engine",
"name": "gbrain",
"version": "0.32.3.0",
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
+2 -2
View File
@@ -23,7 +23,7 @@ health_checks:
label: "Auth provider"
checks:
- type: http
url: "$CLAWVISOR_URL/ready"
url: "$CLAWVISOR_URL/health"
label: "ClawVisor"
- type: env_exists
name: GOOGLE_CLIENT_ID
@@ -135,7 +135,7 @@ Tell the user:
Validate:
```bash
curl -sf "$CLAWVISOR_URL/ready" && echo "PASS: ClawVisor reachable" || echo "FAIL"
curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL"
```
**STOP until ClawVisor validates.**
-308
View File
@@ -1,308 +0,0 @@
---
id: contacts-to-brain
name: Contacts-to-Brain
version: 0.1.0
description: Google Contacts become canonical people/ pages, enriching brain entities with ground-truth name/email/phone/org data.
category: sense
requires: [credential-gateway]
secrets:
- name: CLAWVISOR_URL
description: ClawVisor gateway URL (Option A — recommended, handles OAuth for you)
where: https://clawvisor.com — create an agent, activate Google Contacts service
- name: CLAWVISOR_AGENT_TOKEN
description: ClawVisor agent token (Option A)
where: https://clawvisor.com — agent settings, copy the agent token
- name: GOOGLE_CLIENT_ID
description: Google OAuth2 client ID (Option B — direct API access, you manage tokens)
where: https://console.cloud.google.com/apis/credentials — create OAuth 2.0 Client ID
- name: GOOGLE_CLIENT_SECRET
description: Google OAuth2 client secret (Option B)
where: https://console.cloud.google.com/apis/credentials — same page as client ID
health_checks:
- type: any_of
label: "Auth provider"
checks:
- type: http
url: "$CLAWVISOR_URL/ready"
label: "ClawVisor"
- type: env_exists
name: GOOGLE_CLIENT_ID
label: "Google OAuth"
setup_time: 15 min
cost_estimate: "$0 (both options are free)"
---
# Contacts-to-Brain: Your Address Book Becomes Ground Truth
Calendar attendees and email senders are the bulk of `people/<slug>` brain pages.
Your Google Contacts is the ground-truth directory — canonical name, email, phone,
organization — for those same entities. Syncing it closes the "who is this person"
loop automatically and feeds enrichment.
## IMPORTANT: Instructions for the Agent
**You are the installer.** Follow these steps precisely.
**Why this matters:** email-to-brain and calendar-to-brain create people pages
from whatever name string the API returned — sometimes an email prefix, sometimes
a nickname. Contacts carries the authoritative record. After this recipe runs,
"j.smith@acme-example.com" and "Jon S" resolve to the same person page with the
right display name, phone, and company.
**The output is staging files, not direct writes:** the deterministic collector
dumps contact records to `brain/contacts/.staging/`; YOU (the agent) merge them
into `people/<slug>` pages using judgment — the Notability Gate in
`skills/_brain-filing-rules.md` applies. Not every contact deserves a page.
**Do not skip steps. Verify after each step.**
## Architecture
```
Google Contacts (People API, paginated)
↓ (ClawVisor credential gateway: list_contacts / get_contact / search_contacts)
Contacts Sync Script (deterministic Node.js)
↓ Outputs:
├── brain/contacts/.raw/contacts-{date}.json (raw API responses, provenance)
└── brain/contacts/.staging/{slug}.md (one markdown record per contact)
Agent reads staging files
↓ Judgment calls (Notability Gate):
├── Merge into existing people/<slug> pages (name/email/phone/org enrichment)
├── Create new pages ONLY for notable contacts not already in brain
└── Skip the rest (staging is not the brain)
```
## Opinionated Defaults
**Staging record format** (one file per contact, deterministic):
```markdown
# Alice Example
- **Emails:** alice@acme-example.com, alice@gmail.com
- **Phone:** +1 555 0100
- **Organization:** Acme Example — VP Engineering
- **Source:** Google Contacts (resourceName people/c123, synced 2026-07-21)
```
**Enrichment, not duplication:** if `people/alice-example.md` already exists,
append missing fields to it with a `[Source: Google Contacts]` citation. Do NOT
create a second page. Slug-match by normalized name, then by email against
existing page content.
**Notability Gate (from `skills/_brain-filing-rules.md`):** a contact with no
brain presence gets a new page only if they appear elsewhere in the brain
(calendar attendee, email correspondent) or the user confirms they matter.
When in doubt, DON'T create — a junk page degrades search quality.
## Prerequisites
1. **GBrain installed and configured** (`gbrain doctor` passes)
2. **Node.js 18+** (for the sync script)
3. **Google Contacts access** via ONE of:
- **Option A: ClawVisor** (recommended, handles OAuth for you, no token management)
- **Option B: Google OAuth2 directly** (you manage tokens, no extra service needed)
## Setup Flow
### Step 1: Choose and Configure Contacts Access
Ask the user: "How do you want to connect to Google Contacts?
**Option A: ClawVisor (recommended)**
ClawVisor handles OAuth, token refresh, and encryption. If you already use
ClawVisor for email-to-brain or calendar-to-brain, this uses the same setup —
just activate the Google Contacts service on your existing agent.
**Option B: Google OAuth2 directly**
Connect to the Google People API directly. No extra service needed, but you
manage OAuth tokens yourself."
#### Option A: ClawVisor Setup
Tell the user:
"I need your ClawVisor URL and agent token.
1. Go to https://clawvisor.com
2. Create an agent (or use existing)
3. Activate the **Google Contacts** service
4. Create a standing task with purpose: 'Full contacts access for people
enrichment: list contacts, read contact details, search contacts across
all connected Google accounts.'
IMPORTANT: Be EXPANSIVE in the task purpose. Narrow purposes block requests.
5. Copy the gateway URL and agent token"
Validate:
```bash
curl -sf "$CLAWVISOR_URL/ready" && echo "PASS: ClawVisor reachable" || echo "FAIL"
```
**STOP until ClawVisor validates.**
#### Option B: Google OAuth2 Setup
Same flow as `recipes/credential-gateway.md` Option B, with the contacts scope:
1. https://console.cloud.google.com/apis/credentials — create an OAuth client ID
(Desktop app), consent screen scope: `https://www.googleapis.com/auth/contacts.readonly`
2. Enable the People API: https://console.cloud.google.com/apis/library/people.googleapis.com
3. Run the OAuth flow; store tokens in `~/.gbrain/google-tokens.json` (auto-refresh on expiry)
Validate:
```bash
[ -n "$GOOGLE_CLIENT_ID" ] && [ -n "$GOOGLE_CLIENT_SECRET" ] \
&& echo "PASS: Google OAuth credentials set" \
|| echo "FAIL: Missing GOOGLE_CLIENT_ID or GOOGLE_CLIENT_SECRET"
```
**STOP until OAuth flow completes and tokens are stored.**
### Step 2: Set Up the Contacts Sync Script
```bash
mkdir -p contacts-sync
cd contacts-sync
npm init -y
```
The sync script needs these capabilities:
1. **Paginated retrieval**`list_contacts` (People API `people.connections.list`)
returns pages of up to 100; follow `nextPageToken` until exhausted. Request
fields: names, emailAddresses, phoneNumbers, organizations, metadata.
2. **Deterministic staging output** — one markdown file per contact at
`brain/contacts/.staging/{slug}.md`, slug from normalized display name
(fall back to email prefix). Same contact = same file on every run (idempotent).
3. **Raw JSON preservation** — save raw API responses to
`brain/contacts/.raw/contacts-{date}.json` for provenance.
4. **Skip empty records** — contacts with no name AND no email are noise; drop them.
### Step 3: Run the Full Sync
```bash
node contacts-sync.mjs
```
Verify:
```bash
ls brain/contacts/.staging/ | head -10
```
Should show one file per contact, e.g. `alice-example.md`, `charlie-example.md`.
### Step 4: Enrich People Pages (Agent Judgment)
This is YOUR job (the agent). For each staging record:
1. **Check brain**: `gbrain search "contact name"` — do they have a
`people/<slug>` page? Also search by email address.
2. **Existing page** → merge the ground-truth fields (canonical name, emails,
phone, organization) into the page, each with a
`[Source: Google Contacts]` citation. Fix a wrong/partial display name.
3. **No page** → apply the Notability Gate: create a page only if the contact
already appears in the brain (calendar, email) or is clearly relevant.
Otherwise skip.
4. **Back-link** per the Iron Law in `skills/_brain-filing-rules.md`: an
organization with a brain page gets a link from the person's page and back.
After enrichment, import and embed:
```bash
gbrain sync --no-pull --no-embed && gbrain embed --stale
```
Verify:
```bash
gbrain search "alice-example" --limit 3
```
Should return the enriched people page with contact details.
### Step 5: Set Up Weekly Sync
Contacts change slowly; once a week is plenty:
```bash
# Cron: every Sunday at 9 AM
0 9 * * 0 cd /path/to/contacts-sync && node contacts-sync.mjs
```
After each sync, re-run the Step 4 enrichment pass over CHANGED staging files
only (compare mtime or diff against git), then:
```bash
gbrain sync --no-pull --no-embed && gbrain embed --stale
```
### Step 6: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/contacts-to-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.1.0","status":"ok","details":{"contacts":"CONTACT_COUNT"}}' >> ~/.gbrain/integrations/contacts-to-brain/heartbeat.jsonl
```
Tell the user: "Contacts-to-brain is set up. [N] contacts staged; [M] people
pages enriched with ground-truth contact data. Weekly sync keeps it current."
## Implementation Guide
### Pagination
```
list_all_contacts():
contacts = []
token = null
do:
page = list_contacts({ pageSize: 100, pageToken: token,
personFields: 'names,emailAddresses,phoneNumbers,organizations,metadata' })
contacts += page.connections
token = page.nextPageToken
while token
return contacts
```
### Slug Normalization
```
slugify(contact):
name = contact.names?[0]?.displayName
if not name:
name = contact.emailAddresses?[0]?.value.split('@')[0]
return name.toLowerCase().normalize('NFD')
.replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '')
```
Collision (two contacts, same slug): suffix with the email domain
(`alice-example-acme-example`) rather than overwrite.
### What the Agent Should Test After Setup
1. **Idempotency:** run the sync twice. `git status` on `brain/contacts/.staging/`
shows no changes on the second run.
2. **Pagination:** with 250+ contacts, verify the staging count matches the
Google Contacts count (not capped at 100).
3. **Notability Gate:** verify a one-off contact with no brain presence did NOT
get a `people/` page.
4. **Enrichment merge:** verify an existing people page gained contact fields
without losing its prior content, each with a `[Source: Google Contacts]`
citation.
## Cost Estimate
| Component | Monthly Cost |
|-----------|-------------|
| ClawVisor (free tier) | $0 |
| Google People API | $0 (within free quota) |
| **Total** | **$0** |
## Troubleshooting
**No contacts returned:**
- Check ClawVisor has the Google Contacts service activated
- Check the standing task purpose is expansive enough
- Option B: verify the People API is enabled and the token carries
`contacts.readonly`
**Duplicate people pages after enrichment:**
- The agent matched by name but the brain page slug differs — search by email
address too before creating, then merge and delete the duplicate
**Contacts with no name:**
- The sync script falls back to the email prefix; records with neither name
nor email are dropped as noise
+4 -4
View File
@@ -23,7 +23,7 @@ health_checks:
label: "Auth provider"
checks:
- type: http
url: "$CLAWVISOR_URL/ready"
url: "$CLAWVISOR_URL/health"
label: "ClawVisor"
- type: env_exists
name: GOOGLE_CLIENT_ID
@@ -75,7 +75,7 @@ Tell the user:
3. Activate the services you need:
- **Gmail** (for email-to-brain)
- **Google Calendar** (for calendar-to-brain)
- **Google Contacts** (for contacts-to-brain)
- **Google Contacts** (for enrichment)
4. Create a standing task with a broad purpose. CRITICAL: be EXPANSIVE.
Good purpose: 'Full executive assistant access to Gmail, Calendar, and
@@ -88,7 +88,7 @@ Tell the user:
Validate:
```bash
curl -sf "$CLAWVISOR_URL/ready" \
curl -sf "$CLAWVISOR_URL/health" \
&& echo "PASS: ClawVisor reachable" \
|| echo "FAIL: ClawVisor not reachable — check the URL"
```
@@ -171,7 +171,7 @@ can now access your Google services."
## How to Verify
1. **ClawVisor:** `curl $CLAWVISOR_URL/ready` returns OK.
1. **ClawVisor:** `curl $CLAWVISOR_URL/health` returns OK.
2. **Google OAuth:** Tokens exist at `~/.gbrain/google-tokens.json`.
3. **Gmail access:** Run the email collector — it should pull recent messages.
4. **Calendar access:** Run the calendar sync — it should pull today's events.
+3 -3
View File
@@ -23,7 +23,7 @@ health_checks:
label: "Auth provider"
checks:
- type: http
url: "$CLAWVISOR_URL/ready"
url: "$CLAWVISOR_URL/health"
label: "ClawVisor"
- type: env_exists
name: GOOGLE_CLIENT_ID
@@ -130,7 +130,7 @@ Tell the user:
Validate:
```bash
curl -sf "$CLAWVISOR_URL/ready" && echo "PASS: ClawVisor reachable" || echo "FAIL"
curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL"
```
**STOP until ClawVisor validates.**
@@ -328,7 +328,7 @@ threads you already replied to. Sent mail acts as a negative filter.
## Troubleshooting
**No emails collected:**
- Check ClawVisor health: `curl $CLAWVISOR_URL/ready`
- Check ClawVisor health: `curl $CLAWVISOR_URL/health`
- Check standing task is active and has Gmail service enabled
- Check task purpose is expansive enough (narrow purposes block requests)
+1 -2
View File
@@ -266,5 +266,4 @@ editorial pass.
(e.g. `src/commands/<slug>.ts` if the host SKILL.md declares it
in frontmatter)
- gbrain's `openclaw.plugin.json` — adds the slug to `skills:`
array, sorted alphabetically, without removing OpenClaw-native plugin fields
like `id`, `configSchema`, or `contracts`
array, sorted alphabetically
+1 -3
View File
@@ -57,8 +57,6 @@ This mode guarantees:
- `skills/manifest.json` lists every skill directory
- `skills/RESOLVER.md` references every skill in the manifest
- `openclaw.plugin.json` `skills[]` round-trips with both
- `openclaw.plugin.json` keeps OpenClaw install-required native plugin fields
(`id`, object `configSchema`, and `contracts.contextEngines` when applicable)
- No MECE violations (duplicate triggers across skills)
### Phases
@@ -74,7 +72,7 @@ This mode guarantees:
### Automation
```bash
bun test test/skills-conformance.test.ts test/resolver.test.ts test/openclaw-plugin-manifest.test.ts
bun test test/skills-conformance.test.ts test/resolver.test.ts
```
The CI-gated check is the package.json `test` script.
+1 -8
View File
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -78,8 +78,6 @@ const CLI_ONLY_SELF_HELP = new Set([
'capture',
// v0.42 self-upgrade ships its own usage (flags + the agent-skill story).
'self-upgrade',
// maintain (#3015) prints its own usage block (modes + not-auto-applied list).
'maintain',
// v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol).
'watch',
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
@@ -1759,11 +1757,6 @@ async function handleCliOnly(command: string, args: string[]) {
await runOrphans(engine, args);
break;
}
case 'maintain': {
const { runMaintain } = await import('./commands/maintain.ts');
await runMaintain(engine, args);
break;
}
// v0.32.7 CJK wave — post-upgrade markdown re-chunk sweep.
// v0.36 Phase 3 wave — `gbrain reindex --multimodal` re-embeds content_chunks
// into the unified Voyage multimodal-3 column.
+4 -34
View File
@@ -581,7 +581,7 @@ async function embedPage(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
const updated: ChunkInput[] = chunks.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
@@ -605,31 +605,6 @@ async function embedPage(
slog(`${slug}: embedded ${toEmbed.length} chunks`);
}
/**
* Carry code-chunk metadata (language, symbol_name, symbol_type, line range,
* parent scope, doc comment, qualified name) from a loaded Chunk back into a
* ChunkInput destined for upsertChunks.
*
* Issue #769: every re-embed used to strip these fields, and upsertChunks
* overwrites (does not COALESCE) the metadata columns from EXCLUDED, so
* each pass clobbered code-def's primary index to NULL. Pulling the
* preservation into one helper keeps the three re-embed call sites
* (embedPage, embedAll non-stale, embedAllStale) in lock-step.
*/
function preserveCodeMetadata(loaded: any, base: ChunkInput): ChunkInput {
return {
...base,
language: loaded.language ?? undefined,
symbol_name: loaded.symbol_name ?? undefined,
symbol_type: loaded.symbol_type ?? undefined,
start_line: loaded.start_line ?? undefined,
end_line: loaded.end_line ?? undefined,
parent_symbol_path: loaded.parent_symbol_path ?? undefined,
doc_comment: loaded.doc_comment ?? undefined,
symbol_name_qualified: loaded.symbol_name_qualified ?? undefined,
};
}
async function embedAll(
engine: BrainEngine,
staleOnly: boolean,
@@ -742,10 +717,8 @@ async function embedAll(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
// Preserve ALL chunks, only update embeddings for stale ones.
// preserveCodeMetadata threads code-chunk metadata (#769) so re-embed
// doesn't clobber language/symbol_name/symbol_type to NULL.
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
// Preserve ALL chunks, only update embeddings for stale ones
const updated: ChunkInput[] = chunks.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
@@ -1039,10 +1012,7 @@ async function embedAllStale(
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
}
// preserveCodeMetadata threads code-chunk metadata (#769) so the
// autopilot --stale path doesn't clobber language/symbol_name/etc
// to NULL on every cycle.
const merged: ChunkInput[] = existing.map(c => preserveCodeMetadata(c, {
const merged: ChunkInput[] = existing.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
+6 -26
View File
@@ -21,7 +21,8 @@ import { join } from 'path';
import { tmpdir } from 'os';
import { createHash } from 'crypto';
import { gbrainPath, loadConfig } from '../core/config.ts';
import { gbrainPath, loadConfig, type GBrainConfig } from '../core/config.ts';
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
import { configureGateway, isAvailable } from '../core/ai/gateway.ts';
import { runWithLimit } from '../core/worker-pool.ts';
import { resolveCycleDefault, cycleDefaultSuffix } from '../core/eval/cycle-default.ts';
@@ -264,32 +265,11 @@ function isTTY(): boolean {
* Returns true on success; false (and prints a hint) when no config is found.
*/
function configureGatewayForCli(): boolean {
// Route through buildGatewayConfig (the single adapter seam) so file-plane
// API keys, env base URLs, and provider_chat_options follow the same
// precedence as the runtime path. No config file is fine — env alone serves.
const config = loadConfig();
if (!config) {
// No config file is fine for the eval command — env vars alone may serve.
// We still call configureGateway so gateway recipes can read the env map.
configureGateway({
embedding_model: undefined,
embedding_dimensions: undefined,
expansion_model: undefined,
chat_model: undefined,
chat_fallback_chain: undefined,
base_urls: undefined,
provider_chat_options: undefined,
env: { ...process.env },
});
return true;
}
configureGateway({
embedding_model: config.embedding_model,
embedding_dimensions: config.embedding_dimensions,
expansion_model: config.expansion_model,
chat_model: config.chat_model,
chat_fallback_chain: config.chat_fallback_chain,
base_urls: config.provider_base_urls,
provider_chat_options: config.provider_chat_options,
env: { ...process.env },
});
configureGateway(buildGatewayConfig(config ?? ({} as GBrainConfig)));
return true;
}
+7 -2
View File
@@ -21,7 +21,8 @@
*/
import type { BrainEngine } from '../core/engine.ts';
import { configureGateway } from '../core/ai/gateway.ts';
import { loadConfig } from '../core/config.ts';
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
import { loadConfig, type GBrainConfig } from '../core/config.ts';
import { runEval, DEFAULT_MODEL_PANEL } from '../core/takes-quality-eval/runner.ts';
import { resolveCycleDefault, cycleDefaultSuffix } from '../core/eval/cycle-default.ts';
import { writeReceipt } from '../core/takes-quality-eval/receipt-write.ts';
@@ -127,8 +128,12 @@ export async function runReplayNoBrain(argv: string[]): Promise<number> {
export async function runEvalTakesQuality(engine: BrainEngine, args: string[]): Promise<void> {
// Self-configure the AI gateway (mirrors eval-cross-modal pattern). The
// gateway needs config.ai_gateway + env vars; configureGateway reads both.
// Route through buildGatewayConfig: the old `{ ...cfg, ...process.env }`
// spread never populated the gateway's `env` field (the gateway NEVER reads
// process.env at call time), so availability checks saw no keys at all and
// file-plane API keys / provider base URLs were dropped.
const cfg = loadConfig();
configureGateway({ ...cfg, ...(process.env as Record<string, string>) } as any);
configureGateway(buildGatewayConfig(cfg ?? ({} as GBrainConfig)));
const { subcmd, argv, json } = parseSubcmd(args);
+1 -1
View File
@@ -1651,7 +1651,7 @@ async function extractTimelineFromDB(
* make re-extraction idempotent). EVERY processed page is stamped, including
* zero-link pages they WERE processed.
*/
export async function extractStaleFromDB(
async function extractStaleFromDB(
engine: BrainEngine,
opts: {
dryRun: boolean;
+12 -14
View File
@@ -7,6 +7,7 @@ import { homedir } from 'os';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts';
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
import { createEngine } from '../core/engine-factory.ts';
import { discoverOAuth, mintClientCredentialsToken, smokeTestMcp } from '../core/remote-mcp-probe.ts';
import { runInitEmbedCheck } from '../core/init-embed-check.ts';
@@ -722,7 +723,8 @@ async function configureGatewayWithMergedPrecedence(
// pollutes config.json.
const envOverlay = loadConfig() ?? ({} as GBrainConfig);
const merged = {
const merged: GBrainConfig = {
...envOverlay,
embedding_model: aiOpts?.embedding_model ?? envOverlay.embedding_model ?? existingFile.embedding_model,
embedding_dimensions: aiOpts?.embedding_dimensions ?? envOverlay.embedding_dimensions ?? existingFile.embedding_dimensions,
expansion_model: aiOpts?.expansion_model ?? envOverlay.expansion_model ?? existingFile.expansion_model,
@@ -730,13 +732,9 @@ async function configureGatewayWithMergedPrecedence(
};
const { configureGateway, getEmbeddingModel, getEmbeddingDimensions, getExpansionModel, getChatModel } = await import('../core/ai/gateway.ts');
configureGateway({
embedding_model: merged.embedding_model,
embedding_dimensions: merged.embedding_dimensions,
expansion_model: merged.expansion_model,
chat_model: merged.chat_model,
env: { ...process.env },
});
// buildGatewayConfig (the single adapter seam) so file-plane API keys and
// env base URLs reach the gateway — a hand-rolled config here dropped them.
configureGateway(buildGatewayConfig(merged));
// Read back resolved values — gateway applies internal defaults for unset
// fields, so these are the values that actually shaped the schema.
@@ -831,13 +829,13 @@ async function initPGLite(opts: {
// resolveAIOptions above: CLI flags > env vars > existing file > gateway
// defaults.
const { configureGateway } = await import('../core/ai/gateway.ts');
configureGateway({
configureGateway(buildGatewayConfig({
...(loadConfig() ?? ({} as GBrainConfig)),
embedding_model: resolvedModel ?? opts.aiOpts?.embedding_model,
embedding_dimensions: resolvedDim ?? opts.aiOpts?.embedding_dimensions,
expansion_model: opts.aiOpts?.expansion_model,
chat_model: opts.aiOpts?.chat_model,
env: { ...process.env },
});
} as GBrainConfig));
if (resolvedModel) console.log(` Embedding: ${resolvedModel} (${resolvedDim}d)`);
if (opts.aiOpts?.expansion_model) console.log(` Expansion: ${opts.aiOpts.expansion_model}`);
if (opts.aiOpts?.chat_model) console.log(` Chat: ${opts.aiOpts.chat_model}`);
@@ -1046,13 +1044,13 @@ async function initPostgres(opts: {
// T6: unconditional configureGateway BEFORE initSchema.
const { configureGateway } = await import('../core/ai/gateway.ts');
configureGateway({
configureGateway(buildGatewayConfig({
...(loadConfig() ?? ({} as GBrainConfig)),
embedding_model: resolvedModel ?? opts.aiOpts?.embedding_model,
embedding_dimensions: resolvedDim ?? opts.aiOpts?.embedding_dimensions,
expansion_model: opts.aiOpts?.expansion_model,
chat_model: opts.aiOpts?.chat_model,
env: { ...process.env },
});
} as GBrainConfig));
if (resolvedModel) console.log(` Embedding: ${resolvedModel} (${resolvedDim}d)`);
if (opts.aiOpts?.expansion_model) console.log(` Expansion: ${opts.aiOpts.expansion_model}`);
if (opts.aiOpts?.chat_model) console.log(` Chat: ${opts.aiOpts.chat_model}`);
+1 -11
View File
@@ -334,10 +334,6 @@ export async function scanIntegrity(
if (!page) continue;
// Skip grandfathered pages (opted out of brain-integrity enforcement)
if ((page.frontmatter as Record<string, unknown> | undefined)?.validate === false) continue;
// Skip code pages: indexed source files aren't prose. 'tweet' in an
// identifier or comment is not a bare-tweet citation gap, and auto-repair
// would inject wikilink brackets into source code.
if (page.type === 'code') continue;
pagesScanned++;
bareHits.push(...findBareTweetHits(page.compiled_truth, slug));
externalHits.push(...findExternalLinks(page.compiled_truth, slug));
@@ -367,9 +363,6 @@ async function scanIntegrityBatch(
// YAML) diverges from the sequential path's strict === false check. Intentional
// — gbrain lint should reject stringly-typed validate at write time.
const validateCondition = sql`AND (frontmatter->>'validate' IS NULL OR frontmatter->>'validate' != 'false')`;
// Mirror of the sequential path's `page.type === 'code'` skip: code pages
// (indexed source files) are never prose-integrity candidates.
const codeCondition = sql`AND type IS DISTINCT FROM 'code'`;
// v0.32.8: scan ONE row per (source_id, slug) pair, not one per slug.
// Pre-fix used DISTINCT ON (slug) which collapsed multi-source rows into
@@ -379,7 +372,7 @@ async function scanIntegrityBatch(
const rows = await sql`
SELECT slug, compiled_truth, frontmatter
FROM pages
WHERE 1=1 ${typeCondition} ${validateCondition} ${codeCondition}
WHERE 1=1 ${typeCondition} ${validateCondition}
ORDER BY source_id, slug
LIMIT ${limit}
`;
@@ -468,9 +461,6 @@ async function cmdAuto(args: string[]): Promise<void> {
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
// Never auto-repair code pages — injecting tweet citations into
// indexed source files corrupts them. Same gate as scanIntegrity.
if (page.type === 'code') continue;
pagesProcessed++;
progress.tick(1, slug);
-224
View File
@@ -1,224 +0,0 @@
/**
* gbrain maintain conservative self-healing maintenance.
*
* This command automates the safe parts of the operator runbook:
* - stale link/timeline extraction
* - stale per-source dream cycles when doctor reports cycle_freshness
*
* It deliberately does NOT mutate source files, apply schema-pack upgrades, or
* invent semantic hub links. Those need review or a separate command with an
* auditable proposal surface.
*/
import { existsSync } from 'fs';
import type { BrainEngine } from '../core/engine.ts';
import type { BrainHealth } from '../core/types.ts';
import { buildChecks, computeDoctorReport, type DoctorReport, type Check } from './doctor.ts';
import { extractStaleFromDB } from './extract.ts';
import { runCycle, type CycleReport } from '../core/cycle.ts';
type ActionStatus = 'ok' | 'would_apply' | 'applied' | 'blocked' | 'skipped';
export interface MaintenanceAction {
name: string;
status: ActionStatus;
message: string;
details?: Record<string, unknown>;
}
export interface MaintainOptions {
json: boolean;
safe: boolean;
dryRun: boolean;
help: boolean;
}
export interface MaintainReport {
mode: 'dry-run' | 'safe';
before: {
health: BrainHealth;
doctor: DoctorReport;
};
actions: MaintenanceAction[];
after: {
health: BrainHealth;
doctor: DoctorReport;
};
}
export function parseMaintainArgs(args: string[]): MaintainOptions {
const safe = args.includes('--safe');
return {
json: args.includes('--json'),
safe,
dryRun: args.includes('--dry-run') || !safe,
help: args.includes('--help') || args.includes('-h'),
};
}
export function extractCycleFreshnessSourceIds(checks: Check[]): string[] {
const ids = new Set<string>();
for (const check of checks) {
if (check.name !== 'cycle_freshness' || check.status === 'ok') continue;
const re = /Source '([^']+)' last cycled/g;
for (const match of check.message.matchAll(re)) {
const id = match[1]?.trim();
if (id) ids.add(id);
}
}
return [...ids].sort();
}
async function buildDoctorReport(engine: BrainEngine): Promise<DoctorReport> {
const checks = await buildChecks(engine, ['--json', '--scope=brain']);
return computeDoctorReport(checks);
}
async function runStaleExtraction(
engine: BrainEngine,
beforeHealth: BrainHealth,
dryRun: boolean,
): Promise<MaintenanceAction> {
if (beforeHealth.stale_pages <= 0) {
return { name: 'extract_stale', status: 'ok', message: 'No stale pages.' };
}
if (dryRun) {
return {
name: 'extract_stale',
status: 'would_apply',
message: `Would run DB-backed stale extraction for ${beforeHealth.stale_pages} page(s).`,
details: { stale_pages: beforeHealth.stale_pages },
};
}
const result = await extractStaleFromDB(engine, {
dryRun: false,
jsonMode: false,
includeFrontmatter: false,
catchUp: false,
});
return {
name: 'extract_stale',
status: 'applied',
message: `Processed ${result.pagesProcessed} stale page(s); ${result.staleRemaining} remain.`,
details: {
links_created: result.linksCreated,
timeline_created: result.timelineCreated,
pages_processed: result.pagesProcessed,
stale_remaining: result.staleRemaining,
},
};
}
async function runCycleFreshnessMaintenance(
engine: BrainEngine,
beforeDoctor: DoctorReport,
dryRun: boolean,
): Promise<MaintenanceAction[]> {
const sourceIds = extractCycleFreshnessSourceIds(beforeDoctor.checks);
if (sourceIds.length === 0) {
return [{ name: 'cycle_freshness', status: 'ok', message: 'All sources cycled recently.' }];
}
if (dryRun) {
return sourceIds.map((sourceId) => ({
name: 'cycle_freshness',
status: 'would_apply',
message: `Would run source-scoped dream cycle for ${sourceId}.`,
details: { source_id: sourceId },
}));
}
const sources = await engine.listAllSources();
const actions: MaintenanceAction[] = [];
for (const sourceId of sourceIds) {
const source = sources.find((s) => s.id === sourceId);
const localPath = source?.local_path ?? null;
const brainDir = localPath && existsSync(localPath) ? localPath : null;
const report: CycleReport = await runCycle(engine, {
brainDir,
dryRun: false,
pull: false,
sourceId,
});
actions.push({
name: 'cycle_freshness',
status: report.status === 'failed' ? 'blocked' : 'applied',
message: `Ran source-scoped dream cycle for ${sourceId}: ${report.status}.`,
details: {
source_id: sourceId,
brain_dir: brainDir,
cycle_status: report.status,
phases: report.phases.map((p) => ({ phase: p.phase, status: p.status })),
},
});
}
return actions;
}
export async function runMaintain(engine: BrainEngine, args: string[]): Promise<MaintainReport | void> {
const opts = parseMaintainArgs(args);
if (opts.help) {
console.log(`Usage: gbrain maintain [--safe] [--dry-run] [--json]
Conservative self-healing maintenance.
Modes:
--dry-run Preview safe actions without writes. Default when --safe is absent.
--safe Apply safe actions: stale extraction and source cycle freshness.
--json Emit a structured before/action/after report.
Not auto-applied:
source-file frontmatter fixes, schema-pack upgrades, atom-pack changes,
semantic hub-link guesses, and destructive cleanup.
`);
return;
}
const beforeHealth = await engine.getHealth();
const beforeDoctor = await buildDoctorReport(engine);
const actions: MaintenanceAction[] = [];
actions.push(await runStaleExtraction(engine, beforeHealth, opts.dryRun));
actions.push(...await runCycleFreshnessMaintenance(engine, beforeDoctor, opts.dryRun));
const afterHealth = await engine.getHealth();
const afterDoctor = await buildDoctorReport(engine);
const report: MaintainReport = {
mode: opts.dryRun ? 'dry-run' : 'safe',
before: { health: beforeHealth, doctor: beforeDoctor },
actions,
after: { health: afterHealth, doctor: afterDoctor },
};
if (opts.json) {
console.log(JSON.stringify(report, null, 2));
} else {
printMaintainReport(report);
}
return report;
}
function printMaintainReport(report: MaintainReport): void {
console.log(`GBrain maintain (${report.mode})`);
console.log(
`Before: brain_score=${Math.round(report.before.health.brain_score)}/100 ` +
`stale=${report.before.health.stale_pages} islands=${report.before.health.orphan_pages} ` +
`doctor=${report.before.doctor.status}`,
);
for (const action of report.actions) {
console.log(` ${action.status}: ${action.name}${action.message}`);
}
console.log(
`After: brain_score=${Math.round(report.after.health.brain_score)}/100 ` +
`stale=${report.after.health.stale_pages} islands=${report.after.health.orphan_pages} ` +
`doctor=${report.after.doctor.status}`,
);
if (report.mode === 'dry-run') {
console.log('Run `gbrain maintain --safe` to apply safe actions.');
}
}
+4 -8
View File
@@ -66,15 +66,11 @@ export async function runMigrateOnlyCore(opts?: { timeoutMs?: number }): Promise
// configureGateway BEFORE initSchema (init.ts B.3): a schema bump on a brain
// whose file config is missing embedding fields must not fall through to
// stale hardcoded fallbacks. loadConfig already merged env; propagate it.
// stale hardcoded fallbacks. Route through buildGatewayConfig so file-plane
// API keys and provider base URLs follow the same precedence as runtime.
const { configureGateway } = await import('../../core/ai/gateway.ts');
configureGateway({
embedding_model: config.embedding_model,
embedding_dimensions: config.embedding_dimensions,
expansion_model: config.expansion_model,
chat_model: config.chat_model,
env: { ...process.env },
});
const { buildGatewayConfig } = await import('../../core/ai/build-gateway-config.ts');
configureGateway(buildGatewayConfig(config));
const timeoutMs = opts?.timeoutMs ?? MIGRATE_ONLY_TIMEOUT_MS;
const engine = await createEngine(toEngineConfig(config));
+55 -10
View File
@@ -15,11 +15,6 @@
import type { BrainEngine } from '../core/engine.ts';
import { createProgress, startHeartbeat } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import {
shouldExcludeFromOrphanReporting,
loadOrphanPolicyOverrides,
type OrphanPolicyOverrides,
} from '../core/orphan-policy.ts';
// --- Types ---
@@ -37,14 +32,65 @@ export interface OrphanResult {
excluded: number;
}
// --- Filter constants ---
/** Slug suffixes that are always auto-generated root files */
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
/** Page slugs that are pseudo-pages by convention */
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
/** Slug segment that marks raw sources */
const RAW_SEGMENT = '/raw/';
/** Slug prefixes where no inbound links is expected */
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'openclaw/config/',
];
/** First slug segments where no inbound links is expected */
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
]);
// --- Filter logic ---
/**
* Returns true if a slug should be excluded from orphan reporting by default.
* These are pages where having no inbound links is expected / not a content problem.
*/
export function shouldExclude(slug: string, overrides?: OrphanPolicyOverrides): boolean {
return shouldExcludeFromOrphanReporting(slug, overrides);
export function shouldExclude(slug: string): boolean {
// Pseudo-pages (exact match)
if (PSEUDO_SLUGS.has(slug)) return true;
// Auto-generated suffix patterns
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
// Raw source slugs
if (slug.includes(RAW_SEGMENT)) return true;
// Deny-prefix slugs
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
// First-segment exclusions
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
return false;
}
/**
@@ -110,7 +156,6 @@ export async function findOrphans(
let allOrphans: { slug: string; title: string; domain: string | null }[];
let total: number;
let excludedAll: number;
const overrides = includePseudo ? undefined : await loadOrphanPolicyOverrides(engine);
try {
allOrphans = await engine.findOrphanPages(
sourceIds ? { sourceIds } : sourceId ? { sourceId } : undefined,
@@ -139,7 +184,7 @@ export async function findOrphans(
total = liveRows.length;
excludedAll = includePseudo
? 0
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug, overrides) ? 1 : 0), 0);
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug) ? 1 : 0), 0);
} finally {
stopHb();
progress.finish();
@@ -147,7 +192,7 @@ export async function findOrphans(
const filtered = includePseudo
? allOrphans
: allOrphans.filter(row => !shouldExclude(row.slug, overrides));
: allOrphans.filter(row => !shouldExclude(row.slug));
const orphans: OrphanPage[] = filtered.map(row => ({
slug: row.slug,
+2 -2
View File
@@ -12,8 +12,8 @@ import { parseModelId } from './ai/model-resolver.ts';
* `resolveKey` closure without re-parsing recipes.
*
* Only OPENAI_API_KEY and ZEROENTROPY_API_KEY appear here because those are the
* only embedding keys `buildGatewayConfig` (src/cli.ts) folds from config into
* the gateway env. VOYAGE_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY are deliberately
* only embedding keys `buildGatewayConfig` (src/core/ai/build-gateway-config.ts)
* folds from config into the gateway env. VOYAGE_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY are deliberately
* absent: their config fields are NOT threaded to the gateway today, so the
* producer closures fall through to checking `process.env` ONLY for them. That
* matches what the gateway can actually use (the recipes read those keys from
+5 -22
View File
@@ -54,7 +54,6 @@ import {
import {
generatePerChunkSynopsis,
SYNOPSIS_PROMPT_VERSION,
SYNOPSIS_DOC_MAX_CHARS,
type GeneratePerChunkSynopsisResult,
} from './page-summary.ts';
import {
@@ -104,17 +103,8 @@ function getEmbeddingModelTag(): string {
export function computeCorpusGeneration(args: {
crMode: CRMode;
haikuModel: string;
/**
* Resolved `SYNOPSIS_DOC_MAX_CHARS` for per_chunk_synopsis runs. When
* present, folded into the hash so changes to
* `GBRAIN_SYNOPSIS_DOC_MAX_CHARS` invalidate the prior cache cleanly.
* Omit for `crMode !== 'per_chunk_synopsis'` title / none modes
* don't consult the cap and the field stays out of the hash for
* back-compat with pre-cap embeddings.
*/
synopsisDocMaxChars?: number;
}): string {
const h = createHash('sha256')
return createHash('sha256')
.update(args.crMode)
.update('|')
.update(String(SYNOPSIS_PROMPT_VERSION))
@@ -123,11 +113,9 @@ export function computeCorpusGeneration(args: {
.update('|')
.update(String(TITLE_WRAPPER_VERSION))
.update('|')
.update(getEmbeddingModelTag());
if (args.synopsisDocMaxChars !== undefined) {
h.update('|doc_cap=').update(String(args.synopsisDocMaxChars));
}
return h.digest('hex').slice(0, 16);
.update(getEmbeddingModelTag())
.digest('hex')
.slice(0, 16);
}
/**
@@ -265,11 +253,7 @@ export async function reembedPageWithContextualRetrieval(
args.pageSlug,
args.sourceId,
resolution.mode,
computeCorpusGeneration({
crMode: resolution.mode,
haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL,
synopsisDocMaxChars: resolution.mode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
}),
computeCorpusGeneration({ crMode: resolution.mode, haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL }),
);
return { kind: 'skipped', reason: 'no_chunks' };
}
@@ -298,7 +282,6 @@ export async function reembedPageWithContextualRetrieval(
const corpus_generation = computeCorpusGeneration({
crMode: attemptMode,
haikuModel,
synopsisDocMaxChars: attemptMode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
});
// ── PHASE 2: single DB transaction ───────────────────────────
-5
View File
@@ -733,11 +733,6 @@ export async function importFromContent(
: computeCorpusGeneration({
crMode: effectiveCRMode,
haikuModel: 'anthropic:claude-haiku-4-5-20251001',
// Inline import-file path never uses per_chunk_synopsis (refuses
// upstream); pass undefined so the doc-cap field stays out of
// the hash here. Per_chunk_synopsis runs through the Minion
// backfill handler which threads SYNOPSIS_DOC_MAX_CHARS through
// the service layer.
});
// Transaction wraps all DB writes. Every per-page tx call carries the
+1 -16
View File
@@ -489,22 +489,7 @@ export async function extractPageLinks(
// text inside `[[...]]` before any `|`), NOT the display alias
// (ref.name = match[2]). `[[struktura|the project]]` must resolve
// `struktura`, not "the project". The display text is for context only.
//
// The literal may be path-qualified (`[[notes/struktura]]`). The FS
// path (resolveSlugAll) strips the dirname before its basename lookup,
// but this path passed the raw literal to an index keyed by final
// segments only — so every slash-containing wikilink outside
// DIR_PATTERN silently resolved to nothing. Query by the final
// segment, then use the written path as a disambiguation filter
// (the analogue of the FS ancestor walk honoring the written path):
// a match must end with the literal, so `[[notes/struktura]]` can
// resolve to `vault/notes/struktura` but never to `wiki/struktura`.
const slashIdx = ref.slug.lastIndexOf('/');
const basename = slashIdx === -1 ? ref.slug : ref.slug.slice(slashIdx + 1);
let matches = await resolver.resolveBasenameMatches(basename);
if (slashIdx !== -1) {
matches = matches.filter(m => m === ref.slug || m.endsWith(`/${ref.slug}`));
}
const matches = await resolver.resolveBasenameMatches(ref.slug);
if (matches.length === 0) continue;
const idx = content.indexOf(ref.slug);
const context = idx >= 0 ? excerpt(content, idx, 240) : ref.name;
-116
View File
@@ -1,116 +0,0 @@
/**
* Shared orphan-reporting exclusion policy.
*
* These are pages where "no inbound links" is expected and should not count
* against health. Keep this in core so the CLI orphan report and engine health
* dashboard cannot drift.
*
* Defaults are GBrain-wide conventions only. Brain-specific exclusions
* (private folder names, one-off fixture slugs) belong in the brain's own
* config, not here:
*
* gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
* gbrain config set orphans.exclude_slugs "some-one-off-page"
*/
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
const RAW_SEGMENT = '/raw/';
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'_templates/',
'openclaw/config/',
'extracts/',
];
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
'dreaming',
'daily',
]);
const ROOT_DATE_SLUG = /^\d{4}-\d{2}-\d{2}(?:-.+)?$/;
function isAgentWorkspaceConvention(slug: string): boolean {
if (!slug.startsWith('agents/')) return false;
if (slug.includes('/memory/dreaming/')) return true;
return /^agents\/[^/]+\/(?:agents|identity|soul|tools|user|heartbeat|dreams|dormant)$/.test(slug);
}
/** Per-brain additions to the convention defaults (from config). */
export interface OrphanPolicyOverrides {
excludePrefixes?: string[];
excludeSlugs?: string[];
}
/** Config keys for per-brain orphan exclusions (comma-separated values). */
export const ORPHAN_EXCLUDE_PREFIXES_KEY = 'orphans.exclude_prefixes';
export const ORPHAN_EXCLUDE_SLUGS_KEY = 'orphans.exclude_slugs';
function parseList(value: string | null): string[] {
if (!value) return [];
return value.split(',').map(s => s.trim()).filter(Boolean);
}
/**
* Load per-brain orphan exclusions from the brain config table. Callers with
* an engine in hand (getHealth, `gbrain orphans`) pass the result as the
* second argument to shouldExcludeFromOrphanReporting.
*/
export async function loadOrphanPolicyOverrides(
engine: { getConfig(key: string): Promise<string | null> },
): Promise<OrphanPolicyOverrides> {
const [prefixes, slugs] = await Promise.all([
engine.getConfig(ORPHAN_EXCLUDE_PREFIXES_KEY),
engine.getConfig(ORPHAN_EXCLUDE_SLUGS_KEY),
]);
return { excludePrefixes: parseList(prefixes), excludeSlugs: parseList(slugs) };
}
export function shouldExcludeFromOrphanReporting(
slug: string,
overrides?: OrphanPolicyOverrides,
): boolean {
if (PSEUDO_SLUGS.has(slug)) return true;
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
if (slug.includes(RAW_SEGMENT)) return true;
if (slug.includes('/daily/')) return true;
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
if (ROOT_DATE_SLUG.test(slug)) return true;
if (slug.startsWith('_brain-')) return true;
if (isAgentWorkspaceConvention(slug)) return true;
if (overrides) {
if (overrides.excludeSlugs?.includes(slug)) return true;
for (const prefix of overrides.excludePrefixes ?? []) {
if (slug.startsWith(prefix)) return true;
}
}
return false;
}
+1 -36
View File
@@ -44,33 +44,6 @@ const HAIKU_MAX_TOKENS = 200;
/** Default model when caller doesn't override. Resolves through the gateway. */
const DEFAULT_SYNOPSIS_MODEL = 'anthropic:claude-haiku-4-5-20251001';
/**
* Hard cap on `documentText` length (chars) before send.
*
* 2026-05-25 fix wave: small local chat models (Gemma 4 E2B, Qwen3 4B) get
* dramatically slower on long contexts even with 131K-token windows declared.
* A 73K-char page synopsis on Gemma 4 E2B takes 60-120s, exceeding the
* worker's default 30s `lockDuration` and tripping `lock-lost` errors.
*
* Truncate to a budget that fits a small model's effective throughput while
* preserving enough document context for the synopsis to be useful. Truncates
* the TAIL because the head (title, frontmatter, intro) carries the
* document-level anchor the synopsis needs.
*
* Override per workload via `GBRAIN_SYNOPSIS_DOC_MAX_CHARS`. Default 32768
* (~8K tokens at 4 chars/tok) keeps small-model synopsis under ~30s.
* Anthropic Haiku is unaffected at this cap; bump higher when running
* frontier models if you want richer document anchoring.
*/
export const SYNOPSIS_DOC_MAX_CHARS = (() => {
const env = process.env.GBRAIN_SYNOPSIS_DOC_MAX_CHARS;
if (env && /^\d+$/.test(env)) {
const n = parseInt(env, 10);
if (n >= 512 && n <= 1_048_576) return n;
}
return 32768;
})();
/**
* Synopsis prompt version. Folded into corpus_generation so prompt edits
* invalidate prior embeddings via the v0.40.3.0 query_cache.page_generations
@@ -215,19 +188,11 @@ function buildUserPrompt(
documentText: string,
chunkText: string,
): string {
// Tail-truncate `documentText` to `SYNOPSIS_DOC_MAX_CHARS` so small local
// chat models don't stall on >100KB pages. Head preserved (title block,
// frontmatter, intro paragraphs carry the document-level anchor).
let trimmedDoc = documentText;
if (documentText.length > SYNOPSIS_DOC_MAX_CHARS) {
trimmedDoc = documentText.slice(0, SYNOPSIS_DOC_MAX_CHARS) +
`\n\n[... ${documentText.length - SYNOPSIS_DOC_MAX_CHARS} chars truncated for synopsis budget ...]`;
}
return [
`<page_title>${pageTitle}</page_title>`,
'',
'<full_document>',
trimmedDoc,
documentText,
'</full_document>',
'',
'<chunk>',
+23 -51
View File
@@ -57,8 +57,6 @@ import { finalizeLastSeen } from './chronicle/last-seen.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
import {
normalizeEngineColumn,
buildVectorCastFragment,
@@ -2324,10 +2322,6 @@ export class PGLiteEngine implements BrainEngine {
// v0.40.3.0 D24 NULL→non-NULL race fix mirrors postgres-engine.ts. Two writers
// racing on the same chunk previously raced last-write-wins; the fix lets the
// fresher `embedded_at` win in the text-unchanged branch.
//
// Code-chunk metadata columns follow the same chunk_text-gated CASE pattern as `embedding`
// (#769). Re-chunk trusts EXCLUDED outright; pure re-embed COALESCEs so a caller carrying
// only embedding-shaped fields doesn't clobber metadata to NULL.
await this.db.query(
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
@@ -2351,14 +2345,14 @@ export class PGLiteEngine implements BrainEngine {
THEN EXCLUDED.embedded_at
ELSE content_chunks.embedded_at
END,
language = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.language ELSE COALESCE(EXCLUDED.language, content_chunks.language) END,
symbol_name = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name ELSE COALESCE(EXCLUDED.symbol_name, content_chunks.symbol_name) END,
symbol_type = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_type ELSE COALESCE(EXCLUDED.symbol_type, content_chunks.symbol_type) END,
start_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.start_line ELSE COALESCE(EXCLUDED.start_line, content_chunks.start_line) END,
end_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.end_line ELSE COALESCE(EXCLUDED.end_line, content_chunks.end_line) END,
parent_symbol_path = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.parent_symbol_path ELSE COALESCE(EXCLUDED.parent_symbol_path, content_chunks.parent_symbol_path) END,
doc_comment = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.doc_comment ELSE COALESCE(EXCLUDED.doc_comment, content_chunks.doc_comment) END,
symbol_name_qualified = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name_qualified ELSE COALESCE(EXCLUDED.symbol_name_qualified, content_chunks.symbol_name_qualified) END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified,
modality = EXCLUDED.modality,
embedding_image = COALESCE(EXCLUDED.embedding_image, content_chunks.embedding_image)`,
params
@@ -5208,31 +5202,26 @@ export class PGLiteEngine implements BrainEngine {
const { rows: [h] } = await this.db.query(`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
),
narrative_pages AS (
-- Composition-aware score: code source files and calendar daily
-- files are orphans-by-design (no inbound wikilinks, no Timeline
-- fence). Excluding them from the orphan/link-density/timeline
-- denominators keeps a bulk code/calendar import from cratering
-- brain_score.
SELECT id FROM pages WHERE type IS NULL OR type NOT IN ('code', 'calendar-index')
)
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM narrative_pages) as narrative_page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound). The raw
-- list is filtered in TS using the shared orphan-reporting policy.
0 as orphan_pages,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound).
-- See BrainHealth.orphan_pages docstring; docs updated to match this.
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
(SELECT count(*) FROM links) as link_count,
(SELECT count(DISTINCT te.page_id) FROM timeline_entries te
WHERE te.page_id IN (SELECT id FROM narrative_pages)) as pages_with_timeline,
(SELECT count(DISTINCT page_id) FROM timeline_entries) as pages_with_timeline,
(SELECT count(*) FROM entity_pages e
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as link_coverage,
@@ -5251,34 +5240,17 @@ export class PGLiteEngine implements BrainEngine {
LIMIT 5
`);
const { rows: islandedRows } = await this.db.query(`
SELECT p.slug
FROM pages p
-- Narrative pages only (same type filter as the narrative_pages CTE):
-- code/calendar-index pages are orphans-by-design and must not count
-- against the noOrphans component (#1144).
WHERE (p.type IS NULL OR p.type NOT IN ('code', 'calendar-index'))
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
`);
const r = h as Record<string, unknown>;
const pageCount = Number(r.page_count);
// Composition-aware denominators (excludes code/calendar-index pages);
// a code-only brain has nothing narrative to penalize → full marks.
const narrativePageCount = Number(r.narrative_page_count);
const embedCoverage = Number(r.embed_coverage);
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
const orphanOverrides = await loadOrphanPolicyOverrides(this);
const orphanPages = (islandedRows as { slug: string }[])
.filter(row => !shouldExcludeFromOrphanReporting(row.slug, orphanOverrides)).length;
const orphanPages = Number(r.orphan_pages);
const deadLinks = Number(r.dead_links);
const linkCount = Number(r.link_count);
const pagesWithTimeline = Number(r.pages_with_timeline);
const linkDensity = narrativePageCount > 0 ? Math.min(linkCount / narrativePageCount, 1) : 1;
const timelineCoverageDensity = narrativePageCount > 0 ? Math.min(pagesWithTimeline / narrativePageCount, 1) : 1;
const noOrphans = narrativePageCount > 0 ? 1 - (orphanPages / narrativePageCount) : 1;
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
const timelineCoverageDensity = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
// Bug 11 — per-component points. Sum equals brainScore by construction
// so `doctor` can render a breakdown that adds up to the total.
@@ -5299,7 +5271,7 @@ export class PGLiteEngine implements BrainEngine {
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: stalePages,
stale_pages: Number(r.stale_pages),
orphan_pages: orphanPages,
missing_embeddings: Number(r.missing_embeddings),
brain_score: brainScore,
+26 -54
View File
@@ -67,8 +67,6 @@ import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
function escapeSqlStringLiteral(value: string): string {
return value.replace(/'/g, "''");
@@ -2475,13 +2473,6 @@ export class PostgresEngine implements BrainEngine {
// - new is fresher (embedded_at > existing.embedded_at) → take new
// - otherwise → keep existing (slower writer with stale embedding loses)
// Mirrored in pglite-engine.ts; pinned by test/e2e/concurrent-embed-race.test.ts.
//
// Code-chunk metadata columns (language / symbol_name / symbol_type / line range /
// parent_symbol_path / doc_comment / symbol_name_qualified) follow the SAME chunk_text-gated
// CASE pattern as `embedding` (#769). Re-chunk (chunk_text changed) trusts EXCLUDED outright;
// pure re-embed (chunk_text unchanged) COALESCEs so a caller that only carries embedding
// doesn't clobber metadata to NULL. Without this, every embed --stale pass nuked code-def's
// primary index for thousands of chunks at once.
await sql.unsafe(
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
@@ -2505,14 +2496,14 @@ export class PostgresEngine implements BrainEngine {
THEN EXCLUDED.embedded_at
ELSE content_chunks.embedded_at
END,
language = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.language ELSE COALESCE(EXCLUDED.language, content_chunks.language) END,
symbol_name = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name ELSE COALESCE(EXCLUDED.symbol_name, content_chunks.symbol_name) END,
symbol_type = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_type ELSE COALESCE(EXCLUDED.symbol_type, content_chunks.symbol_type) END,
start_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.start_line ELSE COALESCE(EXCLUDED.start_line, content_chunks.start_line) END,
end_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.end_line ELSE COALESCE(EXCLUDED.end_line, content_chunks.end_line) END,
parent_symbol_path = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.parent_symbol_path ELSE COALESCE(EXCLUDED.parent_symbol_path, content_chunks.parent_symbol_path) END,
doc_comment = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.doc_comment ELSE COALESCE(EXCLUDED.doc_comment, content_chunks.doc_comment) END,
symbol_name_qualified = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name_qualified ELSE COALESCE(EXCLUDED.symbol_name_qualified, content_chunks.symbol_name_qualified) END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified,
modality = EXCLUDED.modality,
embedding_image = COALESCE(EXCLUDED.embedding_image, content_chunks.embedding_image)`,
params as Parameters<typeof sql.unsafe>[1],
@@ -5322,35 +5313,32 @@ export class PostgresEngine implements BrainEngine {
async getHealth(): Promise<BrainHealth> {
const sql = this.sql;
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
// no outbound links). The raw islanded list is filtered through the same
// policy as `gbrain orphans` so convention pages do not count against
// dashboard health.
// no outbound links), aligning both engines with the user-facing
// definition. The type comment previously said "no inbound" but the
// SQL required both — docs now match code so users can trust the
// number. A hub page that links out to many but has no back-references
// is working as intended, not an orphan.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
),
narrative_pages AS (
-- Composition-aware score: code source files and calendar daily
-- files are orphans-by-design (no inbound wikilinks, no Timeline
-- fence). Excluding them from the orphan/link-density/timeline
-- denominators keeps a bulk code/calendar import from cratering
-- brain_score.
SELECT id FROM pages WHERE type IS NULL OR type NOT IN ('code', 'calendar-index')
)
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM narrative_pages) as narrative_page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
0 as orphan_pages,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
(SELECT count(*) FROM links) as link_count,
(SELECT count(DISTINCT te.page_id) FROM timeline_entries te
WHERE te.page_id IN (SELECT id FROM narrative_pages)) as pages_with_timeline,
(SELECT count(DISTINCT page_id) FROM timeline_entries) as pages_with_timeline,
(SELECT count(*) FROM entity_pages e
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as link_coverage,
@@ -5368,33 +5356,17 @@ export class PostgresEngine implements BrainEngine {
LIMIT 5
`;
const islandedRows = await sql<{ slug: string }[]>`
SELECT p.slug
FROM pages p
-- Narrative pages only (same type filter as the narrative_pages CTE):
-- code/calendar-index pages are orphans-by-design and must not count
-- against the noOrphans component (#1144).
WHERE (p.type IS NULL OR p.type NOT IN ('code', 'calendar-index'))
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
`;
const pageCount = Number(h.page_count);
// Composition-aware denominators (excludes code/calendar-index pages);
// a code-only brain has nothing narrative to penalize → full marks.
const narrativePageCount = Number(h.narrative_page_count);
const embedCoverage = Number(h.embed_coverage);
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
const orphanOverrides = await loadOrphanPolicyOverrides(this);
const orphanPages = islandedRows.filter(row => !shouldExcludeFromOrphanReporting(row.slug, orphanOverrides)).length;
const orphanPages = Number(h.orphan_pages);
const deadLinks = Number(h.dead_links);
const linkCount = Number(h.link_count);
const pagesWithTimeline = Number(h.pages_with_timeline);
// brain_score: 0-100 weighted average
const linkDensity = narrativePageCount > 0 ? Math.min(linkCount / narrativePageCount, 1) : 1;
const timelineCoverageWhole = narrativePageCount > 0 ? Math.min(pagesWithTimeline / narrativePageCount, 1) : 1;
const noOrphans = narrativePageCount > 0 ? 1 - (orphanPages / narrativePageCount) : 1;
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
const timelineCoverageWhole = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
// Per-component points. Sum equals brainScore by construction.
//
@@ -5414,7 +5386,7 @@ export class PostgresEngine implements BrainEngine {
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: stalePages,
stale_pages: Number(h.stale_pages),
orphan_pages: orphanPages,
missing_embeddings: Number(h.missing_embeddings),
brain_score: brainScore,
+4 -10
View File
@@ -93,13 +93,7 @@ import { resolveLrSchedule } from './lr-schedule.ts';
import { preflight, formatPreflightReport } from './preflight.ts';
import { isRejected, loadRejectedBuffer, makeRejectedEntry, saveRejectedBuffer } from './rejected-buffer.ts';
import { runReflect, runOneShotRewrite, describeJudges } from './reflect.ts';
import {
acceptCandidate,
proposedPath as proposedFilePath,
revertAllPending,
skillPath,
writeProposed,
} from './version-store.ts';
import { acceptCandidate, bestPath, revertAllPending, skillPath, writeProposed } from './version-store.ts';
import { runValidationGate, scoreSkillOnTasks } from './validate-gate.ts';
import { ROLLOUT_SUCCESS_THRESHOLD } from './types.ts';
import type { SkillOptOpts, EditOp, RunReceipt, BenchmarkTask } from './types.ts';
@@ -708,9 +702,9 @@ async function runOptimizationLoop(
// to the catch's assignment values only (it can't prove the async callback ran).
const finalOutcome = outcome as 'accepted' | 'no_improvement' | 'aborted' | 'errored';
if (!mutateDecision.mutate && finalOutcome === 'accepted') {
// writeProposed() emitted both the best pointer and the stable review
// artifact in the accept branch. SKILL.md remains untouched.
proposedPath = proposedFilePath(skillsDir, skillName);
// best.md was written by writeProposed() in the accept branch (no-mutate
// path); it doubles as proposed.md for human review. SKILL.md untouched.
proposedPath = bestPath(skillsDir, skillName);
} else if (mutateDecision.mutate) {
mutatedSkillFile = finalOutcome === 'accepted';
}
+9 -15
View File
@@ -23,7 +23,6 @@
*
* history.json
* best.md
* proposed.md
* versions/
* v0001_e1_s1.md
* v0002_e1_s2.md
@@ -53,10 +52,6 @@ export function bestPath(skillsDir: string, skillName: string): string {
return path.join(skilloptDir(skillsDir, skillName), 'best.md');
}
export function proposedPath(skillsDir: string, skillName: string): string {
return path.join(skilloptDir(skillsDir, skillName), 'proposed.md');
}
export function skillPath(skillsDir: string, skillName: string): string {
return path.join(skillsDir, skillName, 'SKILL.md');
}
@@ -176,18 +171,17 @@ export function acceptCandidate(input: AcceptInput): AcceptResult {
}
/**
* Write the candidate to both `best.md` and `proposed.md` WITHOUT touching
* SKILL.md or the history ledger. `best.md` remains the optimizer's current
* best pointer; `proposed.md` is the stable human-review artifact promised by
* `--no-mutate`. Returns the proposal path. Each write is atomic (.tmp + rename).
* Write the candidate to `best.md` (which doubles as `proposed.md`) WITHOUT
* touching SKILL.md or the history ledger. Used by the `--no-mutate` /
* bundled-without-allow paths: the optimizer found a better candidate but the
* caller opted out of in-place mutation, so we surface it for human review.
* Returns the path written. Atomic (.tmp + rename).
*/
export function writeProposed(skillsDir: string, skillName: string, candidateText: string): string {
const best = bestPath(skillsDir, skillName);
const proposed = proposedPath(skillsDir, skillName);
fs.mkdirSync(path.dirname(best), { recursive: true });
atomicWrite(best, candidateText);
atomicWrite(proposed, candidateText);
return proposed;
const p = bestPath(skillsDir, skillName);
fs.mkdirSync(path.dirname(p), { recursive: true });
atomicWrite(p, candidateText);
return p;
}
/**
+15 -16
View File
@@ -63,26 +63,25 @@ interface PluginCtx {
[key: string]: unknown;
}
export function register(api: PluginApi) {
api.registerContextEngine(ENGINE_ID, (ctx: PluginCtx) => {
const hostResolver =
typeof ctx.resolveEntities === 'function'
? ctx.resolveEntities
: typeof ctx.brainQuery === 'function'
? ctx.brainQuery
: undefined;
return createGBrainContextEngine({
workspaceDir: ctx.workspaceDir,
resolveEntities: hostResolver,
});
});
}
const entry: PluginEntry = {
id: 'gbrain-context-engine',
name: 'GBrain Context Engine',
description: 'Deterministic temporal/spatial context injection on every turn',
register,
register(api: PluginApi) {
api.registerContextEngine(ENGINE_ID, (ctx: PluginCtx) => {
const hostResolver =
typeof ctx.resolveEntities === 'function'
? ctx.resolveEntities
: typeof ctx.brainQuery === 'function'
? ctx.brainQuery
: undefined;
return createGBrainContextEngine({
workspaceDir: ctx.workspaceDir,
resolveEntities: hostResolver,
});
});
},
};
export default entry;
+1
View File
@@ -25,6 +25,7 @@ import { withEnv } from '../helpers/with-env.ts';
const PASSTHROUGHS: Array<{ envVar: string; recipeId: string }> = [
{ envVar: 'LLAMA_SERVER_BASE_URL', recipeId: 'llama-server' },
{ envVar: 'LLAMA_SERVER_RERANKER_BASE_URL', recipeId: 'llama-server-reranker' },
{ envVar: 'OLLAMA_BASE_URL', recipeId: 'ollama' },
{ envVar: 'LMSTUDIO_BASE_URL', recipeId: 'lmstudio' },
{ envVar: 'LITELLM_BASE_URL', recipeId: 'litellm' },
-34
View File
@@ -119,40 +119,6 @@ describe('Bug 11 — orphan_pages is "no inbound links"', () => {
});
});
describe('#1144 — brain_score is composition-aware', () => {
test('code and calendar-index pages do not count as orphans or dilute density metrics', async () => {
// Two linked narrative pages + a flood of orphan-by-design pages.
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: 'a', frontmatter: {} });
await engine.putPage('people/bob', { type: 'person', title: 'Bob', compiled_truth: 'b', frontmatter: {} });
const aId = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug='people/alice'`)).rows[0].id;
const bId = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug='people/bob'`)).rows[0].id;
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
[aId, bId],
);
for (let i = 0; i < 10; i++) {
await engine.putPage(`code/src/f${i}.py`, { type: 'code', title: `f${i}.py`, compiled_truth: 'def f(): pass', frontmatter: {} });
}
await engine.putPage('daily/calendar/2026-01-01', { type: 'calendar-index', title: '2026-01-01', compiled_truth: 'events', frontmatter: {} });
const h = await engine.getHealth();
// 11 unlinked code/calendar pages exist, but both narrative pages are
// linked → 0 orphans, full no-orphans marks.
expect(h.orphan_pages).toBe(0);
expect(h.no_orphans_score).toBe(15);
// Link density: 1 link / 2 narrative pages, not 1 / 13 total pages.
expect(h.link_density_score).toBe(Math.round(0.5 * 25));
});
test('a brain with ONLY non-narrative pages gets full composition marks', async () => {
await engine.putPage('code/src/only.py', { type: 'code', title: 'only.py', compiled_truth: 'x = 1', frontmatter: {} });
const h = await engine.getHealth();
expect(h.no_orphans_score).toBe(15);
expect(h.link_density_score).toBe(25);
expect(h.timeline_coverage_score).toBe(15);
});
});
describe('Bug 11 — doctor renders brain_score breakdown', () => {
test('doctor source contains brain_score breakdown rendering', async () => {
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
+2 -20
View File
@@ -11,32 +11,13 @@
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { loadConfigWithEngine, type GBrainConfig } from '../src/core/config.ts';
import { buildGatewayConfig } from '../src/core/ai/build-gateway-config.ts';
import {
configureGateway,
getEmbeddingModel,
getMultimodalModel,
resetGateway,
} from '../src/core/ai/gateway.ts';
import type { AIGatewayConfig } from '../src/core/ai/types.ts';
// Mirror the cli.ts buildGatewayConfig helper exactly. Keeping a copy here
// (instead of exporting from cli.ts) is intentional: the test asserts the
// shape of the contract, not the helper's identity. If cli.ts drifts, the
// e2e behavior these tests care about (DB-set value lands in gateway) still
// holds, but a helper-shape test would also catch the drift in PR review.
function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
return {
embedding_model: c.embedding_model,
embedding_dimensions: c.embedding_dimensions,
embedding_multimodal_model: c.embedding_multimodal_model,
expansion_model: c.expansion_model,
chat_model: c.chat_model,
chat_fallback_chain: c.chat_fallback_chain,
base_urls: c.provider_base_urls,
provider_chat_options: c.provider_chat_options,
env: { ...process.env },
};
}
let engine: PGLiteEngine;
@@ -47,6 +28,7 @@ beforeAll(async () => {
});
afterAll(async () => {
resetGateway(); // don't leak this file's gateway config into shard siblings
await engine.disconnect();
});
-45
View File
@@ -763,48 +763,3 @@ describeBoth('Engine parity — federated sourceIds[] secondary reads (#2200)',
}
});
});
// #1144 — brain_score is composition-aware: code / calendar-index pages are
// orphans-by-design and must not feed the orphan/link-density/timeline
// denominators. Both engines must agree.
async function seedComposition(eng: BrainEngine) {
await eng.putPage('people/comp-alice', { type: 'person', title: 'Alice', compiled_truth: 'a', timeline: '' });
await eng.putPage('people/comp-bob', { type: 'person', title: 'Bob', compiled_truth: 'b', timeline: '' });
await eng.addLink('people/comp-alice', 'people/comp-bob', 'knows', 'mentions', 'markdown');
for (let i = 0; i < 5; i++) {
await eng.putPage(`code/src/comp-f${i}.py`, { type: 'code', title: `f${i}.py`, compiled_truth: 'def f(): pass', timeline: '' });
}
await eng.putPage('daily/calendar/2026-01-01', { type: 'calendar-index', title: '2026-01-01', compiled_truth: 'events', timeline: '' });
}
describeBoth('Engine parity — getHealth composition-aware brain_score (#1144)', () => {
let pgEngine: BrainEngine;
let pgliteEngine: PGLiteEngine;
beforeAll(async () => {
pgEngine = await setupDB();
await seedComposition(pgEngine);
pgliteEngine = new PGLiteEngine();
await pgliteEngine.connect({});
await pgliteEngine.initSchema();
await seedComposition(pgliteEngine);
}, 90_000);
afterAll(async () => {
await pgliteEngine.disconnect();
await teardownDB();
}, 30_000);
test('orphan/link-density/timeline components identical and exclude code + calendar-index pages', async () => {
const pg = await pgEngine.getHealth();
const pglite = await pgliteEngine.getHealth();
for (const k of ['orphan_pages', 'no_orphans_score', 'link_density_score', 'timeline_coverage_score', 'brain_score'] as const) {
expect(pg[k]).toBe(pglite[k]);
}
// 6 non-narrative pages exist but both narrative pages are linked → 0 orphans.
expect(pg.orphan_pages).toBe(0);
expect(pg.no_orphans_score).toBe(15);
// 1 link / 2 narrative pages, not / 8 total pages.
expect(pg.link_density_score).toBe(Math.round(0.5 * 25));
});
});
-53
View File
@@ -172,59 +172,6 @@ describe('issue #972 — DB-source (gbrain extract links --source db)', () => {
expect(strk!.link_type).toBe('wikilink_basename');
});
test('flag ON → path-qualified wikilink outside DIR_PATTERN resolves via DB path', async () => {
// `[[notes/struktura]]` — `notes` is not in DIR_PATTERN, so the ref
// reaches the generic pass with its dirname intact. Regression: the DB
// path queried the basename index with the raw literal (which is keyed
// by final segments only), so path-qualified wikilinks outside
// DIR_PATTERN silently produced zero edges while the FS path resolved
// the identical content.
await engine.putPage('notes/struktura', {
type: 'concept' as any, title: 'Struktura Notes',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/knowledge-graph', {
type: 'concept', title: 'Knowledge Graph',
compiled_truth: 'Background in [[notes/struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/knowledge-graph');
const strk = outLinks.find(l => l.to_slug === 'notes/struktura');
expect(strk).toBeDefined();
expect(strk!.link_type).toBe('wikilink_basename');
expect(strk!.link_source).toBe('wikilink-resolved');
});
test('path-qualified wikilink never attaches to a basename-only sibling', async () => {
// Both notes/struktura and wiki/struktura exist. The author wrote
// `[[notes/struktura]]` — the written path must exclude wiki/struktura
// (a bare `[[struktura]]` would legitimately match both).
await engine.putPage('notes/struktura', {
type: 'concept' as any, title: 'Struktura Notes',
compiled_truth: '', timeline: '',
});
await engine.putPage('wiki/struktura', {
type: 'concept' as any, title: 'Struktura Wiki',
compiled_truth: '', timeline: '',
});
await engine.putPage('concepts/x', {
type: 'concept', title: 'X',
compiled_truth: 'See [[notes/struktura]].', timeline: '',
});
await engine.setConfig('link_resolution.global_basename', 'true');
await runExtract(engine, ['links', '--source', 'db']);
const outLinks = await engine.getLinks('concepts/x');
const basenameLinks = outLinks
.filter(l => l.link_type === 'wikilink_basename')
.map(l => l.to_slug);
expect(basenameLinks).toEqual(['notes/struktura']);
});
test('flag OFF → no basename edges via DB path (back-compat)', async () => {
await engine.putPage('projects/struktura', {
type: 'project', title: 'Struktura',
-29
View File
@@ -140,35 +140,6 @@ describeE2E('scanIntegrity batch parity (E2E, Postgres-only)', () => {
});
});
describe('code pages', () => {
test('type:code page is skipped on both paths (#1143)', async () => {
const engine = getEngine();
await engine.putPage('people/alice', {
type: 'person',
title: 'Alice',
compiled_truth: 'Alice tweeted about something.',
timeline: '',
frontmatter: {},
});
await engine.putPage('code/src/bot.py', {
type: 'code',
title: 'src/bot.py (python)',
compiled_truth: '# the user tweeted about this feature\ndef send_tweet():\n pass',
timeline: '',
frontmatter: { language: 'python', file: 'src/bot.py' },
});
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
expect(batchResult.pagesScanned).toBe(seqResult.pagesScanned);
expect(batchResult.pagesScanned).toBe(1);
expect(batchResult.bareHits.map(h => h.slug)).not.toContain('code/src/bot.py');
expect(seqResult.bareHits.map(h => h.slug)).not.toContain('code/src/bot.py');
});
});
describe('topPages', () => {
test('topPages ordering matches between paths', async () => {
const engine = getEngine();
+4 -4
View File
@@ -39,7 +39,6 @@ import { runSkillOpt } from '../../src/core/skillopt/orchestrator.ts';
import {
bestPath,
loadHistory,
proposedPath,
skillPath,
} from '../../src/core/skillopt/version-store.ts';
import { loadRejectedBuffer } from '../../src/core/skillopt/rejected-buffer.ts';
@@ -742,7 +741,7 @@ describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', ()
} finally { fixture.cleanup(); }
});
test('--no-mutate writes proposed.md and best.md, leaves SKILL.md untouched', async () => {
test('--no-mutate writes proposed.md (best.md), leaves SKILL.md untouched', async () => {
const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
try {
installStub({
@@ -754,9 +753,10 @@ describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', ()
const result = await runOnce(fixture, { noMutate: true });
expect(result.outcome).toBe('accepted');
expect(result.mutatedSkillFile).toBe(false);
expect(result.proposedPath).toBe(proposedPath(fixture.skillsDir, SKILL));
expect(result.proposedPath).toBeDefined();
// proposed.md (best.md) exists and carries the improvement.
expect(fs.existsSync(result.proposedPath!)).toBe(true);
expect(fs.readFileSync(result.proposedPath!, 'utf8')).toContain('## Citations');
expect(fs.readFileSync(bestPath(fixture.skillsDir, SKILL), 'utf8')).toContain('## Citations');
// SKILL.md on disk is UNCHANGED (still People-only).
const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
expect(skill).not.toContain('## Citations');
-104
View File
@@ -803,107 +803,3 @@ describe('embedAllStale --source threading (D7)', () => {
expect((firstCallOpts as { sourceId?: string }).sourceId).toBe('media-corpus');
});
});
// ────────────────────────────────────────────────────────────────
// Code metadata preservation across re-embed (regression for #769)
// ────────────────────────────────────────────────────────────────
//
// gbrain v0.30.1 and earlier silently clobbered code-chunk metadata
// (language, symbol_name, symbol_type, start_line, end_line,
// parent_symbol_path, doc_comment, symbol_name_qualified) on every
// re-embed pass. The chunker populated those columns at import time,
// but embed.ts loaded chunks via getChunks then mapped them to a
// stripped ChunkInput carrying only 5 fields. upsertChunks then
// OVERWROTE (not COALESCEd) the metadata columns from EXCLUDED, so
// re-embed wiped them to NULL. End result on a real brain: 4875 code
// pages, 47866 chunks, all with NULL language/symbol_name/symbol_type;
// code-def returned 0 hits across every indexed repo.
//
// All three runEmbed paths (--stale autopilot, --all, --slugs) must
// thread metadata through the re-upsert. Tests below assert that the
// engine.upsertChunks call carries the same metadata it loaded.
describe('runEmbed preserves code-chunk metadata across re-embed (regression for #769)', () => {
const fullCodeChunk = {
chunk_index: 0,
chunk_text: '[Java] foo/Bar.java:10-20 method baz',
chunk_source: 'compiled_truth' as const,
embedded_at: null,
token_count: 12,
language: 'java',
symbol_name: 'baz',
symbol_type: 'function',
start_line: 10,
end_line: 20,
parent_symbol_path: ['Bar'],
doc_comment: 'does the thing',
symbol_name_qualified: 'Bar.baz',
};
function metadataOf(chunk: any) {
return {
language: chunk.language,
symbol_name: chunk.symbol_name,
symbol_type: chunk.symbol_type,
start_line: chunk.start_line,
end_line: chunk.end_line,
parent_symbol_path: chunk.parent_symbol_path,
doc_comment: chunk.doc_comment,
symbol_name_qualified: chunk.symbol_name_qualified,
};
}
test('--stale (autopilot path) carries code metadata into upsertChunks', async () => {
const stale = [{
slug: 'code-page',
chunk_index: 0,
chunk_text: fullCodeChunk.chunk_text,
chunk_source: 'compiled_truth',
model: null,
token_count: 12,
}];
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
countStaleChunks: async () => 1,
listStaleChunks: async () => stale,
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--stale']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
test('--all (full re-embed) carries code metadata into upsertChunks', async () => {
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
listPages: async () => [{ slug: 'code-page' }],
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--all']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
test('--slugs (per-page embed) carries code metadata into upsertChunks', async () => {
let upsertChunkArgs: any[] | null = null;
const engine = mockEngine({
getPage: async () => ({ slug: 'code-page', compiled_truth: 'x', timeline: '' }),
getChunks: async () => [fullCodeChunk],
upsertChunks: async (_slug: string, chunks: any[]) => { upsertChunkArgs = chunks; },
});
await runEmbed(engine, ['--slugs', 'code-page']);
expect(upsertChunkArgs).not.toBeNull();
expect(upsertChunkArgs!).toHaveLength(1);
expect(metadataOf(upsertChunkArgs![0])).toEqual(metadataOf(fullCodeChunk));
});
});
+57
View File
@@ -0,0 +1,57 @@
/**
* eval-takes-quality gateway self-config adapter-boundary regression
* (takeover of PR #2430).
*
* The old callsite spread `{ ...cfg, ...process.env }` straight into
* configureGateway. The gateway NEVER reads process.env at call time it
* reads `_config.env` and that spread never populated an `env` field at
* all, so every availability/diagnose check dereferenced `undefined.env[k]`
* and file-plane API keys (config.json `openai_api_key` etc.) were dropped.
* Routing through buildGatewayConfig fixes both. This test fails (throws)
* on the old code path.
*/
import { afterAll, describe, expect, test } from 'bun:test';
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { runEvalTakesQuality } from '../src/commands/eval-takes-quality.ts';
import { isAvailable, resetGateway } from '../src/core/ai/gateway.ts';
import { withEnv } from './helpers/with-env.ts';
import type { BrainEngine } from '../src/core/engine.ts';
afterAll(() => {
resetGateway();
});
describe('runEvalTakesQuality — gateway self-config routes through buildGatewayConfig', () => {
test('file-plane openai_api_key reaches the gateway env (help path, engine untouched)', async () => {
const home = mkdtempSync(join(tmpdir(), 'gbrain-etq-gw-'));
try {
mkdirSync(join(home, '.gbrain'), { recursive: true });
writeFileSync(
join(home, '.gbrain', 'config.json'),
JSON.stringify({
engine: 'pglite',
database_path: join(home, '.gbrain', 'brain'),
openai_api_key: 'sk-file-plane-test',
}),
);
await withEnv(
{
GBRAIN_HOME: home,
OPENAI_API_KEY: undefined,
DATABASE_URL: undefined,
GBRAIN_DATABASE_URL: undefined,
},
async () => {
// 'help' returns before touching the engine, but the gateway is
// configured first — exactly the seam under test.
await runEvalTakesQuality({} as BrainEngine, ['--help']);
expect(isAvailable('embedding', 'openai:text-embedding-3-small')).toBe(true);
},
);
} finally {
rmSync(home, { recursive: true, force: true });
}
});
});
-16
View File
@@ -201,15 +201,6 @@ describe('scanIntegrity', () => {
timeline: '',
frontmatter: { validate: false },
});
// Indexed source file — 'tweeted about' in a comment must NOT be flagged
// as a bare-tweet citation gap (#1143: auto-repair would corrupt source).
await engine.putPage('code/src/bot.py', {
type: 'code',
title: 'src/bot.py (python)',
compiled_truth: '# the user tweeted about this feature\ndef send_tweet():\n pass',
timeline: '',
frontmatter: { language: 'python', file: 'src/bot.py' },
});
}, 60_000);
afterAll(async () => {
@@ -232,13 +223,6 @@ describe('scanIntegrity', () => {
expect(slugs).not.toContain('people/legacy');
});
test('skips type:code pages (#1143 — bare-tweet false positives on source files)', async () => {
const res = await scanIntegrity(engine);
const slugs = res.bareHits.map(h => h.slug);
expect(slugs).not.toContain('code/src/bot.py');
expect(res.pagesScanned).toBe(2);
});
test('honors limit', async () => {
const res = await scanIntegrity(engine, { limit: 1 });
expect(res.pagesScanned).toBe(1);
-71
View File
@@ -403,77 +403,6 @@ describe('extractPageLinks', () => {
expect(candidates).toEqual([]);
});
test('path-qualified wikilink outside DIR_PATTERN queries by final segment', async () => {
// `[[notes/struktura]]` (dir not in DIR_PATTERN) falls to the generic
// pass. The resolver's basename index is keyed by final path segments,
// so the lookup must strip the dirname — mirroring the FS path
// (resolveSlugAll). Regression: the raw literal was passed through,
// which never matched, so these links silently dropped.
const seen: string[] = [];
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) => {
seen.push(name);
return name === 'struktura' ? ['notes/struktura'] : [];
},
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(seen).toContain('struktura');
expect(seen).not.toContain('notes/struktura');
expect(candidates.map(c => c.targetSlug)).toEqual(['notes/struktura']);
expect(candidates[0].linkType).toBe('wikilink_basename');
expect(candidates[0].linkSource).toBe('wikilink-resolved');
});
test('path-qualified wikilink keeps only matches ending with the written path', async () => {
// The written path disambiguates: `[[notes/struktura]]` must never
// attach to `wiki/struktura` even though both share the basename.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['notes/struktura', 'wiki/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.map(c => c.targetSlug)).toEqual(['notes/struktura']);
});
test('path-qualified wikilink matches a deeper real slug by path suffix', async () => {
// The page lives at vault/notes/struktura; the author wrote the shorter
// tail `[[notes/struktura]]`. Suffix matching connects them, while the
// basename-only sibling `wiki/struktura` stays excluded.
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['vault/notes/struktura', 'wiki/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'concepts/x', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates.map(c => c.targetSlug)).toEqual(['vault/notes/struktura']);
});
test('path-qualified self-link is dropped like the bare form', async () => {
// `[[notes/struktura]]` written on notes/struktura itself must not
// produce a self-loop (same guard as the bare `[[own-tail]]` case).
const resolver: SlugResolver = {
resolve: async () => null,
resolveBasenameMatches: async (name) =>
name === 'struktura' ? ['notes/struktura'] : [],
};
const { candidates } = await extractPageLinks(
'notes/struktura', 'See [[notes/struktura]].',
{}, 'concept', resolver, { globalBasename: true },
);
expect(candidates).toEqual([]);
});
test('bare wikilink resolution does not interfere with DIR_PATTERN wikilinks', async () => {
// 2b refs (people/alice) take the verb-inferred type;
// 2c refs (struktura) take wikilink_basename. Same call.
-62
View File
@@ -1,62 +0,0 @@
import { describe, expect, test } from 'bun:test';
import {
extractCycleFreshnessSourceIds,
parseMaintainArgs,
} from '../src/commands/maintain.ts';
import type { Check } from '../src/commands/doctor.ts';
describe('maintain args', () => {
test('defaults to dry-run unless --safe is explicit', () => {
expect(parseMaintainArgs([])).toMatchObject({
safe: false,
dryRun: true,
json: false,
});
});
test('--safe enables mutating safe mode', () => {
expect(parseMaintainArgs(['--safe', '--json'])).toMatchObject({
safe: true,
dryRun: false,
json: true,
});
});
test('--dry-run wins over --safe', () => {
expect(parseMaintainArgs(['--safe', '--dry-run'])).toMatchObject({
safe: true,
dryRun: true,
});
});
});
describe('cycle freshness source extraction', () => {
test('extracts stale source ids from doctor messages', () => {
const checks: Check[] = [
{
name: 'cycle_freshness',
status: 'fail',
message: "Source 'brain-sync-remote-teffur' last cycled 40h ago. Run `gbrain dream --source <id>`.",
},
{
name: 'cycle_freshness',
status: 'fail',
message: "Source 'wiki' last cycled 25h ago. Source 'wiki' last cycled 25h ago.",
},
];
expect(extractCycleFreshnessSourceIds(checks)).toEqual([
'brain-sync-remote-teffur',
'wiki',
]);
});
test('ignores ok and unrelated checks', () => {
const checks: Check[] = [
{ name: 'cycle_freshness', status: 'ok', message: "Source 'fresh' last cycled recently." },
{ name: 'frontmatter_integrity', status: 'warn', message: "Source 'wiki' has frontmatter issues." },
];
expect(extractCycleFreshnessSourceIds(checks)).toEqual([]);
});
});
-17
View File
@@ -1,17 +0,0 @@
import { describe, expect, it } from 'bun:test';
import { readFileSync } from 'fs';
import { join } from 'path';
describe('root OpenClaw plugin manifest', () => {
it('declares the id required by OpenClaw plugin installs', () => {
const manifest = JSON.parse(readFileSync(join(import.meta.dir, '..', 'openclaw.plugin.json'), 'utf8'));
const entrySource = readFileSync(join(import.meta.dir, '..', 'src', 'openclaw-context-engine.ts'), 'utf8');
const entryId = entrySource.match(/id:\s*'([^']+)'/)?.[1];
expect(manifest.id).toBe(entryId);
expect(manifest.configSchema).toBeDefined();
expect(typeof manifest.configSchema).toBe('object');
expect(manifest.contracts?.contextEngines).toContain('gbrain-context');
expect(entrySource).toContain('export function register');
});
});
-56
View File
@@ -186,67 +186,11 @@ describe('shouldExclude — orphan filter regression (preserve curation)', () =>
expect(shouldExclude('entities/anonymous')).toBe(true);
expect(shouldExclude('atoms/fact-123')).toBe(true);
expect(shouldExclude('skills/gbrain-operations')).toBe(true);
expect(shouldExclude('dreaming/light/2026-07-20')).toBe(true);
expect(shouldExclude('daily/2026-07-20')).toBe(true);
expect(shouldExclude('agent-openclaw/daily/2026-07-20')).toBe(true);
});
test('workspace convention slugs are excluded', () => {
expect(shouldExclude('_brain-conventions')).toBe(true);
expect(shouldExclude('_templates/decision')).toBe(true);
expect(shouldExclude('extracts/2026-06-30/takes.proposed/round-single')).toBe(true);
expect(shouldExclude('2026-07-20')).toBe(true);
expect(shouldExclude('2026-07-20-qa-sweep')).toBe(true);
expect(shouldExclude('agents/arya/identity')).toBe(true);
expect(shouldExclude('agents/arya/memory/dreaming/deep/2026-07-20')).toBe(true);
});
test('regular slugs are NOT excluded', () => {
expect(shouldExclude('people/alice')).toBe(false);
expect(shouldExclude('companies/acme')).toBe(false);
expect(shouldExclude('writing/post-1')).toBe(false);
expect(shouldExclude('agents/arya/qa-reports/launch-review')).toBe(false);
});
});
describe('getHealth orphan_pages uses shared exclusion policy', () => {
test('excluded convention islands do not count against health', async () => {
await engine.putPage('_templates/decision', {
type: 'template', title: 'Decision', compiled_truth: 'template', timeline: '', frontmatter: {},
});
await engine.putPage('skills/arya/source-check', {
type: 'concept', title: 'Skill', compiled_truth: 'skill', timeline: '', frontmatter: {},
});
await engine.putPage('agents/arya/identity', {
type: 'note', title: 'Identity', compiled_truth: 'identity', timeline: '', frontmatter: {},
});
await engine.putPage('people/alice', {
type: 'person', title: 'Alice', compiled_truth: 'real island', timeline: '', frontmatter: {},
});
const health = await engine.getHealth();
expect(health.orphan_pages).toBe(1);
});
test('per-brain config overrides (orphans.exclude_*) also apply to health', async () => {
await engine.putPage('my-private-folder/secret-ref', {
type: 'note', title: 'Ref', compiled_truth: 'ref', timeline: '', frontmatter: {},
});
await engine.putPage('one-off-fixture-page', {
type: 'note', title: 'Fixture', compiled_truth: 'fixture', timeline: '', frontmatter: {},
});
await engine.putPage('people/alice', {
type: 'person', title: 'Alice', compiled_truth: 'real island', timeline: '', frontmatter: {},
});
expect((await engine.getHealth()).orphan_pages).toBe(3);
await engine.setConfig('orphans.exclude_prefixes', 'my-private-folder/');
await engine.setConfig('orphans.exclude_slugs', 'one-off-fixture-page');
expect((await engine.getHealth()).orphan_pages).toBe(1);
await engine.unsetConfig('orphans.exclude_prefixes');
await engine.unsetConfig('orphans.exclude_slugs');
});
});
-38
View File
@@ -66,10 +66,6 @@ describe('shouldExclude', () => {
expect(shouldExclude('templates/meeting-note')).toBe(true);
});
test('excludes deny-prefix: _templates/', () => {
expect(shouldExclude('_templates/meeting-note')).toBe(true);
});
test('excludes deny-prefix: openclaw/config/', () => {
expect(shouldExclude('openclaw/config/agent')).toBe(true);
});
@@ -90,44 +86,10 @@ describe('shouldExclude', () => {
expect(shouldExclude('entities/product-hunt')).toBe(true);
});
test('excludes first-segment: skills, dreaming, and daily', () => {
expect(shouldExclude('skills/arya/source-check')).toBe(true);
expect(shouldExclude('dreaming/light/2026-07-20')).toBe(true);
expect(shouldExclude('daily/2026-07-20')).toBe(true);
expect(shouldExclude('agent-openclaw/daily/2026-07-20')).toBe(true);
});
test('excludes root date logs and agent workspace conventions', () => {
expect(shouldExclude('_brain-conventions')).toBe(true);
expect(shouldExclude('2026-07-20')).toBe(true);
expect(shouldExclude('2026-07-20-qa-sweep')).toBe(true);
expect(shouldExclude('agents/arya/identity')).toBe(true);
expect(shouldExclude('agents/arya/memory/dreaming/deep/2026-07-20')).toBe(true);
});
test('excludes generated extracts', () => {
expect(shouldExclude('extracts/2026-06-30/takes.proposed/round-single')).toBe(true);
});
test('brain-specific exclusions come from config overrides, not global defaults', () => {
// No baked-in defaults for these:
expect(shouldExclude('my-private-folder/some-secret-ref.md')).toBe(false);
expect(shouldExclude('one-off-fixture-page')).toBe(false);
// The per-brain config plane (orphans.exclude_prefixes / exclude_slugs):
const overrides = {
excludePrefixes: ['my-private-folder/'],
excludeSlugs: ['one-off-fixture-page'],
};
expect(shouldExclude('my-private-folder/some-secret-ref.md', overrides)).toBe(true);
expect(shouldExclude('one-off-fixture-page', overrides)).toBe(true);
expect(shouldExclude('people/jane-doe', overrides)).toBe(false);
});
test('does NOT exclude a normal content page', () => {
expect(shouldExclude('companies/acme')).toBe(false);
expect(shouldExclude('people/jane-doe')).toBe(false);
expect(shouldExclude('projects/gbrain')).toBe(false);
expect(shouldExclude('agents/arya/qa-reports/launch-review')).toBe(false);
});
test('does NOT exclude a page ending with log-like text that is not /log', () => {
-34
View File
@@ -1,34 +0,0 @@
/**
* #1098 ClawVisor health checks must hit /ready (surfaces db/vault
* degradation), never /health (bare liveness, masks those failure modes).
* #1099 the credential-gateway recipe tells users to activate Google
* Contacts; the canonical consumer recipe must actually ship.
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync, existsSync } from 'fs';
import { join } from 'path';
const RECIPES = join(import.meta.dir, '../recipes');
describe('#1098 — ClawVisor health checks use /ready', () => {
for (const f of ['email-to-brain.md', 'calendar-to-brain.md', 'credential-gateway.md', 'contacts-to-brain.md']) {
test(`${f} references $CLAWVISOR_URL/ready, never /health`, () => {
const text = readFileSync(join(RECIPES, f), 'utf-8');
expect(text).not.toContain('$CLAWVISOR_URL/health');
expect(text).toContain('$CLAWVISOR_URL/ready');
});
}
});
describe('#1099 — canonical contacts-to-brain recipe ships', () => {
test('recipes/contacts-to-brain.md exists with the sibling frontmatter shape', () => {
const p = join(RECIPES, 'contacts-to-brain.md');
expect(existsSync(p)).toBe(true);
const text = readFileSync(p, 'utf-8');
expect(text).toContain('id: contacts-to-brain');
expect(text).toContain('requires: [credential-gateway]');
expect(text).toContain('health_checks:');
expect(text).toContain('contacts.readonly');
});
});
-15
View File
@@ -12,11 +12,9 @@ import {
bestPath,
historyPath,
loadHistory,
proposedPath,
revertAllPending,
skillPath,
versionsDir,
writeProposed,
} from '../../src/core/skillopt/version-store.ts';
let tmpDir: string;
@@ -81,19 +79,6 @@ describe('acceptCandidate (D8 two-phase commit)', () => {
});
});
describe('writeProposed', () => {
test('writes distinct best and proposed artifacts without mutating SKILL.md (#2635)', () => {
const candidate = '---\nname: test\n---\nproposed body\n';
const written = writeProposed(tmpDir, SKILL, candidate);
expect(written).toBe(proposedPath(tmpDir, SKILL));
expect(fs.readFileSync(bestPath(tmpDir, SKILL), 'utf8')).toBe(candidate);
expect(fs.readFileSync(proposedPath(tmpDir, SKILL), 'utf8')).toBe(candidate);
expect(fs.readFileSync(skillPath(tmpDir, SKILL), 'utf8')).toContain('baseline body');
});
});
describe('revertAllPending (D8 crash recovery)', () => {
test('no-op when no pending rows', () => {
const reverted = revertAllPending(tmpDir, SKILL);