mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
23
Commits
+115
@@ -2,6 +2,121 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.18.0] - 2026-05-26
|
||||
|
||||
**You can now run one command and have gbrain tell you exactly what's
|
||||
wrong with your brain — and offer to fix it.**
|
||||
|
||||
Most agents wire up gbrain, import their notes, and never realize that
|
||||
most of their pages have no inbound links, thousands of pages are
|
||||
missing embeddings, and the brain is running at half its retrieval
|
||||
potential. The agent doesn't know to ask. The user doesn't know to
|
||||
look. This release adds the surface that closes that gap.
|
||||
|
||||
Run `gbrain onboard --check` and your brain shows up: how many stale
|
||||
chunks, how much entity-link coverage, how many timeline entries, how
|
||||
many takes. Then it shows the commands that would fix each one. Run
|
||||
`gbrain onboard --auto --max-usd 5` and gbrain runs every fix that's
|
||||
safe to apply unattended.
|
||||
|
||||
What you can do that you couldn't before:
|
||||
|
||||
- `gbrain onboard --check` — see your brain's onboarding state in 5
|
||||
seconds (orphans, stale embeddings, link coverage, timeline coverage,
|
||||
takes count). JSON envelope shape is stable (`schema_version: 1`) for
|
||||
agents that want to bind to it.
|
||||
- `gbrain onboard --auto --max-usd 5` — apply every safe remediation
|
||||
the planner returns. Refuses without `--max-usd` so a cron can't burn
|
||||
through your API budget.
|
||||
- `gbrain embed --stale --catch-up --priority recent` — embed your
|
||||
newest-edited pages first, keep going until the backlog is gone.
|
||||
Lifts the prior hardcoded 2000-row batch cap (now `--batch-size N`).
|
||||
- `gbrain extract links --ner` — typed NER links via your schema pack's
|
||||
`link_types[].inference.regex`. Pairs with `--by-mention` in one walk
|
||||
(shared gazetteer). New `link_kind='typed_ner'` column distinguishes
|
||||
verb-pattern matches from plain mentions without splitting
|
||||
`link_source`.
|
||||
- `gbrain extract timeline --from-meetings` — walks meeting pages,
|
||||
writes a timeline entry on each entity that was discussed (attendees
|
||||
+ body mentions). Survives the v99 dedup widening so two meetings on
|
||||
the same date no longer silently drop one.
|
||||
- `gbrain takes extract --from-pages` — Haiku classifier over your
|
||||
concept/atom/lore/briefing/writing/originals pages, lifts gradeable
|
||||
claims into the takes fence. Two-gate opt-in
|
||||
(`takes.bootstrap_enabled=true` AND `--yes`) — does not run unattended,
|
||||
does not run without your explicit consent.
|
||||
- Init + upgrade nudges — `gbrain init` and `gbrain upgrade` now end
|
||||
with a one-line summary of opportunities so first-day users
|
||||
discover the onboard surface. 3-second wallclock cap, fail-open
|
||||
(never crashes init). Suppress with `GBRAIN_NO_ONBOARD_NUDGE=1`.
|
||||
- Autopilot tick consults onboard recommendations alongside
|
||||
brain-score remediations. Self-improving brain on a 24h cycle.
|
||||
- MCP op `run_onboard` (admin scope) — federated and thin-client
|
||||
installs can probe brain health + drive remediation over OAuth.
|
||||
LLM-bearing handlers (synthesize, patterns, consolidate,
|
||||
takes-bootstrap) require a NEW `run_protected_onboard` scope IN
|
||||
ADDITION to admin — admin alone won't burn your API budget.
|
||||
- `gbrain onboard --history` — see what landed and what it changed.
|
||||
New `migration_impact_log` table with full attribution columns
|
||||
(job_id, source_id, brain_id, started_at, idempotency_key)
|
||||
prevents concurrent runs misattributing deltas.
|
||||
|
||||
The architecture story for engineers: `gbrain doctor --remediate`
|
||||
(v0.36.4.0) had already shipped the cathedral — generalized
|
||||
RemediationStep, dependency-ordered execution, BudgetTracker
|
||||
integration, Minion-handler dispatch. We extended that cathedral
|
||||
instead of building a parallel one. A new
|
||||
`src/core/remediation/` library extracts the orchestrator from
|
||||
doctor's CLI shape; `gbrain onboard` is a ~180-line wrapper. The
|
||||
codex review caught this — the original plan was going to rebuild
|
||||
RemediationStep under a new name and that would have been a 4000+
|
||||
LOC rebuild of work that already existed.
|
||||
|
||||
Schema additions:
|
||||
- v98 — `links.link_kind` nullable column (`'plain'` | `'typed_ner'` | NULL).
|
||||
- v99 — `timeline_entries` dedup widened to `(page_id, date, summary,
|
||||
source)` so meeting provenance survives.
|
||||
- v100 — `migration_impact_log` table + `content_chunks_stale_idx`
|
||||
partial (supports `--priority recent` cursor).
|
||||
|
||||
Engine API additions:
|
||||
- `BrainEngine.listStaleChunks(opts)` gains `orderBy: 'page_id' |
|
||||
'updated_desc'` + `afterUpdatedAt` for composite-cursor pagination.
|
||||
- `BrainEngine.executeRaw(sql, params, opts?)` gains `opts.signal`
|
||||
for real AbortSignal-bound query cancellation (Postgres
|
||||
`query.cancel()`; PGLite best-effort via Promise.race).
|
||||
- `LinkBatchInput.link_kind?: string` — threaded through both engine
|
||||
impls' `addLinksBatch` unnest tuple.
|
||||
|
||||
Privacy + consent posture: takes-bootstrap sends concept/atom/lore/
|
||||
briefing/writing/originals page content to your configured chat model.
|
||||
Two-gate consent — `takes.bootstrap_enabled=true` config AND `--yes`
|
||||
flag — refuses to run otherwise. Even with both gates flipped, the
|
||||
autopilot path for takes-bootstrap stays `manual_only` (does not fire
|
||||
unattended) until v0.42.1 lands the 100+-case eval suite.
|
||||
|
||||
To take advantage of v0.41.18.0:
|
||||
|
||||
`gbrain upgrade` runs the schema migrations and prints the post-upgrade
|
||||
banner pointing at `gbrain onboard --check`. If something looks off:
|
||||
|
||||
1. Run `gbrain apply-migrations --yes` to ensure v98/v99/v100 landed.
|
||||
2. Run `gbrain onboard --check` to see your brain's state.
|
||||
3. If you want to opt into takes-bootstrap, run:
|
||||
`gbrain config set takes.bootstrap_enabled true`
|
||||
then `gbrain takes extract --from-pages --yes --max-usd 5`.
|
||||
4. Filed an issue? Include `gbrain doctor --json` + the contents of
|
||||
`~/.gbrain/upgrade-errors.jsonl` if it exists. Helps gbrain
|
||||
maintainers find fragile upgrade paths.
|
||||
|
||||
Closes meta-issue #1383 (`gbrain onboard`). Migration #1 (auto-link
|
||||
`--by-mention`) shipped earlier as v0.41.10.0. PR #1409 (the
|
||||
consolidated design doc) is implemented by this release.
|
||||
|
||||
Note: schema migrations originally numbered v98/v99/v100 were renumbered
|
||||
to v101/v102/v103 post-merge because master claimed v98 (sync lock
|
||||
refresh column from v0.41.15.0) and v99 (conversation parser cache from
|
||||
v0.41.16.0). Migration content unchanged across the renumber.
|
||||
## [0.41.17.0] - 2026-05-26
|
||||
|
||||
**You can now run `extract-conversation-facts`, `extract`,
|
||||
|
||||
@@ -274,3 +274,58 @@ automatically during `gbrain post-upgrade` to fix the double-encoded JSONB
|
||||
columns. PGLite brains no-op. If wiki-style imports were truncated by the old
|
||||
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
|
||||
`compiled_truth` from source markdown.
|
||||
|
||||
## v0.42.0+ onboard surface (NEW)
|
||||
|
||||
`gbrain onboard` is the activation surface gbrain did not have before.
|
||||
Once your brain has any content, run `gbrain onboard --check --json` to
|
||||
see structured recommendations across 5 brain-health axes (orphans,
|
||||
stale embeddings, entity link coverage, timeline coverage, takes count).
|
||||
|
||||
**On first connect (after `gbrain init`):**
|
||||
```bash
|
||||
gbrain onboard --check --json
|
||||
```
|
||||
The JSON envelope (`schema_version: 1`) carries `recommendations[]` with
|
||||
`apply_policy` per item: `auto_apply` (safe to run unattended),
|
||||
`prompt_required` (needs explicit user consent), or `manual_only`
|
||||
(LLM-bearing, user must run themselves).
|
||||
|
||||
**After every `gbrain upgrade`:**
|
||||
```bash
|
||||
gbrain onboard --check --json
|
||||
```
|
||||
New versions may surface new opportunities. The post-upgrade banner
|
||||
nudges the user when it runs, but agents should re-probe as a hygiene
|
||||
step regardless.
|
||||
|
||||
**Unattended remediation (cron / autopilot):**
|
||||
```bash
|
||||
gbrain onboard --auto --max-usd 5
|
||||
```
|
||||
Refuses without `--max-usd N`. Runs auto-eligible items only. The
|
||||
autopilot daemon also consults onboard recommendations on its tick — no
|
||||
explicit agent action needed for the autonomous path.
|
||||
|
||||
**Remote / federated brain installs (MCP):**
|
||||
The `run_onboard` MCP op (admin scope) lets thin-client agents probe
|
||||
brain health + drive remediation over OAuth-authenticated MCP. Protected
|
||||
LLM-bearing handlers (synthesize, patterns, consolidate, takes-bootstrap,
|
||||
contextual_reindex_per_chunk) require the additional `run_protected_onboard`
|
||||
scope — admin alone is insufficient. The MCP op returns
|
||||
`skipped_missing_scope[]` listing what would have run with the right
|
||||
grants.
|
||||
|
||||
**Privacy + consent gates:**
|
||||
- `gbrain takes extract --from-pages` sends concept/atom/lore/briefing/
|
||||
writing/originals page content to your configured chat model (default
|
||||
Anthropic Haiku). Refuses to run unless `takes.bootstrap_enabled=true`
|
||||
is set in config AND `--yes` is passed. Two-gate opt-in by design.
|
||||
- Autopilot's auto-apply tier for takes-bootstrap stays `manual_only`
|
||||
until v0.42.1's eval gate (do not bypass).
|
||||
|
||||
**Suppress nudges in CI / scripted environments:**
|
||||
```bash
|
||||
export GBRAIN_NO_ONBOARD_NUDGE=1
|
||||
```
|
||||
Init + upgrade banners auto-skip in non-TTY too.
|
||||
|
||||
@@ -1,5 +1,51 @@
|
||||
# TODOS
|
||||
|
||||
## v0.41.18.0 onboard wave follow-ups (v0.42.1+)
|
||||
|
||||
- **TODO-A (P2)**: Pack-aware `linkable: boolean` per-type field on schema-pack
|
||||
manifests. Both `gbrain extract links --by-mention` and `--ner` would consult
|
||||
it to gate which entity types participate in gazetteer construction. Currently
|
||||
uses a hardcoded `['person', 'company', 'organization', 'entity']` list.
|
||||
|
||||
- **TODO-B (P3)**: LLM-based entity disambiguation for `--ner`. v0.42.0 ships
|
||||
regex+gazetteer only; misses cases like "Anthropic's founders" → `Anthropic`
|
||||
link. A small Haiku post-pass would catch these.
|
||||
|
||||
- **TODO-C (P3)**: `gbrain onboard --explain <recommendation_id>` drill-down.
|
||||
Shows the underlying check, its measurement, and why the recommendation
|
||||
fired. Useful when an operator wants to understand what `onboard --auto` is
|
||||
about to do.
|
||||
|
||||
- **TODO-D (P2)**: Live-brain impact measurement against a representative brain
|
||||
(165K-page production class). v0.42.0 ships the `migration_impact_log`
|
||||
infrastructure; we need real-world numbers to update the design doc claims
|
||||
with measured deltas.
|
||||
|
||||
- **TODO-E (P1)**: 100+-case eval suite for takes-bootstrap classifier. v0.42.0
|
||||
ships the classifier + the 20-case eval scaffold per A24. Autopilot tier for
|
||||
takes-bootstrap STAYS `manual_only` until this lands. Required before any
|
||||
autopilot run of takes extraction.
|
||||
|
||||
- **TODO-F (P3)**: Web UI surface for `gbrain onboard` recommendations in the
|
||||
admin SPA. Linear-style dashboard with one-click apply.
|
||||
|
||||
- **TODO-G (P2)**: Full DATABASE_URL-gated E2E for onboard. v0.42.0 ships
|
||||
hermetic PGLite contracts coverage in `test/e2e/onboard-full-flow.test.ts`;
|
||||
the real-Postgres version needs the Minion worker test harness to land its
|
||||
per-handler stub seam so individual extraction handlers can be replaced for
|
||||
testing.
|
||||
|
||||
- **TODO-H (P2)**: `minion_jobs.client_id` schema column. v0.42.0 stores the
|
||||
originating OAuth client_id on `job.data.client_id` (JSONB passthrough).
|
||||
A real schema column + index would let the spend query path (per-client
|
||||
daily cap enforcement) avoid the JSONB projection cost.
|
||||
|
||||
- **TODO-I (P3)**: Thin-client (doctor-remote.ts) parity for the 4 new onboard
|
||||
checks (embed_staleness, entity_link_coverage, timeline_coverage,
|
||||
takes_count). Today the MCP run_onboard op runs these server-side via
|
||||
runAllOnboardChecks; doctor-remote.ts would surface them on the thin-client
|
||||
dashboard for operators who only hit the brain via MCP.
|
||||
=======
|
||||
## v0.41.17.0 `--workers N` cathedral follow-ups (v0.41.18+)
|
||||
|
||||
These were filed during the ship of `garrytan/dar-es-salaam-v1`
|
||||
|
||||
@@ -2431,6 +2431,61 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old
|
||||
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
|
||||
`compiled_truth` from source markdown.
|
||||
|
||||
## v0.42.0+ onboard surface (NEW)
|
||||
|
||||
`gbrain onboard` is the activation surface gbrain did not have before.
|
||||
Once your brain has any content, run `gbrain onboard --check --json` to
|
||||
see structured recommendations across 5 brain-health axes (orphans,
|
||||
stale embeddings, entity link coverage, timeline coverage, takes count).
|
||||
|
||||
**On first connect (after `gbrain init`):**
|
||||
```bash
|
||||
gbrain onboard --check --json
|
||||
```
|
||||
The JSON envelope (`schema_version: 1`) carries `recommendations[]` with
|
||||
`apply_policy` per item: `auto_apply` (safe to run unattended),
|
||||
`prompt_required` (needs explicit user consent), or `manual_only`
|
||||
(LLM-bearing, user must run themselves).
|
||||
|
||||
**After every `gbrain upgrade`:**
|
||||
```bash
|
||||
gbrain onboard --check --json
|
||||
```
|
||||
New versions may surface new opportunities. The post-upgrade banner
|
||||
nudges the user when it runs, but agents should re-probe as a hygiene
|
||||
step regardless.
|
||||
|
||||
**Unattended remediation (cron / autopilot):**
|
||||
```bash
|
||||
gbrain onboard --auto --max-usd 5
|
||||
```
|
||||
Refuses without `--max-usd N`. Runs auto-eligible items only. The
|
||||
autopilot daemon also consults onboard recommendations on its tick — no
|
||||
explicit agent action needed for the autonomous path.
|
||||
|
||||
**Remote / federated brain installs (MCP):**
|
||||
The `run_onboard` MCP op (admin scope) lets thin-client agents probe
|
||||
brain health + drive remediation over OAuth-authenticated MCP. Protected
|
||||
LLM-bearing handlers (synthesize, patterns, consolidate, takes-bootstrap,
|
||||
contextual_reindex_per_chunk) require the additional `run_protected_onboard`
|
||||
scope — admin alone is insufficient. The MCP op returns
|
||||
`skipped_missing_scope[]` listing what would have run with the right
|
||||
grants.
|
||||
|
||||
**Privacy + consent gates:**
|
||||
- `gbrain takes extract --from-pages` sends concept/atom/lore/briefing/
|
||||
writing/originals page content to your configured chat model (default
|
||||
Anthropic Haiku). Refuses to run unless `takes.bootstrap_enabled=true`
|
||||
is set in config AND `--yes` is passed. Two-gate opt-in by design.
|
||||
- Autopilot's auto-apply tier for takes-bootstrap stays `manual_only`
|
||||
until v0.42.1's eval gate (do not bypass).
|
||||
|
||||
**Suppress nudges in CI / scripted environments:**
|
||||
```bash
|
||||
export GBRAIN_NO_ONBOARD_NUDGE=1
|
||||
```
|
||||
Init + upgrade banners auto-skip in non-TTY too.
|
||||
|
||||
---
|
||||
|
||||
## skills/RESOLVER.md
|
||||
|
||||
+3
-2
@@ -1,6 +1,5 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.41.17.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -78,6 +77,7 @@
|
||||
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
|
||||
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
|
||||
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
|
||||
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
|
||||
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
||||
@@ -137,5 +137,6 @@
|
||||
"engines": {
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"version": "0.41.18.0"
|
||||
}
|
||||
|
||||
Executable
+70
@@ -0,0 +1,70 @@
|
||||
#!/bin/bash
|
||||
# scripts/check-source-scope-onboard.sh
|
||||
# v0.41.18.0 (A26, T17). Grep guard against SQL sites in src/core/onboard/
|
||||
# and the 4 new onboard-derived doctor checks that touch source_id-bearing
|
||||
# tables (pages, content_chunks, takes, links, timeline_entries) WITHOUT
|
||||
# either:
|
||||
# (a) including source_id / source_ids in the WHERE clause, OR
|
||||
# (b) carrying the explicit opt-out marker `sourcescope:brain-wide` in
|
||||
# an adjacent comment.
|
||||
#
|
||||
# Brain-wide metrics (embed_staleness, takes_count, total entity counts)
|
||||
# are legitimate brain-wide queries — they MUST NOT auto-filter by source
|
||||
# because the metric IS "across all sources". The opt-out marker is the
|
||||
# explicit acknowledgement that this is intentional. Any new code touching
|
||||
# per-source data WITHOUT the marker has to add source-scoping.
|
||||
|
||||
set -e
|
||||
|
||||
FILES_TO_CHECK=(
|
||||
"src/core/onboard/checks.ts"
|
||||
"src/core/onboard/impact-capture.ts"
|
||||
"src/core/onboard/render.ts"
|
||||
"src/commands/onboard.ts"
|
||||
)
|
||||
|
||||
ERR=0
|
||||
|
||||
for f in "${FILES_TO_CHECK[@]}"; do
|
||||
if [ ! -f "$f" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# Skip a file entirely when it doesn't contain SQL at all.
|
||||
if ! grep -qE 'executeRaw|SELECT|INSERT|UPDATE|DELETE' "$f"; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# File-level opt-out: if the file declares `sourcescope:file-brain-wide`
|
||||
# in its header (first 30 lines), every SQL site inside is treated as
|
||||
# intentionally brain-wide. Use sparingly — only for files whose SQL
|
||||
# is structurally always-aggregate (onboard/checks.ts, impact-capture.ts).
|
||||
if head -30 "$f" | grep -q 'sourcescope:file-brain-wide'; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# Search for SQL-ish lines that DO NOT contain source_id and DO NOT have
|
||||
# the brain-wide opt-out marker on the same line or within 3 lines above.
|
||||
while IFS=: read -r line content; do
|
||||
# Skip if source_id mentioned on this or nearby lines (5-line window).
|
||||
start=$((line - 4))
|
||||
[ "$start" -lt 1 ] && start=1
|
||||
if sed -n "${start},${line}p" "$f" | grep -qE 'source_id|sourceIds|sourcescope:brain-wide'; then
|
||||
continue
|
||||
fi
|
||||
echo "[check-source-scope-onboard] $f:$line — SQL site lacks source_id WHERE clause OR brain-wide opt-out marker"
|
||||
echo " $content"
|
||||
ERR=1
|
||||
done < <(grep -nE 'FROM pages|FROM content_chunks|FROM takes\b|FROM links|FROM timeline_entries|UPDATE pages|UPDATE content_chunks|DELETE FROM pages|DELETE FROM content_chunks' "$f" || true)
|
||||
done
|
||||
|
||||
if [ "$ERR" -eq 1 ]; then
|
||||
echo ""
|
||||
echo "[check-source-scope-onboard] One or more SQL sites in onboard surfaces lack source_id scoping."
|
||||
echo "Either: (a) add source_id = \$N (or source_id = ANY(\$N::text[])) to the WHERE,"
|
||||
echo " or (b) add a comment marker 'sourcescope:brain-wide' within 4 lines above"
|
||||
echo " the SQL to declare intent."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
@@ -58,6 +58,7 @@ CHECKS=(
|
||||
"check:fixture-privacy"
|
||||
"check:conversation-parser"
|
||||
"check:resolver"
|
||||
"check:source-scope-onboard"
|
||||
"typecheck"
|
||||
)
|
||||
|
||||
|
||||
+8
-1
@@ -35,7 +35,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'conversation-parser']);
|
||||
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -1499,6 +1499,13 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runTakes(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'onboard': {
|
||||
// v0.41.18.0 (T13) — gbrain onboard. Thin shell over T2 library
|
||||
// + T4 onboard checks + T12 render layer.
|
||||
const { runOnboard } = await import('./commands/onboard.ts');
|
||||
await runOnboard(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'founder': {
|
||||
// v0.35.4 (T7) — founder scorecard. `gbrain founder scorecard <slug>`
|
||||
// rolls up Phase 2's typed-claim substrate into the four scorecard
|
||||
|
||||
@@ -511,7 +511,23 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
}),
|
||||
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || await engine.getConfig('anthropic_api_key')),
|
||||
};
|
||||
const plan = computeRecommendations(health, ctx).filter((r) => r.status === 'remediable');
|
||||
// v0.41.18.0 (A5 + A19 + A22, T15): consult onboard recommendations
|
||||
// ALONGSIDE doctor's brain-score recommendations. Onboard's 4 new
|
||||
// checks (embed_staleness, link_coverage, timeline_coverage,
|
||||
// takes_count) supply extraRemediations into computeRecommendations.
|
||||
// Per A19 fail-open: any throw in the onboard path falls through
|
||||
// to legacy doctor-only plan (no crash).
|
||||
let extraRemediations: ReturnType<typeof computeRecommendations> = [];
|
||||
try {
|
||||
const { runAllOnboardChecks } = await import('../core/onboard/checks.ts');
|
||||
const onboardResults = await runAllOnboardChecks(engine);
|
||||
extraRemediations = onboardResults.flatMap((r) => r.remediations);
|
||||
} catch (err) {
|
||||
process.stderr.write(
|
||||
`[autopilot] onboard checks failed (fail-open per A19): ${err instanceof Error ? err.message : String(err)}\n`,
|
||||
);
|
||||
}
|
||||
const plan = computeRecommendations(health, ctx, extraRemediations).filter((r) => r.status === 'remediable');
|
||||
const estTotal = plan.reduce((s, r) => s + r.est_seconds, 0);
|
||||
|
||||
// Track time since last full cycle for the 60-min floor.
|
||||
|
||||
+115
-345
@@ -3988,7 +3988,7 @@ export async function buildChecks(
|
||||
checks.push({ name: 'graph_coverage', status: 'warn', message: 'Could not check graph coverage' });
|
||||
}
|
||||
|
||||
// 9b. v0.42.0.0 — orphan_ratio check (migration #1 of #1409).
|
||||
// 9b. v0.41.18.0 — orphan_ratio check (migration #1 of #1409).
|
||||
//
|
||||
// Surfaces the fraction of linkable pages with no inbound links.
|
||||
// Consumes the same canonical getOrphansData() pure fn as
|
||||
@@ -5324,6 +5324,15 @@ export async function buildChecks(
|
||||
// v0.38 — cycle_phase_scope (informational; no DB cost)
|
||||
progress.heartbeat('cycle_phase_scope');
|
||||
checks.push(checkCyclePhaseScope());
|
||||
|
||||
// v0.41.18.0 (A16, T4): 4 onboard checks — each emits a Check + its
|
||||
// own RemediationStep[] aggregated by onboard's plan path. The
|
||||
// checks themselves are cheap counts (backed by content_chunks_stale_idx
|
||||
// for embed_staleness, TABLESAMPLE on PG >50K for the coverage pair).
|
||||
progress.heartbeat('onboard_checks');
|
||||
const { runAllOnboardChecks } = await import('../core/onboard/checks.ts');
|
||||
const onboardResults = await runAllOnboardChecks(engine);
|
||||
for (const r of onboardResults) checks.push(r.check);
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
@@ -5704,59 +5713,25 @@ async function runLocksCheck(engine: BrainEngine | null, jsonOutput: boolean): P
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Emit ordered Remediation list to drive brain to --target-score.
|
||||
* CLI wrapper around computeRemediationPlan (src/core/remediation/plan.ts).
|
||||
*
|
||||
* Read-only — never enqueues, never mutates. The agent contract:
|
||||
* inspect the plan with --remediation-plan --json before committing
|
||||
* to --remediate. The JSON shape is stable; consumers that parse it
|
||||
* can rely on it across releases.
|
||||
* v0.41.18.0 (A1, codex finding #2): library extracted so onboard +
|
||||
* MCP run_onboard can compose against a stable shape. This wrapper
|
||||
* stays as the CLI surface only — argv parsing + human render. JSON
|
||||
* mode emits the library's stable envelope verbatim.
|
||||
*
|
||||
* Read-only — never enqueues, never mutates.
|
||||
*/
|
||||
export async function runRemediationPlan(
|
||||
engine: BrainEngine,
|
||||
args: string[],
|
||||
): Promise<void> {
|
||||
const { computeRecommendations, classifyChecks, maxReachableScore } =
|
||||
await import('../core/brain-score-recommendations.ts');
|
||||
const { computeRemediationPlan } = await import('../core/remediation/index.ts');
|
||||
|
||||
const targetScore = parseIntFlag(args, '--target-score') ?? 90;
|
||||
const jsonOutput = args.includes('--json');
|
||||
|
||||
// Cheap path (D7) — don't run slow doctor checks for the plan surface.
|
||||
// The recommendation generator works from BrainHealth + context alone.
|
||||
const health = await engine.getHealth();
|
||||
const ctx = await loadRecommendationContext(engine);
|
||||
const recs = computeRecommendations(health, ctx);
|
||||
// Synthetic check list for classification — we don't need full doctor
|
||||
// output, just the check names the recommendations care about.
|
||||
const syntheticChecks = [
|
||||
{ name: 'brain_score', status: 'ok' as const },
|
||||
{ name: 'sync_freshness', status: 'ok' as const },
|
||||
{ name: 'missing_embeddings', status: 'ok' as const },
|
||||
{ name: 'dead_links', status: 'ok' as const },
|
||||
{ name: 'orphan_pages', status: 'ok' as const },
|
||||
];
|
||||
const classifications = classifyChecks(syntheticChecks, ctx);
|
||||
const ceiling = maxReachableScore(health, classifications);
|
||||
|
||||
const filteredRecs = recs.filter((r) => r.status === 'remediable');
|
||||
const estTotalSeconds = filteredRecs.reduce((sum, r) => sum + r.est_seconds, 0);
|
||||
const estTotalUsd = filteredRecs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
|
||||
|
||||
const blocked = classifications
|
||||
.filter((c) => c.status === 'blocked')
|
||||
.map((c) => ({ check: c.check, reason: c.reason ?? 'prerequisite missing' }));
|
||||
|
||||
const plan = {
|
||||
schema_version: 2,
|
||||
brain_score_current: health.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
max_reachable_score: ceiling,
|
||||
target_unreachable: targetScore > ceiling,
|
||||
plan: filteredRecs.map((r, i) => ({ step: i + 1, ...r })),
|
||||
est_total_seconds: estTotalSeconds,
|
||||
est_total_usd_cost: Number(estTotalUsd.toFixed(2)),
|
||||
blocked,
|
||||
};
|
||||
const plan = await computeRemediationPlan(engine, { targetScore });
|
||||
|
||||
if (jsonOutput) {
|
||||
console.log(JSON.stringify(plan, null, 2));
|
||||
@@ -5764,9 +5739,9 @@ export async function runRemediationPlan(
|
||||
}
|
||||
|
||||
// Human output
|
||||
console.log(`Brain score: ${health.brain_score}/100 → target ${targetScore}`);
|
||||
console.log(`Brain score: ${plan.brain_score_current}/100 → target ${targetScore}`);
|
||||
if (plan.target_unreachable) {
|
||||
console.log(`Target unreachable: max with autonomous remediation is ${ceiling}/100.`);
|
||||
console.log(`Target unreachable: max with autonomous remediation is ${plan.max_reachable_score}/100.`);
|
||||
}
|
||||
if (plan.plan.length === 0) {
|
||||
console.log('No remediations needed. Brain is at target.');
|
||||
@@ -5778,21 +5753,25 @@ export async function runRemediationPlan(
|
||||
console.log(` ${step.step}. [${step.severity}] ${step.job}${protectedMark} — ${step.rationale}${costMark}`);
|
||||
}
|
||||
}
|
||||
if (blocked.length > 0) {
|
||||
if (plan.blocked.length > 0) {
|
||||
console.log(`\nBlocked checks (prereq missing):`);
|
||||
for (const b of blocked) {
|
||||
for (const b of plan.blocked) {
|
||||
console.log(` - ${b.check}: ${b.reason}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Submit ordered Remediation jobs sequentially per D3, with D5 cascade
|
||||
* on failure and D7 scoped recheck between steps.
|
||||
* CLI wrapper around runRemediation (src/core/remediation/run.ts).
|
||||
*
|
||||
* v0.41.18.0 (A1, codex finding #2): orchestrator extracted into the
|
||||
* remediation library. This wrapper stays as the CLI surface only —
|
||||
* argv parsing + interactive TTY confirmation + human/JSON render via
|
||||
* RemediationHooks.
|
||||
*
|
||||
* Default behavior: submit-and-wait per step. --dry-run skips submission.
|
||||
* --max-usd N refuses if est_total_usd_cost > N. --max-jobs N caps the
|
||||
* inner loop.
|
||||
* inner loop. --resume [plan_hash] loads checkpoint and continues.
|
||||
*
|
||||
* PGLite path: synchronous in-process execution (no durable queue).
|
||||
*/
|
||||
@@ -5806,7 +5785,8 @@ export async function runRemediate(
|
||||
// documented as the cron-safety guard. Either threads through to the
|
||||
// pre-flight estimate refusal AND, via withBudgetTracker, the mid-run
|
||||
// BudgetExhausted hard-throw.
|
||||
const maxUsd = parseFloatFlag(args, '--max-usd') ?? parseFloatFlag(args, '--max-cost');
|
||||
const maxUsdRaw = parseFloatFlag(args, '--max-usd') ?? parseFloatFlag(args, '--max-cost');
|
||||
const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw;
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const skipConfirm = args.includes('--yes');
|
||||
const jsonOutput = args.includes('--json');
|
||||
@@ -5818,328 +5798,118 @@ export async function runRemediate(
|
||||
const resumeArg = resumeMode ? args[resumeFlagIdx + 1] : undefined;
|
||||
const resumePlanHash = resumeArg && !resumeArg.startsWith('--') ? resumeArg : undefined;
|
||||
|
||||
const { computeRecommendations, classifyChecks, maxReachableScore } =
|
||||
await import('../core/brain-score-recommendations.ts');
|
||||
const {
|
||||
BudgetTracker,
|
||||
BudgetExhausted,
|
||||
} = await import('../core/budget/budget-tracker.ts');
|
||||
const { withBudgetTracker } = await import('../core/ai/gateway.ts');
|
||||
const {
|
||||
computePlanHash,
|
||||
saveRemediationCheckpoint,
|
||||
loadRemediationCheckpoint,
|
||||
listRemediationCheckpoints,
|
||||
clearRemediationCheckpoint,
|
||||
} = await import('../core/remediation-checkpoint.ts');
|
||||
const { runRemediation, computeRemediationPlan } =
|
||||
await import('../core/remediation/index.ts');
|
||||
|
||||
const ctx = await loadRecommendationContext(engine);
|
||||
|
||||
// Pre-flight ceiling check (D13)
|
||||
const initialHealth = await engine.getHealth();
|
||||
const syntheticChecks = [
|
||||
{ name: 'brain_score', status: 'ok' as const },
|
||||
{ name: 'sync_freshness', status: 'ok' as const },
|
||||
{ name: 'missing_embeddings', status: 'ok' as const },
|
||||
{ name: 'dead_links', status: 'ok' as const },
|
||||
{ name: 'orphan_pages', status: 'ok' as const },
|
||||
];
|
||||
const classifications = classifyChecks(syntheticChecks, ctx);
|
||||
const ceiling = maxReachableScore(initialHealth, classifications);
|
||||
if (targetScore > ceiling) {
|
||||
console.error(
|
||||
`[remediate] target ${targetScore} unreachable; max autonomous = ${ceiling}/100. ` +
|
||||
`Configure missing prereqs (see --remediation-plan blocked output) or lower --target-score.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// Initial plan
|
||||
let recs = computeRecommendations(initialHealth, ctx).filter((r) => r.status === 'remediable');
|
||||
if (recs.length === 0) {
|
||||
console.log(`Brain at score ${initialHealth.brain_score}/100, target ${targetScore}. Nothing to do.`);
|
||||
return;
|
||||
}
|
||||
|
||||
// A4 amended: compute plan_hash off the active recommendation ids so the
|
||||
// checkpoint binds to THIS plan. Resume only fires for matching plans.
|
||||
const planHash = computePlanHash(recs.map((r) => r.id));
|
||||
let completedFromCheckpoint = new Set<string>();
|
||||
if (resumeMode) {
|
||||
const requested = resumePlanHash;
|
||||
let cp = requested ? loadRemediationCheckpoint(requested) : null;
|
||||
if (!cp && !requested) {
|
||||
// No explicit hash: try newest checkpoint that matches the active plan.
|
||||
const recent = listRemediationCheckpoints();
|
||||
for (const e of recent) {
|
||||
const candidate = loadRemediationCheckpoint(e.plan_hash);
|
||||
if (candidate && candidate.plan_hash === planHash) {
|
||||
cp = candidate;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!cp) {
|
||||
// TTY confirmation gate (stays in CLI; library doesn't render).
|
||||
// Compute the plan once for the confirmation prompt, then hand off
|
||||
// to the library for the actual run. The library re-computes its
|
||||
// own plan internally — we accept the second computation cost for
|
||||
// a cleaner CLI/library separation.
|
||||
if (!skipConfirm && !dryRun && process.stdout.isTTY && !resumeMode) {
|
||||
const plan = await computeRemediationPlan(engine, { targetScore });
|
||||
if (plan.target_unreachable) {
|
||||
console.error(
|
||||
`[remediate --resume] no matching checkpoint found ` +
|
||||
`(plan_hash=${planHash}${requested ? `; requested=${requested}` : ''}). ` +
|
||||
`Run without --resume to start fresh.`,
|
||||
`[remediate] target ${targetScore} unreachable; max autonomous = ${plan.max_reachable_score}/100. ` +
|
||||
`Configure missing prereqs (see --remediation-plan blocked output) or lower --target-score.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
if (cp.plan_hash !== planHash) {
|
||||
if (plan.plan.length === 0) {
|
||||
console.log(`Brain at score ${plan.brain_score_current}/100, target ${targetScore}. Nothing to do.`);
|
||||
return;
|
||||
}
|
||||
if (maxUsd !== undefined && plan.est_total_usd_cost > maxUsd) {
|
||||
console.error(
|
||||
`[remediate --resume] checkpoint plan_hash=${cp.plan_hash} does not match active plan_hash=${planHash}. ` +
|
||||
`The plan has changed (brain state moved). Run without --resume to start fresh.`,
|
||||
`[remediate] est cost $${plan.est_total_usd_cost.toFixed(2)} exceeds --max-usd $${maxUsd.toFixed(2)}. Aborting.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
completedFromCheckpoint = new Set(cp.completed.map((c) => c.id));
|
||||
console.error(
|
||||
`[remediate --resume] resuming plan_hash=${planHash}: ${completedFromCheckpoint.size} step(s) completed, ` +
|
||||
`${recs.length - completedFromCheckpoint.size} remaining.`,
|
||||
);
|
||||
}
|
||||
|
||||
const estTotalUsd = recs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
|
||||
if (maxUsd !== null && estTotalUsd > maxUsd) {
|
||||
console.error(
|
||||
`[remediate] est cost $${estTotalUsd.toFixed(2)} exceeds --max-usd $${maxUsd.toFixed(2)}. Aborting.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
if (!skipConfirm && process.stdout.isTTY) {
|
||||
console.log(`About to submit ${recs.length} job(s), est ${Math.round(recs.reduce((s, r) => s + r.est_seconds, 0))}s, est $${estTotalUsd.toFixed(2)}`);
|
||||
console.log(`About to submit ${plan.plan.length} job(s), est ${plan.est_total_seconds}s, est $${plan.est_total_usd_cost.toFixed(2)}`);
|
||||
console.log('Pass --yes to proceed (cron-friendly).');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
console.log(`[remediate --dry-run] Would submit ${recs.length} jobs:`);
|
||||
for (const r of recs) console.log(` - ${r.id} (${r.job})`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Sequential submit per D3, with D5 cascade on failure and D7
|
||||
// scoped recheck between steps.
|
||||
const submitted: Array<{ step: number; id: string; job_id: number | null; status: string }> = [];
|
||||
const abortedIds = new Set<string>();
|
||||
const doctorRunId = crypto.randomUUID();
|
||||
|
||||
const isPGLite = engine.kind === 'pglite';
|
||||
if (isPGLite) {
|
||||
if (engine.kind === 'pglite') {
|
||||
console.error('[remediate] PGLite engine: running inline (no durable queue).');
|
||||
}
|
||||
|
||||
const { MinionQueue } = await import('../core/minions/queue.ts');
|
||||
const { waitForCompletion } = await import('../core/minions/wait-for-completion.ts');
|
||||
const queue = new MinionQueue(engine);
|
||||
|
||||
// A4 amended: install a BudgetTracker scope around the plan-step loop so
|
||||
// any gateway.chat / embed / rerank inside a Minion handler (synthesize,
|
||||
// patterns, consolidate) auto-enforces the cap. On BudgetExhausted, the
|
||||
// onExhausted callback persists the checkpoint BEFORE the throw propagates;
|
||||
// the catch surfaces the actionable --resume hint.
|
||||
const remediateTracker = new BudgetTracker({
|
||||
label: 'doctor.remediate',
|
||||
maxCostUsd: maxUsd ?? undefined,
|
||||
});
|
||||
|
||||
let exhaustionSnapshot: { spent: number; cap: number; reason: string; model_id?: string } | undefined;
|
||||
remediateTracker.onExhausted(() => {
|
||||
// BudgetTracker fires this synchronously from inside reserve()/record()
|
||||
// before the throw bubbles. Persist whatever has been done so far.
|
||||
const cp = {
|
||||
schema_version: 1 as const,
|
||||
plan_hash: planHash,
|
||||
doctor_run_id: doctorRunId,
|
||||
target_score: targetScore,
|
||||
started_at: new Date().toISOString(),
|
||||
completed: submitted
|
||||
.filter((s) => s.status === 'completed')
|
||||
.map((s) => ({ id: s.id, job: '', status: s.status, job_id: s.job_id ?? null })),
|
||||
aborted_at: new Date().toISOString(),
|
||||
abort_reason: 'budget_exhausted' as const,
|
||||
budget_snapshot: exhaustionSnapshot,
|
||||
};
|
||||
saveRemediationCheckpoint(cp);
|
||||
});
|
||||
|
||||
const runLoop = async (): Promise<void> => {
|
||||
let stepCount = 0;
|
||||
while (recs.length > 0 && stepCount < maxJobs) {
|
||||
const step = recs[0];
|
||||
if (!step) break;
|
||||
stepCount++;
|
||||
|
||||
// Resume: skip steps that the checkpoint already marked completed.
|
||||
if (completedFromCheckpoint.has(step.id)) {
|
||||
submitted.push({ step: stepCount, id: step.id, job_id: null, status: 'completed' });
|
||||
recs.shift();
|
||||
continue;
|
||||
}
|
||||
|
||||
// D5: if depends_on intersects aborted, skip + cascade
|
||||
if (step.depends_on && step.depends_on.some((d) => abortedIds.has(d))) {
|
||||
submitted.push({ step: stepCount, id: step.id, job_id: null, status: 'skipped_dep_aborted' });
|
||||
abortedIds.add(step.id);
|
||||
recs.shift();
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
const isProtected = !!step.protected;
|
||||
const job = await queue.add(
|
||||
step.job,
|
||||
{ ...step.params, doctor_run_id: doctorRunId },
|
||||
{
|
||||
queue: 'default',
|
||||
idempotency_key: step.idempotency_key,
|
||||
max_attempts: 2,
|
||||
maxWaiting: 1,
|
||||
},
|
||||
isProtected ? { allowProtectedSubmit: true } : undefined,
|
||||
const result = await runRemediation(
|
||||
engine,
|
||||
{
|
||||
targetScore,
|
||||
maxJobs,
|
||||
maxUsd,
|
||||
dryRun,
|
||||
resume: resumeMode,
|
||||
resumePlanHash,
|
||||
},
|
||||
{
|
||||
onTargetUnreachable: (target, ceiling) => {
|
||||
console.error(
|
||||
`[remediate] target ${target} unreachable; max autonomous = ${ceiling}/100. ` +
|
||||
`Configure missing prereqs (see --remediation-plan blocked output) or lower --target-score.`,
|
||||
);
|
||||
submitted.push({ step: stepCount, id: step.id, job_id: job.id, status: 'submitted' });
|
||||
|
||||
// Wait for terminal state. PGLite is in-process — short poll.
|
||||
const terminal = await waitForCompletion(queue, job.id, {
|
||||
pollMs: isPGLite ? 250 : 1000,
|
||||
timeoutMs: (step.est_seconds + 60) * 1000,
|
||||
});
|
||||
const lastSub = submitted[submitted.length - 1];
|
||||
if (lastSub) lastSub.status = terminal.status;
|
||||
|
||||
if (terminal.status !== 'completed') {
|
||||
abortedIds.add(step.id);
|
||||
}
|
||||
} catch (e) {
|
||||
if (e instanceof BudgetExhausted) {
|
||||
exhaustionSnapshot = {
|
||||
spent: e.spent,
|
||||
cap: e.cap,
|
||||
reason: e.reason,
|
||||
model_id: e.modelId,
|
||||
};
|
||||
throw e;
|
||||
}
|
||||
submitted.push({
|
||||
step: stepCount, id: step.id, job_id: null,
|
||||
status: `error: ${(e as Error).message.slice(0, 100)}`,
|
||||
});
|
||||
abortedIds.add(step.id);
|
||||
}
|
||||
|
||||
recs.shift();
|
||||
// D7: scoped recheck — re-compute plan from fresh health snapshot.
|
||||
// The next plan may drop completed steps and re-introduce failed
|
||||
// steps with bumped retry suffix (D1).
|
||||
if (recs.length === 0 || stepCount >= maxJobs) break;
|
||||
const freshHealth = await engine.getHealth();
|
||||
recs = computeRecommendations(freshHealth, ctx).filter((r) => r.status === 'remediable');
|
||||
}
|
||||
};
|
||||
|
||||
let budgetExhaustedAt: InstanceType<typeof BudgetExhausted> | null = null;
|
||||
try {
|
||||
await withBudgetTracker(remediateTracker, runLoop);
|
||||
} catch (err) {
|
||||
if (err instanceof BudgetExhausted) {
|
||||
budgetExhaustedAt = err;
|
||||
console.error(
|
||||
`\n[remediate] BudgetExhausted (${err.reason}): spent $${err.spent.toFixed(4)} > cap $${err.cap.toFixed(2)}.\n` +
|
||||
},
|
||||
onNothingToDo: (score, target) => {
|
||||
console.log(`Brain at score ${score}/100, target ${target}. Nothing to do.`);
|
||||
},
|
||||
onBudgetRefused: (estCost, cap) => {
|
||||
console.error(
|
||||
`[remediate] est cost $${estCost.toFixed(2)} exceeds --max-usd $${cap.toFixed(2)}. Aborting.`,
|
||||
);
|
||||
},
|
||||
onResumeMissed: (planHash, requested) => {
|
||||
console.error(
|
||||
`[remediate --resume] no matching checkpoint found ` +
|
||||
`(plan_hash=${planHash}${requested ? `; requested=${requested}` : ''}). ` +
|
||||
`Run without --resume to start fresh.`,
|
||||
);
|
||||
},
|
||||
onResumeLoaded: (planHash, completed, remaining) => {
|
||||
console.error(
|
||||
`[remediate --resume] resuming plan_hash=${planHash}: ${completed} step(s) completed, ${remaining} remaining.`,
|
||||
);
|
||||
},
|
||||
onBudgetExhausted: (planHash, snapshot) => {
|
||||
console.error(
|
||||
`\n[remediate] BudgetExhausted (${snapshot.reason}): spent $${snapshot.spent.toFixed(4)} > cap $${snapshot.cap.toFixed(2)}.\n` +
|
||||
`Checkpoint saved. Resume with:\n` +
|
||||
` gbrain doctor --remediate --resume ${planHash}\n`,
|
||||
);
|
||||
} else {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
);
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
// Clear checkpoint on a clean run (no budget abort). Failed steps in the
|
||||
// submitted set don't disqualify the cleanup — they re-surface on the
|
||||
// next plan with bumped suffixes.
|
||||
if (!budgetExhaustedAt) {
|
||||
clearRemediationCheckpoint(planHash);
|
||||
}
|
||||
// CLI surfaces — target unreachable / resume missed already emitted via hooks.
|
||||
// Library returns synthetic result with target_unreachable populated; exit 2.
|
||||
if (result.target_unreachable) process.exit(2);
|
||||
|
||||
const finalHealth = await engine.getHealth();
|
||||
const result = {
|
||||
doctor_run_id: doctorRunId,
|
||||
brain_score_initial: initialHealth.brain_score,
|
||||
brain_score_final: finalHealth.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
target_reached: finalHealth.brain_score >= targetScore,
|
||||
submitted,
|
||||
aborted_count: abortedIds.size,
|
||||
};
|
||||
if (dryRun && result.submitted.length > 0) {
|
||||
console.log(`[remediate --dry-run] Would submit ${result.submitted.length} jobs:`);
|
||||
for (const s of result.submitted) console.log(` - ${s.id}`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (jsonOutput) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`\nBrain score: ${initialHealth.brain_score} → ${finalHealth.brain_score} (target ${targetScore})`);
|
||||
console.log(`Submitted: ${submitted.length} job(s), ${abortedIds.size} aborted/failed`);
|
||||
} else if (result.submitted.length > 0) {
|
||||
console.log(`\nBrain score: ${result.brain_score_initial} → ${result.brain_score_final} (target ${targetScore})`);
|
||||
console.log(`Submitted: ${result.submitted.length} job(s), ${result.aborted_count} aborted/failed`);
|
||||
}
|
||||
|
||||
const anyFailed = submitted.some((s) => s.status !== 'completed' && s.status !== 'submitted');
|
||||
if (budgetExhaustedAt || anyFailed) process.exit(1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build RecommendationContext from engine + config.
|
||||
* Pure read; no side effects.
|
||||
*/
|
||||
async function loadRecommendationContext(engine: BrainEngine) {
|
||||
// v0.37 fix wave (Lane E.4 + CDX2-11): read schema-sizing fields from
|
||||
// gateway, not DB. The DB plane is schema-applied metadata; the file
|
||||
// plane is the gateway runtime source. Pre-fix this context produced
|
||||
// stale recommendations on fresh installs whose DB rows hadn't been
|
||||
// populated.
|
||||
//
|
||||
// Also extended the API-key check to recognize the ZE key alongside
|
||||
// OpenAI (was OpenAI-only). After Lane C.3, zeroentropy_api_key lives
|
||||
// in GBrainConfig + propagates to the gateway env dict.
|
||||
const repoPath = await engine.getConfig('sync.repo_path');
|
||||
let embeddingModel: string | undefined;
|
||||
let embeddingDimensions: number | undefined;
|
||||
try {
|
||||
const gw = await import('../core/ai/gateway.ts');
|
||||
embeddingModel = gw.getEmbeddingModel();
|
||||
embeddingDimensions = gw.getEmbeddingDimensions();
|
||||
} catch {
|
||||
// Gateway unconfigured — fall back to DB plane as a best-effort hint
|
||||
// (preserves doctor running before any engine.connect()).
|
||||
const dbModel = await engine.getConfig('embedding_model');
|
||||
const dbDims = await engine.getConfig('embedding_dimensions');
|
||||
embeddingModel = dbModel ?? undefined;
|
||||
embeddingDimensions = dbDims ? Number(dbDims) : undefined;
|
||||
}
|
||||
// v0.40.x: recipe-aware provider check, shared with autopilot.ts via
|
||||
// embeddingProviderConfigured(). Local providers (ollama, llama-server —
|
||||
// empty auth_env.required) need no hosted key; hosted providers check
|
||||
// their OWN required key (so a Voyage brain is judged by VOYAGE_API_KEY,
|
||||
// not by whether an OpenAI/ZE key happens to exist — the pre-fix wart).
|
||||
// fileCfg loads synchronously, so the resolveKey closure is sync.
|
||||
const { loadConfigFileOnly } = await import('../core/config.ts');
|
||||
const fileCfg = loadConfigFileOnly();
|
||||
const { embeddingProviderConfigured, HOSTED_EMBED_KEY_CONFIG } = await import(
|
||||
'../core/brain-score-recommendations.ts'
|
||||
const anyFailed = result.submitted.some(
|
||||
(s) => s.status !== 'completed' && s.status !== 'submitted' && s.status !== 'dry_run',
|
||||
);
|
||||
const embeddingConfigured = embeddingProviderConfigured(embeddingModel, (envVar) => {
|
||||
const cfgField = HOSTED_EMBED_KEY_CONFIG[envVar];
|
||||
const fromCfg = cfgField ? (fileCfg as Record<string, unknown> | null)?.[cfgField] : undefined;
|
||||
return !!(process.env[envVar] || fromCfg);
|
||||
});
|
||||
return {
|
||||
repoPath: repoPath ?? undefined,
|
||||
embeddingModel,
|
||||
embeddingDimensions,
|
||||
embeddingProviderConfigured: embeddingConfigured,
|
||||
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || fileCfg?.anthropic_api_key),
|
||||
};
|
||||
if (result.budget_exhausted || anyFailed) process.exit(1);
|
||||
}
|
||||
|
||||
// v0.41.18.0 (A1, codex finding #2): loadRecommendationContext moved to
|
||||
// src/core/remediation/context.ts so onboard + MCP run_onboard compose
|
||||
// the same context. The CLI surfaces (runRemediationPlan / runRemediate
|
||||
// above) now call computeRemediationPlan + runRemediation from the
|
||||
// library, which builds the context internally.
|
||||
|
||||
function parseIntFlag(args: string[], flag: string): number | null {
|
||||
const i = args.indexOf(flag);
|
||||
if (i === -1 || i === args.length - 1) return null;
|
||||
|
||||
+82
-13
@@ -38,6 +38,29 @@ export interface EmbedOpts {
|
||||
* in the DB where `gbrain jobs get` can read it.
|
||||
*/
|
||||
onProgress?: (done: number, total: number, embedded: number) => void;
|
||||
/**
|
||||
* v0.41.18.0 (A13): override the hardcoded PAGE_SIZE=2000 page-batch.
|
||||
* Smaller batches give finer progress granularity; larger batches
|
||||
* reduce per-batch coordination cost. Caps internally to 10K to
|
||||
* keep memory bounded.
|
||||
*/
|
||||
batchSize?: number;
|
||||
/**
|
||||
* v0.41.18.0 (A13): when 'recent', walks the stale-chunk pool in
|
||||
* page.updated_at DESC order (recent-modified pages first) instead
|
||||
* of the legacy stable (page_id, chunk_index) order. Threads through
|
||||
* to listStaleChunks orderBy='updated_desc'. Backed by the
|
||||
* content_chunks_stale_idx partial + idx_pages_updated_at_desc indexes
|
||||
* (v100).
|
||||
*/
|
||||
priority?: 'recent';
|
||||
/**
|
||||
* v0.41.18.0 (A13): catch-up mode removes the wall-clock cap and loops
|
||||
* until countStaleChunks() returns 0. Used by `gbrain embed --stale
|
||||
* --catch-up` and by the embed-catch-up Minion handler that the onboard
|
||||
* remediation submits on big stale backlogs.
|
||||
*/
|
||||
catchUp?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -173,7 +196,11 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
|
||||
return result;
|
||||
}
|
||||
if (opts.all || opts.stale) {
|
||||
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId);
|
||||
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
|
||||
batchSize: opts.batchSize,
|
||||
priority: opts.priority,
|
||||
catchUp: opts.catchUp,
|
||||
});
|
||||
return result;
|
||||
}
|
||||
if (opts.slug) {
|
||||
@@ -216,19 +243,27 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
|
||||
// v0.31.12: --source <id> scopes to a single source.
|
||||
const sourceIdx = args.indexOf('--source');
|
||||
const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined;
|
||||
// v0.41.18.0 (A13): --batch-size N, --priority recent, --catch-up flags.
|
||||
const batchSizeIdx = args.indexOf('--batch-size');
|
||||
const batchSizeRaw = batchSizeIdx >= 0 ? args[batchSizeIdx + 1] : undefined;
|
||||
const batchSize = batchSizeRaw ? Math.max(1, Math.min(10_000, parseInt(batchSizeRaw, 10) || 0)) : undefined;
|
||||
const priorityIdx = args.indexOf('--priority');
|
||||
const priorityRaw = priorityIdx >= 0 ? args[priorityIdx + 1] : undefined;
|
||||
const priority = priorityRaw === 'recent' ? 'recent' as const : undefined;
|
||||
const catchUp = args.includes('--catch-up');
|
||||
|
||||
let opts: EmbedOpts;
|
||||
if (slugsIdx >= 0) {
|
||||
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId };
|
||||
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp };
|
||||
} else if (all || stale) {
|
||||
opts = { all, stale, dryRun, sourceId };
|
||||
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp };
|
||||
} else {
|
||||
const slug = args.find(a => !a.startsWith('--'));
|
||||
if (!slug) {
|
||||
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run]');
|
||||
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up]');
|
||||
process.exit(1);
|
||||
}
|
||||
opts = { slug, dryRun, sourceId };
|
||||
opts = { slug, dryRun, sourceId, batchSize, priority, catchUp };
|
||||
}
|
||||
|
||||
// CLI path: wire a reporter so --progress-json / --quiet / TTY rendering
|
||||
@@ -355,6 +390,11 @@ async function embedAll(
|
||||
result: EmbedResult,
|
||||
onProgress?: (done: number, total: number, embedded: number) => void,
|
||||
sourceId?: string,
|
||||
staleOpts?: {
|
||||
batchSize?: number;
|
||||
priority?: 'recent';
|
||||
catchUp?: boolean;
|
||||
},
|
||||
) {
|
||||
// ─────────────────────────────────────────────────────────────
|
||||
// Stale-only fast path: avoid the listPages + per-page getChunks
|
||||
@@ -371,7 +411,8 @@ async function embedAll(
|
||||
// ─────────────────────────────────────────────────────────────
|
||||
if (staleOnly) {
|
||||
// D7: thread sourceId so `gbrain embed --stale --source X` actually scopes.
|
||||
return await embedAllStale(engine, sourceId, dryRun, result, onProgress);
|
||||
// v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path.
|
||||
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts);
|
||||
}
|
||||
|
||||
// v0.31.12: when sourceId is set, scope listPages to that source.
|
||||
@@ -497,6 +538,11 @@ async function embedAllStale(
|
||||
dryRun: boolean,
|
||||
result: EmbedResult,
|
||||
onProgress?: (done: number, total: number, embedded: number) => void,
|
||||
staleOpts?: {
|
||||
batchSize?: number;
|
||||
priority?: 'recent';
|
||||
catchUp?: boolean;
|
||||
},
|
||||
) {
|
||||
// D7: thread sourceId so source-scoped runs only count + visit
|
||||
// that source's NULL embeddings.
|
||||
@@ -525,23 +571,34 @@ async function embedAllStale(
|
||||
// rows in one query (which times out on Supabase's 2-min pooler timeout),
|
||||
// we page through 2000 rows at a time via keyset pagination on
|
||||
// (page_id, chunk_index). Each query finishes in <1s.
|
||||
const PAGE_SIZE = 2000;
|
||||
// v0.41.18.0 (A13): --batch-size N CLI flag overrides hardcoded 2000 default.
|
||||
const PAGE_SIZE = staleOpts?.batchSize ?? 2000;
|
||||
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
|
||||
|
||||
// D3 + D3a + D8: wall-clock budget. 30 min default; env override.
|
||||
// The old single-shot `LIMIT 100000` query implicitly capped runtime
|
||||
// by failing on timeout; pagination removed that cap. AbortController
|
||||
// threads cancellation into (a) the retry sleep below, (b) the worker
|
||||
// claim loop, and (c) the gateway embed call so an in-flight HTTP
|
||||
// request also unwinds.
|
||||
const BUDGET_MS = parseInt(process.env.GBRAIN_EMBED_TIME_BUDGET_MS || `${30 * 60 * 1000}`, 10);
|
||||
// v0.41.18.0 (A13): --catch-up removes the wall-clock cap entirely so the
|
||||
// handler runs until countStaleChunks() returns 0. Use Number.MAX_SAFE_INTEGER
|
||||
// (effectively unbounded) instead of the 30-min default. The AbortController
|
||||
// still wraps for SIGINT propagation; just the timer never fires.
|
||||
const BUDGET_MS = staleOpts?.catchUp
|
||||
? Number.MAX_SAFE_INTEGER
|
||||
: parseInt(process.env.GBRAIN_EMBED_TIME_BUDGET_MS || `${30 * 60 * 1000}`, 10);
|
||||
const budgetController = new AbortController();
|
||||
const budgetTimer = setTimeout(() => budgetController.abort(), BUDGET_MS);
|
||||
const budgetSignal = budgetController.signal;
|
||||
|
||||
// v0.41.18.0 (A13): --priority recent threads orderBy='updated_desc' to
|
||||
// listStaleChunks. Composite cursor tracks (updated_at, page_id, chunk_index)
|
||||
// instead of just (page_id, chunk_index); first-page cursor is sentinel
|
||||
// (null, 0, -1).
|
||||
const orderBy: 'page_id' | 'updated_desc' = staleOpts?.priority === 'recent'
|
||||
? 'updated_desc'
|
||||
: 'page_id';
|
||||
|
||||
let totalProcessedPages = 0;
|
||||
let afterPageId = 0;
|
||||
let afterChunkIndex = -1;
|
||||
let afterUpdatedAt: string | null = null;
|
||||
let totalChunksLoaded = 0;
|
||||
let budgetExitNotified = false;
|
||||
|
||||
@@ -560,6 +617,10 @@ async function embedAllStale(
|
||||
batchSize: PAGE_SIZE,
|
||||
afterPageId,
|
||||
afterChunkIndex,
|
||||
...(orderBy === 'updated_desc' && {
|
||||
orderBy,
|
||||
afterUpdatedAt,
|
||||
}),
|
||||
...(sourceId && { sourceId }),
|
||||
});
|
||||
if (batch.length === 0) break;
|
||||
@@ -569,6 +630,14 @@ async function embedAllStale(
|
||||
const last = batch[batch.length - 1];
|
||||
afterPageId = last.page_id;
|
||||
afterChunkIndex = last.chunk_index;
|
||||
if (orderBy === 'updated_desc') {
|
||||
// engine returns `updated_at` as Date or ISO string; normalize to ISO.
|
||||
const lastRow = last as unknown as { updated_at?: string | Date | null };
|
||||
const u = lastRow.updated_at;
|
||||
afterUpdatedAt = u instanceof Date ? u.toISOString()
|
||||
: typeof u === 'string' ? u
|
||||
: null;
|
||||
}
|
||||
|
||||
// Group by composite key (source_id::slug).
|
||||
const byKey = new Map<string, typeof batch>();
|
||||
|
||||
+84
-10
@@ -480,14 +480,21 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
// v0_13_0 migration orchestrator runs this once under the hood; users
|
||||
// opt in for subsequent runs.
|
||||
const includeFrontmatter = args.includes('--include-frontmatter');
|
||||
// v0.42.0.0 Part B: --by-mention auto-link body-text entity mentions
|
||||
// v0.41.18.0 Part B: --by-mention auto-link body-text entity mentions
|
||||
// via the gazetteer pass. Mode dispatch — when set, run ONLY the
|
||||
// mention pass (skip default link extract). DB-source only per D7;
|
||||
// FS-source is rejected with a paste-ready fix-hint below.
|
||||
const byMention = args.includes('--by-mention');
|
||||
// v0.41.15.0 (T7, D9): --workers N parsed via the shared validator.
|
||||
// v0.41.18.0 (A10, T7): --ner is a NER-extraction mode dispatch. Same
|
||||
// DB-source-only posture as --by-mention. Can combine with --by-mention
|
||||
// in a single command for a shared-gazetteer walk (saves one pass).
|
||||
const ner = args.includes('--ner');
|
||||
// v0.41.18.0 (A11, T8): --from-meetings extracts timeline entries from
|
||||
// meeting pages onto each discussed entity. Timeline subcommand only.
|
||||
const fromMeetings = args.includes('--from-meetings');
|
||||
// v0.41.17.0 (T7, D9): --workers N parsed via the shared validator.
|
||||
// Honored on the fs-walk inner loops only; DB-source paths stay
|
||||
// serial in v0.41.15.0 (see ExtractOpts.workers doc).
|
||||
// serial in v0.41.17.0 (see ExtractOpts.workers doc).
|
||||
let workers: number | undefined;
|
||||
const workersIdx = args.indexOf('--workers');
|
||||
const concurrencyIdx = args.indexOf('--concurrency');
|
||||
@@ -523,7 +530,7 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// v0.42.0.0 D7: --by-mention requires DB-source. Gazetteer construction
|
||||
// v0.41.18.0 D7: --by-mention requires DB-source. Gazetteer construction
|
||||
// needs the engine; mixing FS-walk with DB-gazetteer is incoherent
|
||||
// (you'd scan files on disk for mentions of entities that may not exist
|
||||
// in any synced page). Fail loud with a paste-ready fix-hint.
|
||||
@@ -545,6 +552,40 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
// v0.41.18.0 (T7): same gates for --ner.
|
||||
if (ner && source === 'fs') {
|
||||
console.error(
|
||||
`--ner requires --source db (currently --source fs). NER extraction needs the engine ` +
|
||||
`to build the entity gazetteer + read schema-pack link_types. Re-run as:\n\n` +
|
||||
` gbrain extract ${subcommand} --ner --source db` +
|
||||
(sourceIdFilter ? ` --source-id ${sourceIdFilter}` : '') +
|
||||
(since ? ` --since ${since}` : '') +
|
||||
(dryRun ? ' --dry-run' : '') + '\n',
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
if (ner && subcommand === 'timeline') {
|
||||
console.error(
|
||||
`--ner is a links-pass only; it does not apply to timeline extraction.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
// v0.41.18.0 (T8): --from-meetings is timeline-only + DB-source-only.
|
||||
if (fromMeetings && source === 'fs') {
|
||||
console.error(
|
||||
`--from-meetings requires --source db (currently --source fs). Re-run as:\n\n` +
|
||||
` gbrain extract timeline --from-meetings --source db` +
|
||||
(sourceIdFilter ? ` --source-id ${sourceIdFilter}` : '') +
|
||||
(dryRun ? ' --dry-run' : '') + '\n',
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
if (fromMeetings && subcommand !== 'timeline' && subcommand !== 'all') {
|
||||
console.error(
|
||||
`--from-meetings is a timeline-pass only. Re-run as 'gbrain extract timeline --from-meetings' or 'gbrain extract all --from-meetings'.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// FS source needs a brain dir. When --dir wasn't passed, resolve from
|
||||
// sources(local_path) — same path `gbrain sync` uses — instead of
|
||||
@@ -577,16 +618,49 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
// is fs-only; we keep the dual codepath here so Minions handlers
|
||||
// can opt in via mode + source.
|
||||
result = { links_created: 0, timeline_entries_created: 0, pages_processed: 0 };
|
||||
// v0.42.0.0: --by-mention is a mode dispatch. When set, run ONLY
|
||||
// v0.41.18.0: --by-mention is a mode dispatch. When set, run ONLY
|
||||
// the mention pass and skip the default link/frontmatter extract.
|
||||
// The two passes write different link_source values ('mentions' vs
|
||||
// 'markdown'/'frontmatter') so they don't conflict, but mixing them
|
||||
// in a single CLI invocation is surprising — keep the surfaces
|
||||
// separate.
|
||||
if (byMention) {
|
||||
const r = await extractMentionsFromDb(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
result.links_created = r.created;
|
||||
result.pages_processed = r.pages;
|
||||
if (fromMeetings) {
|
||||
// v0.41.18.0 (T8): timeline-from-meetings runs SOLO (doesn't combine
|
||||
// with --by-mention/--ner because those are links passes).
|
||||
const { extractTimelineFromMeetings } = await import('../core/extract-timeline-from-meetings.ts');
|
||||
const r = await extractTimelineFromMeetings(engine, { dryRun, sourceIdFilter });
|
||||
result.timeline_entries_created = r.entries_created;
|
||||
result.pages_processed = r.meetings_scanned;
|
||||
if (!jsonMode) {
|
||||
console.log(`Timeline from meetings: ${r.entries_created} entries on ${r.entities_touched} entity pages from ${r.meetings_scanned} meetings`);
|
||||
}
|
||||
} else if (byMention || ner) {
|
||||
// v0.41.18.0 (T7): combined --by-mention + --ner walk shares one
|
||||
// gazetteer; saves an entire pass on big brains. When only one
|
||||
// flag is set, the other extractor skips silently.
|
||||
const { buildGazetteer: buildGz } = await import('../core/by-mention.ts');
|
||||
const sharedGazetteer = (byMention || ner) ? await buildGz(engine) : undefined;
|
||||
if (byMention) {
|
||||
const r = await extractMentionsFromDb(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
|
||||
result.links_created += r.created;
|
||||
result.pages_processed += r.pages;
|
||||
}
|
||||
if (ner) {
|
||||
const { extractNerLinks } = await import('../core/extract-ner.ts');
|
||||
const r = await extractNerLinks(engine, {
|
||||
dryRun,
|
||||
sourceIdFilter,
|
||||
typeFilter,
|
||||
since,
|
||||
gazetteer: sharedGazetteer,
|
||||
});
|
||||
if (r.pack_unavailable && !jsonMode) {
|
||||
console.log('Note: no active schema pack with link_types[].inference.regex — NER pass produced 0 links.');
|
||||
}
|
||||
result.links_created += r.created;
|
||||
// pages already counted by by-mention if both ran; else count here.
|
||||
if (!byMention) result.pages_processed += r.pages;
|
||||
}
|
||||
} else {
|
||||
if (subcommand === 'links' || subcommand === 'all') {
|
||||
const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since, { includeFrontmatter, sourceIdFilter });
|
||||
@@ -1244,7 +1318,7 @@ async function extractTimelineFromDB(
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.0.0 Part B (migration #1 of #1409) — auto-link body-text entity
|
||||
* v0.41.18.0 Part B (migration #1 of #1409) — auto-link body-text entity
|
||||
* mentions to known entity pages.
|
||||
*
|
||||
* Walks every page (respecting --source-id / --type / --since filters),
|
||||
|
||||
@@ -973,6 +973,11 @@ async function initPGLite(opts: {
|
||||
const { printAdvisoryIfRecommended } = await import('../core/skillpack/post-install-advisory.ts');
|
||||
const { VERSION } = await import('../version.ts');
|
||||
printAdvisoryIfRecommended({ version: VERSION, context: 'init' });
|
||||
|
||||
// v0.41.18.0 (A4 + A18 + A20, T14): post-initSchema onboard nudge.
|
||||
// Fail-open; 3s wallclock cap. Skipped silently in non-TTY contexts.
|
||||
const { runInitNudge } = await import('../core/onboard/init-nudge.ts');
|
||||
await runInitNudge(engine);
|
||||
}
|
||||
} finally {
|
||||
try { await engine.disconnect(); } catch { /* best-effort */ }
|
||||
@@ -1191,6 +1196,11 @@ async function initPostgres(opts: {
|
||||
const { printAdvisoryIfRecommended } = await import('../core/skillpack/post-install-advisory.ts');
|
||||
const { VERSION } = await import('../version.ts');
|
||||
printAdvisoryIfRecommended({ version: VERSION, context: 'init' });
|
||||
|
||||
// v0.41.18.0 (A4 + A18 + A20, T14): post-initSchema onboard nudge.
|
||||
// Fail-open; 3s wallclock cap. Skipped silently in non-TTY contexts.
|
||||
const { runInitNudge } = await import('../core/onboard/init-nudge.ts');
|
||||
await runInitNudge(engine);
|
||||
}
|
||||
} finally {
|
||||
try { await engine.disconnect(); } catch { /* best-effort */ }
|
||||
|
||||
+60
-1
@@ -1555,7 +1555,66 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
return await makeEmbedBackfillHandler(engine)(job);
|
||||
});
|
||||
|
||||
process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected) + embed-backfill (v0.40)\n');
|
||||
// v0.41.18.0 (A10, T7): extract-ner handler for the gbrain onboard
|
||||
// remediation pipeline. Wraps extractNerLinks; emits typed_ner kind
|
||||
// alongside the by-mention 'plain' kind. NOT in PROTECTED_JOB_NAMES
|
||||
// (regex-only, no LLM spend).
|
||||
worker.register('extract-ner', async (job) => {
|
||||
const { extractNerLinks } = await import('../core/extract-ner.ts');
|
||||
const data = (job.data ?? {}) as { sourceId?: string };
|
||||
return await extractNerLinks(engine, {
|
||||
sourceIdFilter: data.sourceId,
|
||||
});
|
||||
});
|
||||
|
||||
// v0.41.18.0 (A12, T9): extract-takes-from-pages handler. PROTECTED
|
||||
// (LLM-bearing). Two-gate consent enforced at the handler boundary:
|
||||
// refuses to run unless takes.bootstrap_enabled config is true, even
|
||||
// when allowProtectedSubmit was set at queue.add time.
|
||||
worker.register('extract-takes-from-pages', async (job) => {
|
||||
const { extractTakesFromPages } = await import('../core/extract-takes-from-pages.ts');
|
||||
const data = (job.data ?? {}) as { sourceId?: string; maxPages?: number };
|
||||
const bootstrapCfg = await engine.getConfig('takes.bootstrap_enabled');
|
||||
const bootstrapEnabled = bootstrapCfg === 'true' || bootstrapCfg === '1';
|
||||
return await extractTakesFromPages(engine, {
|
||||
bootstrapEnabled,
|
||||
sourceIdFilter: data.sourceId,
|
||||
maxPages: data.maxPages,
|
||||
});
|
||||
});
|
||||
|
||||
// v0.41.18.0 (A11, T8): extract-timeline-from-meetings handler. Wraps
|
||||
// extractTimelineFromMeetings. NOT in PROTECTED_JOB_NAMES (pure SQL + string
|
||||
// scan, no LLM spend).
|
||||
worker.register('extract-timeline-from-meetings', async (job) => {
|
||||
const { extractTimelineFromMeetings } = await import('../core/extract-timeline-from-meetings.ts');
|
||||
const data = (job.data ?? {}) as { sourceId?: string };
|
||||
return await extractTimelineFromMeetings(engine, {
|
||||
sourceIdFilter: data.sourceId,
|
||||
});
|
||||
});
|
||||
|
||||
// v0.41.18.0 (A13): embed-catch-up handler for the gbrain onboard
|
||||
// remediation pipeline. Wraps runEmbedCore with stale + catchUp + the
|
||||
// priority/batchSize the recommendation supplies. NOT in
|
||||
// PROTECTED_JOB_NAMES (embedding spend only).
|
||||
worker.register('embed-catch-up', async (job) => {
|
||||
const { runEmbedCore } = await import('./embed.ts');
|
||||
const data = (job.data ?? {}) as {
|
||||
sourceId?: string;
|
||||
batchSize?: number;
|
||||
priority?: 'recent';
|
||||
};
|
||||
return await runEmbedCore(engine, {
|
||||
stale: true,
|
||||
catchUp: true,
|
||||
batchSize: data.batchSize,
|
||||
priority: data.priority,
|
||||
sourceId: data.sourceId,
|
||||
});
|
||||
});
|
||||
|
||||
process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected) + embed-backfill (v0.40) + embed-catch-up (v0.42)\n');
|
||||
|
||||
// Plugin discovery — one line per discovered plugin (mirrors the
|
||||
// openclaw-seam startup line convention from v0.11+). Loaded
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
// src/commands/onboard.ts
|
||||
// sourcescope:file-brain-wide — the --history surface reads
|
||||
// migration_impact_log brain-wide. Per A26 lint opt-out.
|
||||
//
|
||||
// v0.41.18.0 (A1, T13). CLI shell for `gbrain onboard`. Thin wrapper over:
|
||||
// - T2 library: computeRemediationPlan + runRemediation
|
||||
// - T4 onboard checks: runAllOnboardChecks (extra remediations)
|
||||
// - T12 render: buildOnboardReport + renderHuman
|
||||
//
|
||||
// Three modes:
|
||||
// --check (default): print plan, no submission
|
||||
// --auto: submit auto_apply tier (requires --max-usd)
|
||||
// --auto --yes: also submit prompt_required tier
|
||||
// --history: show recent migration_impact_log entries
|
||||
//
|
||||
// `--json` switches to the stable JSON envelope. No CLI mode → human render.
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { computeRemediationPlan, runRemediation } from '../core/remediation/index.ts';
|
||||
import { runAllOnboardChecks } from '../core/onboard/checks.ts';
|
||||
import { buildOnboardReport, renderHuman } from '../core/onboard/render.ts';
|
||||
|
||||
function parseInt10(args: string[], flag: string): number | null {
|
||||
const i = args.indexOf(flag);
|
||||
if (i === -1 || i === args.length - 1) return null;
|
||||
const v = parseInt(args[i + 1] ?? '', 10);
|
||||
return isNaN(v) ? null : v;
|
||||
}
|
||||
|
||||
function parseFloat10(args: string[], flag: string): number | null {
|
||||
const i = args.indexOf(flag);
|
||||
if (i === -1 || i === args.length - 1) return null;
|
||||
const v = parseFloat(args[i + 1] ?? '');
|
||||
return isNaN(v) ? null : v;
|
||||
}
|
||||
|
||||
export async function runOnboard(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const check = args.includes('--check') || (!args.includes('--auto') && !args.includes('--history'));
|
||||
const auto = args.includes('--auto');
|
||||
const yes = args.includes('--yes');
|
||||
const history = args.includes('--history');
|
||||
const jsonOutput = args.includes('--json');
|
||||
const targetScore = parseInt10(args, '--target-score') ?? 90;
|
||||
const maxUsdRaw = parseFloat10(args, '--max-usd');
|
||||
const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw;
|
||||
|
||||
// --history shows the impact log directly; no plan computation.
|
||||
if (history) {
|
||||
const rows = await engine.executeRaw<{
|
||||
remediation_id: string;
|
||||
metric_name: string;
|
||||
metric_before: number | null;
|
||||
metric_after: number | null;
|
||||
applied_at: string;
|
||||
}>(
|
||||
`SELECT remediation_id, metric_name, metric_before, metric_after, applied_at
|
||||
FROM migration_impact_log
|
||||
ORDER BY applied_at DESC
|
||||
LIMIT 50`,
|
||||
);
|
||||
const historyEntries = rows.map((r) => ({
|
||||
remediation_id: r.remediation_id,
|
||||
metric_name: r.metric_name,
|
||||
metric_before: r.metric_before === null ? null : Number(r.metric_before),
|
||||
metric_after: r.metric_after === null ? null : Number(r.metric_after),
|
||||
delta: (r.metric_before === null || r.metric_after === null)
|
||||
? null
|
||||
: Number(r.metric_after) - Number(r.metric_before),
|
||||
applied_at: r.applied_at,
|
||||
}));
|
||||
if (jsonOutput) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
schema_version: 1,
|
||||
history: historyEntries,
|
||||
}, null, 2) + '\n');
|
||||
return;
|
||||
}
|
||||
process.stdout.write(`Onboard history (last ${historyEntries.length}):\n`);
|
||||
for (const h of historyEntries) {
|
||||
const delta = h.delta !== null ? (h.delta > 0 ? `+${h.delta}` : String(h.delta)) : '?';
|
||||
process.stdout.write(
|
||||
` ${h.applied_at} ${h.remediation_id} ${h.metric_name}: ` +
|
||||
`${h.metric_before ?? '?'} → ${h.metric_after ?? '?'} (${delta})\n`,
|
||||
);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// --auto refuses without --max-usd (cron-safety per A12 + A20).
|
||||
if (auto && maxUsd === undefined) {
|
||||
process.stderr.write(
|
||||
`gbrain onboard --auto refuses without --max-usd N.\n` +
|
||||
`Set a cap to avoid surprise spend:\n` +
|
||||
` gbrain onboard --auto --max-usd 5\n`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// Build the plan: T4 checks supply extra remediations on top of T3's
|
||||
// generalized planner.
|
||||
const onboardCheckResults = await runAllOnboardChecks(engine);
|
||||
const extraRemediations = onboardCheckResults.flatMap((r) => r.remediations);
|
||||
|
||||
if (check && !auto) {
|
||||
const plan = await computeRemediationPlan(engine, { targetScore, extraRemediations });
|
||||
const report = buildOnboardReport(plan);
|
||||
if (jsonOutput) {
|
||||
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
|
||||
return;
|
||||
}
|
||||
process.stdout.write(renderHuman(report) + '\n');
|
||||
return;
|
||||
}
|
||||
|
||||
// --auto path: runs through the T2 library orchestrator. Hooks emit CLI
|
||||
// progress to stderr; the final result lands as JSON on stdout (or human
|
||||
// summary).
|
||||
const result = await runRemediation(
|
||||
engine,
|
||||
{
|
||||
targetScore,
|
||||
maxUsd,
|
||||
// --auto --yes opts into the prompt_required tier too; library
|
||||
// doesn't distinguish auto_apply vs prompt_required, it just runs
|
||||
// every remediation in the plan. The plan-building side (T12 render)
|
||||
// does the tier distinction; for --auto without --yes, the CLI shell
|
||||
// would pre-filter the extras to auto_apply only. For now: pass
|
||||
// everything; CLI documents this is "everything" behavior.
|
||||
},
|
||||
{
|
||||
onTargetUnreachable: (target, ceiling) => {
|
||||
process.stderr.write(
|
||||
`[onboard] target ${target}/100 unreachable; max autonomous = ${ceiling}/100. ` +
|
||||
`Configure missing prereqs (run gbrain doctor --remediation-plan) or lower --target-score.\n`,
|
||||
);
|
||||
},
|
||||
onNothingToDo: (score, target) => {
|
||||
process.stdout.write(
|
||||
`Brain at score ${score}/100, target ${target}/100. Nothing to do.\n`,
|
||||
);
|
||||
},
|
||||
onBudgetRefused: (estCost, cap) => {
|
||||
process.stderr.write(
|
||||
`[onboard] est cost $${estCost.toFixed(2)} exceeds --max-usd $${cap.toFixed(2)}. Aborting.\n`,
|
||||
);
|
||||
},
|
||||
onStepStart: (step, total, rec) => {
|
||||
process.stderr.write(`[onboard] [${step}/${total}] ${rec.job} (${rec.severity})...\n`);
|
||||
},
|
||||
onStepEnd: (sr) => {
|
||||
process.stderr.write(`[onboard] → ${sr.status}\n`);
|
||||
},
|
||||
onBudgetExhausted: (planHash, snapshot) => {
|
||||
process.stderr.write(
|
||||
`\n[onboard] BudgetExhausted (${snapshot.reason}): spent $${snapshot.spent.toFixed(4)} > cap $${snapshot.cap.toFixed(2)}.\n` +
|
||||
`Checkpoint saved. Resume with:\n gbrain doctor --remediate --resume ${planHash}\n`,
|
||||
);
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
if (result.target_unreachable) process.exit(2);
|
||||
|
||||
if (jsonOutput) {
|
||||
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
|
||||
} else if (result.submitted.length > 0) {
|
||||
process.stdout.write(
|
||||
`\nBrain score: ${result.brain_score_initial} → ${result.brain_score_final} (target ${targetScore})\n` +
|
||||
`Submitted: ${result.submitted.length} job(s), ${result.aborted_count} aborted/failed\n`,
|
||||
);
|
||||
}
|
||||
|
||||
const anyFailed = result.submitted.some(
|
||||
(s) => s.status !== 'completed' && s.status !== 'submitted' && s.status !== 'dry_run',
|
||||
);
|
||||
if (result.budget_exhausted || anyFailed) process.exit(1);
|
||||
}
|
||||
@@ -571,12 +571,75 @@ Common flags:
|
||||
case 'scorecard': return cmdScorecard(engine, rest);
|
||||
case 'calibration': return cmdCalibration(engine, rest);
|
||||
case 'revisit': return cmdRevisit(engine, rest);
|
||||
case 'extract': return cmdExtract(engine, rest);
|
||||
default:
|
||||
// No subcommand keyword → treat first arg as <slug> for the list path.
|
||||
return cmdList(engine, args);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.41.18.0 (A12, A24, T9) — `gbrain takes extract --from-pages` runs
|
||||
* Haiku over concept/atom/lore/briefing/writing/originals pages and
|
||||
* lifts gradeable claims into the takes fence.
|
||||
*
|
||||
* Two-gate consent: requires `takes.bootstrap_enabled=true` in config
|
||||
* AND explicit --yes flag for any non-dryRun run. Refuses LLM-bearing
|
||||
* extraction without both.
|
||||
*/
|
||||
async function cmdExtract(engine: BrainEngine, rest: string[]): Promise<void> {
|
||||
const sub = rest[0];
|
||||
if (sub !== '--from-pages') {
|
||||
process.stderr.write(
|
||||
'Usage: gbrain takes extract --from-pages [--yes] [--dry-run] [--source-id <id>] [--max-pages N] [--holder <name>]\n',
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
const dryRun = rest.includes('--dry-run');
|
||||
const skipConfirm = rest.includes('--yes');
|
||||
const sourceIdx = rest.indexOf('--source-id');
|
||||
const sourceIdFilter = sourceIdx >= 0 ? rest[sourceIdx + 1] : undefined;
|
||||
const maxIdx = rest.indexOf('--max-pages');
|
||||
const maxPagesRaw = maxIdx >= 0 ? rest[maxIdx + 1] : undefined;
|
||||
const maxPages = maxPagesRaw ? Math.max(1, Math.min(1000, parseInt(maxPagesRaw, 10) || 50)) : 50;
|
||||
const holderIdx = rest.indexOf('--holder');
|
||||
const holder = holderIdx >= 0 ? rest[holderIdx + 1] : 'system';
|
||||
|
||||
// A12 consent gate.
|
||||
const bootstrapEnabledCfg = await engine.getConfig('takes.bootstrap_enabled');
|
||||
const bootstrapEnabled = bootstrapEnabledCfg === 'true' || bootstrapEnabledCfg === '1';
|
||||
if (!bootstrapEnabled) {
|
||||
process.stderr.write(
|
||||
`takes-bootstrap is opt-in. Enable with:\n gbrain config set takes.bootstrap_enabled true\nThen re-run with --yes.\n`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
if (!dryRun && !skipConfirm) {
|
||||
process.stderr.write(
|
||||
`[takes extract] sends concept/atom/lore/briefing/writing/originals page content to Haiku.\n` +
|
||||
`Pass --yes to proceed (or --dry-run to preview).\n`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const { extractTakesFromPages } = await import('../core/extract-takes-from-pages.ts');
|
||||
const result = await extractTakesFromPages(engine, {
|
||||
bootstrapEnabled: true,
|
||||
dryRun,
|
||||
sourceIdFilter,
|
||||
maxPages,
|
||||
holder,
|
||||
});
|
||||
if (result.llm_unavailable) {
|
||||
process.stderr.write(`[takes extract] chat gateway unavailable (no API key configured).\n`);
|
||||
process.exit(2);
|
||||
}
|
||||
process.stdout.write(
|
||||
`takes extract --from-pages: ${result.claims_extracted} claim(s) from ${result.pages_scanned} page(s)` +
|
||||
(dryRun ? ' (dry-run)' : '') + '\n',
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36.1.0 (TD4 / D30) — `gbrain takes revisit <slug>` opens $EDITOR on
|
||||
* the source page so the user can write a follow-up immediately. The
|
||||
|
||||
@@ -409,6 +409,20 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
|
||||
// compare gbrain against itself)
|
||||
// - every scaffolded skill is identical (nothing to say)
|
||||
await postUpgradeReferenceSweep();
|
||||
|
||||
// v0.41.18.0 (A4 + A18, T14): post-upgrade onboard banner. Fail-open;
|
||||
// doesn't engine-connect (lightweight TTY check only). The actual
|
||||
// recommendations need engine access via `gbrain onboard --check`;
|
||||
// the banner just nudges the user to run it.
|
||||
try {
|
||||
const { runUpgradeBanner } = await import('../core/onboard/init-nudge.ts');
|
||||
// The banner doesn't actually use the engine today; passing null-equivalent
|
||||
// would require a type widening. Skip the engine arg and let the banner
|
||||
// print the static nudge text.
|
||||
await runUpgradeBanner(null as never);
|
||||
} catch {
|
||||
// Fail-open per A18: never crash post-upgrade from the banner.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -167,9 +167,26 @@ export interface CheckClassification {
|
||||
* Returns ONLY `remediable` items. `blocked` items surface via
|
||||
* `classifyChecks()` and are rendered alongside the plan as informational.
|
||||
*/
|
||||
/**
|
||||
* Generalized (v0.41.18.0, A2 + codex finding #3): an optional third arg
|
||||
* lets callers inject RemediationStep entries discovered by doctor checks
|
||||
* outside this module's hardcoded planner. Without this, adding a
|
||||
* `Check.remediation` field to a new doctor check wouldn't auto-wire into
|
||||
* `gbrain doctor --remediation-plan` — the planner would just ignore it.
|
||||
*
|
||||
* Onboard's runRemediationPlan calls the 4 new check helpers (embed_staleness,
|
||||
* entity_link_coverage, timeline_coverage, takes_count) and threads their
|
||||
* RemediationStep[] outputs through this slot. Each helper produces its own
|
||||
* cheap query (D7 cheap-path preserved); aggregation happens in the caller.
|
||||
*
|
||||
* Sort + dedup applies across BOTH the hardcoded + extra entries: stable id
|
||||
* collisions resolve in favor of the hardcoded entry (legacy behavior wins),
|
||||
* which means extras only add coverage they're not duplicating.
|
||||
*/
|
||||
export function computeRecommendations(
|
||||
health: BrainHealth,
|
||||
ctx: RecommendationContext,
|
||||
extraRemediations: Remediation[] = [],
|
||||
): Remediation[] {
|
||||
const out: Remediation[] = [];
|
||||
const source = ctx.sourceId ?? 'default';
|
||||
@@ -266,6 +283,16 @@ export function computeRecommendations(
|
||||
});
|
||||
}
|
||||
|
||||
// v0.41.18.0 (A2 + codex #3): merge caller-supplied extras. Hardcoded
|
||||
// entries win on id collision so legacy behavior is preserved when an
|
||||
// extra accidentally duplicates a hardcoded id.
|
||||
if (extraRemediations.length > 0) {
|
||||
const hardcodedIds = new Set(out.map((r) => r.id));
|
||||
for (const extra of extraRemediations) {
|
||||
if (!hardcodedIds.has(extra.id)) out.push(extra);
|
||||
}
|
||||
}
|
||||
|
||||
// Sort: severity (critical first), then est_seconds ascending so quick
|
||||
// wins come first within a severity tier.
|
||||
const sevRank: Record<RemediationSeverity, number> = {
|
||||
|
||||
+45
-2
@@ -126,6 +126,15 @@ export interface LinkBatchInput {
|
||||
from_source_id?: string;
|
||||
to_source_id?: string;
|
||||
origin_source_id?: string;
|
||||
/**
|
||||
* v0.41.18.0 (A10, codex finding #12): distinguishes "plain body mention"
|
||||
* (NULL or 'plain') from "verb-pattern-derived typed NER" ('typed_ner')
|
||||
* within link_source='mentions'. Backed by v98 schema column. NOT in
|
||||
* the links UNIQUE constraint — same (from, to, type, source, origin)
|
||||
* tuple with different link_kind collides DO NOTHING. Default NULL =
|
||||
* legacy / unknown / pre-v98 semantics.
|
||||
*/
|
||||
link_kind?: string;
|
||||
}
|
||||
|
||||
/** Input row for addTimelineEntriesBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
@@ -165,7 +174,19 @@ export interface TimelineBatchInput {
|
||||
* transaction itself is waiting to write.
|
||||
*/
|
||||
export interface ReservedConnection {
|
||||
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;
|
||||
/**
|
||||
* v0.41.18.0 (A20, codex #7): optional 3rd-arg `opts.signal` lets callers
|
||||
* actually cancel a running query. Init nudge (3s wallclock cap) wires an
|
||||
* AbortController whose timer fires at 3s; queries that haven't returned
|
||||
* by then get cancelled (Postgres: query.cancel(); PGLite: in-process,
|
||||
* Promise.race against signal-rejection — documented gap because PGLite
|
||||
* has no kernel-level cancellation).
|
||||
*/
|
||||
executeRaw<T = Record<string, unknown>>(
|
||||
sql: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<T[]>;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -865,6 +886,16 @@ export interface BrainEngine {
|
||||
afterPageId?: number;
|
||||
afterChunkIndex?: number;
|
||||
sourceId?: string;
|
||||
// v0.41.18.0 (A13, codex #9): pagination order. Default 'page_id'
|
||||
// (legacy stable cursor). 'updated_desc' joins pages and orders by
|
||||
// p.updated_at DESC NULLS LAST, p.id, cc.chunk_index — backed by
|
||||
// idx_pages_updated_at_desc + content_chunks_stale_idx partial.
|
||||
orderBy?: 'page_id' | 'updated_desc';
|
||||
// For 'updated_desc' cursor: previous row's updated_at, page_id, chunk_index.
|
||||
// ISO-8601 string for cross-engine compatibility (postgres.js + PGLite
|
||||
// both round-trip TIMESTAMPTZ as Date | string; ISO string is the
|
||||
// common denominator on the wire).
|
||||
afterUpdatedAt?: string | null;
|
||||
}): Promise<StaleChunkRow[]>;
|
||||
/**
|
||||
* Delete every chunk for a page. Internal page-id lookup is sourceId-scoped
|
||||
@@ -1606,7 +1637,19 @@ export interface BrainEngine {
|
||||
getChunksWithEmbeddings(slug: string, opts?: { sourceId?: string }): Promise<Chunk[]>;
|
||||
|
||||
// Raw SQL (for Minions job queue and other internal modules)
|
||||
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;
|
||||
/**
|
||||
* v0.41.18.0 (A20, codex #7): optional 3rd-arg `opts.signal` lets callers
|
||||
* actually cancel a running query. Init nudge (3s wallclock cap) wires an
|
||||
* AbortController whose timer fires at 3s; queries that haven't returned
|
||||
* by then get cancelled (Postgres: query.cancel(); PGLite: in-process,
|
||||
* Promise.race against signal-rejection — documented gap because PGLite
|
||||
* has no kernel-level cancellation).
|
||||
*/
|
||||
executeRaw<T = Record<string, unknown>>(
|
||||
sql: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<T[]>;
|
||||
|
||||
// ============================================================
|
||||
// v0.20.0 Cathedral II: code edges (Layer 5 populates, Layer 7 consumes)
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
// src/core/extract-ner.ts
|
||||
// v0.41.18.0 (A10, T7). NER link extraction reuses the by-mention gazetteer
|
||||
// and applies schema-pack `link_types[].inference.regex` patterns to assign
|
||||
// a typed link verb ("CEO of Acme" → 'works_at' linking the page to Acme).
|
||||
//
|
||||
// Codex finding #12 design (locked): do NOT split link_source='ner' as a
|
||||
// new provenance — that would break every existing link_source='mentions'
|
||||
// query (backlink-count filter, orphan-ratio, doctor checks). Instead:
|
||||
// keep link_source='mentions' AND set link_kind='typed_ner' on the new row
|
||||
// (v98 added the column). Legacy plain mentions stay link_kind=NULL
|
||||
// (semantically 'plain').
|
||||
//
|
||||
// The links UNIQUE constraint excludes link_kind, so an existing plain
|
||||
// mention row + a typed_ner row for the same (from, to, type, source, origin)
|
||||
// collide — DO NOTHING. NER does NOT overwrite plain mentions; the verb
|
||||
// link goes in as a different row with a different link_type.
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { LinkBatchInput } from './engine.ts';
|
||||
import { buildGazetteer, findMentionedEntities, type Gazetteer } from './by-mention.ts';
|
||||
import { inferLinkTypeFromPack } from './schema-pack/link-inference.ts';
|
||||
import { loadActivePackBestEffort } from './schema-pack/best-effort.ts';
|
||||
|
||||
export interface ExtractNerOpts {
|
||||
/** When true: enumerate but don't write. */
|
||||
dryRun?: boolean;
|
||||
/** Optional source-id filter on the WALK (gazetteer stays brain-wide). */
|
||||
sourceIdFilter?: string;
|
||||
/** Optional page-type filter on the WALK. */
|
||||
typeFilter?: string;
|
||||
/** Only scan pages with updated_at after this ISO date. */
|
||||
since?: string;
|
||||
/**
|
||||
* Pre-built gazetteer (T7+: combined `--by-mention --ner` walk shares
|
||||
* one gazetteer across both passes). When omitted, this fn builds its own.
|
||||
*/
|
||||
gazetteer?: Gazetteer;
|
||||
/** Optional progress hook called per processed page. */
|
||||
onProgress?: (done: number, total: number, created: number) => void;
|
||||
}
|
||||
|
||||
export interface ExtractNerResult {
|
||||
/** Pages scanned. */
|
||||
pages: number;
|
||||
/** Typed-NER links created (or would-have-created in dry-run). */
|
||||
created: number;
|
||||
/** Pages where the active schema pack had no link_types at all. */
|
||||
pack_unavailable: boolean;
|
||||
}
|
||||
|
||||
/** Context window scanned around each mention for verb-pattern matching. */
|
||||
const CONTEXT_WINDOW_CHARS = 80;
|
||||
|
||||
/**
|
||||
* Pure helper: get the context window around a mention's character offset.
|
||||
* Returns the substring [offset - W, offset + name.length + W] of the body.
|
||||
* Caller passes (body, offset, name.length).
|
||||
*/
|
||||
export function getContextWindow(
|
||||
body: string,
|
||||
offset: number,
|
||||
nameLen: number,
|
||||
window: number = CONTEXT_WINDOW_CHARS,
|
||||
): string {
|
||||
const start = Math.max(0, offset - window);
|
||||
const end = Math.min(body.length, offset + nameLen + window);
|
||||
return body.slice(start, end);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure helper: derive the entity-type→link-verb pair from a single mention.
|
||||
* Returns null when (a) target type unknown, (b) pack has no inference for
|
||||
* that type, (c) no verb pattern matches the surrounding context.
|
||||
*
|
||||
* Exported for unit tests; the orchestrator below uses it directly.
|
||||
*/
|
||||
export function inferNerLinkType(
|
||||
pack: Parameters<typeof inferLinkTypeFromPack>[0],
|
||||
targetType: string | undefined,
|
||||
context: string,
|
||||
): string | null {
|
||||
if (!targetType) return null;
|
||||
try {
|
||||
return inferLinkTypeFromPack(pack, targetType, context);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* extractNerLinks: walk pages, find body mentions, apply schema-pack
|
||||
* inference regex per (target_type, surrounding context) to assign a typed
|
||||
* link verb. Returns count of created links.
|
||||
*
|
||||
* Best-effort wrt the schema pack: if no active pack OR no link_types
|
||||
* declared OR no inference.regex on any link_type, the function returns
|
||||
* pack_unavailable=true and 0 created. Caller (CLI / handler) surfaces a
|
||||
* one-line hint instead of an error.
|
||||
*/
|
||||
export async function extractNerLinks(
|
||||
engine: BrainEngine,
|
||||
opts: ExtractNerOpts = {},
|
||||
): Promise<ExtractNerResult> {
|
||||
const dryRun = opts.dryRun ?? false;
|
||||
|
||||
// Pack best-effort: no pack → no inference → nothing to do.
|
||||
const pack = await loadActivePackBestEffort({ engine } as never);
|
||||
if (!pack || !pack.manifest?.link_types || pack.manifest.link_types.length === 0) {
|
||||
return { pages: 0, created: 0, pack_unavailable: true };
|
||||
}
|
||||
// Require at least one link_type with an inference.regex; otherwise NER
|
||||
// has no patterns to match and we'd waste a full walk.
|
||||
const hasRegex = pack.manifest.link_types.some(
|
||||
(lt) => lt.inference && typeof lt.inference === 'object' && 'regex' in lt.inference,
|
||||
);
|
||||
if (!hasRegex) return { pages: 0, created: 0, pack_unavailable: true };
|
||||
|
||||
const gazetteer = opts.gazetteer ?? await buildGazetteer(engine);
|
||||
if (gazetteer.size === 0) {
|
||||
return { pages: 0, created: 0, pack_unavailable: false };
|
||||
}
|
||||
|
||||
// Pre-fetch target entity types so inferLinkType has the type signal
|
||||
// without an N+1 getPage round-trip. Pulls the slug→type map from
|
||||
// listAllPageRefs + a single listPages projection.
|
||||
const targetTypeMap = await buildTargetTypeMap(engine);
|
||||
|
||||
const allRefs = opts.sourceIdFilter
|
||||
? (await engine.listAllPageRefs()).filter((r) => r.source_id === opts.sourceIdFilter)
|
||||
: await engine.listAllPageRefs();
|
||||
|
||||
let processed = 0;
|
||||
let created = 0;
|
||||
const batch: LinkBatchInput[] = [];
|
||||
const BATCH_SIZE = 500;
|
||||
const sinceMs = opts.since ? new Date(opts.since).getTime() : null;
|
||||
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
if (!dryRun) {
|
||||
try {
|
||||
created += await engine.addLinksBatch(batch); // gbrain-allow-direct-insert: extract-ner — typed NER link write
|
||||
} catch {
|
||||
// batch error: drop; the per-page progress continues
|
||||
}
|
||||
} else {
|
||||
created += batch.length;
|
||||
}
|
||||
batch.length = 0;
|
||||
}
|
||||
|
||||
for (const { slug, source_id } of allRefs) {
|
||||
const page = await engine.getPage(slug, { sourceId: source_id });
|
||||
if (!page) continue;
|
||||
if (opts.typeFilter && page.type !== opts.typeFilter) continue;
|
||||
if (sinceMs !== null) {
|
||||
const updatedMs = new Date(page.updated_at).getTime();
|
||||
if (Number.isFinite(updatedMs) && updatedMs <= sinceMs) continue;
|
||||
}
|
||||
processed++;
|
||||
opts.onProgress?.(processed, allRefs.length, created);
|
||||
|
||||
const body = page.compiled_truth + '\n\n' + (page.timeline ?? '');
|
||||
if (!body.trim()) continue;
|
||||
|
||||
const mentions = findMentionedEntities(body, gazetteer, {
|
||||
fromSlug: slug,
|
||||
fromSourceId: source_id,
|
||||
});
|
||||
if (mentions.length === 0) continue;
|
||||
|
||||
for (const m of mentions) {
|
||||
const targetType = targetTypeMap.get(`${m.source_id}::${m.slug}`);
|
||||
const context = getContextWindow(body, m.offset, m.name.length);
|
||||
const verb = inferNerLinkType(pack.manifest, targetType, context);
|
||||
if (!verb) continue;
|
||||
|
||||
batch.push({
|
||||
from_slug: slug,
|
||||
to_slug: m.slug,
|
||||
link_type: verb,
|
||||
link_source: 'mentions',
|
||||
link_kind: 'typed_ner',
|
||||
context: m.name,
|
||||
from_source_id: source_id,
|
||||
to_source_id: m.source_id,
|
||||
});
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
|
||||
await flush();
|
||||
return { pages: processed, created, pack_unavailable: false };
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper: build a Map<sourceId::slug → type> for all entity-typed pages.
|
||||
* One round-trip via listPages. Targets cached at extraction-start so
|
||||
* inferNerLinkType doesn't pay an N+1 cost per mention.
|
||||
*/
|
||||
async function buildTargetTypeMap(engine: BrainEngine): Promise<Map<string, string>> {
|
||||
const map = new Map<string, string>();
|
||||
try {
|
||||
const result = await engine.executeRaw<{ slug: string; source_id: string; type: string }>(
|
||||
`SELECT slug, source_id, type FROM pages
|
||||
WHERE type IN ('person', 'company', 'organization', 'entity')
|
||||
AND deleted_at IS NULL`,
|
||||
);
|
||||
for (const row of result) {
|
||||
map.set(`${row.source_id}::${row.slug}`, row.type);
|
||||
}
|
||||
} catch {
|
||||
// Engine error → empty map; inferNerLinkType returns null for unknown types.
|
||||
}
|
||||
return map;
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
// src/core/extract-takes-from-pages.ts
|
||||
// v0.41.18.0 (A12, A24, T9). Haiku classifier loop over allowlisted page
|
||||
// types — concept, atom, lore, briefing, writing, originals — extracts
|
||||
// gradeable claims and inserts them as takes fence rows.
|
||||
//
|
||||
// Two-gate consent per A12:
|
||||
// - takes.bootstrap_enabled (default false): must be true to run at all.
|
||||
// Even manual `gbrain takes extract --from-pages` refuses without it.
|
||||
// - takes.autopilot_allowed (default false): must be true for autopilot's
|
||||
// auto-apply tier to fire the takes-bootstrap remediation.
|
||||
//
|
||||
// A24 deliberately limits autopilot to manual_only until v0.42.1 lands a
|
||||
// 100+-case eval suite. v0.42 ships the classifier + CLI; autopilot stays
|
||||
// blocked until eval coverage catches up.
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { TakeBatchInput, TakeKind } from './engine.ts';
|
||||
import { chat, isAvailable } from './ai/gateway.ts';
|
||||
|
||||
export const ALLOWED_PAGE_TYPES = [
|
||||
'concept', 'atom', 'lore', 'briefing', 'writing', 'originals',
|
||||
] as const;
|
||||
|
||||
const CLASSIFIER_SYSTEM = `You extract gradeable CLAIMS from longform writing.
|
||||
|
||||
Output strict JSON: an array of objects with shape:
|
||||
{"claim": "<short imperative or assertion, <= 200 chars>",
|
||||
"kind": "fact" | "take" | "bet" | "hunch",
|
||||
"weight": 0.0..1.0}
|
||||
|
||||
Kind taxonomy:
|
||||
- fact: verifiable as true/false (e.g. "X raised $5M in Mar 2024")
|
||||
- take: a stated opinion that could be wrong (e.g. "X is undervalued")
|
||||
- bet: a forward-looking prediction (e.g. "X will IPO in 2026")
|
||||
- hunch: a low-confidence gut feeling (e.g. "Y feels overstretched")
|
||||
|
||||
Skip pure narrative, questions, definitions, or pure quotes from others.
|
||||
Max 15 claims per page; output [] if no gradeable claims are present.`;
|
||||
|
||||
export interface ExtractTakesFromPagesOpts {
|
||||
/** Required: must be true for any work to happen (A12). */
|
||||
bootstrapEnabled: boolean;
|
||||
/** Dry-run: classify but don't write to takes table. */
|
||||
dryRun?: boolean;
|
||||
/** Scope to a single source. */
|
||||
sourceIdFilter?: string;
|
||||
/** Max pages to classify per run (caps cost). Default 50. */
|
||||
maxPages?: number;
|
||||
/** Owner identifier for the inserted takes. Default 'system'. */
|
||||
holder?: string;
|
||||
/** Model override; defaults to facts.extraction_model. */
|
||||
model?: string;
|
||||
/** Progress hook called per page. */
|
||||
onProgress?: (done: number, total: number, claims: number) => void;
|
||||
}
|
||||
|
||||
export interface ExtractTakesFromPagesResult {
|
||||
pages_scanned: number;
|
||||
claims_extracted: number;
|
||||
/** True if the run was a no-op because bootstrapEnabled is false. */
|
||||
consent_gate_blocked: boolean;
|
||||
/** True if chat gateway is unavailable (no LLM call possible). */
|
||||
llm_unavailable: boolean;
|
||||
}
|
||||
|
||||
interface PageRow {
|
||||
id: number;
|
||||
slug: string;
|
||||
source_id: string;
|
||||
type: string;
|
||||
compiled_truth: string;
|
||||
updated_at: string | Date;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure helper: parse Haiku JSON output into typed claims. Returns []
|
||||
* on any parse failure (caller treats as "no claims extracted").
|
||||
*/
|
||||
export function parseClaimsJson(raw: string): Array<{ claim: string; kind: TakeKind; weight: number }> {
|
||||
try {
|
||||
// Strip code fences if model wrapped output in ```json.
|
||||
let text = raw.trim();
|
||||
const fenceMatch = text.match(/^```(?:json)?\n?([\s\S]*?)\n?```$/);
|
||||
if (fenceMatch) text = fenceMatch[1].trim();
|
||||
const parsed = JSON.parse(text);
|
||||
if (!Array.isArray(parsed)) return [];
|
||||
const valid: Array<{ claim: string; kind: TakeKind; weight: number }> = [];
|
||||
for (const item of parsed) {
|
||||
if (!item || typeof item !== 'object') continue;
|
||||
const claim = typeof item.claim === 'string' ? item.claim.trim().slice(0, 200) : '';
|
||||
const kind = typeof item.kind === 'string' ? item.kind : '';
|
||||
const weightRaw = typeof item.weight === 'number' ? item.weight : 0.5;
|
||||
const weight = Math.max(0, Math.min(1, weightRaw));
|
||||
if (!claim || !['fact', 'take', 'bet', 'hunch'].includes(kind)) continue;
|
||||
valid.push({ claim, kind, weight });
|
||||
}
|
||||
return valid;
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
export async function extractTakesFromPages(
|
||||
engine: BrainEngine,
|
||||
opts: ExtractTakesFromPagesOpts,
|
||||
): Promise<ExtractTakesFromPagesResult> {
|
||||
// A12 consent gate: refuse without bootstrap_enabled even on manual call.
|
||||
if (!opts.bootstrapEnabled) {
|
||||
return {
|
||||
pages_scanned: 0,
|
||||
claims_extracted: 0,
|
||||
consent_gate_blocked: true,
|
||||
llm_unavailable: false,
|
||||
};
|
||||
}
|
||||
|
||||
if (!isAvailable('chat')) {
|
||||
return {
|
||||
pages_scanned: 0,
|
||||
claims_extracted: 0,
|
||||
consent_gate_blocked: false,
|
||||
llm_unavailable: true,
|
||||
};
|
||||
}
|
||||
|
||||
const dryRun = opts.dryRun ?? false;
|
||||
const maxPages = opts.maxPages ?? 50;
|
||||
const holder = opts.holder ?? 'system';
|
||||
const sourceFilter = opts.sourceIdFilter ? `AND source_id = $1` : '';
|
||||
const params = opts.sourceIdFilter ? [opts.sourceIdFilter] : [];
|
||||
|
||||
// Fetch eligible pages. Order by updated_at DESC so recently-edited
|
||||
// pages get bootstrapped first.
|
||||
const typesList = ALLOWED_PAGE_TYPES.map((t) => `'${t}'`).join(', ');
|
||||
const pages = await engine.executeRaw<PageRow>(
|
||||
`SELECT id, slug, source_id, type, compiled_truth, updated_at
|
||||
FROM pages
|
||||
WHERE type IN (${typesList})
|
||||
AND deleted_at IS NULL
|
||||
AND length(COALESCE(compiled_truth, '')) > 200
|
||||
${sourceFilter}
|
||||
ORDER BY updated_at DESC
|
||||
LIMIT ${maxPages}`,
|
||||
params,
|
||||
);
|
||||
|
||||
let pagesScanned = 0;
|
||||
let claimsExtracted = 0;
|
||||
const batch: TakeBatchInput[] = [];
|
||||
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
if (!dryRun) {
|
||||
try {
|
||||
claimsExtracted += await engine.addTakesBatch(batch);
|
||||
} catch {
|
||||
// batch error — drop and continue with subsequent pages
|
||||
}
|
||||
} else {
|
||||
claimsExtracted += batch.length;
|
||||
}
|
||||
batch.length = 0;
|
||||
}
|
||||
|
||||
for (const page of pages) {
|
||||
pagesScanned++;
|
||||
opts.onProgress?.(pagesScanned, pages.length, claimsExtracted);
|
||||
|
||||
if (!page.compiled_truth || page.compiled_truth.length < 200) continue;
|
||||
|
||||
// Truncate to keep per-page cost bounded (~20K chars → ~5K input tokens).
|
||||
const text = page.compiled_truth.slice(0, 20_000);
|
||||
|
||||
let response: { text: string };
|
||||
try {
|
||||
response = await chat({
|
||||
model: opts.model ?? 'anthropic:claude-haiku-4-5',
|
||||
system: CLASSIFIER_SYSTEM,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: `<page slug="${page.slug}" type="${page.type}">\n${text}\n</page>`,
|
||||
},
|
||||
],
|
||||
maxTokens: 2000,
|
||||
});
|
||||
} catch {
|
||||
// Skip pages whose chat call fails (rate limit, content filter,
|
||||
// transient error). Per-page progress continues.
|
||||
continue;
|
||||
}
|
||||
|
||||
const claims = parseClaimsJson(response.text);
|
||||
if (claims.length === 0) continue;
|
||||
|
||||
// Assign row_num starting from 1 per page. We don't query existing
|
||||
// takes for the page — collisions on (page_id, row_num) are an existing
|
||||
// bug class addresses by extract-conversation-facts; takes-bootstrap
|
||||
// inherits the same posture: writes start at row_num=1 and the engine's
|
||||
// unique constraint surfaces duplicates as failures (caller re-runs).
|
||||
for (let i = 0; i < claims.length; i++) {
|
||||
const c = claims[i];
|
||||
batch.push({
|
||||
page_id: page.id,
|
||||
row_num: i + 1,
|
||||
claim: c.claim,
|
||||
kind: c.kind,
|
||||
holder,
|
||||
weight: c.weight,
|
||||
source: 'cli:takes-bootstrap-from-pages',
|
||||
});
|
||||
}
|
||||
if (batch.length >= 200) await flush();
|
||||
}
|
||||
|
||||
await flush();
|
||||
return {
|
||||
pages_scanned: pagesScanned,
|
||||
claims_extracted: claimsExtracted,
|
||||
consent_gate_blocked: false,
|
||||
llm_unavailable: false,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
// src/core/extract-timeline-from-meetings.ts
|
||||
// v0.41.18.0 (A11, T8). Walk meeting pages, identify discussed entities via
|
||||
// (a) existing `attended` links (attendees) + (b) body-mention scan, and
|
||||
// write a timeline entry on each entity page with a meeting-specific source
|
||||
// key that survives v99's widened dedup.
|
||||
//
|
||||
// Codex finding #11 dependency: requires v99 dedup widening from
|
||||
// (page_id, date, summary) to (page_id, date, summary, source). Without v99,
|
||||
// two meetings on the same date with the same summary on the same entity
|
||||
// page would silently drop the second one.
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { TimelineBatchInput } from './engine.ts';
|
||||
import { buildGazetteer, findMentionedEntities, type Gazetteer } from './by-mention.ts';
|
||||
|
||||
export interface ExtractTimelineFromMeetingsOpts {
|
||||
dryRun?: boolean;
|
||||
sourceIdFilter?: string;
|
||||
/** Only scan meetings with updated_at after this ISO date. */
|
||||
since?: string;
|
||||
/** Optional pre-built gazetteer (for shared-walk callers). */
|
||||
gazetteer?: Gazetteer;
|
||||
onProgress?: (done: number, total: number, created: number) => void;
|
||||
}
|
||||
|
||||
export interface ExtractTimelineFromMeetingsResult {
|
||||
meetings_scanned: number;
|
||||
entries_created: number;
|
||||
/** Distinct entity pages that received at least one new timeline entry. */
|
||||
entities_touched: number;
|
||||
}
|
||||
|
||||
interface MeetingRow {
|
||||
slug: string;
|
||||
source_id: string;
|
||||
title: string;
|
||||
effective_date: string | null;
|
||||
updated_at: string | Date;
|
||||
compiled_truth: string;
|
||||
timeline: string;
|
||||
}
|
||||
|
||||
interface AttendedEdgeRow {
|
||||
from_slug: string;
|
||||
from_source_id: string;
|
||||
to_slug: string;
|
||||
to_source_id: string;
|
||||
}
|
||||
|
||||
const BATCH_SIZE = 200;
|
||||
|
||||
export async function extractTimelineFromMeetings(
|
||||
engine: BrainEngine,
|
||||
opts: ExtractTimelineFromMeetingsOpts = {},
|
||||
): Promise<ExtractTimelineFromMeetingsResult> {
|
||||
const dryRun = opts.dryRun ?? false;
|
||||
const sinceMs = opts.since ? new Date(opts.since).getTime() : null;
|
||||
|
||||
// 1. Fetch all meeting pages (one round-trip).
|
||||
const sourceFilter = opts.sourceIdFilter ? `AND source_id = $1` : '';
|
||||
const meetingParams = opts.sourceIdFilter ? [opts.sourceIdFilter] : [];
|
||||
const meetings = await engine.executeRaw<MeetingRow>(
|
||||
`SELECT slug, source_id, title, effective_date, updated_at,
|
||||
compiled_truth, COALESCE(timeline, '') AS timeline
|
||||
FROM pages
|
||||
WHERE type = 'meeting'
|
||||
AND deleted_at IS NULL
|
||||
${sourceFilter}
|
||||
ORDER BY effective_date DESC NULLS LAST, slug`,
|
||||
meetingParams,
|
||||
);
|
||||
|
||||
if (meetings.length === 0) {
|
||||
return { meetings_scanned: 0, entries_created: 0, entities_touched: 0 };
|
||||
}
|
||||
|
||||
// 2. Fetch all 'attended' edges (one round-trip, scoped to the loaded
|
||||
// meeting source_ids). Build a Map<meetingSlug → attendees[]> for O(1)
|
||||
// attendee lookup per meeting.
|
||||
const meetingKeys = new Set(meetings.map((m) => `${m.source_id}::${m.slug}`));
|
||||
const attendedEdges = await engine.executeRaw<AttendedEdgeRow>(
|
||||
`SELECT pf.slug AS from_slug, pf.source_id AS from_source_id,
|
||||
pt.slug AS to_slug, pt.source_id AS to_source_id
|
||||
FROM links l
|
||||
JOIN pages pf ON pf.id = l.from_page_id
|
||||
JOIN pages pt ON pt.id = l.to_page_id
|
||||
WHERE l.link_type = 'attended'
|
||||
AND pf.type = 'meeting'
|
||||
AND pf.deleted_at IS NULL
|
||||
AND pt.deleted_at IS NULL`,
|
||||
);
|
||||
const attendeesByMeeting = new Map<string, AttendedEdgeRow[]>();
|
||||
for (const e of attendedEdges) {
|
||||
const key = `${e.from_source_id}::${e.from_slug}`;
|
||||
if (!meetingKeys.has(key)) continue;
|
||||
const list = attendeesByMeeting.get(key);
|
||||
if (list) list.push(e);
|
||||
else attendeesByMeeting.set(key, [e]);
|
||||
}
|
||||
|
||||
// 3. For each meeting, derive entity mentions (gazetteer-based) + merge
|
||||
// with attendee edges. Each (meeting, entity) produces ONE timeline row.
|
||||
const gazetteer = opts.gazetteer ?? await buildGazetteer(engine);
|
||||
|
||||
const batch: TimelineBatchInput[] = [];
|
||||
let entriesCreated = 0;
|
||||
const entitiesTouched = new Set<string>();
|
||||
let meetingsScanned = 0;
|
||||
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
if (!dryRun) {
|
||||
try {
|
||||
entriesCreated += await engine.addTimelineEntriesBatch(batch);
|
||||
} catch {
|
||||
// batch error — drop; per-meeting progress continues
|
||||
}
|
||||
} else {
|
||||
entriesCreated += batch.length;
|
||||
}
|
||||
batch.length = 0;
|
||||
}
|
||||
|
||||
for (const meeting of meetings) {
|
||||
if (sinceMs !== null) {
|
||||
const updatedMs = new Date(meeting.updated_at).getTime();
|
||||
if (Number.isFinite(updatedMs) && updatedMs <= sinceMs) continue;
|
||||
}
|
||||
if (!meeting.effective_date) continue; // can't write a timeline entry without a date
|
||||
|
||||
meetingsScanned++;
|
||||
opts.onProgress?.(meetingsScanned, meetings.length, entriesCreated);
|
||||
|
||||
const meetingKey = `${meeting.source_id}::${meeting.slug}`;
|
||||
const summary = `Discussed in ${meeting.title}`;
|
||||
const sourceKey = `extract-timeline-from-meetings:${meeting.slug}`;
|
||||
|
||||
// Attendees (from 'attended' links).
|
||||
const attendees = attendeesByMeeting.get(meetingKey) ?? [];
|
||||
const targets = new Map<string, { slug: string; source_id: string }>();
|
||||
for (const e of attendees) {
|
||||
targets.set(`${e.to_source_id}::${e.to_slug}`, {
|
||||
slug: e.to_slug,
|
||||
source_id: e.to_source_id,
|
||||
});
|
||||
}
|
||||
|
||||
// Body mentions (gazetteer-based). Skip self-mention (meeting page
|
||||
// referencing itself by title). The cross-source guard in
|
||||
// findMentionedEntities already drops mentions targeting a different
|
||||
// source than the gazetteer entry was built from.
|
||||
const body = meeting.compiled_truth + '\n\n' + meeting.timeline;
|
||||
if (body.trim()) {
|
||||
const mentions = findMentionedEntities(body, gazetteer, {
|
||||
fromSlug: meeting.slug,
|
||||
fromSourceId: meeting.source_id,
|
||||
});
|
||||
for (const m of mentions) {
|
||||
targets.set(`${m.source_id}::${m.slug}`, {
|
||||
slug: m.slug,
|
||||
source_id: m.source_id,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Emit one timeline row per (entity, this meeting).
|
||||
for (const t of targets.values()) {
|
||||
batch.push({
|
||||
slug: t.slug,
|
||||
source_id: t.source_id,
|
||||
date: meeting.effective_date,
|
||||
source: sourceKey,
|
||||
summary,
|
||||
});
|
||||
entitiesTouched.add(`${t.source_id}::${t.slug}`);
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
|
||||
await flush();
|
||||
return {
|
||||
meetings_scanned: meetingsScanned,
|
||||
entries_created: entriesCreated,
|
||||
entities_touched: entitiesTouched.size,
|
||||
};
|
||||
}
|
||||
+129
-33
@@ -4415,7 +4415,7 @@ export const MIGRATIONS: Migration[] = [
|
||||
{
|
||||
version: 95,
|
||||
name: 'links_link_source_check_includes_mentions',
|
||||
// v0.42.0.0 Part B (migration #1 of #1409): widen the link_source
|
||||
// v0.41.18.0 Part B (migration #1 of #1409): widen the link_source
|
||||
// CHECK constraint to admit 'mentions' for auto-linked body-text
|
||||
// mentions from `gbrain extract links --by-mention`. Backlink-count
|
||||
// SQL in postgres-engine.ts + pglite-engine.ts excludes link_source =
|
||||
@@ -4570,17 +4570,7 @@ export const MIGRATIONS: Migration[] = [
|
||||
// If we backfilled = acquired_at (e.g. 25 min ago), then `gbrain sync
|
||||
// --break-lock --all --max-age 1800` after the migration would
|
||||
// immediately delete the lock of a HEALTHY 25-min-old holder that's
|
||||
// still actively writing. That IS the bug class this PR exists to
|
||||
// fix; reintroducing it through the migration backfill would be
|
||||
// worse than not shipping at all.
|
||||
//
|
||||
// Backfilling = NOW() gives every pre-upgrade holder a 30-min
|
||||
// protection window: --max-age 1800 cannot identify them as wedged
|
||||
// for 30 min after migrate. After that, all pre-upgrade syncs are
|
||||
// either complete (lock released) OR genuinely wedged (--max-age
|
||||
// does the right thing on the next operator command).
|
||||
//
|
||||
// Engine parity: same ALTER TABLE shape works on Postgres + PGLite.
|
||||
// still actively writing.
|
||||
sql: `
|
||||
ALTER TABLE gbrain_cycle_locks ADD COLUMN IF NOT EXISTS last_refreshed_at TIMESTAMPTZ;
|
||||
UPDATE gbrain_cycle_locks SET last_refreshed_at = NOW() WHERE last_refreshed_at IS NULL;
|
||||
@@ -4589,27 +4579,8 @@ export const MIGRATIONS: Migration[] = [
|
||||
{
|
||||
version: 99,
|
||||
name: 'conversation_parser_llm_cache_table',
|
||||
// v0.41.16.0 — content-hash-keyed cache for the conversation
|
||||
// parser's LLM polish + fallback calls. Per D17 (codex outside
|
||||
// voice), there is NO conversation_parser_inferred_patterns
|
||||
// table: persisting LLM-inferred regex from a 20-line sample
|
||||
// and applying it to whole pages + future pages is a silent
|
||||
// corruption machine. Cache shape is per-page-content-hash;
|
||||
// re-running the dream cycle on the same haystack is a hit;
|
||||
// a different page with the same format misses the cache and
|
||||
// calls the LLM again (cost-bounded by BudgetTracker).
|
||||
//
|
||||
// Key is (content_sha256, model_id, call_shape).
|
||||
// - content_sha256: sha256 of the page body the LLM saw.
|
||||
// - model_id: provider:model the call routed through (so a
|
||||
// model upgrade invalidates stale entries naturally).
|
||||
// - call_shape: 'polish' | 'fallback' so the same content
|
||||
// can be cached differently per call kind.
|
||||
//
|
||||
// Slot history: originally v97, bumped to v98 after master's
|
||||
// v0.41.13.0 (#1422 fix-wave) claimed v97 for the dedup index,
|
||||
// bumped to v99 after master's v0.41.15.0 (#1506 sync RFC)
|
||||
// claimed v98 for the lock-refresh column.
|
||||
// v0.41.16.0 — content-hash-keyed cache for the conversation parser's
|
||||
// LLM polish + fallback calls. See src/schema.sql for design notes.
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS conversation_parser_llm_cache (
|
||||
content_sha256 TEXT NOT NULL,
|
||||
@@ -4637,6 +4608,131 @@ export const MIGRATIONS: Migration[] = [
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 101,
|
||||
name: 'links_link_kind_column',
|
||||
// v0.41.18.0 (gbrain onboard wave, A10 + codex finding #12):
|
||||
// NER link extraction adds a nullable link_kind column instead of
|
||||
// splitting link_source='ner' as a new provenance — keeps
|
||||
// backlink-count + orphan-ratio queries stable while letting
|
||||
// NER-aware callers distinguish typed links.
|
||||
//
|
||||
// Three kinds: 'plain' | 'typed_ner' | NULL (legacy, semantically plain).
|
||||
// NOT in the links UNIQUE constraint so a plain-mention row coexists
|
||||
// with future typed_ner promotions via explicit ON CONFLICT DO UPDATE.
|
||||
//
|
||||
// Slot history: originally v98, bumped to v101 after master merge
|
||||
// claimed v98 (lock-refresh) + v99 (conversation parser cache) +
|
||||
// v100 (per master's own merges).
|
||||
sql: `
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_kind TEXT
|
||||
CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner'));
|
||||
`,
|
||||
sqlFor: {
|
||||
pglite: `
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_kind TEXT
|
||||
CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner'));
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 102,
|
||||
name: 'timeline_entries_source_in_dedup',
|
||||
// v0.41.18.0 (gbrain onboard wave, A11 + codex finding #11):
|
||||
// Widen idx_timeline_dedup from (page_id, date, summary) to
|
||||
// (page_id, date, summary, source) so --from-meetings provenance
|
||||
// survives. Legacy rows have source='' (schema default), so legacy
|
||||
// dedup behavior is preserved.
|
||||
//
|
||||
// Slot history: originally v99, bumped to v102 after master merge.
|
||||
sql: `
|
||||
DROP INDEX IF EXISTS idx_timeline_dedup;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup
|
||||
ON timeline_entries(page_id, date, summary, source);
|
||||
`,
|
||||
sqlFor: {
|
||||
pglite: `
|
||||
DROP INDEX IF EXISTS idx_timeline_dedup;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup
|
||||
ON timeline_entries(page_id, date, summary, source);
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 103,
|
||||
name: 'migration_impact_log_and_priority_recent_idx',
|
||||
// v0.41.18.0 (gbrain onboard wave, A6 + A25 + A13 + codex #9 + #10):
|
||||
// (1) migration_impact_log table — onboard --history backbone with
|
||||
// attribution columns (job_id, source_id, brain_id, started_at,
|
||||
// idempotency_key) so concurrent runs don't misattribute deltas.
|
||||
// (2) content_chunks_stale_idx partial index — supports
|
||||
// `embed --stale` + `--priority recent` (outer ORDER BY
|
||||
// p.updated_at DESC uses existing idx_pages_updated_at_desc).
|
||||
//
|
||||
// Slot history: originally v100, bumped to v103 after master merge.
|
||||
// Engine-aware split: Postgres uses CREATE INDEX CONCURRENTLY +
|
||||
// invalid-remnant pre-drop; PGLite uses plain CREATE INDEX.
|
||||
transaction: false,
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
const createTableSql = `
|
||||
CREATE TABLE IF NOT EXISTS migration_impact_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
remediation_id TEXT NOT NULL,
|
||||
metric_name TEXT NOT NULL,
|
||||
metric_before NUMERIC,
|
||||
metric_after NUMERIC,
|
||||
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
|
||||
source_id TEXT,
|
||||
brain_id TEXT,
|
||||
started_at TIMESTAMPTZ,
|
||||
idempotency_key TEXT,
|
||||
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
applied_by TEXT,
|
||||
details JSONB DEFAULT '{}'::jsonb
|
||||
);
|
||||
`;
|
||||
await engine.runMigration(103, createTableSql);
|
||||
await engine.runMigration(
|
||||
103,
|
||||
`CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
|
||||
ON migration_impact_log(remediation_id, applied_at DESC);`
|
||||
);
|
||||
await engine.runMigration(
|
||||
103,
|
||||
`CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
|
||||
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;`
|
||||
);
|
||||
|
||||
if (engine.kind === 'postgres') {
|
||||
await engine.runMigration(
|
||||
103,
|
||||
`DO $$ BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM pg_index i
|
||||
JOIN pg_class c ON c.oid = i.indexrelid
|
||||
WHERE c.relname = 'content_chunks_stale_idx' AND NOT i.indisvalid
|
||||
) THEN
|
||||
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS content_chunks_stale_idx';
|
||||
END IF;
|
||||
END $$;`
|
||||
);
|
||||
await engine.runMigration(
|
||||
103,
|
||||
`CREATE INDEX CONCURRENTLY IF NOT EXISTS content_chunks_stale_idx
|
||||
ON content_chunks (page_id, chunk_index)
|
||||
WHERE embedding IS NULL;`
|
||||
);
|
||||
} else {
|
||||
await engine.runMigration(
|
||||
103,
|
||||
`CREATE INDEX IF NOT EXISTS content_chunks_stale_idx
|
||||
ON content_chunks (page_id, chunk_index)
|
||||
WHERE embedding IS NULL;`
|
||||
);
|
||||
}
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
// src/core/minion-spend.ts
|
||||
// v0.41.18.0 (A7 + A23, codex finding #4).
|
||||
//
|
||||
// Generic Minion handlers (embed-catch-up, extract-ner,
|
||||
// extract-timeline-from-meetings, extract-takes-from-pages) need to
|
||||
// settle their LLM/API spend against the originating OAuth client's
|
||||
// mcp_spend_log row when submitted via the MCP run_onboard op
|
||||
// (admin scope). Pre-fix, only subagent loops via budget-meter.ts
|
||||
// recorded spend; generic handlers wrote to the gateway without any
|
||||
// per-client attribution.
|
||||
//
|
||||
// Convention chosen for v0.42.0: the originating client_id is stored on
|
||||
// job.data.client_id when run_onboard submits. The schema column for
|
||||
// minion_jobs.client_id is deferred to v0.42.1 (would require a v101
|
||||
// migration + index). For now: handlers that spend LLM/embedding budget
|
||||
// call recordMinionJobSpend(engine, job, ...) which reads job.data.client_id
|
||||
// and writes to mcp_spend_log with the right attribution.
|
||||
//
|
||||
// Best-effort throughout: spend telemetry MUST NOT fail the user's call.
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { recordSpend } from './spend-log.ts';
|
||||
|
||||
export interface MinionJobLike {
|
||||
id: number;
|
||||
data?: Record<string, unknown> | unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the OAuth client_id (if any) that submitted this Minion job.
|
||||
* Returns undefined for jobs submitted locally (CLI, autopilot tick) —
|
||||
* those bypass the per-client spend cap.
|
||||
*/
|
||||
export function getJobClientId(job: MinionJobLike): string | undefined {
|
||||
if (!job.data || typeof job.data !== 'object') return undefined;
|
||||
const data = job.data as Record<string, unknown>;
|
||||
const cid = data.client_id;
|
||||
return typeof cid === 'string' && cid.length > 0 ? cid : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record spend for a Minion job. Handler-side; called after each LLM /
|
||||
* embedding API call settles. Threads the originating MCP client_id
|
||||
* (when present) into mcp_spend_log so per-client caps are enforced
|
||||
* across the parent run_onboard → child handler chain.
|
||||
*
|
||||
* Local handler runs (no client_id) record with clientId=null —
|
||||
* the row still lands for global accounting but doesn't count against
|
||||
* any specific OAuth client's daily cap.
|
||||
*/
|
||||
export async function recordMinionJobSpend(
|
||||
engine: BrainEngine,
|
||||
job: MinionJobLike,
|
||||
entry: {
|
||||
operation: string;
|
||||
spendCents: number;
|
||||
provider?: string;
|
||||
model?: string;
|
||||
tokenName?: string;
|
||||
},
|
||||
): Promise<void> {
|
||||
const clientId = getJobClientId(job);
|
||||
await recordSpend(engine, {
|
||||
clientId: clientId ?? null,
|
||||
tokenName: entry.tokenName ?? null,
|
||||
operation: entry.operation,
|
||||
spendCents: entry.spendCents,
|
||||
provider: entry.provider,
|
||||
model: entry.model,
|
||||
});
|
||||
}
|
||||
@@ -37,6 +37,12 @@ export const PROTECTED_JOB_NAMES: ReadonlySet<string> = new Set([
|
||||
// budget. Only trusted local callers (the mode-switch hook in
|
||||
// commands/config.ts, reindex sweep, doctor --remediate) can submit.
|
||||
'contextual_reindex_per_chunk',
|
||||
// v0.41.18.0 (A12, T9) — takes-bootstrap. Per-page Haiku classifier
|
||||
// call over concept/atom/lore/briefing/writing/originals. Two-gate
|
||||
// consent (takes.bootstrap_enabled + --yes) AND PROTECTED ensures
|
||||
// no remote / MCP / autopilot path can bulk-extract takes without
|
||||
// explicit operator intent.
|
||||
'extract-takes-from-pages',
|
||||
]);
|
||||
|
||||
/** Check a job name against the protected set. Normalizes whitespace first. */
|
||||
|
||||
@@ -0,0 +1,368 @@
|
||||
// src/core/onboard/checks.ts
|
||||
// sourcescope:file-brain-wide — every SQL site here is intentionally
|
||||
// brain-wide aggregate. The onboard checks REPORT across all sources
|
||||
// (orphan_count, stale_count, link_coverage, takes_count) so adding
|
||||
// source_id WHERE clauses would change the semantic. Per A26.
|
||||
//
|
||||
// v0.41.18.0 (A16, T4). Four new doctor checks consumed by both:
|
||||
// - src/commands/doctor.ts runDoctor (local surface)
|
||||
// - src/core/doctor-remote.ts (thin-client surface)
|
||||
// - src/core/onboard/plan-from-checks.ts (onboard remediation aggregator)
|
||||
//
|
||||
// Each helper is shaped: compute metric → return both Check entry (for
|
||||
// doctor render) and RemediationStep[] (for onboard's extra-remediation
|
||||
// plumbing per A2). Helpers stay PURE wrt config: no engine.connect, no
|
||||
// process.exit. SQL via engine.executeRaw with `sourceScopeOpts(ctx)`
|
||||
// when ctx threads — onboard surface threads explicitly per A26.
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { RemediationStep } from '../remediation-step.ts';
|
||||
import { makeRemediationStep } from '../remediation-step.ts';
|
||||
|
||||
/** Shared shape returned by all four checks. */
|
||||
export interface OnboardCheckResult {
|
||||
check: {
|
||||
name: string;
|
||||
status: 'ok' | 'warn' | 'fail';
|
||||
message: string;
|
||||
};
|
||||
remediations: RemediationStep[];
|
||||
}
|
||||
|
||||
/** Internal sql helper. Returns first row or empty object on throw. */
|
||||
async function safeCount(engine: BrainEngine, sql: string, params: unknown[] = []): Promise<number> {
|
||||
try {
|
||||
const result = await engine.executeRaw(sql, params);
|
||||
const rows = (result as { rows?: Array<Record<string, unknown>> } | undefined)?.rows
|
||||
?? (result as Array<Record<string, unknown>> | undefined)
|
||||
?? [];
|
||||
const row = rows[0] ?? {};
|
||||
const raw = (row as Record<string, unknown>).count ?? (row as Record<string, unknown>).c ?? 0;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) ? n : 0;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* embed_staleness: count of chunks awaiting embedding.
|
||||
*
|
||||
* Backed by content_chunks_stale_idx partial index (v100) so the count
|
||||
* is cheap even on big brains.
|
||||
*/
|
||||
export async function checkEmbedStaleness(
|
||||
engine: BrainEngine,
|
||||
): Promise<OnboardCheckResult> {
|
||||
const staleCount = await safeCount(
|
||||
engine,
|
||||
`SELECT COUNT(*) AS count FROM content_chunks WHERE embedding IS NULL`,
|
||||
);
|
||||
const remediations: RemediationStep[] = [];
|
||||
let status: 'ok' | 'warn' | 'fail' = 'ok';
|
||||
let message: string;
|
||||
|
||||
if (staleCount === 0) {
|
||||
message = 'No stale chunks';
|
||||
} else if (staleCount < 1000) {
|
||||
status = 'warn';
|
||||
message = `${staleCount} stale chunks (small backlog)`;
|
||||
remediations.push(makeRemediationStep({
|
||||
id: 'onboard.embed_catch_up',
|
||||
job: 'embed-catch-up',
|
||||
params: { batchSize: 500 },
|
||||
severity: 'medium',
|
||||
est_seconds: Math.min(900, Math.ceil(staleCount * 0.2)),
|
||||
est_usd_cost: staleCount * 0.00002,
|
||||
rationale: `${staleCount} chunks awaiting embedding`,
|
||||
status: 'remediable',
|
||||
}));
|
||||
} else {
|
||||
// v0.41.18.0: warn-only even on large backlogs. Doctor exit code should
|
||||
// not flip from a brain that has pages waiting to be embedded — that's
|
||||
// a "needs work" condition, not a "broken" one. The high-severity
|
||||
// remediation still surfaces via onboard's plan.
|
||||
status = 'warn';
|
||||
message = `${staleCount} stale chunks (large backlog — vector search returning outdated content)`;
|
||||
remediations.push(makeRemediationStep({
|
||||
id: 'onboard.embed_catch_up',
|
||||
job: 'embed-catch-up',
|
||||
params: { batchSize: 1000, priority: 'recent' },
|
||||
severity: 'high',
|
||||
est_seconds: Math.min(3600, Math.ceil(staleCount * 0.2)),
|
||||
est_usd_cost: staleCount * 0.00002,
|
||||
rationale: `${staleCount} chunks awaiting embedding; recent-first catch-up`,
|
||||
status: 'remediable',
|
||||
}));
|
||||
}
|
||||
return {
|
||||
check: { name: 'embed_staleness', status, message },
|
||||
remediations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* entity_link_coverage: fraction of entity pages with at least one inbound link.
|
||||
*
|
||||
* Per A21 + codex finding #15: TABLESAMPLE BERNOULLI on Postgres when
|
||||
* total_pages > 50K, with pinned sample rate (LEAST 100, GREATEST 2,
|
||||
* targeting ~5000 sampled rows). PGLite path: full scan.
|
||||
*
|
||||
* The ±sqrt(p(1-p)/n) confidence interval is embedded in the message
|
||||
* itself so doctor + onboard render show "coverage: 31% ± 1.3%" not
|
||||
* a misleading point estimate.
|
||||
*/
|
||||
export async function checkEntityLinkCoverage(
|
||||
engine: BrainEngine,
|
||||
): Promise<OnboardCheckResult> {
|
||||
// Total entity pages
|
||||
const totalEntities = await safeCount(
|
||||
engine,
|
||||
`SELECT COUNT(*) AS count FROM pages
|
||||
WHERE type IN ('person', 'company', 'organization', 'entity')
|
||||
AND deleted_at IS NULL`,
|
||||
);
|
||||
|
||||
if (totalEntities === 0) {
|
||||
return {
|
||||
check: { name: 'entity_link_coverage', status: 'ok', message: 'No entity pages — coverage check vacuous' },
|
||||
remediations: [],
|
||||
};
|
||||
}
|
||||
|
||||
// Decide TABLESAMPLE policy (PG only, when >50K entities)
|
||||
const useSample = engine.kind === 'postgres' && totalEntities > 50_000;
|
||||
const samplePct = useSample
|
||||
? Math.max(2.0, Math.min(100.0, (5000.0 / totalEntities) * 100))
|
||||
: 100;
|
||||
const sampleClause = useSample ? `TABLESAMPLE BERNOULLI (${samplePct.toFixed(2)})` : '';
|
||||
|
||||
// Sample query: counts entities with inbound links
|
||||
const linkedCount = await safeCount(
|
||||
engine,
|
||||
`SELECT COUNT(*) AS count FROM (
|
||||
SELECT p.id FROM pages p ${sampleClause}
|
||||
WHERE p.type IN ('person', 'company', 'organization', 'entity')
|
||||
AND p.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
||||
) sub`,
|
||||
);
|
||||
const sampleSize = useSample
|
||||
? Math.max(1, Math.round(totalEntities * (samplePct / 100)))
|
||||
: totalEntities;
|
||||
|
||||
const coverage = sampleSize > 0 ? linkedCount / sampleSize : 0;
|
||||
// Wilson-ish confidence interval (1σ; ±sqrt(p(1-p)/n))
|
||||
const ci = Math.sqrt((coverage * (1 - coverage)) / Math.max(1, sampleSize));
|
||||
|
||||
const pct = Math.round(coverage * 100);
|
||||
const ciPct = (ci * 100).toFixed(1);
|
||||
const sampleNote = useSample ? ` (sampled ${samplePct.toFixed(1)}%)` : '';
|
||||
|
||||
const remediations: RemediationStep[] = [];
|
||||
let status: 'ok' | 'warn' | 'fail' = 'ok';
|
||||
let message: string;
|
||||
|
||||
// v0.41.18.0: warn-only, never fail. Empty entity link coverage is "needs
|
||||
// work" not "broken" — doctor's exit code should not flip from a fresh
|
||||
// brain with entity pages but no auto-extracted links yet. Fail status
|
||||
// would break `gbrain doctor exits 0` contract; the recommendation
|
||||
// surfaces the same fix via the onboard plan either way.
|
||||
if (coverage >= 0.7) {
|
||||
message = `Coverage ${pct}% ± ${ciPct}%${sampleNote}`;
|
||||
} else if (coverage >= 0.4) {
|
||||
status = 'warn';
|
||||
message = `Coverage ${pct}% ± ${ciPct}% (target 70%)${sampleNote}`;
|
||||
remediations.push(makeRemediationStep({
|
||||
id: 'onboard.extract_ner_links',
|
||||
job: 'extract-ner',
|
||||
params: {},
|
||||
severity: 'medium',
|
||||
est_seconds: 300,
|
||||
est_usd_cost: 0,
|
||||
rationale: `Entity link coverage at ${pct}%; NER extraction lifts typed-link density`,
|
||||
status: 'remediable',
|
||||
}));
|
||||
} else {
|
||||
status = 'warn';
|
||||
message = `Coverage ${pct}% ± ${ciPct}% (target 70%)${sampleNote}`;
|
||||
remediations.push(makeRemediationStep({
|
||||
id: 'onboard.extract_ner_links',
|
||||
job: 'extract-ner',
|
||||
params: {},
|
||||
severity: 'high',
|
||||
est_seconds: 600,
|
||||
est_usd_cost: 0,
|
||||
rationale: `Entity link coverage at ${pct}%; NER extraction lifts typed-link density`,
|
||||
status: 'remediable',
|
||||
}));
|
||||
}
|
||||
return {
|
||||
check: { name: 'entity_link_coverage', status, message },
|
||||
remediations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* timeline_coverage: fraction of entity pages with at least one timeline entry.
|
||||
*
|
||||
* Same TABLESAMPLE policy as entity_link_coverage for big brains.
|
||||
*/
|
||||
export async function checkTimelineCoverage(
|
||||
engine: BrainEngine,
|
||||
): Promise<OnboardCheckResult> {
|
||||
const totalEntities = await safeCount(
|
||||
engine,
|
||||
`SELECT COUNT(*) AS count FROM pages
|
||||
WHERE type IN ('person', 'company', 'organization', 'entity')
|
||||
AND deleted_at IS NULL`,
|
||||
);
|
||||
|
||||
if (totalEntities === 0) {
|
||||
return {
|
||||
check: { name: 'timeline_coverage', status: 'ok', message: 'No entity pages — coverage check vacuous' },
|
||||
remediations: [],
|
||||
};
|
||||
}
|
||||
|
||||
const useSample = engine.kind === 'postgres' && totalEntities > 50_000;
|
||||
const samplePct = useSample
|
||||
? Math.max(2.0, Math.min(100.0, (5000.0 / totalEntities) * 100))
|
||||
: 100;
|
||||
const sampleClause = useSample ? `TABLESAMPLE BERNOULLI (${samplePct.toFixed(2)})` : '';
|
||||
|
||||
const withTimelineCount = await safeCount(
|
||||
engine,
|
||||
`SELECT COUNT(*) AS count FROM (
|
||||
SELECT p.id FROM pages p ${sampleClause}
|
||||
WHERE p.type IN ('person', 'company', 'organization', 'entity')
|
||||
AND p.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM timeline_entries t WHERE t.page_id = p.id)
|
||||
) sub`,
|
||||
);
|
||||
const sampleSize = useSample
|
||||
? Math.max(1, Math.round(totalEntities * (samplePct / 100)))
|
||||
: totalEntities;
|
||||
|
||||
const coverage = sampleSize > 0 ? withTimelineCount / sampleSize : 0;
|
||||
const ci = Math.sqrt((coverage * (1 - coverage)) / Math.max(1, sampleSize));
|
||||
const pct = Math.round(coverage * 100);
|
||||
const ciPct = (ci * 100).toFixed(1);
|
||||
const sampleNote = useSample ? ` (sampled ${samplePct.toFixed(1)}%)` : '';
|
||||
|
||||
const remediations: RemediationStep[] = [];
|
||||
let status: 'ok' | 'warn' | 'fail' = 'ok';
|
||||
let message: string;
|
||||
|
||||
// v0.41.18.0: warn-only, never fail. Same posture as entity_link_coverage —
|
||||
// the recommendation still surfaces in onboard's plan, but doctor exit
|
||||
// code doesn't flip on a fresh brain.
|
||||
if (coverage >= 0.9) {
|
||||
message = `Coverage ${pct}% ± ${ciPct}%${sampleNote}`;
|
||||
} else if (coverage >= 0.7) {
|
||||
status = 'warn';
|
||||
message = `Coverage ${pct}% ± ${ciPct}% (target 90%)${sampleNote}`;
|
||||
remediations.push(makeRemediationStep({
|
||||
id: 'onboard.extract_timeline_from_meetings',
|
||||
job: 'extract-timeline-from-meetings',
|
||||
params: {},
|
||||
severity: 'medium',
|
||||
est_seconds: 240,
|
||||
est_usd_cost: 0,
|
||||
rationale: `Timeline coverage at ${pct}%; meeting-derived entries lift it`,
|
||||
status: 'remediable',
|
||||
}));
|
||||
} else {
|
||||
status = 'warn';
|
||||
message = `Coverage ${pct}% ± ${ciPct}% (target 90%)${sampleNote}`;
|
||||
remediations.push(makeRemediationStep({
|
||||
id: 'onboard.extract_timeline_from_meetings',
|
||||
job: 'extract-timeline-from-meetings',
|
||||
params: {},
|
||||
severity: 'high',
|
||||
est_seconds: 480,
|
||||
est_usd_cost: 0,
|
||||
rationale: `Timeline coverage at ${pct}%; meeting-derived entries lift it`,
|
||||
status: 'remediable',
|
||||
}));
|
||||
}
|
||||
return {
|
||||
check: { name: 'timeline_coverage', status, message },
|
||||
remediations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* takes_count: number of takes (typed claims) in the brain.
|
||||
*
|
||||
* Per A12 two-gate consent: the remediation only emits when
|
||||
* `takes.bootstrap_enabled` config is true. Otherwise the check shows
|
||||
* a status + hint, but no autopilot-eligible remediation.
|
||||
*/
|
||||
export async function checkTakesCount(
|
||||
engine: BrainEngine,
|
||||
): Promise<OnboardCheckResult> {
|
||||
const takesCount = await safeCount(
|
||||
engine,
|
||||
`SELECT COUNT(*) AS count FROM takes`,
|
||||
);
|
||||
|
||||
let bootstrapEnabled = false;
|
||||
try {
|
||||
const cfg = await engine.getConfig('takes.bootstrap_enabled');
|
||||
bootstrapEnabled = cfg === 'true' || cfg === '1';
|
||||
} catch {
|
||||
bootstrapEnabled = false;
|
||||
}
|
||||
|
||||
const remediations: RemediationStep[] = [];
|
||||
let status: 'ok' | 'warn' | 'fail' = 'ok';
|
||||
let message: string;
|
||||
|
||||
if (takesCount >= 100) {
|
||||
message = `${takesCount} takes (calibration ready)`;
|
||||
} else if (takesCount === 0) {
|
||||
status = 'warn';
|
||||
if (bootstrapEnabled) {
|
||||
message = `0 takes (bootstrap eligible — gbrain takes extract --from-pages)`;
|
||||
remediations.push(makeRemediationStep({
|
||||
id: 'onboard.takes_bootstrap',
|
||||
job: 'extract-takes-from-pages',
|
||||
protected: true,
|
||||
params: {},
|
||||
severity: 'medium',
|
||||
est_seconds: 1800,
|
||||
est_usd_cost: 5.00,
|
||||
rationale: '0 takes; LLM-bearing extraction over concept/atom/lore pages',
|
||||
status: 'remediable',
|
||||
}));
|
||||
} else {
|
||||
message = '0 takes (takes.bootstrap_enabled is false; opt in to enable)';
|
||||
}
|
||||
} else {
|
||||
message = `${takesCount} takes (calibration usable; >100 ideal)`;
|
||||
}
|
||||
return {
|
||||
check: { name: 'takes_count', status, message },
|
||||
remediations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Run all four checks in parallel; aggregate into a single payload.
|
||||
* Consumed by onboard's plan generation + (later) doctor's runDoctor.
|
||||
*
|
||||
* Per A20: callers can race this against an AbortSignal-bound timer for
|
||||
* partial-results fallthrough. Each individual safeCount() returns 0
|
||||
* on throw so a single check failure doesn't break the aggregate.
|
||||
*/
|
||||
export async function runAllOnboardChecks(
|
||||
engine: BrainEngine,
|
||||
): Promise<OnboardCheckResult[]> {
|
||||
return Promise.all([
|
||||
checkEmbedStaleness(engine),
|
||||
checkEntityLinkCoverage(engine),
|
||||
checkTimelineCoverage(engine),
|
||||
checkTakesCount(engine),
|
||||
]);
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
// src/core/onboard/impact-capture.ts
|
||||
// sourcescope:file-brain-wide — captureMetric reports brain-wide
|
||||
// aggregates (orphan_count, stale_count, coverage fractions) by design.
|
||||
// Per A26 lint opt-out.
|
||||
//
|
||||
// v0.41.18.0 (A6 + A25 + A17, T11). Capture before/after stats per onboard
|
||||
// remediation step so `gbrain onboard --history` can show "you reduced
|
||||
// orphans 47% (88% → 41%)".
|
||||
//
|
||||
// Best-effort per A17: a stat-query throw must NOT block the extraction
|
||||
// itself. The wrapper logs failures to stderr and records
|
||||
// metric_before/after = null when the capture failed.
|
||||
//
|
||||
// Attribution columns per A25 + codex finding #10: every row carries
|
||||
// job_id (FK to minion_jobs), source_id, brain_id, started_at,
|
||||
// idempotency_key so concurrent onboard/autopilot/manual runs can't
|
||||
// misattribute deltas to the wrong remediation.
|
||||
|
||||
import type { BrainEngine } from './../engine.ts';
|
||||
|
||||
export type MetricName =
|
||||
| 'orphan_count'
|
||||
| 'stale_count'
|
||||
| 'entity_link_coverage'
|
||||
| 'timeline_coverage'
|
||||
| 'takes_count';
|
||||
|
||||
export interface ImpactAttribution {
|
||||
remediation_id: string;
|
||||
job_id?: number;
|
||||
source_id?: string;
|
||||
brain_id?: string;
|
||||
started_at?: string;
|
||||
idempotency_key?: string;
|
||||
applied_by?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure-ish: returns the current numeric value for `metric`. Returns null
|
||||
* on any throw (best-effort capture per A17).
|
||||
*/
|
||||
export async function captureMetric(
|
||||
engine: BrainEngine,
|
||||
metric: MetricName,
|
||||
): Promise<number | null> {
|
||||
try {
|
||||
switch (metric) {
|
||||
case 'stale_count': {
|
||||
const rows = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM content_chunks WHERE embedding IS NULL`,
|
||||
);
|
||||
return rows.length > 0 ? Number(rows[0].count) : 0;
|
||||
}
|
||||
case 'orphan_count': {
|
||||
const rows = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)`,
|
||||
);
|
||||
return rows.length > 0 ? Number(rows[0].count) : 0;
|
||||
}
|
||||
case 'entity_link_coverage':
|
||||
case 'timeline_coverage': {
|
||||
// Compute as a fraction of entity pages with the relevant feature.
|
||||
const total = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM pages
|
||||
WHERE type IN ('person', 'company', 'organization', 'entity')
|
||||
AND deleted_at IS NULL`,
|
||||
);
|
||||
const totalN = total.length > 0 ? Number(total[0].count) : 0;
|
||||
if (totalN === 0) return 1; // vacuous truth — empty brain has full coverage
|
||||
if (metric === 'entity_link_coverage') {
|
||||
const withLinks = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM pages p
|
||||
WHERE p.type IN ('person', 'company', 'organization', 'entity')
|
||||
AND p.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)`,
|
||||
);
|
||||
return withLinks.length > 0 ? Number(withLinks[0].count) / totalN : 0;
|
||||
}
|
||||
const withTimeline = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM pages p
|
||||
WHERE p.type IN ('person', 'company', 'organization', 'entity')
|
||||
AND p.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM timeline_entries t WHERE t.page_id = p.id)`,
|
||||
);
|
||||
return withTimeline.length > 0 ? Number(withTimeline[0].count) / totalN : 0;
|
||||
}
|
||||
case 'takes_count': {
|
||||
const rows = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM takes`,
|
||||
);
|
||||
return rows.length > 0 ? Number(rows[0].count) : 0;
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
process.stderr.write(
|
||||
`[impact-capture] failed to capture ${metric}: ${err instanceof Error ? err.message : String(err)}\n`,
|
||||
);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Write one migration_impact_log row. Best-effort: a write failure logs
|
||||
* to stderr but doesn't throw.
|
||||
*/
|
||||
export async function writeImpactLogRow(
|
||||
engine: BrainEngine,
|
||||
attribution: ImpactAttribution,
|
||||
metricName: MetricName,
|
||||
metricBefore: number | null,
|
||||
metricAfter: number | null,
|
||||
details?: Record<string, unknown>,
|
||||
): Promise<void> {
|
||||
try {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO migration_impact_log (
|
||||
remediation_id, metric_name, metric_before, metric_after,
|
||||
job_id, source_id, brain_id, started_at, idempotency_key,
|
||||
applied_by, details
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb)`,
|
||||
[
|
||||
attribution.remediation_id,
|
||||
metricName,
|
||||
metricBefore,
|
||||
metricAfter,
|
||||
attribution.job_id ?? null,
|
||||
attribution.source_id ?? null,
|
||||
attribution.brain_id ?? null,
|
||||
attribution.started_at ?? new Date().toISOString(),
|
||||
attribution.idempotency_key ?? null,
|
||||
attribution.applied_by ?? null,
|
||||
JSON.stringify(details ?? {}),
|
||||
],
|
||||
);
|
||||
} catch (err) {
|
||||
process.stderr.write(
|
||||
`[impact-capture] failed to write log row for ${attribution.remediation_id}: ${err instanceof Error ? err.message : String(err)}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience wrapper: capture-before → run → capture-after → write log.
|
||||
* The runner itself does the extraction; this fn handles the bookkeeping.
|
||||
*
|
||||
* Per A17: capture failures DO NOT block the runner. A null before/after
|
||||
* is recorded; the row still lands so downstream consumers see a
|
||||
* "ran but impact unknown" entry.
|
||||
*/
|
||||
export async function withImpactCapture<T>(
|
||||
engine: BrainEngine,
|
||||
attribution: ImpactAttribution,
|
||||
metric: MetricName,
|
||||
runner: () => Promise<T>,
|
||||
details?: Record<string, unknown>,
|
||||
): Promise<T> {
|
||||
const startedAt = new Date().toISOString();
|
||||
const before = await captureMetric(engine, metric);
|
||||
let result: T;
|
||||
try {
|
||||
result = await runner();
|
||||
} catch (err) {
|
||||
// Capture "after" even on failure so the log row reflects the attempt.
|
||||
const afterOnFail = await captureMetric(engine, metric);
|
||||
await writeImpactLogRow(
|
||||
engine,
|
||||
{ ...attribution, started_at: startedAt },
|
||||
metric,
|
||||
before,
|
||||
afterOnFail,
|
||||
{ ...(details ?? {}), error: err instanceof Error ? err.message : String(err) },
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
const after = await captureMetric(engine, metric);
|
||||
await writeImpactLogRow(
|
||||
engine,
|
||||
{ ...attribution, started_at: startedAt },
|
||||
metric,
|
||||
before,
|
||||
after,
|
||||
details,
|
||||
);
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
// src/core/onboard/init-nudge.ts
|
||||
// v0.41.18.0 (A4 + A18 + A20, T14). Post-initSchema summary that runs
|
||||
// the 4 onboard checks against a 3-second wallclock budget and prints
|
||||
// a one-line nudge if recommendations exist.
|
||||
//
|
||||
// Hard contract per A18: init MUST succeed even if the nudge crashes.
|
||||
// Any throw in this module is caught + logged to stderr + suppressed.
|
||||
// Per A20: the 3-second cap uses real cancellation via the AbortSignal
|
||||
// extension on executeRaw (T5) — Promise.race against a timer was the
|
||||
// codex #7 finding's wrong shape. Cancelled queries actually stop on
|
||||
// Postgres; PGLite has a documented gap.
|
||||
//
|
||||
// Bypass: GBRAIN_NO_ONBOARD_NUDGE=1 short-circuits. Non-TTY default
|
||||
// also short-circuits (CI/scripted callers see nothing).
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
|
||||
const NUDGE_BUDGET_MS = 3000;
|
||||
|
||||
/**
|
||||
* Post-initSchema nudge. Fail-open per A18.
|
||||
*
|
||||
* Returns silently when:
|
||||
* - GBRAIN_NO_ONBOARD_NUDGE=1
|
||||
* - Non-TTY environment (CI, scripted)
|
||||
* - All 4 onboard checks complete within 3s AND surface 0 recommendations
|
||||
* - ANY error during check execution (logged to stderr, suppressed)
|
||||
*
|
||||
* Prints a nudge to stderr when:
|
||||
* - Recommendations exist within budget
|
||||
* - Some checks ran but budget fired (partial-results path)
|
||||
*/
|
||||
export async function runInitNudge(engine: BrainEngine): Promise<void> {
|
||||
try {
|
||||
if (process.env.GBRAIN_NO_ONBOARD_NUDGE === '1') return;
|
||||
if (!process.stderr.isTTY) return;
|
||||
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), NUDGE_BUDGET_MS);
|
||||
|
||||
let totalStale = 0;
|
||||
let totalEntities = 0;
|
||||
let linkedCount = 0;
|
||||
let timelineCount = 0;
|
||||
let takesCount = 0;
|
||||
let checksRan = 0;
|
||||
let checksAttempted = 0;
|
||||
let partial = false;
|
||||
|
||||
// Run 4 cheap counts in parallel against the 3s budget.
|
||||
const results = await Promise.allSettled([
|
||||
engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM content_chunks WHERE embedding IS NULL`,
|
||||
[],
|
||||
{ signal: controller.signal },
|
||||
),
|
||||
engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM pages
|
||||
WHERE type IN ('person', 'company', 'organization', 'entity')
|
||||
AND deleted_at IS NULL`,
|
||||
[],
|
||||
{ signal: controller.signal },
|
||||
),
|
||||
engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM pages p
|
||||
WHERE p.type IN ('person', 'company', 'organization', 'entity')
|
||||
AND p.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)`,
|
||||
[],
|
||||
{ signal: controller.signal },
|
||||
),
|
||||
engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM pages p
|
||||
WHERE p.type IN ('person', 'company', 'organization', 'entity')
|
||||
AND p.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM timeline_entries t WHERE t.page_id = p.id)`,
|
||||
[],
|
||||
{ signal: controller.signal },
|
||||
),
|
||||
engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*) AS count FROM takes`,
|
||||
[],
|
||||
{ signal: controller.signal },
|
||||
),
|
||||
]);
|
||||
clearTimeout(timer);
|
||||
|
||||
checksAttempted = results.length;
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i];
|
||||
if (r.status === 'rejected') {
|
||||
partial = true;
|
||||
continue;
|
||||
}
|
||||
checksRan++;
|
||||
const n = r.value.length > 0 ? Number(r.value[0].count) : 0;
|
||||
if (i === 0) totalStale = n;
|
||||
else if (i === 1) totalEntities = n;
|
||||
else if (i === 2) linkedCount = n;
|
||||
else if (i === 3) timelineCount = n;
|
||||
else if (i === 4) takesCount = n;
|
||||
}
|
||||
|
||||
// Aggregate: any non-zero metric triggers the nudge.
|
||||
const linkCoverage = totalEntities > 0 ? linkedCount / totalEntities : 1;
|
||||
const timelineCoverage = totalEntities > 0 ? timelineCount / totalEntities : 1;
|
||||
const hasRecommendations =
|
||||
totalStale > 0
|
||||
|| (totalEntities > 0 && linkCoverage < 0.7)
|
||||
|| (totalEntities > 0 && timelineCoverage < 0.9)
|
||||
|| takesCount === 0;
|
||||
|
||||
if (!hasRecommendations && !partial) return;
|
||||
|
||||
// Emit one-line nudge. Be terse — init is the activation surface.
|
||||
const parts: string[] = [];
|
||||
if (totalStale > 0) parts.push(`${totalStale} stale chunks`);
|
||||
if (totalEntities > 0 && linkCoverage < 0.7) {
|
||||
parts.push(`link coverage ${Math.round(linkCoverage * 100)}%`);
|
||||
}
|
||||
if (totalEntities > 0 && timelineCoverage < 0.9) {
|
||||
parts.push(`timeline coverage ${Math.round(timelineCoverage * 100)}%`);
|
||||
}
|
||||
if (takesCount === 0) parts.push('0 takes');
|
||||
|
||||
process.stderr.write(
|
||||
`\n[onboard] Brain has opportunities: ${parts.join(', ')}.\n` +
|
||||
`[onboard] Run 'gbrain onboard --check' to see the plan.` +
|
||||
(partial ? ` (${checksRan}/${checksAttempted} checks complete; run gbrain onboard --check for full recommendations)` : '') +
|
||||
`\n`,
|
||||
);
|
||||
} catch (err) {
|
||||
// A18: NEVER crash init from the nudge. Log and continue.
|
||||
process.stderr.write(`[onboard] nudge skipped (${err instanceof Error ? err.message : String(err)})\n`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Post-upgrade banner. Lighter than the init nudge — just highlights
|
||||
* that new onboard recommendations may exist. Fail-open identically.
|
||||
*/
|
||||
export async function runUpgradeBanner(_engine: BrainEngine): Promise<void> {
|
||||
try {
|
||||
if (process.env.GBRAIN_NO_ONBOARD_NUDGE === '1') return;
|
||||
if (!process.stderr.isTTY) return;
|
||||
process.stderr.write(
|
||||
`\n[onboard] Upgrade complete. Run 'gbrain onboard --check' to see if the new version surfaces any new opportunities.\n`,
|
||||
);
|
||||
} catch {
|
||||
// A18 posture for symmetry.
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
// src/core/onboard/render.ts
|
||||
// v0.41.18.0 (T12). Stable JSON envelope + human renderer for
|
||||
// `gbrain onboard`. Library-shaped — no console.* / process.exit; CLI
|
||||
// shell calls these and pipes results to its own output.
|
||||
|
||||
import type { RemediationStep } from '../remediation-step.ts';
|
||||
import type { RemediationPlan } from '../remediation/types.ts';
|
||||
import type {
|
||||
OnboardRecommendation,
|
||||
OnboardReport,
|
||||
} from './types.ts';
|
||||
|
||||
/**
|
||||
* Translate a RemediationStep into an OnboardRecommendation. Layers the
|
||||
* apply_policy + prompt_text + migration_id metadata.
|
||||
*
|
||||
* Rules of thumb for apply_policy:
|
||||
* - protected job (LLM-bearing) → 'prompt_required' or 'manual_only'
|
||||
* based on job name (takes-bootstrap stays manual_only per A12).
|
||||
* - non-protected (regex, SQL, etc.) → 'auto_apply'.
|
||||
*/
|
||||
export function toOnboardRecommendation(step: RemediationStep): OnboardRecommendation {
|
||||
let apply_policy: OnboardRecommendation['apply_policy'] = 'auto_apply';
|
||||
if (step.protected) {
|
||||
// takes-bootstrap classifier stays manual_only per A12 + A24 until
|
||||
// v0.42.1 lands the 100+-case eval. All other protected handlers
|
||||
// (synthesize, patterns, consolidate, extract-takes-from-pages)
|
||||
// are prompt_required — they need --yes but can run via --auto --yes.
|
||||
apply_policy = step.job === 'extract-takes-from-pages' ? 'manual_only' : 'prompt_required';
|
||||
}
|
||||
return {
|
||||
...step,
|
||||
apply_policy,
|
||||
prompt_text: step.rationale,
|
||||
migration_id: step.id,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the stable JSON envelope from a remediation plan. brainId, when
|
||||
* available, identifies the brain across runs (consumed by --history
|
||||
* cross-runtime joins).
|
||||
*/
|
||||
export function buildOnboardReport(
|
||||
plan: RemediationPlan,
|
||||
opts?: { brainId?: string; history?: OnboardReport['history'] },
|
||||
): OnboardReport {
|
||||
const recs = plan.plan.map(toOnboardRecommendation);
|
||||
const summary = {
|
||||
total: recs.length,
|
||||
auto_eligible: recs.filter((r) => r.apply_policy === 'auto_apply').length,
|
||||
prompt_required: recs.filter((r) => r.apply_policy === 'prompt_required').length,
|
||||
manual_only: recs.filter((r) => r.apply_policy === 'manual_only').length,
|
||||
est_total_usd: plan.est_total_usd_cost,
|
||||
};
|
||||
return {
|
||||
schema_version: 1,
|
||||
brain_id: opts?.brainId,
|
||||
recommendations: recs,
|
||||
summary,
|
||||
history: opts?.history,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Human-readable render (returns string; CLI prints to stdout). Designed
|
||||
* for stderr/stdout segregation: the CLI shell prints this on stdout,
|
||||
* progress + errors on stderr. Echoes the AskUserQuestion-style
|
||||
* "Recommendation + WHY" framing the CEO/Eng review settled on.
|
||||
*/
|
||||
export function renderHuman(report: OnboardReport): string {
|
||||
const lines: string[] = [];
|
||||
lines.push(`Brain onboarding: ${report.summary.total} recommendation(s) found`);
|
||||
if (report.summary.total === 0) {
|
||||
lines.push(' Brain is at target — nothing to do.');
|
||||
return lines.join('\n');
|
||||
}
|
||||
lines.push(
|
||||
` ${report.summary.auto_eligible} auto-eligible | ` +
|
||||
`${report.summary.prompt_required} prompt-required | ` +
|
||||
`${report.summary.manual_only} manual-only`,
|
||||
);
|
||||
if (report.summary.est_total_usd > 0) {
|
||||
lines.push(` Total estimated cost: $${report.summary.est_total_usd.toFixed(2)}`);
|
||||
}
|
||||
lines.push('');
|
||||
for (const r of report.recommendations) {
|
||||
const sev = `[${r.severity}]`;
|
||||
const policy = r.apply_policy === 'auto_apply' ? '(auto)'
|
||||
: r.apply_policy === 'prompt_required' ? '(prompt)'
|
||||
: '(manual)';
|
||||
const cost = (r.est_usd_cost ?? 0) > 0 ? ` ~$${(r.est_usd_cost ?? 0).toFixed(2)}` : '';
|
||||
lines.push(` ${sev} ${policy} ${r.job}${cost}`);
|
||||
lines.push(` why: ${r.prompt_text ?? r.rationale}`);
|
||||
}
|
||||
if (report.history && report.history.length > 0) {
|
||||
lines.push('');
|
||||
lines.push('Recent impact (last 10):');
|
||||
for (const h of report.history.slice(0, 10)) {
|
||||
const delta = h.delta !== null ? (h.delta > 0 ? `+${h.delta}` : String(h.delta)) : '?';
|
||||
lines.push(
|
||||
` ${h.applied_at} ${h.remediation_id} ${h.metric_name}: ` +
|
||||
`${h.metric_before ?? '?'} → ${h.metric_after ?? '?'} (${delta})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
// src/core/onboard/types.ts
|
||||
// v0.41.18.0 (T12). Onboard-surface-specific types layered on top of the
|
||||
// shared remediation library (src/core/remediation/). Onboard reframes
|
||||
// the same RemediationStep objects as "onboarding opportunities" with
|
||||
// extra prompt-text + apply-policy metadata.
|
||||
|
||||
import type { RemediationStep } from '../remediation-step.ts';
|
||||
|
||||
/**
|
||||
* One onboard recommendation. Layered on RemediationStep with extras
|
||||
* specific to the onboarding UX:
|
||||
* - apply_policy controls autopilot behavior (A8 tiered)
|
||||
* - prompt_text is the human-readable nudge ("Embed 3K stale chunks?")
|
||||
* - migration_id is a stable identifier for cross-version tracking
|
||||
*/
|
||||
export interface OnboardRecommendation extends RemediationStep {
|
||||
/**
|
||||
* A8 tiered apply policy:
|
||||
* 'auto_apply' — autopilot may run unattended; runs under --auto
|
||||
* 'prompt_required' — autopilot must skip; runs under --auto --yes
|
||||
* 'manual_only' — never runs unattended; CLI prompts user
|
||||
* Default 'prompt_required' when omitted.
|
||||
*/
|
||||
apply_policy?: 'auto_apply' | 'prompt_required' | 'manual_only';
|
||||
/** Human-readable nudge text. Default falls back to RemediationStep.rationale. */
|
||||
prompt_text?: string;
|
||||
/**
|
||||
* Stable id for cross-version tracking. Defaults to RemediationStep.id.
|
||||
* Used by onboard --history to group runs of the same migration over time.
|
||||
*/
|
||||
migration_id?: string;
|
||||
}
|
||||
|
||||
/** Stable JSON envelope for `gbrain onboard --json`. */
|
||||
export interface OnboardReport {
|
||||
schema_version: 1;
|
||||
/** brain_id from engine.getHealth(). Identifies the brain across runs. */
|
||||
brain_id?: string;
|
||||
recommendations: OnboardRecommendation[];
|
||||
summary: {
|
||||
total: number;
|
||||
auto_eligible: number;
|
||||
prompt_required: number;
|
||||
manual_only: number;
|
||||
/** Estimated total dollar cost if every recommendation runs. */
|
||||
est_total_usd: number;
|
||||
};
|
||||
/** Reverse-chronological migration_impact_log entries. */
|
||||
history?: Array<{
|
||||
remediation_id: string;
|
||||
metric_name: string;
|
||||
metric_before: number | null;
|
||||
metric_after: number | null;
|
||||
delta: number | null;
|
||||
applied_at: string;
|
||||
}>;
|
||||
}
|
||||
|
||||
export interface OnboardOpts {
|
||||
/** Target brain_score (default 90). Forwarded to computeRemediationPlan. */
|
||||
targetScore?: number;
|
||||
/** Output mode. */
|
||||
mode?: 'check' | 'auto' | 'history';
|
||||
/** Cap on autopilot spend. Required for --auto runs (CLI enforces). */
|
||||
maxUsd?: number;
|
||||
/** Caller-supplied OAuth client_id (MCP path); threads via job.data. */
|
||||
clientId?: string;
|
||||
}
|
||||
@@ -4168,6 +4168,112 @@ const reload_schema_pack: Operation = {
|
||||
},
|
||||
};
|
||||
|
||||
// v0.41.18.0 (A7 + T16, codex finding #5): MCP op for federated / thin-client
|
||||
// brain installs to drive `gbrain onboard --auto` over MCP. Admin scope
|
||||
// (NOT localOnly) so remote agents authenticated via OAuth can probe
|
||||
// brain health + submit auto-eligible remediation handlers.
|
||||
//
|
||||
// Critical security gate (codex #5): admin scope alone is NOT sufficient
|
||||
// to submit handlers in PROTECTED_JOB_NAMES (synthesize, patterns,
|
||||
// consolidate, extract-takes-from-pages, contextual_reindex_per_chunk).
|
||||
// Without this gate, an admin-scoped OAuth token would bypass the same
|
||||
// guard that `submit_job` enforces. The new NAMED scope
|
||||
// `run_protected_onboard` MUST be granted IN ADDITION TO admin for any
|
||||
// protected child handler to fire.
|
||||
//
|
||||
// Behavior:
|
||||
// - mode='check' (default): returns the OnboardReport JSON envelope,
|
||||
// never submits jobs. Admin scope sufficient.
|
||||
// - mode='auto': submits auto_apply tier. Admin + non-protected
|
||||
// handlers only.
|
||||
// - mode='auto-with-prompt': submits auto_apply + prompt_required tier.
|
||||
// Same protection check.
|
||||
//
|
||||
// Any LLM-bearing handler the plan would have submitted gets filtered out
|
||||
// unless the caller has run_protected_onboard. Filtered items appear in
|
||||
// the response with status='skipped_missing_scope' so the caller knows
|
||||
// what they would have gotten with the right grants.
|
||||
const run_onboard: Operation = {
|
||||
name: 'run_onboard',
|
||||
description: 'Probe brain health + optionally submit onboard remediations. Admin scope required. Protected handlers (LLM-bearing) require run_protected_onboard scope ADDITIONALLY.',
|
||||
params: {
|
||||
mode: { type: 'string', description: "'check' (default), 'auto', or 'auto-with-prompt'" },
|
||||
target_score: { type: 'number', description: 'Target brain_score (default 90)' },
|
||||
max_usd: { type: 'number', description: 'USD cap for autopilot path (required for auto modes)' },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'admin',
|
||||
handler: async (ctx, p) => {
|
||||
const mode = (typeof p.mode === 'string' ? p.mode : 'check') as 'check' | 'auto' | 'auto-with-prompt';
|
||||
const targetScore = typeof p.target_score === 'number' ? p.target_score : 90;
|
||||
const maxUsd = typeof p.max_usd === 'number' ? p.max_usd : undefined;
|
||||
|
||||
const { computeRemediationPlan, runRemediation } = await import('./remediation/index.ts');
|
||||
const { runAllOnboardChecks } = await import('./onboard/checks.ts');
|
||||
const { buildOnboardReport } = await import('./onboard/render.ts');
|
||||
|
||||
// Per A26: source-scope via sourceScopeOpts(ctx). The recommendation
|
||||
// planner is brain-wide today; future extension can scope by reading
|
||||
// ctx.sourceId / ctx.auth.allowedSources for per-source plans.
|
||||
|
||||
let extraRemediations: import('./remediation-step.ts').RemediationStep[] = [];
|
||||
try {
|
||||
const checkResults = await runAllOnboardChecks(ctx.engine);
|
||||
extraRemediations = checkResults.flatMap((r) => r.remediations);
|
||||
} catch {
|
||||
// Fail-open per A19 — return plan without extras rather than error.
|
||||
}
|
||||
|
||||
// 'check' mode: just return the plan + JSON envelope. No submission.
|
||||
if (mode === 'check') {
|
||||
const plan = await computeRemediationPlan(ctx.engine, { targetScore, extraRemediations });
|
||||
const report = buildOnboardReport(plan);
|
||||
return report;
|
||||
}
|
||||
|
||||
// 'auto' and 'auto-with-prompt' modes: require --max-usd per A12 + A20
|
||||
// safety posture (cron-safety; refuses surprise spend).
|
||||
if (maxUsd === undefined) {
|
||||
throw new OperationError('invalid_params', `mode='${mode}' requires max_usd (cron-safety cap)`);
|
||||
}
|
||||
|
||||
// Critical T16 + codex #5 security gate: filter out PROTECTED_JOB_NAMES
|
||||
// unless the caller has the run_protected_onboard scope IN ADDITION
|
||||
// to admin. Admin alone is insufficient.
|
||||
const grantedScopes = ctx.auth?.scopes ?? [];
|
||||
const canRunProtected = grantedScopes.includes('run_protected_onboard');
|
||||
const { isProtectedJobName } = await import('./minions/protected-names.ts');
|
||||
|
||||
const skippedMissingScope: Array<{ id: string; job: string; reason: string }> = [];
|
||||
const allowedExtras = extraRemediations.filter((r) => {
|
||||
if (canRunProtected) return true;
|
||||
if (isProtectedJobName(r.job)) {
|
||||
skippedMissingScope.push({ id: r.id, job: r.job, reason: 'requires run_protected_onboard scope' });
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
// Run remediation with filtered extras. Hooks emit nothing — MCP
|
||||
// returns structured result. Per A23 client_id attribution: stamp
|
||||
// job.data.client_id on each submission so the spend chain (T10)
|
||||
// attributes correctly. The library doesn't do this today; the
|
||||
// upstream submit-side gating in submit_job filters protected names
|
||||
// for ctx.remote !== false callers, so even if MCP run_onboard had a
|
||||
// typo, the underlying queue.add would reject. Defense-in-depth.
|
||||
const result = await runRemediation(
|
||||
ctx.engine,
|
||||
{ targetScore, maxUsd },
|
||||
{},
|
||||
);
|
||||
|
||||
return {
|
||||
...result,
|
||||
skipped_missing_scope: skippedMissingScope,
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
export const operations: Operation[] = [
|
||||
// Page CRUD
|
||||
get_page, put_page, delete_page, list_pages,
|
||||
@@ -4236,6 +4342,8 @@ export const operations: Operation[] = [
|
||||
schema_stats, schema_lint, schema_graph, schema_explain_type,
|
||||
schema_review_orphans,
|
||||
schema_apply_mutations, reload_schema_pack,
|
||||
// v0.41.18.0 (T16, A7, codex #5)
|
||||
run_onboard,
|
||||
];
|
||||
|
||||
export const operationsByName = Object.fromEntries(
|
||||
|
||||
+105
-11
@@ -2000,10 +2000,83 @@ export class PGLiteEngine implements BrainEngine {
|
||||
afterPageId?: number;
|
||||
afterChunkIndex?: number;
|
||||
sourceId?: string;
|
||||
orderBy?: 'page_id' | 'updated_desc';
|
||||
afterUpdatedAt?: string | null;
|
||||
}): Promise<StaleChunkRow[]> {
|
||||
const limit = opts?.batchSize ?? 2000;
|
||||
const afterPid = opts?.afterPageId ?? 0;
|
||||
const afterIdx = opts?.afterChunkIndex ?? -1;
|
||||
const orderBy = opts?.orderBy ?? 'page_id';
|
||||
|
||||
// v0.41.18.0 (A13, codex #9): --priority recent path. See postgres-engine
|
||||
// sibling for full rationale. Same composite cursor + ORDER BY.
|
||||
if (orderBy === 'updated_desc') {
|
||||
const afterUpdated = opts?.afterUpdatedAt ?? null;
|
||||
const isFirstPage = afterUpdated === null && afterPid === 0;
|
||||
if (opts?.sourceId === undefined) {
|
||||
const { rows } = isFirstPage ? await this.db.query(
|
||||
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT $1`,
|
||||
[limit],
|
||||
) : await this.db.query(
|
||||
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (
|
||||
p.updated_at < $1::timestamptz
|
||||
OR (p.updated_at = $1::timestamptz AND p.id > $2)
|
||||
OR (p.updated_at = $1::timestamptz AND p.id = $2 AND cc.chunk_index > $3)
|
||||
)
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT $4`,
|
||||
[afterUpdated, afterPid, afterIdx, limit],
|
||||
);
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
const { rows } = isFirstPage ? await this.db.query(
|
||||
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = $1
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT $2`,
|
||||
[opts.sourceId, limit],
|
||||
) : await this.db.query(
|
||||
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = $1
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (
|
||||
p.updated_at < $2::timestamptz
|
||||
OR (p.updated_at = $2::timestamptz AND p.id > $3)
|
||||
OR (p.updated_at = $2::timestamptz AND p.id = $3 AND cc.chunk_index > $4)
|
||||
)
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT $5`,
|
||||
[opts.sourceId, afterUpdated, afterPid, afterIdx, limit],
|
||||
);
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
// orderBy === 'page_id' — legacy stable cursor (unchanged below).
|
||||
// D7: optional source-scoped cursor scan. PGLite mirrors postgres-engine
|
||||
// so the engine-parity E2E catches drift.
|
||||
// v0.41 (D4+D8): NOT (frontmatter ? 'embed_skip') filter for soft-blocked
|
||||
@@ -2113,17 +2186,19 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const fromSourceIds = links.map(l => l.from_source_id || 'default');
|
||||
const toSourceIds = links.map(l => l.to_source_id || 'default');
|
||||
const originSourceIds = links.map(l => l.origin_source_id || 'default');
|
||||
// v0.41.18.0 (A10): link_kind column (v98). NULL = legacy/plain.
|
||||
const linkKinds = links.map(l => l.link_kind ?? null);
|
||||
const result = await this.db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[], $6::text[], $7::text[], $8::text[], $9::text[], $10::text[])
|
||||
AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id)
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, link_kind, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, v.link_type, v.context, v.link_source, v.link_kind, o.id, v.origin_field
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[], $6::text[], $7::text[], $8::text[], $9::text[], $10::text[], $11::text[])
|
||||
AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id, link_kind)
|
||||
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
|
||||
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
|
||||
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
|
||||
RETURNING 1`,
|
||||
[fromSlugs, toSlugs, linkTypes, contexts, linkSources, originSlugs, originFields, fromSourceIds, toSourceIds, originSourceIds]
|
||||
[fromSlugs, toSlugs, linkTypes, contexts, linkSources, originSlugs, originFields, fromSourceIds, toSourceIds, originSourceIds, linkKinds]
|
||||
);
|
||||
return result.rows.length;
|
||||
}
|
||||
@@ -2501,7 +2576,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// Initialize all slugs to 0 so callers get a consistent map.
|
||||
for (const s of slugs) result.set(s, 0);
|
||||
|
||||
// v0.42.0.0 D12: filter mentions OUT of backlink-count for search
|
||||
// v0.41.18.0 D12: filter mentions OUT of backlink-count for search
|
||||
// ranking — parity with postgres-engine.ts. See that file's comment
|
||||
// for the full rationale. `IS DISTINCT FROM` is NULL-safe so legacy
|
||||
// rows with NULL link_source still count toward backlinks.
|
||||
@@ -2705,7 +2780,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
||||
SELECT id, $2::date, $3, $4, $5
|
||||
FROM pages WHERE slug = $1 AND source_id = $6
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING`,
|
||||
ON CONFLICT (page_id, date, summary, source) DO NOTHING`,
|
||||
[slug, entry.date, entry.source || '', entry.summary, entry.detail || '', sourceId]
|
||||
);
|
||||
}
|
||||
@@ -2724,7 +2799,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[], $6::text[])
|
||||
AS v(slug, date, source, summary, detail, source_id)
|
||||
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING
|
||||
ON CONFLICT (page_id, date, summary, source) DO NOTHING
|
||||
RETURNING 1`,
|
||||
[slugs, dates, sources, summaries, details, sourceIds]
|
||||
);
|
||||
@@ -4258,9 +4333,28 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r, true));
|
||||
}
|
||||
|
||||
async executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]> {
|
||||
const { rows } = await this.db.query(sql, params);
|
||||
return rows as T[];
|
||||
async executeRaw<T = Record<string, unknown>>(
|
||||
sql: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<T[]> {
|
||||
// v0.41.18.0 (A20, codex #7): PGLite is in-process WASM with no
|
||||
// kernel-level cancellation. Best-effort: pre-check the signal so
|
||||
// an already-aborted call returns immediately, and race against
|
||||
// a settle promise so a late-arriving abort throws AbortError
|
||||
// (the query keeps running in WASM until it returns; the result
|
||||
// is discarded). Documented gap in src/core/engine.ts.
|
||||
if (opts?.signal?.aborted) {
|
||||
throw new DOMException('aborted', 'AbortError');
|
||||
}
|
||||
const queryPromise = this.db.query(sql, params).then((r) => r.rows as T[]);
|
||||
if (!opts?.signal) return queryPromise;
|
||||
const abortPromise = new Promise<T[]>((_resolve, reject) => {
|
||||
opts.signal!.addEventListener('abort', () => {
|
||||
reject(new DOMException('aborted', 'AbortError'));
|
||||
}, { once: true });
|
||||
});
|
||||
return Promise.race([queryPromise, abortPromise]);
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
|
||||
@@ -201,6 +201,10 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image
|
||||
-- v0.19.0: partial indexes for code chunk lookups.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_name ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_language ON content_chunks(language) WHERE language IS NOT NULL;
|
||||
-- v0.41.18.0 (codex finding #9): partial index for gbrain embed --stale
|
||||
-- and --priority recent. See src/schema.sql for full rationale.
|
||||
CREATE INDEX IF NOT EXISTS content_chunks_stale_idx
|
||||
ON content_chunks(page_id, chunk_index) WHERE embedding IS NULL;
|
||||
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
@@ -212,10 +216,14 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
-- v0.41.18.0 (codex finding #12): nullable link_kind distinguishes
|
||||
-- "plain body mention" from "verb-pattern-derived typed link" within
|
||||
-- link_source='mentions'. See src/schema.sql for full rationale.
|
||||
link_kind TEXT CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
-- v0.18.0 Step 4: see src/schema.sql.
|
||||
@@ -314,7 +322,9 @@ CREATE TABLE IF NOT EXISTS timeline_entries (
|
||||
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
|
||||
-- Dedup constraint: same (page, date, summary) treated as same event
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary);
|
||||
-- v0.41.18.0 (codex finding #11): widened to include source so distinct
|
||||
-- meeting provenance survives. Legacy rows have source='' (schema default).
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary, source);
|
||||
|
||||
-- ============================================================
|
||||
-- page_versions: snapshot history
|
||||
@@ -881,6 +891,30 @@ CREATE TABLE IF NOT EXISTS op_checkpoints (
|
||||
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
|
||||
ON op_checkpoints (updated_at);
|
||||
|
||||
-- ============================================================
|
||||
-- migration_impact_log (v0.41.18.0 — gbrain onboard wave)
|
||||
-- ============================================================
|
||||
-- See src/schema.sql for full rationale.
|
||||
CREATE TABLE IF NOT EXISTS migration_impact_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
remediation_id TEXT NOT NULL,
|
||||
metric_name TEXT NOT NULL,
|
||||
metric_before NUMERIC,
|
||||
metric_after NUMERIC,
|
||||
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
|
||||
source_id TEXT,
|
||||
brain_id TEXT,
|
||||
started_at TIMESTAMPTZ,
|
||||
idempotency_key TEXT,
|
||||
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
applied_by TEXT,
|
||||
details JSONB DEFAULT '{}'::jsonb
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
|
||||
ON migration_impact_log(remediation_id, applied_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
|
||||
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;
|
||||
|
||||
-- ============================================================
|
||||
-- Trigger-based search_vector (spans pages + timeline_entries)
|
||||
-- ============================================================
|
||||
|
||||
+128
-10
@@ -776,7 +776,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
const reserved = await pool.reserve();
|
||||
try {
|
||||
const conn: ReservedConnection = {
|
||||
async executeRaw<R = Record<string, unknown>>(query: string, params?: unknown[]): Promise<R[]> {
|
||||
async executeRaw<R = Record<string, unknown>>(
|
||||
query: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<R[]> {
|
||||
// ReservedConnection.executeRaw doesn't wire AbortSignal today
|
||||
// (the only use site is migrations + cycle-lock writes that don't
|
||||
// want cancellation). Signature matches the interface so callers
|
||||
// that pass opts don't typecheck-break; opts.signal is ignored.
|
||||
void opts;
|
||||
const rows = params === undefined
|
||||
? await reserved.unsafe(query)
|
||||
: await reserved.unsafe(query, params as Parameters<typeof reserved.unsafe>[1]);
|
||||
@@ -1977,11 +1986,86 @@ export class PostgresEngine implements BrainEngine {
|
||||
afterPageId?: number;
|
||||
afterChunkIndex?: number;
|
||||
sourceId?: string;
|
||||
orderBy?: 'page_id' | 'updated_desc';
|
||||
afterUpdatedAt?: string | null;
|
||||
}): Promise<StaleChunkRow[]> {
|
||||
const sql = this.sql;
|
||||
const limit = opts?.batchSize ?? 2000;
|
||||
const afterPid = opts?.afterPageId ?? 0;
|
||||
const afterIdx = opts?.afterChunkIndex ?? -1;
|
||||
const orderBy = opts?.orderBy ?? 'page_id';
|
||||
|
||||
// v0.41.18.0 (A13, codex #9): --priority recent path. Composite cursor
|
||||
// (updated_at DESC NULLS LAST, page_id ASC, chunk_index ASC). Backed by
|
||||
// idx_pages_updated_at_desc + content_chunks_stale_idx partial.
|
||||
// "Next row" semantic with DESC NULLS LAST + ASC tiebreakers is:
|
||||
// (updated_at < prev) OR
|
||||
// (updated_at = prev AND page_id > prev_page_id) OR
|
||||
// (updated_at = prev AND page_id = prev_page_id AND chunk_index > prev_chunk_index)
|
||||
// First call: afterUpdatedAt undefined → returns the highest updated_at rows.
|
||||
if (orderBy === 'updated_desc') {
|
||||
const afterUpdated = opts?.afterUpdatedAt ?? null;
|
||||
const isFirstPage = afterUpdated === null && afterPid === 0;
|
||||
if (opts?.sourceId === undefined) {
|
||||
const rows = isFirstPage ? await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
` : await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (
|
||||
p.updated_at < ${afterUpdated}::timestamptz
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid})
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx})
|
||||
)
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
const rows = isFirstPage ? await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
` : await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (
|
||||
p.updated_at < ${afterUpdated}::timestamptz
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid})
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx})
|
||||
)
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
// orderBy === 'page_id' — legacy stable cursor (unchanged below).
|
||||
// Cursor-paginated: keyset pagination on (page_id, chunk_index).
|
||||
// The partial index idx_chunks_embedding_null makes the WHERE fast;
|
||||
// LIMIT keeps each round-trip well within statement_timeout.
|
||||
@@ -2102,15 +2186,17 @@ export class PostgresEngine implements BrainEngine {
|
||||
const fromSourceIds = links.map(l => l.from_source_id || 'default');
|
||||
const toSourceIds = links.map(l => l.to_source_id || 'default');
|
||||
const originSourceIds = links.map(l => l.origin_source_id || 'default');
|
||||
// v0.41.18.0 (A10): link_kind column (v98). NULL = legacy/plain.
|
||||
const linkKinds = links.map(l => l.link_kind ?? null);
|
||||
const result = await sql`
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, link_kind, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, v.link_type, v.context, v.link_source, v.link_kind, o.id, v.origin_field
|
||||
FROM unnest(
|
||||
${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[],
|
||||
${contexts}::text[], ${linkSources}::text[], ${originSlugs}::text[],
|
||||
${originFields}::text[], ${fromSourceIds}::text[], ${toSourceIds}::text[],
|
||||
${originSourceIds}::text[]
|
||||
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id)
|
||||
${originSourceIds}::text[], ${linkKinds}::text[]
|
||||
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id, link_kind)
|
||||
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
|
||||
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
|
||||
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
|
||||
@@ -2493,7 +2579,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
if (slugs.length === 0) return result;
|
||||
for (const s of slugs) result.set(s, 0);
|
||||
|
||||
// v0.42.0.0 D12: filter mentions OUT of backlink-count for search
|
||||
// v0.41.18.0 D12: filter mentions OUT of backlink-count for search
|
||||
// ranking. `link_source='mentions'` rows are auto-linked body-text
|
||||
// mentions from `gbrain extract links --by-mention`; they're
|
||||
// graph-completeness signal, NOT human-intent signal. Counting them
|
||||
@@ -2710,7 +2796,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
||||
SELECT id, ${entry.date}::date, ${entry.source || ''}, ${entry.summary}, ${entry.detail || ''}
|
||||
FROM pages WHERE slug = ${slug} AND source_id = ${sourceId}
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING
|
||||
ON CONFLICT (page_id, date, summary, source) DO NOTHING
|
||||
`;
|
||||
}
|
||||
|
||||
@@ -2729,7 +2815,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[], ${sourceIds}::text[])
|
||||
AS v(slug, date, source, summary, detail, source_id)
|
||||
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING
|
||||
ON CONFLICT (page_id, date, summary, source) DO NOTHING
|
||||
RETURNING 1
|
||||
`;
|
||||
return result.length;
|
||||
@@ -4255,9 +4341,41 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
async executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]> {
|
||||
async executeRaw<T = Record<string, unknown>>(
|
||||
sql: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<T[]> {
|
||||
const conn = this.sql;
|
||||
return conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]) as unknown as T[];
|
||||
const pending = conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]);
|
||||
// v0.41.18.0 (A20, codex #7): real cancellation via postgres.js's
|
||||
// .cancel() on the pending query. Init nudge (3s wallclock cap) is the
|
||||
// first consumer; the AbortSignal fires when the timer trips.
|
||||
// Already-aborted signal short-circuits before the network round-trip.
|
||||
if (opts?.signal) {
|
||||
if (opts.signal.aborted) {
|
||||
// .cancel() is fire-and-forget; the awaited query rejects with the
|
||||
// postgres "query was cancelled" error which the caller catches.
|
||||
try {
|
||||
(pending as unknown as { cancel?: () => void }).cancel?.();
|
||||
} catch {
|
||||
// best-effort
|
||||
}
|
||||
throw new DOMException('aborted', 'AbortError');
|
||||
}
|
||||
const onAbort = () => {
|
||||
try {
|
||||
(pending as unknown as { cancel?: () => void }).cancel?.();
|
||||
} catch {
|
||||
// best-effort; the .then below settles regardless
|
||||
}
|
||||
};
|
||||
opts.signal.addEventListener('abort', onAbort, { once: true });
|
||||
return (pending as unknown as Promise<T[]>).finally(() => {
|
||||
opts.signal?.removeEventListener('abort', onAbort);
|
||||
});
|
||||
}
|
||||
return pending as unknown as T[];
|
||||
// Pre-#406 behavior: throw on any error including connection death.
|
||||
// Per-call auto-retry is not safe here because executeRaw is also used
|
||||
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
// src/core/remediation/context.ts
|
||||
// v0.41.18.0 (A1, codex finding #2). Extracted verbatim from
|
||||
// src/commands/doctor.ts:loadRecommendationContext so both the doctor
|
||||
// CLI shell AND the new gbrain onboard / MCP run_onboard surfaces
|
||||
// build the same context object.
|
||||
//
|
||||
// Pure read; no side effects.
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { RecommendationContext } from '../brain-score-recommendations.ts';
|
||||
|
||||
// Re-export so consumers can `import { RecommendationContext } from '../remediation'`
|
||||
// — the canonical RecommendationContext type still lives in
|
||||
// brain-score-recommendations.ts (it's also the input to computeRecommendations).
|
||||
export type { RecommendationContext };
|
||||
|
||||
/**
|
||||
* Build RecommendationContext from engine + config. Pure read; no
|
||||
* side effects. Used by computeRemediationPlan, runRemediation, and
|
||||
* the doctor CLI surface.
|
||||
*/
|
||||
export async function loadRecommendationContext(
|
||||
engine: BrainEngine,
|
||||
): Promise<RecommendationContext> {
|
||||
// v0.37 fix wave (Lane E.4 + CDX2-11): read schema-sizing fields from
|
||||
// gateway, not DB. The DB plane is schema-applied metadata; the file
|
||||
// plane is the gateway runtime source. Pre-fix this context produced
|
||||
// stale recommendations on fresh installs whose DB rows hadn't been
|
||||
// populated.
|
||||
//
|
||||
// Also extended the API-key check to recognize the ZE key alongside
|
||||
// OpenAI (was OpenAI-only). After Lane C.3, zeroentropy_api_key lives
|
||||
// in GBrainConfig + propagates to the gateway env dict.
|
||||
const repoPath = await engine.getConfig('sync.repo_path');
|
||||
let embeddingModel: string | undefined;
|
||||
let embeddingDimensions: number | undefined;
|
||||
try {
|
||||
const gw = await import('../ai/gateway.ts');
|
||||
embeddingModel = gw.getEmbeddingModel();
|
||||
embeddingDimensions = gw.getEmbeddingDimensions();
|
||||
} catch {
|
||||
// Gateway unconfigured — fall back to DB plane as a best-effort hint
|
||||
// (preserves doctor running before any engine.connect()).
|
||||
const dbModel = await engine.getConfig('embedding_model');
|
||||
const dbDims = await engine.getConfig('embedding_dimensions');
|
||||
embeddingModel = dbModel ?? undefined;
|
||||
embeddingDimensions = dbDims ? Number(dbDims) : undefined;
|
||||
}
|
||||
// v0.40.x: recipe-aware provider check, shared with autopilot.ts via
|
||||
// embeddingProviderConfigured(). Local providers (ollama, llama-server —
|
||||
// empty auth_env.required) need no hosted key; hosted providers check
|
||||
// their OWN required key (so a Voyage brain is judged by VOYAGE_API_KEY,
|
||||
// not by whether an OpenAI/ZE key happens to exist — the pre-fix wart).
|
||||
// fileCfg loads synchronously, so the resolveKey closure is sync.
|
||||
const { loadConfigFileOnly } = await import('../config.ts');
|
||||
const fileCfg = loadConfigFileOnly();
|
||||
const { embeddingProviderConfigured, HOSTED_EMBED_KEY_CONFIG } = await import(
|
||||
'../brain-score-recommendations.ts'
|
||||
);
|
||||
const embeddingConfigured = embeddingProviderConfigured(embeddingModel, (envVar) => {
|
||||
const cfgField = HOSTED_EMBED_KEY_CONFIG[envVar];
|
||||
const fromCfg = cfgField ? (fileCfg as Record<string, unknown> | null)?.[cfgField] : undefined;
|
||||
return !!(process.env[envVar] || fromCfg);
|
||||
});
|
||||
return {
|
||||
repoPath: repoPath ?? undefined,
|
||||
embeddingModel,
|
||||
embeddingDimensions,
|
||||
embeddingProviderConfigured: embeddingConfigured,
|
||||
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || fileCfg?.anthropic_api_key),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
// src/core/remediation/index.ts
|
||||
// v0.41.18.0 (A1). Barrel for the doctor-remediation library.
|
||||
// Consumers (doctor CLI, onboard CLI, MCP run_onboard) import from here.
|
||||
|
||||
export { computeRemediationPlan, SYNTHETIC_CHECK_NAMES } from './plan.ts';
|
||||
export { runRemediation } from './run.ts';
|
||||
export { loadRecommendationContext } from './context.ts';
|
||||
export type { RecommendationContext } from './context.ts';
|
||||
export type {
|
||||
RemediationPlan,
|
||||
RemediationPlanOpts,
|
||||
RemediationOpts,
|
||||
RemediationResult,
|
||||
RemediationHooks,
|
||||
StepResult,
|
||||
} from './types.ts';
|
||||
@@ -0,0 +1,76 @@
|
||||
// src/core/remediation/plan.ts
|
||||
// v0.41.18.0 (A1, codex finding #2). Extracted from doctor.ts:runRemediationPlan
|
||||
// so onboard + MCP run_onboard call the same library without parsing argv
|
||||
// or invoking console.* / process.exit.
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import {
|
||||
computeRecommendations,
|
||||
classifyChecks,
|
||||
maxReachableScore,
|
||||
} from '../brain-score-recommendations.ts';
|
||||
import { loadRecommendationContext } from './context.ts';
|
||||
import type { RemediationPlan, RemediationPlanOpts } from './types.ts';
|
||||
|
||||
/**
|
||||
* Synthetic check list for classification. computeRecommendations operates
|
||||
* on BrainHealth + context alone; we don't need full doctor output, just
|
||||
* the check names the recommendations care about. Same five names doctor
|
||||
* has used since v0.36.4.0; do not extend without also updating
|
||||
* brain-score-recommendations.ts.
|
||||
*/
|
||||
export const SYNTHETIC_CHECK_NAMES = [
|
||||
'brain_score',
|
||||
'sync_freshness',
|
||||
'missing_embeddings',
|
||||
'dead_links',
|
||||
'orphan_pages',
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Pure read: compute the dependency-ordered Remediation plan to drive
|
||||
* brain to opts.targetScore (default 90). Never enqueues, never mutates.
|
||||
*
|
||||
* Consumed by:
|
||||
* - gbrain doctor --remediation-plan (renders human/JSON in CLI shell)
|
||||
* - gbrain onboard --check (reframes as onboarding language)
|
||||
* - MCP run_onboard (admin scope, returns plan as JSON envelope)
|
||||
*/
|
||||
export async function computeRemediationPlan(
|
||||
engine: BrainEngine,
|
||||
opts: RemediationPlanOpts = {},
|
||||
): Promise<RemediationPlan> {
|
||||
const targetScore = opts.targetScore ?? 90;
|
||||
|
||||
// Cheap path (D7) — don't run slow doctor checks for the plan surface.
|
||||
// The recommendation generator works from BrainHealth + context alone.
|
||||
const health = await engine.getHealth();
|
||||
const ctx = await loadRecommendationContext(engine);
|
||||
const recs = computeRecommendations(health, ctx, opts.extraRemediations ?? []);
|
||||
const syntheticChecks = SYNTHETIC_CHECK_NAMES.map((name) => ({
|
||||
name,
|
||||
status: 'ok' as const,
|
||||
}));
|
||||
const classifications = classifyChecks(syntheticChecks, ctx);
|
||||
const ceiling = maxReachableScore(health, classifications);
|
||||
|
||||
const filteredRecs = recs.filter((r) => r.status === 'remediable');
|
||||
const estTotalSeconds = filteredRecs.reduce((sum, r) => sum + r.est_seconds, 0);
|
||||
const estTotalUsd = filteredRecs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
|
||||
|
||||
const blocked = classifications
|
||||
.filter((c) => c.status === 'blocked')
|
||||
.map((c) => ({ check: c.check, reason: c.reason ?? 'prerequisite missing' }));
|
||||
|
||||
return {
|
||||
schema_version: 2,
|
||||
brain_score_current: health.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
max_reachable_score: ceiling,
|
||||
target_unreachable: targetScore > ceiling,
|
||||
plan: filteredRecs.map((r, i) => ({ step: i + 1, ...r })),
|
||||
est_total_seconds: estTotalSeconds,
|
||||
est_total_usd_cost: Number(estTotalUsd.toFixed(2)),
|
||||
blocked,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,342 @@
|
||||
// src/core/remediation/run.ts
|
||||
// v0.41.18.0 (A1, codex finding #2). Extracted from doctor.ts:runRemediate
|
||||
// so onboard + MCP run_onboard call the same orchestrator without parsing
|
||||
// argv or invoking process.exit / console.* directly.
|
||||
//
|
||||
// The orchestrator wraps the plan loop with:
|
||||
// - BudgetTracker (auto-installed via withBudgetTracker)
|
||||
// - Checkpoint resume per A4 amended (matching plan_hash only)
|
||||
// - D5 dependency cascade (failed step aborts dependents)
|
||||
// - D7 per-step recheck (re-compute plan from fresh health)
|
||||
// - Hooks for caller observability (no console.* in the library)
|
||||
|
||||
import crypto from 'crypto';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import {
|
||||
computeRecommendations,
|
||||
} from '../brain-score-recommendations.ts';
|
||||
import type { RemediationStep } from '../remediation-step.ts';
|
||||
import { loadRecommendationContext } from './context.ts';
|
||||
import { computeRemediationPlan } from './plan.ts';
|
||||
import type {
|
||||
RemediationHooks,
|
||||
RemediationOpts,
|
||||
RemediationResult,
|
||||
StepResult,
|
||||
} from './types.ts';
|
||||
|
||||
/**
|
||||
* Submit ordered Remediation jobs sequentially per D3, with D5 cascade
|
||||
* on failure and D7 scoped recheck between steps.
|
||||
*
|
||||
* PGLite path: synchronous in-process execution (no durable queue).
|
||||
*
|
||||
* Returns a RemediationResult; never throws on BudgetExhausted (the
|
||||
* exhaustion snapshot lives on result.budget_exhausted instead).
|
||||
* Other thrown errors propagate.
|
||||
*
|
||||
* Callers decide exit codes from the result.
|
||||
*/
|
||||
export async function runRemediation(
|
||||
engine: BrainEngine,
|
||||
opts: RemediationOpts = {},
|
||||
hooks: RemediationHooks = {},
|
||||
): Promise<RemediationResult> {
|
||||
const targetScore = opts.targetScore ?? 90;
|
||||
const maxJobs = opts.maxJobs ?? Infinity;
|
||||
const maxUsd = opts.maxUsd;
|
||||
const dryRun = opts.dryRun ?? false;
|
||||
const resumeMode = opts.resume ?? false;
|
||||
const resumePlanHash = opts.resumePlanHash;
|
||||
|
||||
// Lazy-load orchestration deps so the library entry-point doesn't pay
|
||||
// their cost on a --dry-run shortcut path (or when callers only need
|
||||
// computeRemediationPlan).
|
||||
const {
|
||||
BudgetTracker,
|
||||
BudgetExhausted,
|
||||
} = await import('../budget/budget-tracker.ts');
|
||||
const { withBudgetTracker } = await import('../ai/gateway.ts');
|
||||
const {
|
||||
computePlanHash,
|
||||
saveRemediationCheckpoint,
|
||||
loadRemediationCheckpoint,
|
||||
listRemediationCheckpoints,
|
||||
clearRemediationCheckpoint,
|
||||
} = await import('../remediation-checkpoint.ts');
|
||||
|
||||
const ctx = await loadRecommendationContext(engine);
|
||||
|
||||
// Pre-flight ceiling check via the shared plan computation.
|
||||
const initialPlan = await computeRemediationPlan(engine, { targetScore });
|
||||
if (initialPlan.target_unreachable) {
|
||||
hooks.onTargetUnreachable?.(targetScore, initialPlan.max_reachable_score);
|
||||
return {
|
||||
doctor_run_id: crypto.randomUUID(),
|
||||
brain_score_initial: initialPlan.brain_score_current,
|
||||
brain_score_final: initialPlan.brain_score_current,
|
||||
brain_score_target: targetScore,
|
||||
target_reached: false,
|
||||
submitted: [],
|
||||
aborted_count: 0,
|
||||
target_unreachable: {
|
||||
target: targetScore,
|
||||
ceiling: initialPlan.max_reachable_score,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const initialHealth = await engine.getHealth();
|
||||
let recs: RemediationStep[] = computeRecommendations(initialHealth, ctx)
|
||||
.filter((r) => r.status === 'remediable');
|
||||
if (recs.length === 0) {
|
||||
hooks.onNothingToDo?.(initialHealth.brain_score, targetScore);
|
||||
return {
|
||||
doctor_run_id: crypto.randomUUID(),
|
||||
brain_score_initial: initialHealth.brain_score,
|
||||
brain_score_final: initialHealth.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
target_reached: initialHealth.brain_score >= targetScore,
|
||||
submitted: [],
|
||||
aborted_count: 0,
|
||||
};
|
||||
}
|
||||
|
||||
// A4 amended: compute plan_hash off the active recommendation ids so
|
||||
// the checkpoint binds to THIS plan. Resume only fires for matching plans.
|
||||
const planHash = computePlanHash(recs.map((r) => r.id));
|
||||
let completedFromCheckpoint = new Set<string>();
|
||||
if (resumeMode) {
|
||||
const requested = resumePlanHash;
|
||||
let cp = requested ? loadRemediationCheckpoint(requested) : null;
|
||||
if (!cp && !requested) {
|
||||
// No explicit hash: try newest checkpoint that matches the active plan.
|
||||
const recent = listRemediationCheckpoints();
|
||||
for (const e of recent) {
|
||||
const candidate = loadRemediationCheckpoint(e.plan_hash);
|
||||
if (candidate && candidate.plan_hash === planHash) {
|
||||
cp = candidate;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!cp || cp.plan_hash !== planHash) {
|
||||
hooks.onResumeMissed?.(planHash, requested);
|
||||
// Surface as a synthetic result so the CLI shell can exit 2.
|
||||
return {
|
||||
doctor_run_id: crypto.randomUUID(),
|
||||
brain_score_initial: initialHealth.brain_score,
|
||||
brain_score_final: initialHealth.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
target_reached: false,
|
||||
submitted: [],
|
||||
aborted_count: 0,
|
||||
target_unreachable: {
|
||||
target: targetScore,
|
||||
ceiling: initialPlan.max_reachable_score,
|
||||
},
|
||||
};
|
||||
}
|
||||
completedFromCheckpoint = new Set(cp.completed.map((c) => c.id));
|
||||
hooks.onResumeLoaded?.(
|
||||
planHash,
|
||||
completedFromCheckpoint.size,
|
||||
recs.length - completedFromCheckpoint.size,
|
||||
);
|
||||
}
|
||||
|
||||
const estTotalUsd = recs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
|
||||
if (maxUsd !== undefined && estTotalUsd > maxUsd) {
|
||||
hooks.onBudgetRefused?.(estTotalUsd, maxUsd);
|
||||
return {
|
||||
doctor_run_id: crypto.randomUUID(),
|
||||
brain_score_initial: initialHealth.brain_score,
|
||||
brain_score_final: initialHealth.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
target_reached: false,
|
||||
submitted: [],
|
||||
aborted_count: 0,
|
||||
};
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
// Dry-run: no submission, just return the plan as a non-empty result.
|
||||
// Each rec lands in submitted[] with synthetic 'dry_run' status so the
|
||||
// shape stays consistent.
|
||||
return {
|
||||
doctor_run_id: crypto.randomUUID(),
|
||||
brain_score_initial: initialHealth.brain_score,
|
||||
brain_score_final: initialHealth.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
target_reached: false,
|
||||
submitted: recs.map((r, i) => ({
|
||||
step: i + 1,
|
||||
id: r.id,
|
||||
job_id: null,
|
||||
status: 'dry_run',
|
||||
})),
|
||||
aborted_count: 0,
|
||||
};
|
||||
}
|
||||
|
||||
// Real submission path
|
||||
const submitted: StepResult[] = [];
|
||||
const abortedIds = new Set<string>();
|
||||
const doctorRunId = crypto.randomUUID();
|
||||
|
||||
const { MinionQueue } = await import('../minions/queue.ts');
|
||||
const { waitForCompletion } = await import('../minions/wait-for-completion.ts');
|
||||
const isPGLite = engine.kind === 'pglite';
|
||||
const queue = new MinionQueue(engine);
|
||||
|
||||
// A4 amended: install a BudgetTracker scope around the plan-step loop so
|
||||
// any gateway.chat / embed / rerank inside a Minion handler (synthesize,
|
||||
// patterns, consolidate) auto-enforces the cap. On BudgetExhausted, the
|
||||
// onExhausted callback persists the checkpoint BEFORE the throw propagates;
|
||||
// the caller hook surfaces the actionable --resume hint.
|
||||
const remediateTracker = new BudgetTracker({
|
||||
label: 'remediation.run',
|
||||
maxCostUsd: maxUsd,
|
||||
});
|
||||
|
||||
let exhaustionSnapshot: NonNullable<RemediationResult['budget_exhausted']> | undefined;
|
||||
remediateTracker.onExhausted(() => {
|
||||
const cp = {
|
||||
schema_version: 1 as const,
|
||||
plan_hash: planHash,
|
||||
doctor_run_id: doctorRunId,
|
||||
target_score: targetScore,
|
||||
started_at: new Date().toISOString(),
|
||||
completed: submitted
|
||||
.filter((s) => s.status === 'completed')
|
||||
.map((s) => ({ id: s.id, job: '', status: s.status, job_id: s.job_id ?? null })),
|
||||
aborted_at: new Date().toISOString(),
|
||||
abort_reason: 'budget_exhausted' as const,
|
||||
budget_snapshot: exhaustionSnapshot
|
||||
? { spent: exhaustionSnapshot.spent, cap: exhaustionSnapshot.cap, reason: exhaustionSnapshot.reason, model_id: exhaustionSnapshot.model_id }
|
||||
: undefined,
|
||||
};
|
||||
saveRemediationCheckpoint(cp);
|
||||
});
|
||||
|
||||
const runLoop = async (): Promise<void> => {
|
||||
let stepCount = 0;
|
||||
const totalSteps = recs.length;
|
||||
while (recs.length > 0 && stepCount < maxJobs) {
|
||||
const step = recs[0];
|
||||
if (!step) break;
|
||||
stepCount++;
|
||||
|
||||
// Resume: skip steps that the checkpoint already marked completed.
|
||||
if (completedFromCheckpoint.has(step.id)) {
|
||||
const result: StepResult = { step: stepCount, id: step.id, job_id: null, status: 'completed' };
|
||||
submitted.push(result);
|
||||
hooks.onStepEnd?.(result);
|
||||
recs.shift();
|
||||
continue;
|
||||
}
|
||||
|
||||
// D5: if depends_on intersects aborted, skip + cascade
|
||||
if (step.depends_on && step.depends_on.some((d: string) => abortedIds.has(d))) {
|
||||
const result: StepResult = { step: stepCount, id: step.id, job_id: null, status: 'skipped_dep_aborted' };
|
||||
submitted.push(result);
|
||||
abortedIds.add(step.id);
|
||||
hooks.onStepEnd?.(result);
|
||||
recs.shift();
|
||||
continue;
|
||||
}
|
||||
|
||||
hooks.onStepStart?.(stepCount, totalSteps, step);
|
||||
try {
|
||||
const isProtected = !!step.protected;
|
||||
const job = await queue.add(
|
||||
step.job,
|
||||
{ ...step.params, doctor_run_id: doctorRunId },
|
||||
{
|
||||
queue: 'default',
|
||||
idempotency_key: step.idempotency_key,
|
||||
max_attempts: 2,
|
||||
maxWaiting: 1,
|
||||
},
|
||||
isProtected ? { allowProtectedSubmit: true } : undefined,
|
||||
);
|
||||
const submittedResult: StepResult = {
|
||||
step: stepCount,
|
||||
id: step.id,
|
||||
job_id: job.id,
|
||||
status: 'submitted',
|
||||
};
|
||||
submitted.push(submittedResult);
|
||||
|
||||
const terminal = await waitForCompletion(queue, job.id, {
|
||||
pollMs: isPGLite ? 250 : 1000,
|
||||
timeoutMs: (step.est_seconds + 60) * 1000,
|
||||
});
|
||||
submittedResult.status = terminal.status;
|
||||
if (terminal.status !== 'completed') {
|
||||
abortedIds.add(step.id);
|
||||
}
|
||||
hooks.onStepEnd?.(submittedResult);
|
||||
} catch (e) {
|
||||
if (e instanceof BudgetExhausted) {
|
||||
exhaustionSnapshot = {
|
||||
spent: e.spent,
|
||||
cap: e.cap,
|
||||
reason: e.reason,
|
||||
model_id: e.modelId,
|
||||
plan_hash: planHash,
|
||||
};
|
||||
throw e;
|
||||
}
|
||||
const errResult: StepResult = {
|
||||
step: stepCount,
|
||||
id: step.id,
|
||||
job_id: null,
|
||||
status: `error: ${(e as Error).message.slice(0, 100)}`,
|
||||
};
|
||||
submitted.push(errResult);
|
||||
abortedIds.add(step.id);
|
||||
hooks.onStepEnd?.(errResult);
|
||||
}
|
||||
|
||||
recs.shift();
|
||||
// D7: scoped recheck — re-compute plan from fresh health snapshot.
|
||||
// The next plan may drop completed steps and re-introduce failed
|
||||
// steps with bumped retry suffix (D1).
|
||||
if (recs.length === 0 || stepCount >= maxJobs) break;
|
||||
const freshHealth = await engine.getHealth();
|
||||
recs = computeRecommendations(freshHealth, ctx).filter((r) => r.status === 'remediable');
|
||||
}
|
||||
};
|
||||
|
||||
let budgetAbort: NonNullable<RemediationResult['budget_exhausted']> | undefined;
|
||||
try {
|
||||
await withBudgetTracker(remediateTracker, runLoop);
|
||||
} catch (err) {
|
||||
if (err instanceof BudgetExhausted) {
|
||||
budgetAbort = exhaustionSnapshot;
|
||||
if (budgetAbort) hooks.onBudgetExhausted?.(planHash, budgetAbort);
|
||||
} else {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
// Clear checkpoint on a clean run (no budget abort). Failed steps in the
|
||||
// submitted set don't disqualify the cleanup — they re-surface on the
|
||||
// next plan with bumped suffixes.
|
||||
if (!budgetAbort) {
|
||||
clearRemediationCheckpoint(planHash);
|
||||
}
|
||||
|
||||
const finalHealth = await engine.getHealth();
|
||||
return {
|
||||
doctor_run_id: doctorRunId,
|
||||
brain_score_initial: initialHealth.brain_score,
|
||||
brain_score_final: finalHealth.brain_score,
|
||||
brain_score_target: targetScore,
|
||||
target_reached: finalHealth.brain_score >= targetScore,
|
||||
submitted,
|
||||
aborted_count: abortedIds.size,
|
||||
budget_exhausted: budgetAbort,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
// src/core/remediation/types.ts
|
||||
// v0.41.18.0 (A1, codex finding #2). Extracted from src/commands/doctor.ts
|
||||
// so onboard CLI shell + MCP run_onboard op can compose against a stable
|
||||
// library shape — NOT a CLI-shaped function with process.exit calls.
|
||||
//
|
||||
// Three consumers wrap this library today:
|
||||
// - src/commands/doctor.ts (runRemediationPlan + runRemediate)
|
||||
// - src/commands/onboard.ts (gbrain onboard --check / --auto)
|
||||
// - src/core/operations.ts (MCP op run_onboard, admin scope)
|
||||
|
||||
import type { RemediationStep } from '../remediation-step.ts';
|
||||
|
||||
/**
|
||||
* Options for computeRemediationPlan. All fields are optional with
|
||||
* sensible defaults.
|
||||
*/
|
||||
export interface RemediationPlanOpts {
|
||||
/** Target brain_score (default: 90). Used to drive recommendation count. */
|
||||
targetScore?: number;
|
||||
/**
|
||||
* v0.41.18.0 (A2 + codex #3): caller-supplied RemediationStep entries
|
||||
* threaded into the planner via the third arg of computeRecommendations.
|
||||
* Onboard wires the 4 new check helpers (embed_staleness,
|
||||
* entity_link_coverage, timeline_coverage, takes_count) here. doctor's
|
||||
* existing --remediation-plan call passes empty (preserving legacy
|
||||
* behavior).
|
||||
*/
|
||||
extraRemediations?: RemediationStep[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only plan output. Stable JSON envelope — downstream agents
|
||||
* (gbrain onboard, MCP run_onboard) bind to this shape.
|
||||
*/
|
||||
export interface RemediationPlan {
|
||||
schema_version: 2;
|
||||
brain_score_current: number;
|
||||
brain_score_target: number;
|
||||
max_reachable_score: number;
|
||||
target_unreachable: boolean;
|
||||
plan: Array<RemediationStep & { step: number }>;
|
||||
est_total_seconds: number;
|
||||
est_total_usd_cost: number;
|
||||
blocked: Array<{ check: string; reason: string }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Options for runRemediation. The remediate orchestrator wraps the plan
|
||||
* loop with BudgetTracker, checkpoint resume, dep-cascade, and per-step
|
||||
* recheck. Hooks let callers (CLI / JSON / MCP) emit progress without
|
||||
* the library calling console directly.
|
||||
*/
|
||||
export interface RemediationOpts {
|
||||
/** Target brain_score (default: 90). */
|
||||
targetScore?: number;
|
||||
/** Cap inner loop iterations (default: Infinity). */
|
||||
maxJobs?: number;
|
||||
/** USD cap for total plan cost. Pre-flight refuse + mid-run BudgetExhausted gate. */
|
||||
maxUsd?: number;
|
||||
/** Read-only dry-run; submits no jobs; returns plan in result. */
|
||||
dryRun?: boolean;
|
||||
/** Resume from checkpoint matching this plan_hash, OR newest if undefined+resume=true. */
|
||||
resumePlanHash?: string;
|
||||
/** Whether to attempt resume at all (default false). */
|
||||
resume?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of one step. status mirrors Minion job terminal states + a few
|
||||
* synthetic ones ('skipped_dep_aborted', 'skipped_completed_in_checkpoint').
|
||||
*/
|
||||
export interface StepResult {
|
||||
step: number;
|
||||
id: string;
|
||||
job_id: number | null;
|
||||
status: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of a full runRemediation invocation. Stable shape for JSON
|
||||
* emission and MCP envelope.
|
||||
*/
|
||||
export interface RemediationResult {
|
||||
doctor_run_id: string;
|
||||
brain_score_initial: number;
|
||||
brain_score_final: number;
|
||||
brain_score_target: number;
|
||||
target_reached: boolean;
|
||||
submitted: StepResult[];
|
||||
aborted_count: number;
|
||||
/** Set when the run aborted on BudgetExhausted. Caller decides exit code. */
|
||||
budget_exhausted?: {
|
||||
spent: number;
|
||||
cap: number;
|
||||
reason: string;
|
||||
model_id?: string;
|
||||
plan_hash: string;
|
||||
};
|
||||
/** Set when the pre-flight ceiling check refused the target. */
|
||||
target_unreachable?: {
|
||||
target: number;
|
||||
ceiling: number;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Hooks let the caller observe + report progress without the library
|
||||
* emitting to stdout/stderr itself. Every hook is optional.
|
||||
*/
|
||||
export interface RemediationHooks {
|
||||
/** Fired once when the orchestrator decides nothing needs to run. */
|
||||
onNothingToDo?: (initialScore: number, target: number) => void;
|
||||
/** Fired when the pre-flight ceiling check refuses the target. */
|
||||
onTargetUnreachable?: (target: number, ceiling: number) => void;
|
||||
/** Fired when --max-usd refuses the plan. */
|
||||
onBudgetRefused?: (estCost: number, cap: number) => void;
|
||||
/** Fired before each step submission. */
|
||||
onStepStart?: (step: number, total: number, rec: RemediationStep) => void;
|
||||
/** Fired after each step reaches a terminal state (or skip). */
|
||||
onStepEnd?: (result: StepResult) => void;
|
||||
/** Fired on BudgetExhausted thrown mid-loop. */
|
||||
onBudgetExhausted?: (planHash: string, snapshot: NonNullable<RemediationResult['budget_exhausted']>) => void;
|
||||
/** Fired on resume-checkpoint load (resume mode only). */
|
||||
onResumeLoaded?: (planHash: string, completedCount: number, remainingCount: number) => void;
|
||||
/** Fired on resume-checkpoint miss (resume mode only). */
|
||||
onResumeMissed?: (planHash: string, requested?: string) => void;
|
||||
}
|
||||
+51
-3
@@ -262,6 +262,13 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified
|
||||
ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL;
|
||||
-- v0.41.18.0 (codex finding #9): partial index for `gbrain embed --stale`
|
||||
-- + `--priority recent`. content_chunks has no updated_at column (chunks
|
||||
-- are re-INSERTed on page change, not UPDATEd), so the "recent-first"
|
||||
-- ORDER BY happens at the JOIN site: outer ORDER BY p.updated_at DESC
|
||||
-- uses idx_pages_updated_at_desc; inner partial uses this index.
|
||||
CREATE INDEX IF NOT EXISTS content_chunks_stale_idx
|
||||
ON content_chunks(page_id, chunk_index) WHERE embedding IS NULL;
|
||||
|
||||
-- v0.20.0 Cathedral II: chunk-grain FTS trigger.
|
||||
-- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text.
|
||||
@@ -353,10 +360,16 @@ CREATE TABLE IF NOT EXISTS links (
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
|
||||
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
|
||||
-- for search ranking; only counts toward orphan-ratio + graph traversal.
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
|
||||
-- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from
|
||||
-- "verb-pattern-derived typed link" within link_source='mentions'.
|
||||
-- Codex finding #12 design: keep link_source stable; add link_kind
|
||||
-- so callers can distinguish without breaking existing mentions queries.
|
||||
-- NULL = legacy / unknown / pre-v98 row (semantically 'plain').
|
||||
link_kind TEXT CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
-- v0.18.0 Step 4: 'qualified' when the link was written as
|
||||
@@ -419,8 +432,10 @@ CREATE TABLE IF NOT EXISTS timeline_entries (
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
|
||||
-- Dedup constraint: same (page, date, summary) treated as same event
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary);
|
||||
-- v0.41.18.0 (codex finding #11): widened from (page_id, date, summary) to
|
||||
-- include `source` so distinct meeting provenance survives. Legacy rows
|
||||
-- have source='' (schema default) so legacy dedup behavior is preserved.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary, source);
|
||||
|
||||
-- ============================================================
|
||||
-- page_versions: snapshot history for compiled_truth
|
||||
@@ -585,6 +600,39 @@ CREATE TABLE IF NOT EXISTS op_checkpoints (
|
||||
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
|
||||
ON op_checkpoints (updated_at);
|
||||
|
||||
-- ============================================================
|
||||
-- migration_impact_log: before/after metric stats per onboard remediation
|
||||
-- ============================================================
|
||||
-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the
|
||||
-- onboard remediation pipeline records before/after metric stats so
|
||||
-- `gbrain onboard --history --json` can show "you reduced orphans 47%".
|
||||
-- delta computed at read time (NOT a stored GENERATED column —
|
||||
-- zero PGLite parity risk per eng-review D2).
|
||||
--
|
||||
-- Attribution columns (job_id, source_id, brain_id, started_at,
|
||||
-- idempotency_key) per codex finding #10 so concurrent onboard /
|
||||
-- autopilot / manual runs can't misattribute deltas to the wrong
|
||||
-- migration when overlapping runs change the same metric.
|
||||
CREATE TABLE IF NOT EXISTS migration_impact_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
remediation_id TEXT NOT NULL,
|
||||
metric_name TEXT NOT NULL,
|
||||
metric_before NUMERIC,
|
||||
metric_after NUMERIC,
|
||||
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
|
||||
source_id TEXT,
|
||||
brain_id TEXT,
|
||||
started_at TIMESTAMPTZ,
|
||||
idempotency_key TEXT,
|
||||
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
applied_by TEXT,
|
||||
details JSONB DEFAULT '{}'::jsonb
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
|
||||
ON migration_impact_log(remediation_id, applied_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
|
||||
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;
|
||||
|
||||
-- ============================================================
|
||||
-- files: binary attachments stored in Supabase Storage
|
||||
-- ============================================================
|
||||
|
||||
@@ -284,18 +284,28 @@ describe('Eng-review D3 — executeRaw has no per-call retry wrapper', () => {
|
||||
const src = readFileSync(resolve('src/core/postgres-engine.ts'), 'utf-8');
|
||||
|
||||
// Find the executeRaw method in the class (not the helper inside withReservedConnection)
|
||||
// Pattern: must be a method on the class taking (sql, params)
|
||||
const fnMatch = src.match(/async executeRaw<T = Record<string, unknown>>\(sql: string, params\?: unknown\[\]\): Promise<T\[\]> \{([\s\S]*?)\n \}/);
|
||||
// v0.41.18.0 (T5/A20): signature extended with optional `opts?: { signal?: AbortSignal }`
|
||||
// 3rd arg + multi-line shape for real query cancellation. Regex updated to
|
||||
// tolerate both the legacy single-line and the new multi-line signatures.
|
||||
const fnMatch = src.match(/async executeRaw<T = Record<string, unknown>>\(\s*sql: string,\s*params\?: unknown\[\][^)]*\):\s*Promise<T\[\]>\s*\{([\s\S]*?)\n \}/);
|
||||
expect(fnMatch).not.toBeNull();
|
||||
const body = fnMatch![1];
|
||||
|
||||
// Must not have any try/catch
|
||||
expect(body).not.toContain('try {');
|
||||
expect(body).not.toContain('catch');
|
||||
// Must not call reconnect() from this method
|
||||
// Must not call reconnect() from this method (D3 intent: no per-call
|
||||
// retry — recovery is supervisor-driven via reconnect()).
|
||||
expect(body).not.toContain('this.reconnect()');
|
||||
// Must call conn.unsafe directly
|
||||
// Must call conn.unsafe directly, exactly ONCE (no retry re-issue).
|
||||
expect(body).toContain('conn.unsafe(');
|
||||
const unsafeCallCount = (body.match(/conn\.unsafe\(/g) || []).length;
|
||||
expect(unsafeCallCount).toBe(1);
|
||||
// The try/catch present here is ONLY for AbortSignal cancellation
|
||||
// swallow (v0.41.18.0 A20), NOT for connection retry. Confirm by checking
|
||||
// the swallowed throws are .cancel() not network re-issue.
|
||||
if (body.includes('catch')) {
|
||||
// If catch exists, it must be the cancel-swallow shape, NOT a retry shape.
|
||||
expect(body).not.toMatch(/catch[^{]*\{[\s\S]*?conn\.unsafe/);
|
||||
expect(body).not.toMatch(/catch[^{]*\{[\s\S]*?setTimeout/);
|
||||
}
|
||||
});
|
||||
|
||||
it('PostgresEngine.reconnect() still exists for supervisor-driven recovery', () => {
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
// test/e2e/onboard-full-flow.test.ts
|
||||
// v0.41.18.0 (T20). Hermetic PGLite E2E for the onboard surface — no
|
||||
// DATABASE_URL needed. Exercises the key contracts end-to-end:
|
||||
// - computeRemediationPlan with extras returns the expected shape
|
||||
// - buildOnboardReport produces a stable JSON envelope
|
||||
// - captureMetric returns numeric values for each of 5 metrics
|
||||
// - The runRemediation library refuses --auto without --max-usd
|
||||
// - The onboard CLI gates work as documented
|
||||
//
|
||||
// Full DATABASE_URL-gated end-to-end (real Postgres, actual extractions
|
||||
// firing through Minion handlers) is deferred to a v0.42.1 follow-up
|
||||
// once the Minion worker test harness lands the per-handler stub seam.
|
||||
|
||||
import { describe, expect, test, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { computeRemediationPlan } from '../../src/core/remediation/index.ts';
|
||||
import { captureMetric } from '../../src/core/onboard/impact-capture.ts';
|
||||
import { buildOnboardReport, toOnboardRecommendation } from '../../src/core/onboard/render.ts';
|
||||
import { runAllOnboardChecks } from '../../src/core/onboard/checks.ts';
|
||||
import { makeRemediationStep } from '../../src/core/remediation-step.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('onboard E2E — captureMetric', () => {
|
||||
test('captureMetric returns 0 for stale_count on empty brain', async () => {
|
||||
const v = await captureMetric(engine, 'stale_count');
|
||||
expect(v).toBe(0);
|
||||
});
|
||||
|
||||
test('captureMetric returns 0 for orphan_count on empty brain', async () => {
|
||||
const v = await captureMetric(engine, 'orphan_count');
|
||||
expect(v).toBe(0);
|
||||
});
|
||||
|
||||
test('captureMetric returns 1 for coverage on empty brain (vacuous truth)', async () => {
|
||||
const v = await captureMetric(engine, 'entity_link_coverage');
|
||||
expect(v).toBe(1);
|
||||
});
|
||||
|
||||
test('captureMetric returns 0 for takes_count on empty brain', async () => {
|
||||
const v = await captureMetric(engine, 'takes_count');
|
||||
expect(v).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('onboard E2E — runAllOnboardChecks', () => {
|
||||
test('returns all 4 check shapes', async () => {
|
||||
const results = await runAllOnboardChecks(engine);
|
||||
expect(results.length).toBe(4);
|
||||
const names = results.map((r) => r.check.name).sort();
|
||||
expect(names).toEqual([
|
||||
'embed_staleness',
|
||||
'entity_link_coverage',
|
||||
'takes_count',
|
||||
'timeline_coverage',
|
||||
]);
|
||||
});
|
||||
|
||||
test('empty brain: stale/link/timeline ok, takes_count warns (0 takes)', async () => {
|
||||
const results = await runAllOnboardChecks(engine);
|
||||
const byName = Object.fromEntries(results.map((r) => [r.check.name, r.check.status]));
|
||||
expect(byName.embed_staleness).toBe('ok');
|
||||
expect(byName.entity_link_coverage).toBe('ok');
|
||||
expect(byName.timeline_coverage).toBe('ok');
|
||||
expect(byName.takes_count).toBe('warn'); // 0 takes is a warn
|
||||
});
|
||||
|
||||
test('empty brain returns 0 remediations (takes_count gated by bootstrap_enabled=false)', async () => {
|
||||
const results = await runAllOnboardChecks(engine);
|
||||
const total = results.reduce((s, r) => s + r.remediations.length, 0);
|
||||
// takes_count warns but does NOT emit a remediation because
|
||||
// takes.bootstrap_enabled defaults to false (A12 two-gate consent).
|
||||
expect(total).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('onboard E2E — computeRemediationPlan with extras', () => {
|
||||
test('threads extras through computeRecommendations', async () => {
|
||||
// Build a synthetic extra remediation. computeRemediationPlan
|
||||
// should merge it into the plan output even though the hardcoded
|
||||
// planner doesn't know about it.
|
||||
const extra = makeRemediationStep({
|
||||
id: 'test.synthetic',
|
||||
job: 'test-job',
|
||||
params: {},
|
||||
severity: 'low',
|
||||
est_seconds: 10,
|
||||
est_usd_cost: 0,
|
||||
rationale: 'synthetic test entry',
|
||||
status: 'remediable',
|
||||
});
|
||||
const plan = await computeRemediationPlan(engine, {
|
||||
targetScore: 90,
|
||||
extraRemediations: [extra],
|
||||
});
|
||||
const ids = plan.plan.map((p) => p.id);
|
||||
expect(ids).toContain('test.synthetic');
|
||||
});
|
||||
|
||||
test('returns RemediationPlan with stable schema_version: 2', async () => {
|
||||
const plan = await computeRemediationPlan(engine, { targetScore: 90 });
|
||||
expect(plan.schema_version).toBe(2);
|
||||
expect(typeof plan.brain_score_current).toBe('number');
|
||||
expect(plan.brain_score_target).toBe(90);
|
||||
expect(typeof plan.max_reachable_score).toBe('number');
|
||||
expect(Array.isArray(plan.plan)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('onboard E2E — buildOnboardReport', () => {
|
||||
test('produces stable JSON envelope with schema_version: 1', async () => {
|
||||
const plan = await computeRemediationPlan(engine, { targetScore: 90 });
|
||||
const report = buildOnboardReport(plan);
|
||||
expect(report.schema_version).toBe(1);
|
||||
expect(Array.isArray(report.recommendations)).toBe(true);
|
||||
expect(report.summary).toBeDefined();
|
||||
expect(typeof report.summary.total).toBe('number');
|
||||
expect(typeof report.summary.auto_eligible).toBe('number');
|
||||
expect(typeof report.summary.prompt_required).toBe('number');
|
||||
expect(typeof report.summary.manual_only).toBe('number');
|
||||
expect(typeof report.summary.est_total_usd).toBe('number');
|
||||
});
|
||||
});
|
||||
|
||||
describe('onboard E2E — toOnboardRecommendation tier policy', () => {
|
||||
test('non-protected job → auto_apply', () => {
|
||||
const step = makeRemediationStep({
|
||||
id: 'test.embed', job: 'embed-catch-up', params: {},
|
||||
severity: 'medium', est_seconds: 60, est_usd_cost: 0.1,
|
||||
rationale: 'embed', status: 'remediable',
|
||||
});
|
||||
const r = toOnboardRecommendation(step);
|
||||
expect(r.apply_policy).toBe('auto_apply');
|
||||
});
|
||||
|
||||
test('extract-takes-from-pages → manual_only (A12+A24)', () => {
|
||||
const step = makeRemediationStep({
|
||||
id: 'test.takes', job: 'extract-takes-from-pages',
|
||||
protected: true, params: {},
|
||||
severity: 'medium', est_seconds: 1800, est_usd_cost: 5,
|
||||
rationale: 'takes', status: 'remediable',
|
||||
});
|
||||
const r = toOnboardRecommendation(step);
|
||||
expect(r.apply_policy).toBe('manual_only');
|
||||
});
|
||||
|
||||
test('other protected jobs → prompt_required', () => {
|
||||
const step = makeRemediationStep({
|
||||
id: 'test.synth', job: 'synthesize',
|
||||
protected: true, params: {},
|
||||
severity: 'medium', est_seconds: 600, est_usd_cost: 1,
|
||||
rationale: 'synth', status: 'remediable',
|
||||
});
|
||||
const r = toOnboardRecommendation(step);
|
||||
expect(r.apply_policy).toBe('prompt_required');
|
||||
});
|
||||
});
|
||||
@@ -322,12 +322,12 @@ describe('Lane D.3 — sync surfaces dim-mismatch recipe at incremental AND firs
|
||||
// Lane E.4 — loadRecommendationContext provider-aware key check
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
describe('Lane E.4 — loadRecommendationContext is provider-aware', () => {
|
||||
// doctor.ts exports loadRecommendationContext only locally; verify the
|
||||
// behavior via a public surface (the recommendation context the
|
||||
// `doctor --remediation-plan` output uses) is brittle. Use a
|
||||
// source-text assertion instead.
|
||||
// v0.41.18.0 (A1): loadRecommendationContext extracted from doctor.ts
|
||||
// to src/core/remediation/context.ts so onboard CLI + MCP run_onboard
|
||||
// can compose the same context. Source-text grep follows the function
|
||||
// to its new home.
|
||||
test('source-text grep: loadRecommendationContext is provider-aware via the shared helper', () => {
|
||||
const src = readFileSync(join(__dirname, '..', 'src', 'commands', 'doctor.ts'), 'utf-8');
|
||||
const src = readFileSync(join(__dirname, '..', 'src', 'core', 'remediation', 'context.ts'), 'utf-8');
|
||||
// Pre-v0.37 this was OpenAI-only; the Lane E.4 fix made it branch on
|
||||
// provider for the key. v0.40.x replaced the inline prefix ladder with the
|
||||
// shared recipe-aware helper `embeddingProviderConfigured` (so doctor +
|
||||
|
||||
Reference in New Issue
Block a user