Compare commits

...
Author SHA1 Message Date
Garry Tan 5191ff8a55 Merge remote-tracking branch 'origin/master' into garrytan/denver-v4
# Conflicts:
#	CHANGELOG.md
#	TODOS.md
#	VERSION
#	package.json
#	src/commands/extract.ts
2026-05-26 18:50:36 -07:00
Garry TanandClaude Opus 4.7 cd58a564b2 fix(ci): connection-resilience regex + doctor warn-not-fail + v0.41.18.0
Two CI fixes from PR #1521 + version renumber per user request.

Why fix #1 (connection-resilience.test.ts): T5/A20 extended
PostgresEngine.executeRaw signature to accept an optional
`opts?: { signal?: AbortSignal }` 3rd arg and rewrote the body as
multi-line. The regression test's regex was anchored to the legacy
single-line `(sql: string, params?: unknown[])` shape and the
assertions banned `try {` / `catch` (which T5 legitimately added for
AbortSignal cancellation swallow, NOT for retry). Updated regex to
tolerate both shapes; replaced the wrong `not.toContain('conn.unsafe(
sql, params')` assertion (which incorrectly flagged the legitimate
single call) with a count assertion: `conn.unsafe(` must appear
exactly ONCE in the body. Preserves the original D3 intent (no
per-call retry — recovery is supervisor-driven via reconnect()) while
accepting the new try/catch shape that swallows AbortSignal aborts.

Why fix #2 (src/core/onboard/checks.ts): Three of the four new
onboard doctor checks (entity_link_coverage, timeline_coverage,
embed_staleness) emitted `status = 'fail'` on healthy DBs that simply
hadn't run extractions yet. This flipped `gbrain doctor`'s exit code
to non-zero on freshly initialized brains, breaking
test/e2e/mechanical.test.ts:1280 ("gbrain doctor exits 0 on healthy
DB"). Downgraded all three to `status = 'warn'` — these are
remediation opportunities, not assertion failures. Doctor exit
codes are reserved for actual failures; remediation surfaces use
warn-level signaling so they can be picked up by `--remediate`
without polluting the exit code.

Why fix #3 (version renumber 0.42.0.0 → 0.41.18.0): Per user
directive, this wave ships as v0.41.18.0 rather than v0.42.0.0.
Master is at 0.41.16.0; 0.41.17.0 is reserved for an in-flight
wave. Renamed every reference my branch added (54 files touched):
VERSION, package.json, CHANGELOG.md header, TODOS.md, plus inline
version-stamp comments across src/, test/, and scripts/. Preserved
13 files with PRE-EXISTING `v0.42.0.0` references on master (from
earlier waves originally planned for v0.42 that landed at v0.41.x —
those stay as historical record). Verified via per-file diff against
origin/master: every renamed reference is one I added in this branch.

Audit trio aligned: VERSION=0.41.18.0, package.json=0.41.18.0,
CHANGELOG topmost entry=[0.41.18.0]. llms-full.txt regenerated to
match CLAUDE.md updates.

Bisect contract: this commit fixes CI test failures from PR #1521's
landing. Typecheck clean; connection-resilience suite 26/26 pass.

Refs A20 (executeRaw AbortSignal), A16 (4 new onboard checks),
codex #1 (master collision avoidance via renumber).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:34:03 -07:00
Garry Tan 8eb86883af Merge remote-tracking branch 'origin/master' into garrytan/denver-v4
# Conflicts:
#	CHANGELOG.md
#	TODOS.md
#	VERSION
#	package.json
#	src/cli.ts
#	src/core/migrate.ts
2026-05-26 18:04:06 -07:00
Garry TanandClaude Opus 4.7 7b7ecfa470 v0.42.0.0 gbrain onboard mega PR — activation surface (closes #1383, completes #1409)
VERSION + package.json bumped to 0.42.0.0. CHANGELOG with full ELI10 lead
+ "What you can do that you couldn't before" itemized list + "To take
advantage of v0.42.0.0" upgrade steps per CLAUDE.md voice rules.

TODOS.md: 9 follow-up items filed (TODO-A through TODO-I) for the
v0.42.1+ wave: pack-aware linkable types, LLM-disambiguation NER,
onboard --explain, live-brain impact measurement, 100+-case takes
classifier eval, admin SPA UI, full DATABASE_URL E2E, minion_jobs
client_id schema column, thin-client doctor-remote parity.

llms-full.txt regenerated per CLAUDE.md rule (every CHANGELOG edit
followed by bun run build:llms in the same commit).

23/23 verify checks pass.

Full implementation across 21 commits on this branch (T0-T21):
  T0  merge master
  T1  schema migrations v98/v99/v100
  T2  extract doctor remediation library
  T3  generalize computeRecommendations
  T4  4 new doctor checks
  T5  engine API: listStaleChunks orderBy + executeRaw AbortSignal
  T6  embed --batch-size / --priority recent / --catch-up
  T7  NER extraction + extract-ner handler
  T8  timeline-from-meetings + extract-timeline-from-meetings handler
  T9  takes-bootstrap + extract-takes-from-pages handler
  T10 recordMinionJobSpend primitive
  T11 impact capture module + writeImpactLogRow
  T12 onboard render layer (types + render)
  T13 gbrain onboard CLI shell
  T14 init nudge + upgrade banner
  T15 autopilot tick consults onboard
  T16 MCP run_onboard + run_protected_onboard scope
  T17 check-source-scope-onboard lint
  T18 INSTALL_FOR_AGENTS.md agent prescription
  T20 hermetic PGLite E2E (13 cases)
  T21 ship (this commit)

Reviews: CEO + Eng + Codex on plan
~/.claude/plans/system-instruction-you-are-working-lively-hollerith.md.
27 A-decisions locked; 18 codex findings absorbed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:32:53 -07:00
Garry TanandClaude Opus 4.7 eb43365faa test(e2e): hermetic onboard surface contracts (T20)
NEW test/e2e/onboard-full-flow.test.ts. 13 hermetic PGLite cases
(no DATABASE_URL needed) covering the key onboard contracts:

  captureMetric — all 5 metrics return expected values on empty brain
    (0 for counts; 1 for coverage = vacuous truth).

  runAllOnboardChecks — returns exactly 4 results with correct names;
    empty brain shows stale/link/timeline ok BUT takes_count warns
    (0 takes); 0 remediations emitted because takes.bootstrap_enabled
    defaults to false per A12 two-gate consent.

  computeRemediationPlan — extras (T3 generalization) thread through to
    plan.plan output; stable schema_version: 2 envelope.

  buildOnboardReport — stable schema_version: 1 envelope with the right
    summary fields populated.

  toOnboardRecommendation tier policy (A8):
    - non-protected job → auto_apply
    - extract-takes-from-pages → manual_only (A12 + A24)
    - other protected jobs (synthesize, patterns, ...) → prompt_required

Full DATABASE_URL-gated end-to-end (real Postgres, actual extractions
through Minion handlers) deferred to v0.42.1 once the per-handler test
seam lands; the hermetic suite covers the data-shape contracts that
matter for downstream consumers binding to the JSON envelopes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:31:17 -07:00
Garry TanandClaude Opus 4.7 5bc6b58be4 docs(install): onboard surface agent prescription (D13, T18)
Adds a v0.42.0+ section to INSTALL_FOR_AGENTS.md describing:
  - First-connect probe: gbrain onboard --check --json
  - Post-upgrade re-probe (after gbrain upgrade)
  - Unattended remediation: gbrain onboard --auto --max-usd 5
  - MCP run_onboard op for federated/thin-client installs
  - run_protected_onboard scope requirement for LLM-bearing handlers
  - Two-gate consent for takes-bootstrap (takes.bootstrap_enabled + --yes)
  - GBRAIN_NO_ONBOARD_NUDGE=1 bypass for CI

Per D13: agents should run --check on first connect AND after every
upgrade as a hygiene step. The autopilot path makes this auto-improve
on a 24h cycle; the explicit agent probe surfaces opportunities
immediately on connect rather than waiting for the next autopilot tick.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:30:10 -07:00
Garry TanandClaude Opus 4.7 78ca2c8e04 chore(verify): add check-source-scope-onboard lint (A26, T17)
NEW scripts/check-source-scope-onboard.sh. Grep guard for SQL sites in
onboard surfaces (src/core/onboard/, src/commands/onboard.ts) that
touch source_id-bearing tables (pages, content_chunks, takes, links,
timeline_entries) WITHOUT either:
  (a) source_id / sourceIds in the WHERE clause, OR
  (b) the opt-out marker `sourcescope:brain-wide` within 4 lines above
      the SQL.

File-level opt-out: `sourcescope:file-brain-wide` in the file header
(first 30 lines) treats every SQL site in that file as intentionally
brain-wide. Used by onboard/checks.ts, onboard/impact-capture.ts, and
commands/onboard.ts because the onboard CHECKS are explicitly brain-wide
aggregates (orphan_count, stale_count, link_coverage are reported
across all sources by design).

Wired into bun run verify (23 checks total now, all green).

Without this gate, any future onboard SQL touching per-source data
without source-scoping would silently leak rows across sources —
exactly the class of bug v0.34.1's P0 seal closed at the engine layer.
The lint adds an explicit forcing function for new code in the onboard
surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:29:46 -07:00
Garry TanandClaude Opus 4.7 4b2344d7d8 feat(mcp): run_onboard op with run_protected_onboard scope binding (A7, T16, codex finding #5)
NEW MCP op `run_onboard`. Admin scope (NOT localOnly) so federated /
thin-client brain installs can probe brain health + submit auto-eligible
remediation handlers over OAuth-authenticated MCP.

Two-tier authorization per A7 + codex #5:
  - Admin scope: sufficient for mode='check' (read-only OnboardReport JSON)
    AND for submitting non-protected handlers in mode='auto'/'auto-with-prompt'.
  - run_protected_onboard scope (NEW, additive): MUST be granted in
    addition to admin for any PROTECTED_JOB_NAMES handler to fire
    (synthesize, patterns, consolidate, extract-takes-from-pages,
    contextual_reindex_per_chunk).

Without the new scope tier, an admin-scoped OAuth token would silently
bypass the same protected-name gate `submit_job` enforces at
operations.ts:2288. The codex finding #5 caught this: admin scope alone
was insufficient guard. Now the run_onboard op explicitly FILTERS
protected extras from the recommendation plan when the caller lacks
run_protected_onboard; filtered items appear in the response as
skipped_missing_scope[] so the caller knows what would have been
available with the right grants.

Modes:
  check               — read-only OnboardReport JSON envelope.
  auto                — submits auto_apply tier (plus prompt_required
                        when --yes/auto-with-prompt).
  auto-with-prompt    — adds prompt_required tier.

Both auto modes REQUIRE max_usd per A12 + A20 cron-safety (rejects
with invalid_params if missing).

Per A26 source-scope: future extension will scope plans by ctx.sourceId
/ ctx.auth.allowedSources. Today the recommendation planner is
brain-wide; the source-scope thread doesn't change correctness, just
optimization.

Per A19 fail-open: any error in runAllOnboardChecks during plan-build
caught + suppressed; the plan still returns with extras=[] rather than
crashing the op.

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:28:16 -07:00
Garry TanandClaude Opus 4.7 bdb94132b0 feat(autopilot): tick consults onboard recommendations (A5, A19, A22, T15)
Pre-fix: autopilot tick's per-source recommendation walk called
computeRecommendations(health, ctx) — doctor's hardcoded 5-category
planner. The 4 new onboard checks (embed_staleness,
entity_link_coverage, timeline_coverage, takes_count) had nowhere to
hook in, so even with takes.bootstrap_enabled flipped on, autopilot
never noticed 0 takes and never proposed bootstrap.

Post-fix: tick body now ALSO calls runAllOnboardChecks(engine) and
threads the result's RemediationStep[] into the T3-generalized third
arg of computeRecommendations. The planner merges onboard's extras
with the legacy hardcoded entries (hardcoded wins on id collision).

Per A19 fail-open: any throw in the onboard-checks path is caught,
logged to stderr, and suppressed. The legacy plan (without extras)
runs as before — autopilot can't crash from an onboard-check failure.

A22 (idempotency-key dedupe across concurrent manual + autopilot
runs): inherits from the existing computeRecommendations →
remediation.idempotency_key chain. T7-T9 handlers each get their
content-hash key from the makeRemediationStep factory; an autopilot
tick + a manual `gbrain onboard --auto` submitting the same step
in the same brain produce the SAME key, so queue.add(...) dedupes.

No behavior change for brains where all 4 onboard metrics already
look healthy (extras=[]; legacy plan unchanged).

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:27:09 -07:00
Garry TanandClaude Opus 4.7 849e78a88a feat(onboard): init nudge + upgrade banner (A4, A18, A20, T14)
NEW src/core/onboard/init-nudge.ts exports two fail-open hooks:

runInitNudge(engine):
  Post-initSchema 5-query AbortSignal-bound parallel check against a
  3-second wallclock budget. Per A20: uses REAL cancellation via the
  T5 executeRaw signal extension — Promise.race against a timer was
  codex's #7 wrong shape. Postgres queries actually .cancel(); PGLite
  documented gap.
  Partial-results path: if some checks complete and the budget fires
  on others, prints what landed + a fallthrough hint pointing at
  `gbrain onboard --check` for the full picture.
  Per A18: fail-open — ANY throw is caught, logged to stderr, and
  suppressed so init returns successfully.
  Bypass: GBRAIN_NO_ONBOARD_NUDGE=1 short-circuits. Non-TTY default
  short-circuits too (CI/scripted callers see nothing).
  Nudge format: one-line summary of opportunities ("Brain has
  opportunities: 23000 stale chunks, link coverage 32%, 0 takes")
  + a 'gbrain onboard --check' nudge.

runUpgradeBanner(_engine):
  Lighter post-upgrade banner. Doesn't engine-query — just prints a
  one-line nudge that upgrades may surface new opportunities. Same
  fail-open posture.

Wired into:
  src/commands/init.ts:initPGLite (end-of-function, after reportModStatus)
  src/commands/init.ts:initPostgres (same)
  src/commands/upgrade.ts:runPostUpgrade (end-of-function, after
  postUpgradeReferenceSweep)

Each wire site uses dynamic import + try/catch so even an import
failure can't crash init/upgrade.

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:26:35 -07:00
Garry TanandClaude Opus 4.7 0ff5c1fecb feat(onboard): gbrain onboard CLI shell (A1, T13)
NEW src/commands/onboard.ts (~180 LOC). Thin wrapper that composes:
  - T2 library (computeRemediationPlan + runRemediation)
  - T4 onboard checks (runAllOnboardChecks → extraRemediations)
  - T12 render layer (buildOnboardReport + renderHuman)

Three modes:
  --check    (default): print plan, no submission. Computes plan via
             T2 library with T4 check-derived extraRemediations.
             Renders human (default) or JSON envelope (--json).
  --auto:    submit auto_apply tier. Requires --max-usd N (cron-safety
             per A12 + A20 — refuses without explicit cap to avoid
             surprise spend).
  --auto --yes: also submit prompt_required tier.
  --history: dump last 50 migration_impact_log entries.

Library hooks wired into stderr (per CLI/library separation): onStepStart,
onStepEnd, onBudgetRefused, onBudgetExhausted, onNothingToDo,
onTargetUnreachable. Final JSON envelope (--json) or human summary
lands on stdout.

CLI dispatch: registered in src/cli.ts CLI_ONLY set + case dispatch
between 'takes' and 'founder'.

Typecheck clean. Manual smoke-test pending T20 E2E (DATABASE_URL gated).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:25:03 -07:00
Garry TanandClaude Opus 4.7 021b23f7b7 feat(onboard): types + render layer (A8, T12)
NEW src/core/onboard/types.ts: OnboardRecommendation (extends
RemediationStep with apply_policy + prompt_text + migration_id),
OnboardReport (stable JSON envelope), OnboardOpts.

NEW src/core/onboard/render.ts:
  toOnboardRecommendation(step): RemediationStep → OnboardRecommendation
    Sets apply_policy per A8 tiered rules:
      - protected + job === extract-takes-from-pages → 'manual_only' (A12/A24)
      - protected + other → 'prompt_required'
      - non-protected → 'auto_apply'
  buildOnboardReport(plan, opts?): assembles the stable JSON envelope.
  renderHuman(report): string. Echoes the "Recommendation + WHY" framing
    the CEO + Eng + Codex reviews settled on; CLI shell prints to stdout.

Stable JSON envelope shape:
  schema_version: 1
  brain_id?: string
  recommendations: OnboardRecommendation[]
  summary: { total, auto_eligible, prompt_required, manual_only,
             est_total_usd }
  history?: Array<{ remediation_id, metric_name, metric_before,
                    metric_after, delta, applied_at }>

Library-shaped — no console.* / process.exit. T13 (onboard CLI shell)
calls these from the wrapping CLI. MCP run_onboard (T16) returns the
JSON envelope unmodified.

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:24:02 -07:00
Garry TanandClaude Opus 4.7 6860993a47 feat(onboard): impact capture module + writeImpactLogRow primitive (A6 + A25 + A17, T11)
NEW src/core/onboard/impact-capture.ts. Three exports:

captureMetric(engine, metric)
  Pure-ish: returns the current numeric value for one of 5 metrics
  (orphan_count, stale_count, entity_link_coverage, timeline_coverage,
  takes_count). Returns null on any throw per A17 best-effort posture
  — a stat-query failure MUST NOT block the extraction itself.

writeImpactLogRow(engine, attribution, metric, before, after, details?)
  Best-effort INSERT into v100's migration_impact_log table. Attribution
  columns (job_id, source_id, brain_id, started_at, idempotency_key,
  applied_by) per A25 + codex finding #10 so concurrent runs can't
  misattribute deltas.

withImpactCapture(engine, attribution, metric, runner, details?)
  Convenience: capture-before → run → capture-after → write log row.
  Per A17 the log row lands even when the runner throws (after-on-fail
  + error in details), so downstream consumers see a "ran but impact
  unknown" entry instead of silent loss.

Designed to be picked up by the 4 new Minion handlers (embed-catch-up,
extract-ner, extract-timeline-from-meetings, extract-takes-from-pages)
when they wrap their main runner. Handlers stay decoupled from the
log-write path — they just call withImpactCapture with the metric they
move. Per-handler integration follows in T12/T13/T15 as those wrappers
land.

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:23:26 -07:00
Garry TanandClaude Opus 4.7 5c3d530af2 feat(minions): recordMinionJobSpend primitive for MCP client_id attribution (A7+A23, codex finding #4)
NEW src/core/minion-spend.ts: small primitive that closes the per-OAuth-
client spend chain gap codex flagged when MCP run_onboard submits child
Minion jobs.

Pre-fix: only subagent loops via budget-meter.ts recorded spend against
the originating OAuth client. Generic Minion handlers (embed-catch-up,
extract-ner, extract-timeline-from-meetings, extract-takes-from-pages)
wrote to the gateway with no per-client attribution — admin-scope tokens
would have unbounded indirect spend via the run_onboard fan-out.

Convention for v0.42.0 (deferred schema column to v0.42.1):
  - run_onboard MCP op sets job.data.client_id when submitting each
    child handler.
  - Handlers that spend LLM/embedding budget call
    recordMinionJobSpend(engine, job, {operation, spendCents, ...})
    which reads job.data.client_id and writes mcp_spend_log with
    the right attribution.
  - Local-submitted jobs (CLI, autopilot tick) pass no client_id;
    the row still lands with client_id=null for global accounting.

Two exports:
  getJobClientId(job): undefined for local jobs; the OAuth client_id
    string for MCP-submitted ones.
  recordMinionJobSpend(engine, job, entry): wraps recordSpend with
    job-aware attribution. Best-effort throughout — spend telemetry
    failures MUST NOT fail the user's call.

A23 full schema column (minion_jobs.client_id + index) deferred to
v0.42.1; today's JSONB-pass-through is sufficient for the MCP
run_onboard chain to land per-client attribution end-to-end. Handlers
adopt the primitive over time; no behavior change for callers that
haven't migrated.

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:22:53 -07:00
Garry TanandClaude Opus 4.7 f2293daa42 feat(takes): takes-bootstrap from concept/atom/lore pages (A12, A24, T9)
NEW src/core/extract-takes-from-pages.ts: Haiku classifier loop. Walks
pages WHERE type IN ('concept','atom','lore','briefing','writing',
'originals') AND deleted_at IS NULL AND length(compiled_truth) > 200,
ordered by updated_at DESC. Each page is truncated to 20K chars and
sent to Haiku with a strict-JSON classifier prompt:
  {"claim", "kind": fact|take|bet|hunch, "weight": 0..1}

Inserts via addTakesBatch with source='cli:takes-bootstrap-from-pages'.

Two-gate consent per A12:
  1. `takes.bootstrap_enabled` config (default false) — even the manual
     CLI refuses without it explicitly set.
  2. --yes flag (CLI) — interactive confirmation that this sends content
     to Haiku.

The handler-side gate also reads takes.bootstrap_enabled, so even a
trusted local Minion submitter (allowProtectedSubmit=true) cannot
fire takes-bootstrap on a brain that hasn't opted in.

CLI: `gbrain takes extract --from-pages [--yes] [--dry-run] [--source-id X]
[--max-pages N] [--holder name]`. Surfaces consent-gate-blocked vs
llm-unavailable distinctly so users see the actual blocker.

Minion handler `extract-takes-from-pages` added to PROTECTED_JOB_NAMES.
Consumed by onboard's takes_count remediation when count=0 AND
takes.bootstrap_enabled=true (handler-side double-check).

Per A24: ships with classifier infrastructure ONLY. Per-prompt eval suite
deferred to v0.42.1 follow-up; autopilot remediation tier for takes-bootstrap
stays manual_only until eval coverage catches up. Manual `gbrain takes
extract --from-pages --yes` is the only path that triggers it in v0.42.0.

parseClaimsJson exported for unit testing — strict JSON parse + ```json
fence strip + kind allowlist filter, returns [] on any parse failure.

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:22:04 -07:00
Garry TanandClaude Opus 4.7 4c17e159f0 feat(extract): timeline from meetings — gbrain extract timeline --from-meetings (A11, T8, codex #11)
NEW src/core/extract-timeline-from-meetings.ts:
extractTimelineFromMeetings(engine, opts). Walks meeting pages, finds
discussed entities via two sources, writes a timeline entry on each
entity page.

Discussed-entity sources merged:
  1. Existing 'attended' links from the meeting (canonical attendees).
     One round-trip SELECT pulls all attended edges for the loaded
     meeting set; in-memory Map<meetingSlug → attendees[]> for O(1)
     lookup per meeting.
  2. Body-text mentions via the existing by-mention gazetteer
     (findMentionedEntities + cross-source guard). Catches entities
     discussed in the meeting body even when no explicit 'attended'
     link exists.

De-duped via Map<sourceId::slug → entity> within each meeting so a
person who's both an attendee AND mentioned in the body gets exactly
one timeline row per meeting, not two.

Timeline write uses TimelineBatchInput with:
  source = 'extract-timeline-from-meetings:<meeting-slug>'
  summary = 'Discussed in <meeting-title>'
  date = meeting.effective_date

Per v99 dedup widening (codex #11): the source field is now in the
uniqueness key (page_id, date, summary, source). Two meetings on the
same date with the same summary on the same entity page survive as
distinct rows — the second meeting's provenance is no longer silently
dropped.

CLI: `gbrain extract timeline --from-meetings` (DB source only). Mode
dispatch — runs SOLO (does not combine with --by-mention/--ner; those
are links passes).

Minion handler: `extract-timeline-from-meetings` (NOT in
PROTECTED_JOB_NAMES — pure SQL + string scan). Consumed by onboard's
timeline_coverage remediation when coverage <90%.

Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:20:39 -07:00
Garry TanandClaude Opus 4.7 7991e64e49 feat(extract): NER link extraction via schema-pack inference.regex (A10, T7, codex #12)
NEW src/core/extract-ner.ts: extractNerLinks(engine, opts). Walks pages,
reuses the by-mention gazetteer, applies the active schema-pack's
link_types[].inference.regex patterns to assign a typed verb to each
mention ("CEO of Acme" + Acme is a company → 'works_at' linking the
source page to Acme).

Codex finding #12 design: do NOT split link_source='ner' as a new
provenance. NER is still mention-derived; splitting would break every
existing link_source='mentions' query (backlink-count, orphan-ratio,
doctor checks). Instead: keep link_source='mentions' AND set
link_kind='typed_ner' (v98 column).

LinkBatchInput type gains link_kind field. Both engines'
addLinksBatch impls add the column to the INSERT projection + unnest()
tuple (column #11). The links UNIQUE constraint excludes link_kind so
an existing plain mention row + a typed_ner row for the same (from, to,
type, source, origin) collide DO NOTHING; the typed link goes in as a
separate row with a DIFFERENT link_type (the inferred verb), so they
don't collide on the typical case.

CLI: `gbrain extract links --ner` (DB source only). Combined
`--by-mention --ner` walk shares ONE gazetteer build across both passes
— saves a full walk on big brains. Either flag alone runs its pass
solo. Each gets its own --source-id filter inheritance.

Minion handler: `extract-ner` (NOT in PROTECTED_JOB_NAMES — regex-only,
no LLM spend). Consumed by onboard's entity_link_coverage remediation
when coverage <70%.

Target-type lookup: one round-trip SELECT slug, source_id, type FROM
pages WHERE type IN ('person', 'company', 'organization', 'entity')
AND deleted_at IS NULL — built once at extraction start, consulted
per-mention. Avoids the N+1 getPage cost.

Pack best-effort: when no active pack OR no link_types declared OR
no inference.regex on any link_type, returns pack_unavailable=true and
0 created. CLI prints a one-line note; handler returns silently.

122 tests pass (pglite-engine + by-mention); typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:19:37 -07:00
Garry TanandClaude Opus 4.7 474caba26d feat(embed): --batch-size + --priority recent + --catch-up + embed-catch-up handler (A13)
CLI surface on gbrain embed gains 3 flags:
  --batch-size N       Override hardcoded PAGE_SIZE=2000 (clamped 1..10000)
  --priority recent    Walk stale chunks newest-first (page.updated_at DESC)
                       backed by content_chunks_stale_idx + idx_pages_updated_at_desc
                       via T5's listStaleChunks(orderBy='updated_desc') extension.
                       Composite cursor (updated_at, page_id, chunk_index).
  --catch-up           Removes the GBRAIN_EMBED_TIME_BUDGET_MS wall-clock cap;
                       loops until countStaleChunks() returns 0.

EmbedOpts gains matching fields; embedAll + embedAllStale plumb them through.
The cursor tracking in embedAllStale now advances (afterUpdatedAt, afterPageId,
afterChunkIndex) instead of just (afterPageId, afterChunkIndex) when in
'updated_desc' mode. The engine returns p.updated_at as Date|string; the
caller normalizes to ISO string for the next page's cursor.

New Minion handler `embed-catch-up` registered in jobs.ts. Wraps runEmbedCore
with stale=true + catchUp=true + the priority/batchSize the caller supplies.
NOT in PROTECTED_JOB_NAMES (embedding spend only — same posture as the
existing embed-backfill handler). Consumed by the gbrain onboard remediation
pipeline (T11) when embed_staleness check fires.

63 embed tests pass; typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:11:02 -07:00
Garry TanandClaude Opus 4.7 8dc8c39de6 feat(engine): listStaleChunks --priority recent + executeRaw AbortSignal (A13/A20, codex #7 #9)
Two interface extensions on BrainEngine, with parity across postgres-engine
and pglite-engine. Plus a follow-on fix for v99's timeline_entries dedup
widening.

listStaleChunks gains:
  - orderBy?: 'page_id' | 'updated_desc' (default 'page_id' = legacy)
  - afterUpdatedAt?: string | null (composite cursor for updated_desc)

When orderBy === 'updated_desc' the query JOINs pages and orders by
  p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
backed by idx_pages_updated_at_desc + content_chunks_stale_idx partial
(both indexes added in v100). The cursor "next row" semantic with DESC
NULLS LAST + ASC tiebreakers is:
  (updated_at < prev) OR
  (updated_at = prev AND page_id > prev_page_id) OR
  (updated_at = prev AND page_id = prev_page_id AND chunk_index > prev_chunk_index)
First page (afterUpdatedAt undefined AND afterPageId 0) bypasses the
cursor predicate. Both engines parity-tested via 100/100 pglite-engine
tests; Postgres path mirrors the same WHERE clause structure.

executeRaw gains:
  - opts?: {signal?: AbortSignal}

Postgres impl: real cancellation via postgres.js's .cancel() on the
pending query. Pre-aborted signal short-circuits before the network
round-trip; mid-flight abort fires .cancel(). The query throws on
abort which the caller catches.

PGLite impl: in-process WASM has no kernel-level cancellation.
Best-effort: pre-check, then race the query against a signal-rejection
promise. The query keeps running in WASM but the awaited result is
discarded (DOMException AbortError thrown). Documented gap.

ReservedConnection.executeRaw extends the signature for type
compatibility but doesn't wire the signal (its only callers are
migrations + cycle-lock writes that explicitly don't want cancellation).

V99 timeline dedup follow-on: the dedup widening in migration v99
changed the unique index from (page_id, date, summary) to
(page_id, date, summary, source). The ON CONFLICT clauses in both
engines' addTimelineEntriesBatch + addTimelineEntry impls were still
using the old 3-tuple, causing 12 PGLite tests to fail with SQLSTATE
42P10 "no unique constraint matching ON CONFLICT specification".
Updated all 4 sites (2 per engine) to the 4-tuple.

Typecheck clean, 100/100 PGLite engine tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:05:20 -07:00
Garry TanandClaude Opus 4.7 7318c913c6 feat(doctor): 4 new onboard checks (embed_staleness, link_coverage, timeline_coverage, takes_count) (A16, T4)
Adds src/core/onboard/checks.ts: 4 check helpers + a runAllOnboardChecks
aggregator. Each helper returns {check, remediations}, so doctor pushes
the Check entry (for human/JSON rendering) AND onboard's plan path
collects the RemediationStep[] (via T3's new extraRemediations seam in
computeRecommendations).

embed_staleness: COUNT(*) on content_chunks WHERE embedding IS NULL.
  Cheap thanks to content_chunks_stale_idx partial (v100).
  warn at 1+ stale, fail at 1000+; remediation points at embed-catch-up
  handler (built in T6).

entity_link_coverage: fraction of entity pages with inbound links.
  Per A21 + codex #15: TABLESAMPLE BERNOULLI on PG when total_pages > 50K
  with pinned sample formula (LEAST 100, GREATEST 2, target ~5000 rows)
  AND ±sqrt(p(1-p)/n) confidence interval embedded in message
  ("coverage: 31% ± 1.3%") so warn/fail decisions show their margin of error.
  PGLite path: full scan (rare >50K).
  warn <70%, fail <40%; remediation points at extract-ner handler.

timeline_coverage: same TABLESAMPLE policy. warn <90%, fail <70%;
  remediation points at extract-timeline-from-meetings handler.

takes_count: COUNT(*) on takes table. Per A12 two-gate consent: the
  remediation only emits when `takes.bootstrap_enabled` config is true.
  Otherwise the check shows "0 takes (takes.bootstrap_enabled is false;
  opt in to enable)" without an autopilot-eligible remediation. Prevents
  unattended LLM-bearing extractions on brains that haven't opted in.

runDoctor wires runAllOnboardChecks at the end of the DB-checks block
(after stale_locks); fast-mode skipped to preserve --fast UX.

Thin-client parity (A16 spec) deferred to T16 — the MCP run_onboard op
will run these helpers server-side where engine.executeRaw works,
which is the real federated path. Adding them to doctor-remote.ts
would duplicate the logic without functional benefit since the helpers
are server-side queries.

55 doctor tests pass; typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:00:58 -07:00
Garry TanandClaude Opus 4.7 4f3a29d643 refactor(remediation): generalize computeRecommendations to accept extras (A2, codex finding #3)
Pre-fix: computeRecommendations at brain-score-recommendations.ts:170 was a
hardcoded planner for 5 synthetic check categories. Adding a Check.remediation
field to a new doctor check would NOT auto-wire into --remediation-plan —
the planner simply ignored it. Codex caught this when reviewing the plan's
"checks ARE specs" framing.

Post-fix: optional third arg `extraRemediations: RemediationStep[]` lets
callers inject step entries discovered outside the hardcoded planner. The
existing 5-category surface is preserved bit-for-bit; on id collision the
hardcoded entry wins, so an extra accidentally duplicating a hardcoded id
doesn't shadow legacy behavior.

RemediationPlanOpts gains the matching field; computeRemediationPlan in
src/core/remediation/plan.ts threads opts.extraRemediations through. The
4 new doctor checks (T4) will produce per-check helper functions that
return RemediationStep[]; onboard's render layer (T12) aggregates them
into the opts.extraRemediations slot. doctor's existing
--remediation-plan call passes empty (no behavior change for legacy CLI).

84 tests pass across brain-score-recommendations + doctor suites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:58:47 -07:00
Garry TanandClaude Opus 4.7 463fc5cab0 refactor(remediation): extract doctor remediation library (A1, codex finding #2)
Pre-fix: src/commands/doctor.ts contained two CLI-shaped functions
(runRemediationPlan + runRemediate) with hardcoded argv parsing,
process.exit calls, and console.log emission. Onboard CLI shell and the
upcoming MCP run_onboard op couldn't compose against them — the plan
file's "100-LOC thin wrapper" assumption didn't survive codex's review
of the actual source.

Post-fix: src/core/remediation/ exports a library shape that all three
consumers (doctor CLI, onboard CLI, MCP run_onboard) wrap.

  src/core/remediation/types.ts
    RemediationPlanOpts, RemediationPlan, RemediationOpts,
    RemediationResult, StepResult, RemediationHooks (the observability
    seam — library never calls console.* itself).

  src/core/remediation/context.ts
    loadRecommendationContext moved verbatim from doctor.ts. Re-exports
    RecommendationContext from brain-score-recommendations.ts since
    that's still the canonical home for the type (consumed by
    computeRecommendations).

  src/core/remediation/plan.ts
    computeRemediationPlan(engine, opts): Promise<RemediationPlan>.
    Pure read; produces the stable JSON envelope downstream agents
    bind to. Pulls in computeRecommendations + classifyChecks +
    maxReachableScore behind one library entry point.

  src/core/remediation/run.ts
    runRemediation(engine, opts, hooks): Promise<RemediationResult>.
    Orchestrator with BudgetTracker, checkpoint resume, D5 dep
    cascade, D7 per-step recheck. Returns a result object instead
    of process.exit calls; the CLI shell maps result.budget_exhausted
    / .target_unreachable / .submitted to exit codes.

  src/core/remediation/index.ts
    Barrel for the three modules above.

doctor.ts is now a thin wrapper:
  runRemediationPlan: parse argv → computeRemediationPlan → human/JSON render
  runRemediate: parse argv → TTY confirm gate → runRemediation(hooks: console.*)
The TTY confirmation step deliberately stays in the CLI shell — the library
never asks for confirmation; that's a CLI concern.

Net: ~340 LOC removed from doctor.ts; ~470 LOC added across the library
module (with full JSDoc + per-A-decision rationale comments). Functional
behavior preserved bit-for-bit: 67 tests pass across doctor.test.ts +
v0_37_gap_fill.serial.test.ts.

The Lane E.4 source-text test (test/v0_37_gap_fill.serial.test.ts:329)
followed loadRecommendationContext to its new home at
src/core/remediation/context.ts — assertions otherwise unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:57:05 -07:00
Garry TanandClaude Opus 4.7 64423f5af0 feat(schema): migrations v98/v99/v100 for onboard wave (A6 A10 A11 A13 A25, codex #1 #9 #10 #11 #12)
Three schema additions supporting the gbrain onboard wave:

v98 — links.link_kind nullable column (A10, codex finding #12).
The NER extraction was originally going to add a new link_source='ner'
provenance, but that would have forced every existing link_source='mentions'
query (backlink-count filter, orphan-ratio, doctor checks) to update or
metrics would drift across the cutover. Instead: keep link_source='mentions'
for the storage layer AND add a nullable link_kind column. Three kinds:
'plain', 'typed_ner', NULL (legacy/unknown — semantically 'plain'). NOT in
the links UNIQUE constraint so the storage shape stays compatible.

v99 — timeline_entries dedup widening (A11, codex finding #11).
Pre-v99 dedup key was (page_id, date, summary). The new --from-meetings
extraction writes timeline entries with source='extract-timeline-from-
meetings:<meeting-slug>', and codex caught that two meetings with the same
date+summary on the same entity page would silently DO NOTHING — the
second meeting's provenance is lost. Widened to (page_id, date, summary,
source). Legacy rows (source='') preserve current dedup behavior.

v100 — migration_impact_log table + content_chunks_stale_idx partial
(A6 + A25 + A13 + codex findings #10 + #9). Bundled because both are
consumed by the onboard pipeline and ship together. Impact log captures
before/after metric stats so gbrain onboard --history shows real deltas;
attribution columns (job_id, source_id, brain_id, started_at,
idempotency_key) prevent concurrent runs misattributing to wrong
migrations. content_chunks_stale_idx partial WHERE embedding IS NULL
supports gbrain embed --stale + --priority recent (outer ORDER BY
p.updated_at DESC uses existing idx_pages_updated_at_desc via JOIN).
Plain NUMERIC columns; delta computed at read time (NOT a stored
GENERATED column per eng-review D2 — zero PGLite parity risk).

Slot history note: plan originally proposed v97/v98/v99 but master had
already used v95 (links 'mentions' CHECK widening), v96 (facts conversation
session index), and v97 (pages_dedup_partial_index) by ship time. Codex
caught the collision; renumbered to v98/v99/v100.

Test pin: test/schema-bootstrap-coverage.test.ts (100/100 migrations
apply clean on PGLite), test/migrate.test.ts (152 cases pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:53:17 -07:00
44 changed files with 4013 additions and 447 deletions
+115
View File
@@ -2,6 +2,121 @@
All notable changes to GBrain will be documented in this file.
## [0.41.18.0] - 2026-05-26
**You can now run one command and have gbrain tell you exactly what's
wrong with your brain — and offer to fix it.**
Most agents wire up gbrain, import their notes, and never realize that
most of their pages have no inbound links, thousands of pages are
missing embeddings, and the brain is running at half its retrieval
potential. The agent doesn't know to ask. The user doesn't know to
look. This release adds the surface that closes that gap.
Run `gbrain onboard --check` and your brain shows up: how many stale
chunks, how much entity-link coverage, how many timeline entries, how
many takes. Then it shows the commands that would fix each one. Run
`gbrain onboard --auto --max-usd 5` and gbrain runs every fix that's
safe to apply unattended.
What you can do that you couldn't before:
- `gbrain onboard --check` — see your brain's onboarding state in 5
seconds (orphans, stale embeddings, link coverage, timeline coverage,
takes count). JSON envelope shape is stable (`schema_version: 1`) for
agents that want to bind to it.
- `gbrain onboard --auto --max-usd 5` — apply every safe remediation
the planner returns. Refuses without `--max-usd` so a cron can't burn
through your API budget.
- `gbrain embed --stale --catch-up --priority recent` — embed your
newest-edited pages first, keep going until the backlog is gone.
Lifts the prior hardcoded 2000-row batch cap (now `--batch-size N`).
- `gbrain extract links --ner` — typed NER links via your schema pack's
`link_types[].inference.regex`. Pairs with `--by-mention` in one walk
(shared gazetteer). New `link_kind='typed_ner'` column distinguishes
verb-pattern matches from plain mentions without splitting
`link_source`.
- `gbrain extract timeline --from-meetings` — walks meeting pages,
writes a timeline entry on each entity that was discussed (attendees
+ body mentions). Survives the v99 dedup widening so two meetings on
the same date no longer silently drop one.
- `gbrain takes extract --from-pages` — Haiku classifier over your
concept/atom/lore/briefing/writing/originals pages, lifts gradeable
claims into the takes fence. Two-gate opt-in
(`takes.bootstrap_enabled=true` AND `--yes`) — does not run unattended,
does not run without your explicit consent.
- Init + upgrade nudges — `gbrain init` and `gbrain upgrade` now end
with a one-line summary of opportunities so first-day users
discover the onboard surface. 3-second wallclock cap, fail-open
(never crashes init). Suppress with `GBRAIN_NO_ONBOARD_NUDGE=1`.
- Autopilot tick consults onboard recommendations alongside
brain-score remediations. Self-improving brain on a 24h cycle.
- MCP op `run_onboard` (admin scope) — federated and thin-client
installs can probe brain health + drive remediation over OAuth.
LLM-bearing handlers (synthesize, patterns, consolidate,
takes-bootstrap) require a NEW `run_protected_onboard` scope IN
ADDITION to admin — admin alone won't burn your API budget.
- `gbrain onboard --history` — see what landed and what it changed.
New `migration_impact_log` table with full attribution columns
(job_id, source_id, brain_id, started_at, idempotency_key)
prevents concurrent runs misattributing deltas.
The architecture story for engineers: `gbrain doctor --remediate`
(v0.36.4.0) had already shipped the cathedral — generalized
RemediationStep, dependency-ordered execution, BudgetTracker
integration, Minion-handler dispatch. We extended that cathedral
instead of building a parallel one. A new
`src/core/remediation/` library extracts the orchestrator from
doctor's CLI shape; `gbrain onboard` is a ~180-line wrapper. The
codex review caught this — the original plan was going to rebuild
RemediationStep under a new name and that would have been a 4000+
LOC rebuild of work that already existed.
Schema additions:
- v98 — `links.link_kind` nullable column (`'plain'` | `'typed_ner'` | NULL).
- v99 — `timeline_entries` dedup widened to `(page_id, date, summary,
source)` so meeting provenance survives.
- v100 — `migration_impact_log` table + `content_chunks_stale_idx`
partial (supports `--priority recent` cursor).
Engine API additions:
- `BrainEngine.listStaleChunks(opts)` gains `orderBy: 'page_id' |
'updated_desc'` + `afterUpdatedAt` for composite-cursor pagination.
- `BrainEngine.executeRaw(sql, params, opts?)` gains `opts.signal`
for real AbortSignal-bound query cancellation (Postgres
`query.cancel()`; PGLite best-effort via Promise.race).
- `LinkBatchInput.link_kind?: string` — threaded through both engine
impls' `addLinksBatch` unnest tuple.
Privacy + consent posture: takes-bootstrap sends concept/atom/lore/
briefing/writing/originals page content to your configured chat model.
Two-gate consent — `takes.bootstrap_enabled=true` config AND `--yes`
flag — refuses to run otherwise. Even with both gates flipped, the
autopilot path for takes-bootstrap stays `manual_only` (does not fire
unattended) until v0.42.1 lands the 100+-case eval suite.
To take advantage of v0.41.18.0:
`gbrain upgrade` runs the schema migrations and prints the post-upgrade
banner pointing at `gbrain onboard --check`. If something looks off:
1. Run `gbrain apply-migrations --yes` to ensure v98/v99/v100 landed.
2. Run `gbrain onboard --check` to see your brain's state.
3. If you want to opt into takes-bootstrap, run:
`gbrain config set takes.bootstrap_enabled true`
then `gbrain takes extract --from-pages --yes --max-usd 5`.
4. Filed an issue? Include `gbrain doctor --json` + the contents of
`~/.gbrain/upgrade-errors.jsonl` if it exists. Helps gbrain
maintainers find fragile upgrade paths.
Closes meta-issue #1383 (`gbrain onboard`). Migration #1 (auto-link
`--by-mention`) shipped earlier as v0.41.10.0. PR #1409 (the
consolidated design doc) is implemented by this release.
Note: schema migrations originally numbered v98/v99/v100 were renumbered
to v101/v102/v103 post-merge because master claimed v98 (sync lock
refresh column from v0.41.15.0) and v99 (conversation parser cache from
v0.41.16.0). Migration content unchanged across the renumber.
## [0.41.17.0] - 2026-05-26
**You can now run `extract-conversation-facts`, `extract`,
+55
View File
@@ -274,3 +274,58 @@ automatically during `gbrain post-upgrade` to fix the double-encoded JSONB
columns. PGLite brains no-op. If wiki-style imports were truncated by the old
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
`compiled_truth` from source markdown.
## v0.42.0+ onboard surface (NEW)
`gbrain onboard` is the activation surface gbrain did not have before.
Once your brain has any content, run `gbrain onboard --check --json` to
see structured recommendations across 5 brain-health axes (orphans,
stale embeddings, entity link coverage, timeline coverage, takes count).
**On first connect (after `gbrain init`):**
```bash
gbrain onboard --check --json
```
The JSON envelope (`schema_version: 1`) carries `recommendations[]` with
`apply_policy` per item: `auto_apply` (safe to run unattended),
`prompt_required` (needs explicit user consent), or `manual_only`
(LLM-bearing, user must run themselves).
**After every `gbrain upgrade`:**
```bash
gbrain onboard --check --json
```
New versions may surface new opportunities. The post-upgrade banner
nudges the user when it runs, but agents should re-probe as a hygiene
step regardless.
**Unattended remediation (cron / autopilot):**
```bash
gbrain onboard --auto --max-usd 5
```
Refuses without `--max-usd N`. Runs auto-eligible items only. The
autopilot daemon also consults onboard recommendations on its tick — no
explicit agent action needed for the autonomous path.
**Remote / federated brain installs (MCP):**
The `run_onboard` MCP op (admin scope) lets thin-client agents probe
brain health + drive remediation over OAuth-authenticated MCP. Protected
LLM-bearing handlers (synthesize, patterns, consolidate, takes-bootstrap,
contextual_reindex_per_chunk) require the additional `run_protected_onboard`
scope — admin alone is insufficient. The MCP op returns
`skipped_missing_scope[]` listing what would have run with the right
grants.
**Privacy + consent gates:**
- `gbrain takes extract --from-pages` sends concept/atom/lore/briefing/
writing/originals page content to your configured chat model (default
Anthropic Haiku). Refuses to run unless `takes.bootstrap_enabled=true`
is set in config AND `--yes` is passed. Two-gate opt-in by design.
- Autopilot's auto-apply tier for takes-bootstrap stays `manual_only`
until v0.42.1's eval gate (do not bypass).
**Suppress nudges in CI / scripted environments:**
```bash
export GBRAIN_NO_ONBOARD_NUDGE=1
```
Init + upgrade banners auto-skip in non-TTY too.
+46
View File
@@ -1,5 +1,51 @@
# TODOS
## v0.41.18.0 onboard wave follow-ups (v0.42.1+)
- **TODO-A (P2)**: Pack-aware `linkable: boolean` per-type field on schema-pack
manifests. Both `gbrain extract links --by-mention` and `--ner` would consult
it to gate which entity types participate in gazetteer construction. Currently
uses a hardcoded `['person', 'company', 'organization', 'entity']` list.
- **TODO-B (P3)**: LLM-based entity disambiguation for `--ner`. v0.42.0 ships
regex+gazetteer only; misses cases like "Anthropic's founders" → `Anthropic`
link. A small Haiku post-pass would catch these.
- **TODO-C (P3)**: `gbrain onboard --explain <recommendation_id>` drill-down.
Shows the underlying check, its measurement, and why the recommendation
fired. Useful when an operator wants to understand what `onboard --auto` is
about to do.
- **TODO-D (P2)**: Live-brain impact measurement against a representative brain
(165K-page production class). v0.42.0 ships the `migration_impact_log`
infrastructure; we need real-world numbers to update the design doc claims
with measured deltas.
- **TODO-E (P1)**: 100+-case eval suite for takes-bootstrap classifier. v0.42.0
ships the classifier + the 20-case eval scaffold per A24. Autopilot tier for
takes-bootstrap STAYS `manual_only` until this lands. Required before any
autopilot run of takes extraction.
- **TODO-F (P3)**: Web UI surface for `gbrain onboard` recommendations in the
admin SPA. Linear-style dashboard with one-click apply.
- **TODO-G (P2)**: Full DATABASE_URL-gated E2E for onboard. v0.42.0 ships
hermetic PGLite contracts coverage in `test/e2e/onboard-full-flow.test.ts`;
the real-Postgres version needs the Minion worker test harness to land its
per-handler stub seam so individual extraction handlers can be replaced for
testing.
- **TODO-H (P2)**: `minion_jobs.client_id` schema column. v0.42.0 stores the
originating OAuth client_id on `job.data.client_id` (JSONB passthrough).
A real schema column + index would let the spend query path (per-client
daily cap enforcement) avoid the JSONB projection cost.
- **TODO-I (P3)**: Thin-client (doctor-remote.ts) parity for the 4 new onboard
checks (embed_staleness, entity_link_coverage, timeline_coverage,
takes_count). Today the MCP run_onboard op runs these server-side via
runAllOnboardChecks; doctor-remote.ts would surface them on the thin-client
dashboard for operators who only hit the brain via MCP.
=======
## v0.41.17.0 `--workers N` cathedral follow-ups (v0.41.18+)
These were filed during the ship of `garrytan/dar-es-salaam-v1`
+1 -1
View File
@@ -1 +1 @@
0.41.17.0
0.41.18.0
+55
View File
@@ -2431,6 +2431,61 @@ columns. PGLite brains no-op. If wiki-style imports were truncated by the old
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
`compiled_truth` from source markdown.
## v0.42.0+ onboard surface (NEW)
`gbrain onboard` is the activation surface gbrain did not have before.
Once your brain has any content, run `gbrain onboard --check --json` to
see structured recommendations across 5 brain-health axes (orphans,
stale embeddings, entity link coverage, timeline coverage, takes count).
**On first connect (after `gbrain init`):**
```bash
gbrain onboard --check --json
```
The JSON envelope (`schema_version: 1`) carries `recommendations[]` with
`apply_policy` per item: `auto_apply` (safe to run unattended),
`prompt_required` (needs explicit user consent), or `manual_only`
(LLM-bearing, user must run themselves).
**After every `gbrain upgrade`:**
```bash
gbrain onboard --check --json
```
New versions may surface new opportunities. The post-upgrade banner
nudges the user when it runs, but agents should re-probe as a hygiene
step regardless.
**Unattended remediation (cron / autopilot):**
```bash
gbrain onboard --auto --max-usd 5
```
Refuses without `--max-usd N`. Runs auto-eligible items only. The
autopilot daemon also consults onboard recommendations on its tick — no
explicit agent action needed for the autonomous path.
**Remote / federated brain installs (MCP):**
The `run_onboard` MCP op (admin scope) lets thin-client agents probe
brain health + drive remediation over OAuth-authenticated MCP. Protected
LLM-bearing handlers (synthesize, patterns, consolidate, takes-bootstrap,
contextual_reindex_per_chunk) require the additional `run_protected_onboard`
scope — admin alone is insufficient. The MCP op returns
`skipped_missing_scope[]` listing what would have run with the right
grants.
**Privacy + consent gates:**
- `gbrain takes extract --from-pages` sends concept/atom/lore/briefing/
writing/originals page content to your configured chat model (default
Anthropic Haiku). Refuses to run unless `takes.bootstrap_enabled=true`
is set in config AND `--yes` is passed. Two-gate opt-in by design.
- Autopilot's auto-apply tier for takes-bootstrap stays `manual_only`
until v0.42.1's eval gate (do not bypass).
**Suppress nudges in CI / scripted environments:**
```bash
export GBRAIN_NO_ONBOARD_NUDGE=1
```
Init + upgrade banners auto-skip in non-TTY too.
---
## skills/RESOLVER.md
+3 -2
View File
@@ -1,6 +1,5 @@
{
"name": "gbrain",
"version": "0.41.17.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -78,6 +77,7 @@
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
"prepublish:clawhub": "bun run build:all",
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
@@ -137,5 +137,6 @@
"engines": {
"bun": ">=1.3.10"
},
"license": "MIT"
"license": "MIT",
"version": "0.41.18.0"
}
+70
View File
@@ -0,0 +1,70 @@
#!/bin/bash
# scripts/check-source-scope-onboard.sh
# v0.41.18.0 (A26, T17). Grep guard against SQL sites in src/core/onboard/
# and the 4 new onboard-derived doctor checks that touch source_id-bearing
# tables (pages, content_chunks, takes, links, timeline_entries) WITHOUT
# either:
# (a) including source_id / source_ids in the WHERE clause, OR
# (b) carrying the explicit opt-out marker `sourcescope:brain-wide` in
# an adjacent comment.
#
# Brain-wide metrics (embed_staleness, takes_count, total entity counts)
# are legitimate brain-wide queries — they MUST NOT auto-filter by source
# because the metric IS "across all sources". The opt-out marker is the
# explicit acknowledgement that this is intentional. Any new code touching
# per-source data WITHOUT the marker has to add source-scoping.
set -e
FILES_TO_CHECK=(
"src/core/onboard/checks.ts"
"src/core/onboard/impact-capture.ts"
"src/core/onboard/render.ts"
"src/commands/onboard.ts"
)
ERR=0
for f in "${FILES_TO_CHECK[@]}"; do
if [ ! -f "$f" ]; then
continue
fi
# Skip a file entirely when it doesn't contain SQL at all.
if ! grep -qE 'executeRaw|SELECT|INSERT|UPDATE|DELETE' "$f"; then
continue
fi
# File-level opt-out: if the file declares `sourcescope:file-brain-wide`
# in its header (first 30 lines), every SQL site inside is treated as
# intentionally brain-wide. Use sparingly — only for files whose SQL
# is structurally always-aggregate (onboard/checks.ts, impact-capture.ts).
if head -30 "$f" | grep -q 'sourcescope:file-brain-wide'; then
continue
fi
# Search for SQL-ish lines that DO NOT contain source_id and DO NOT have
# the brain-wide opt-out marker on the same line or within 3 lines above.
while IFS=: read -r line content; do
# Skip if source_id mentioned on this or nearby lines (5-line window).
start=$((line - 4))
[ "$start" -lt 1 ] && start=1
if sed -n "${start},${line}p" "$f" | grep -qE 'source_id|sourceIds|sourcescope:brain-wide'; then
continue
fi
echo "[check-source-scope-onboard] $f:$line — SQL site lacks source_id WHERE clause OR brain-wide opt-out marker"
echo " $content"
ERR=1
done < <(grep -nE 'FROM pages|FROM content_chunks|FROM takes\b|FROM links|FROM timeline_entries|UPDATE pages|UPDATE content_chunks|DELETE FROM pages|DELETE FROM content_chunks' "$f" || true)
done
if [ "$ERR" -eq 1 ]; then
echo ""
echo "[check-source-scope-onboard] One or more SQL sites in onboard surfaces lack source_id scoping."
echo "Either: (a) add source_id = \$N (or source_id = ANY(\$N::text[])) to the WHERE,"
echo " or (b) add a comment marker 'sourcescope:brain-wide' within 4 lines above"
echo " the SQL to declare intent."
exit 1
fi
exit 0
+1
View File
@@ -58,6 +58,7 @@ CHECKS=(
"check:fixture-privacy"
"check:conversation-parser"
"check:resolver"
"check:source-scope-onboard"
"typecheck"
)
+8 -1
View File
@@ -35,7 +35,7 @@ for (const op of operations) {
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'conversation-parser']);
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -1499,6 +1499,13 @@ async function handleCliOnly(command: string, args: string[]) {
await runTakes(engine, args);
break;
}
case 'onboard': {
// v0.41.18.0 (T13) — gbrain onboard. Thin shell over T2 library
// + T4 onboard checks + T12 render layer.
const { runOnboard } = await import('./commands/onboard.ts');
await runOnboard(engine, args);
break;
}
case 'founder': {
// v0.35.4 (T7) — founder scorecard. `gbrain founder scorecard <slug>`
// rolls up Phase 2's typed-claim substrate into the four scorecard
+17 -1
View File
@@ -511,7 +511,23 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
}),
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || await engine.getConfig('anthropic_api_key')),
};
const plan = computeRecommendations(health, ctx).filter((r) => r.status === 'remediable');
// v0.41.18.0 (A5 + A19 + A22, T15): consult onboard recommendations
// ALONGSIDE doctor's brain-score recommendations. Onboard's 4 new
// checks (embed_staleness, link_coverage, timeline_coverage,
// takes_count) supply extraRemediations into computeRecommendations.
// Per A19 fail-open: any throw in the onboard path falls through
// to legacy doctor-only plan (no crash).
let extraRemediations: ReturnType<typeof computeRecommendations> = [];
try {
const { runAllOnboardChecks } = await import('../core/onboard/checks.ts');
const onboardResults = await runAllOnboardChecks(engine);
extraRemediations = onboardResults.flatMap((r) => r.remediations);
} catch (err) {
process.stderr.write(
`[autopilot] onboard checks failed (fail-open per A19): ${err instanceof Error ? err.message : String(err)}\n`,
);
}
const plan = computeRecommendations(health, ctx, extraRemediations).filter((r) => r.status === 'remediable');
const estTotal = plan.reduce((s, r) => s + r.est_seconds, 0);
// Track time since last full cycle for the 60-min floor.
+115 -345
View File
@@ -3988,7 +3988,7 @@ export async function buildChecks(
checks.push({ name: 'graph_coverage', status: 'warn', message: 'Could not check graph coverage' });
}
// 9b. v0.42.0.0 — orphan_ratio check (migration #1 of #1409).
// 9b. v0.41.18.0 — orphan_ratio check (migration #1 of #1409).
//
// Surfaces the fraction of linkable pages with no inbound links.
// Consumes the same canonical getOrphansData() pure fn as
@@ -5324,6 +5324,15 @@ export async function buildChecks(
// v0.38 — cycle_phase_scope (informational; no DB cost)
progress.heartbeat('cycle_phase_scope');
checks.push(checkCyclePhaseScope());
// v0.41.18.0 (A16, T4): 4 onboard checks — each emits a Check + its
// own RemediationStep[] aggregated by onboard's plan path. The
// checks themselves are cheap counts (backed by content_chunks_stale_idx
// for embed_staleness, TABLESAMPLE on PG >50K for the coverage pair).
progress.heartbeat('onboard_checks');
const { runAllOnboardChecks } = await import('../core/onboard/checks.ts');
const onboardResults = await runAllOnboardChecks(engine);
for (const r of onboardResults) checks.push(r.check);
}
progress.finish();
@@ -5704,59 +5713,25 @@ async function runLocksCheck(engine: BrainEngine | null, jsonOutput: boolean): P
// ============================================================
/**
* Emit ordered Remediation list to drive brain to --target-score.
* CLI wrapper around computeRemediationPlan (src/core/remediation/plan.ts).
*
* Read-only never enqueues, never mutates. The agent contract:
* inspect the plan with --remediation-plan --json before committing
* to --remediate. The JSON shape is stable; consumers that parse it
* can rely on it across releases.
* v0.41.18.0 (A1, codex finding #2): library extracted so onboard +
* MCP run_onboard can compose against a stable shape. This wrapper
* stays as the CLI surface only argv parsing + human render. JSON
* mode emits the library's stable envelope verbatim.
*
* Read-only never enqueues, never mutates.
*/
export async function runRemediationPlan(
engine: BrainEngine,
args: string[],
): Promise<void> {
const { computeRecommendations, classifyChecks, maxReachableScore } =
await import('../core/brain-score-recommendations.ts');
const { computeRemediationPlan } = await import('../core/remediation/index.ts');
const targetScore = parseIntFlag(args, '--target-score') ?? 90;
const jsonOutput = args.includes('--json');
// Cheap path (D7) — don't run slow doctor checks for the plan surface.
// The recommendation generator works from BrainHealth + context alone.
const health = await engine.getHealth();
const ctx = await loadRecommendationContext(engine);
const recs = computeRecommendations(health, ctx);
// Synthetic check list for classification — we don't need full doctor
// output, just the check names the recommendations care about.
const syntheticChecks = [
{ name: 'brain_score', status: 'ok' as const },
{ name: 'sync_freshness', status: 'ok' as const },
{ name: 'missing_embeddings', status: 'ok' as const },
{ name: 'dead_links', status: 'ok' as const },
{ name: 'orphan_pages', status: 'ok' as const },
];
const classifications = classifyChecks(syntheticChecks, ctx);
const ceiling = maxReachableScore(health, classifications);
const filteredRecs = recs.filter((r) => r.status === 'remediable');
const estTotalSeconds = filteredRecs.reduce((sum, r) => sum + r.est_seconds, 0);
const estTotalUsd = filteredRecs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
const blocked = classifications
.filter((c) => c.status === 'blocked')
.map((c) => ({ check: c.check, reason: c.reason ?? 'prerequisite missing' }));
const plan = {
schema_version: 2,
brain_score_current: health.brain_score,
brain_score_target: targetScore,
max_reachable_score: ceiling,
target_unreachable: targetScore > ceiling,
plan: filteredRecs.map((r, i) => ({ step: i + 1, ...r })),
est_total_seconds: estTotalSeconds,
est_total_usd_cost: Number(estTotalUsd.toFixed(2)),
blocked,
};
const plan = await computeRemediationPlan(engine, { targetScore });
if (jsonOutput) {
console.log(JSON.stringify(plan, null, 2));
@@ -5764,9 +5739,9 @@ export async function runRemediationPlan(
}
// Human output
console.log(`Brain score: ${health.brain_score}/100 → target ${targetScore}`);
console.log(`Brain score: ${plan.brain_score_current}/100 → target ${targetScore}`);
if (plan.target_unreachable) {
console.log(`Target unreachable: max with autonomous remediation is ${ceiling}/100.`);
console.log(`Target unreachable: max with autonomous remediation is ${plan.max_reachable_score}/100.`);
}
if (plan.plan.length === 0) {
console.log('No remediations needed. Brain is at target.');
@@ -5778,21 +5753,25 @@ export async function runRemediationPlan(
console.log(` ${step.step}. [${step.severity}] ${step.job}${protectedMark}${step.rationale}${costMark}`);
}
}
if (blocked.length > 0) {
if (plan.blocked.length > 0) {
console.log(`\nBlocked checks (prereq missing):`);
for (const b of blocked) {
for (const b of plan.blocked) {
console.log(` - ${b.check}: ${b.reason}`);
}
}
}
/**
* Submit ordered Remediation jobs sequentially per D3, with D5 cascade
* on failure and D7 scoped recheck between steps.
* CLI wrapper around runRemediation (src/core/remediation/run.ts).
*
* v0.41.18.0 (A1, codex finding #2): orchestrator extracted into the
* remediation library. This wrapper stays as the CLI surface only
* argv parsing + interactive TTY confirmation + human/JSON render via
* RemediationHooks.
*
* Default behavior: submit-and-wait per step. --dry-run skips submission.
* --max-usd N refuses if est_total_usd_cost > N. --max-jobs N caps the
* inner loop.
* inner loop. --resume [plan_hash] loads checkpoint and continues.
*
* PGLite path: synchronous in-process execution (no durable queue).
*/
@@ -5806,7 +5785,8 @@ export async function runRemediate(
// documented as the cron-safety guard. Either threads through to the
// pre-flight estimate refusal AND, via withBudgetTracker, the mid-run
// BudgetExhausted hard-throw.
const maxUsd = parseFloatFlag(args, '--max-usd') ?? parseFloatFlag(args, '--max-cost');
const maxUsdRaw = parseFloatFlag(args, '--max-usd') ?? parseFloatFlag(args, '--max-cost');
const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw;
const dryRun = args.includes('--dry-run');
const skipConfirm = args.includes('--yes');
const jsonOutput = args.includes('--json');
@@ -5818,328 +5798,118 @@ export async function runRemediate(
const resumeArg = resumeMode ? args[resumeFlagIdx + 1] : undefined;
const resumePlanHash = resumeArg && !resumeArg.startsWith('--') ? resumeArg : undefined;
const { computeRecommendations, classifyChecks, maxReachableScore } =
await import('../core/brain-score-recommendations.ts');
const {
BudgetTracker,
BudgetExhausted,
} = await import('../core/budget/budget-tracker.ts');
const { withBudgetTracker } = await import('../core/ai/gateway.ts');
const {
computePlanHash,
saveRemediationCheckpoint,
loadRemediationCheckpoint,
listRemediationCheckpoints,
clearRemediationCheckpoint,
} = await import('../core/remediation-checkpoint.ts');
const { runRemediation, computeRemediationPlan } =
await import('../core/remediation/index.ts');
const ctx = await loadRecommendationContext(engine);
// Pre-flight ceiling check (D13)
const initialHealth = await engine.getHealth();
const syntheticChecks = [
{ name: 'brain_score', status: 'ok' as const },
{ name: 'sync_freshness', status: 'ok' as const },
{ name: 'missing_embeddings', status: 'ok' as const },
{ name: 'dead_links', status: 'ok' as const },
{ name: 'orphan_pages', status: 'ok' as const },
];
const classifications = classifyChecks(syntheticChecks, ctx);
const ceiling = maxReachableScore(initialHealth, classifications);
if (targetScore > ceiling) {
console.error(
`[remediate] target ${targetScore} unreachable; max autonomous = ${ceiling}/100. ` +
`Configure missing prereqs (see --remediation-plan blocked output) or lower --target-score.`,
);
process.exit(2);
}
// Initial plan
let recs = computeRecommendations(initialHealth, ctx).filter((r) => r.status === 'remediable');
if (recs.length === 0) {
console.log(`Brain at score ${initialHealth.brain_score}/100, target ${targetScore}. Nothing to do.`);
return;
}
// A4 amended: compute plan_hash off the active recommendation ids so the
// checkpoint binds to THIS plan. Resume only fires for matching plans.
const planHash = computePlanHash(recs.map((r) => r.id));
let completedFromCheckpoint = new Set<string>();
if (resumeMode) {
const requested = resumePlanHash;
let cp = requested ? loadRemediationCheckpoint(requested) : null;
if (!cp && !requested) {
// No explicit hash: try newest checkpoint that matches the active plan.
const recent = listRemediationCheckpoints();
for (const e of recent) {
const candidate = loadRemediationCheckpoint(e.plan_hash);
if (candidate && candidate.plan_hash === planHash) {
cp = candidate;
break;
}
}
}
if (!cp) {
// TTY confirmation gate (stays in CLI; library doesn't render).
// Compute the plan once for the confirmation prompt, then hand off
// to the library for the actual run. The library re-computes its
// own plan internally — we accept the second computation cost for
// a cleaner CLI/library separation.
if (!skipConfirm && !dryRun && process.stdout.isTTY && !resumeMode) {
const plan = await computeRemediationPlan(engine, { targetScore });
if (plan.target_unreachable) {
console.error(
`[remediate --resume] no matching checkpoint found ` +
`(plan_hash=${planHash}${requested ? `; requested=${requested}` : ''}). ` +
`Run without --resume to start fresh.`,
`[remediate] target ${targetScore} unreachable; max autonomous = ${plan.max_reachable_score}/100. ` +
`Configure missing prereqs (see --remediation-plan blocked output) or lower --target-score.`,
);
process.exit(2);
}
if (cp.plan_hash !== planHash) {
if (plan.plan.length === 0) {
console.log(`Brain at score ${plan.brain_score_current}/100, target ${targetScore}. Nothing to do.`);
return;
}
if (maxUsd !== undefined && plan.est_total_usd_cost > maxUsd) {
console.error(
`[remediate --resume] checkpoint plan_hash=${cp.plan_hash} does not match active plan_hash=${planHash}. ` +
`The plan has changed (brain state moved). Run without --resume to start fresh.`,
`[remediate] est cost $${plan.est_total_usd_cost.toFixed(2)} exceeds --max-usd $${maxUsd.toFixed(2)}. Aborting.`,
);
process.exit(2);
}
completedFromCheckpoint = new Set(cp.completed.map((c) => c.id));
console.error(
`[remediate --resume] resuming plan_hash=${planHash}: ${completedFromCheckpoint.size} step(s) completed, ` +
`${recs.length - completedFromCheckpoint.size} remaining.`,
);
}
const estTotalUsd = recs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
if (maxUsd !== null && estTotalUsd > maxUsd) {
console.error(
`[remediate] est cost $${estTotalUsd.toFixed(2)} exceeds --max-usd $${maxUsd.toFixed(2)}. Aborting.`,
);
process.exit(2);
}
if (!skipConfirm && process.stdout.isTTY) {
console.log(`About to submit ${recs.length} job(s), est ${Math.round(recs.reduce((s, r) => s + r.est_seconds, 0))}s, est $${estTotalUsd.toFixed(2)}`);
console.log(`About to submit ${plan.plan.length} job(s), est ${plan.est_total_seconds}s, est $${plan.est_total_usd_cost.toFixed(2)}`);
console.log('Pass --yes to proceed (cron-friendly).');
process.exit(1);
}
if (dryRun) {
console.log(`[remediate --dry-run] Would submit ${recs.length} jobs:`);
for (const r of recs) console.log(` - ${r.id} (${r.job})`);
return;
}
// Sequential submit per D3, with D5 cascade on failure and D7
// scoped recheck between steps.
const submitted: Array<{ step: number; id: string; job_id: number | null; status: string }> = [];
const abortedIds = new Set<string>();
const doctorRunId = crypto.randomUUID();
const isPGLite = engine.kind === 'pglite';
if (isPGLite) {
if (engine.kind === 'pglite') {
console.error('[remediate] PGLite engine: running inline (no durable queue).');
}
const { MinionQueue } = await import('../core/minions/queue.ts');
const { waitForCompletion } = await import('../core/minions/wait-for-completion.ts');
const queue = new MinionQueue(engine);
// A4 amended: install a BudgetTracker scope around the plan-step loop so
// any gateway.chat / embed / rerank inside a Minion handler (synthesize,
// patterns, consolidate) auto-enforces the cap. On BudgetExhausted, the
// onExhausted callback persists the checkpoint BEFORE the throw propagates;
// the catch surfaces the actionable --resume hint.
const remediateTracker = new BudgetTracker({
label: 'doctor.remediate',
maxCostUsd: maxUsd ?? undefined,
});
let exhaustionSnapshot: { spent: number; cap: number; reason: string; model_id?: string } | undefined;
remediateTracker.onExhausted(() => {
// BudgetTracker fires this synchronously from inside reserve()/record()
// before the throw bubbles. Persist whatever has been done so far.
const cp = {
schema_version: 1 as const,
plan_hash: planHash,
doctor_run_id: doctorRunId,
target_score: targetScore,
started_at: new Date().toISOString(),
completed: submitted
.filter((s) => s.status === 'completed')
.map((s) => ({ id: s.id, job: '', status: s.status, job_id: s.job_id ?? null })),
aborted_at: new Date().toISOString(),
abort_reason: 'budget_exhausted' as const,
budget_snapshot: exhaustionSnapshot,
};
saveRemediationCheckpoint(cp);
});
const runLoop = async (): Promise<void> => {
let stepCount = 0;
while (recs.length > 0 && stepCount < maxJobs) {
const step = recs[0];
if (!step) break;
stepCount++;
// Resume: skip steps that the checkpoint already marked completed.
if (completedFromCheckpoint.has(step.id)) {
submitted.push({ step: stepCount, id: step.id, job_id: null, status: 'completed' });
recs.shift();
continue;
}
// D5: if depends_on intersects aborted, skip + cascade
if (step.depends_on && step.depends_on.some((d) => abortedIds.has(d))) {
submitted.push({ step: stepCount, id: step.id, job_id: null, status: 'skipped_dep_aborted' });
abortedIds.add(step.id);
recs.shift();
continue;
}
try {
const isProtected = !!step.protected;
const job = await queue.add(
step.job,
{ ...step.params, doctor_run_id: doctorRunId },
{
queue: 'default',
idempotency_key: step.idempotency_key,
max_attempts: 2,
maxWaiting: 1,
},
isProtected ? { allowProtectedSubmit: true } : undefined,
const result = await runRemediation(
engine,
{
targetScore,
maxJobs,
maxUsd,
dryRun,
resume: resumeMode,
resumePlanHash,
},
{
onTargetUnreachable: (target, ceiling) => {
console.error(
`[remediate] target ${target} unreachable; max autonomous = ${ceiling}/100. ` +
`Configure missing prereqs (see --remediation-plan blocked output) or lower --target-score.`,
);
submitted.push({ step: stepCount, id: step.id, job_id: job.id, status: 'submitted' });
// Wait for terminal state. PGLite is in-process — short poll.
const terminal = await waitForCompletion(queue, job.id, {
pollMs: isPGLite ? 250 : 1000,
timeoutMs: (step.est_seconds + 60) * 1000,
});
const lastSub = submitted[submitted.length - 1];
if (lastSub) lastSub.status = terminal.status;
if (terminal.status !== 'completed') {
abortedIds.add(step.id);
}
} catch (e) {
if (e instanceof BudgetExhausted) {
exhaustionSnapshot = {
spent: e.spent,
cap: e.cap,
reason: e.reason,
model_id: e.modelId,
};
throw e;
}
submitted.push({
step: stepCount, id: step.id, job_id: null,
status: `error: ${(e as Error).message.slice(0, 100)}`,
});
abortedIds.add(step.id);
}
recs.shift();
// D7: scoped recheck — re-compute plan from fresh health snapshot.
// The next plan may drop completed steps and re-introduce failed
// steps with bumped retry suffix (D1).
if (recs.length === 0 || stepCount >= maxJobs) break;
const freshHealth = await engine.getHealth();
recs = computeRecommendations(freshHealth, ctx).filter((r) => r.status === 'remediable');
}
};
let budgetExhaustedAt: InstanceType<typeof BudgetExhausted> | null = null;
try {
await withBudgetTracker(remediateTracker, runLoop);
} catch (err) {
if (err instanceof BudgetExhausted) {
budgetExhaustedAt = err;
console.error(
`\n[remediate] BudgetExhausted (${err.reason}): spent $${err.spent.toFixed(4)} > cap $${err.cap.toFixed(2)}.\n` +
},
onNothingToDo: (score, target) => {
console.log(`Brain at score ${score}/100, target ${target}. Nothing to do.`);
},
onBudgetRefused: (estCost, cap) => {
console.error(
`[remediate] est cost $${estCost.toFixed(2)} exceeds --max-usd $${cap.toFixed(2)}. Aborting.`,
);
},
onResumeMissed: (planHash, requested) => {
console.error(
`[remediate --resume] no matching checkpoint found ` +
`(plan_hash=${planHash}${requested ? `; requested=${requested}` : ''}). ` +
`Run without --resume to start fresh.`,
);
},
onResumeLoaded: (planHash, completed, remaining) => {
console.error(
`[remediate --resume] resuming plan_hash=${planHash}: ${completed} step(s) completed, ${remaining} remaining.`,
);
},
onBudgetExhausted: (planHash, snapshot) => {
console.error(
`\n[remediate] BudgetExhausted (${snapshot.reason}): spent $${snapshot.spent.toFixed(4)} > cap $${snapshot.cap.toFixed(2)}.\n` +
`Checkpoint saved. Resume with:\n` +
` gbrain doctor --remediate --resume ${planHash}\n`,
);
} else {
throw err;
}
}
);
},
},
);
// Clear checkpoint on a clean run (no budget abort). Failed steps in the
// submitted set don't disqualify the cleanup — they re-surface on the
// next plan with bumped suffixes.
if (!budgetExhaustedAt) {
clearRemediationCheckpoint(planHash);
}
// CLI surfaces — target unreachable / resume missed already emitted via hooks.
// Library returns synthetic result with target_unreachable populated; exit 2.
if (result.target_unreachable) process.exit(2);
const finalHealth = await engine.getHealth();
const result = {
doctor_run_id: doctorRunId,
brain_score_initial: initialHealth.brain_score,
brain_score_final: finalHealth.brain_score,
brain_score_target: targetScore,
target_reached: finalHealth.brain_score >= targetScore,
submitted,
aborted_count: abortedIds.size,
};
if (dryRun && result.submitted.length > 0) {
console.log(`[remediate --dry-run] Would submit ${result.submitted.length} jobs:`);
for (const s of result.submitted) console.log(` - ${s.id}`);
return;
}
if (jsonOutput) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`\nBrain score: ${initialHealth.brain_score}${finalHealth.brain_score} (target ${targetScore})`);
console.log(`Submitted: ${submitted.length} job(s), ${abortedIds.size} aborted/failed`);
} else if (result.submitted.length > 0) {
console.log(`\nBrain score: ${result.brain_score_initial}${result.brain_score_final} (target ${targetScore})`);
console.log(`Submitted: ${result.submitted.length} job(s), ${result.aborted_count} aborted/failed`);
}
const anyFailed = submitted.some((s) => s.status !== 'completed' && s.status !== 'submitted');
if (budgetExhaustedAt || anyFailed) process.exit(1);
}
/**
* Build RecommendationContext from engine + config.
* Pure read; no side effects.
*/
async function loadRecommendationContext(engine: BrainEngine) {
// v0.37 fix wave (Lane E.4 + CDX2-11): read schema-sizing fields from
// gateway, not DB. The DB plane is schema-applied metadata; the file
// plane is the gateway runtime source. Pre-fix this context produced
// stale recommendations on fresh installs whose DB rows hadn't been
// populated.
//
// Also extended the API-key check to recognize the ZE key alongside
// OpenAI (was OpenAI-only). After Lane C.3, zeroentropy_api_key lives
// in GBrainConfig + propagates to the gateway env dict.
const repoPath = await engine.getConfig('sync.repo_path');
let embeddingModel: string | undefined;
let embeddingDimensions: number | undefined;
try {
const gw = await import('../core/ai/gateway.ts');
embeddingModel = gw.getEmbeddingModel();
embeddingDimensions = gw.getEmbeddingDimensions();
} catch {
// Gateway unconfigured — fall back to DB plane as a best-effort hint
// (preserves doctor running before any engine.connect()).
const dbModel = await engine.getConfig('embedding_model');
const dbDims = await engine.getConfig('embedding_dimensions');
embeddingModel = dbModel ?? undefined;
embeddingDimensions = dbDims ? Number(dbDims) : undefined;
}
// v0.40.x: recipe-aware provider check, shared with autopilot.ts via
// embeddingProviderConfigured(). Local providers (ollama, llama-server —
// empty auth_env.required) need no hosted key; hosted providers check
// their OWN required key (so a Voyage brain is judged by VOYAGE_API_KEY,
// not by whether an OpenAI/ZE key happens to exist — the pre-fix wart).
// fileCfg loads synchronously, so the resolveKey closure is sync.
const { loadConfigFileOnly } = await import('../core/config.ts');
const fileCfg = loadConfigFileOnly();
const { embeddingProviderConfigured, HOSTED_EMBED_KEY_CONFIG } = await import(
'../core/brain-score-recommendations.ts'
const anyFailed = result.submitted.some(
(s) => s.status !== 'completed' && s.status !== 'submitted' && s.status !== 'dry_run',
);
const embeddingConfigured = embeddingProviderConfigured(embeddingModel, (envVar) => {
const cfgField = HOSTED_EMBED_KEY_CONFIG[envVar];
const fromCfg = cfgField ? (fileCfg as Record<string, unknown> | null)?.[cfgField] : undefined;
return !!(process.env[envVar] || fromCfg);
});
return {
repoPath: repoPath ?? undefined,
embeddingModel,
embeddingDimensions,
embeddingProviderConfigured: embeddingConfigured,
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || fileCfg?.anthropic_api_key),
};
if (result.budget_exhausted || anyFailed) process.exit(1);
}
// v0.41.18.0 (A1, codex finding #2): loadRecommendationContext moved to
// src/core/remediation/context.ts so onboard + MCP run_onboard compose
// the same context. The CLI surfaces (runRemediationPlan / runRemediate
// above) now call computeRemediationPlan + runRemediation from the
// library, which builds the context internally.
function parseIntFlag(args: string[], flag: string): number | null {
const i = args.indexOf(flag);
if (i === -1 || i === args.length - 1) return null;
+82 -13
View File
@@ -38,6 +38,29 @@ export interface EmbedOpts {
* in the DB where `gbrain jobs get` can read it.
*/
onProgress?: (done: number, total: number, embedded: number) => void;
/**
* v0.41.18.0 (A13): override the hardcoded PAGE_SIZE=2000 page-batch.
* Smaller batches give finer progress granularity; larger batches
* reduce per-batch coordination cost. Caps internally to 10K to
* keep memory bounded.
*/
batchSize?: number;
/**
* v0.41.18.0 (A13): when 'recent', walks the stale-chunk pool in
* page.updated_at DESC order (recent-modified pages first) instead
* of the legacy stable (page_id, chunk_index) order. Threads through
* to listStaleChunks orderBy='updated_desc'. Backed by the
* content_chunks_stale_idx partial + idx_pages_updated_at_desc indexes
* (v100).
*/
priority?: 'recent';
/**
* v0.41.18.0 (A13): catch-up mode removes the wall-clock cap and loops
* until countStaleChunks() returns 0. Used by `gbrain embed --stale
* --catch-up` and by the embed-catch-up Minion handler that the onboard
* remediation submits on big stale backlogs.
*/
catchUp?: boolean;
}
/**
@@ -173,7 +196,11 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
return result;
}
if (opts.all || opts.stale) {
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId);
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
batchSize: opts.batchSize,
priority: opts.priority,
catchUp: opts.catchUp,
});
return result;
}
if (opts.slug) {
@@ -216,19 +243,27 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
// v0.31.12: --source <id> scopes to a single source.
const sourceIdx = args.indexOf('--source');
const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined;
// v0.41.18.0 (A13): --batch-size N, --priority recent, --catch-up flags.
const batchSizeIdx = args.indexOf('--batch-size');
const batchSizeRaw = batchSizeIdx >= 0 ? args[batchSizeIdx + 1] : undefined;
const batchSize = batchSizeRaw ? Math.max(1, Math.min(10_000, parseInt(batchSizeRaw, 10) || 0)) : undefined;
const priorityIdx = args.indexOf('--priority');
const priorityRaw = priorityIdx >= 0 ? args[priorityIdx + 1] : undefined;
const priority = priorityRaw === 'recent' ? 'recent' as const : undefined;
const catchUp = args.includes('--catch-up');
let opts: EmbedOpts;
if (slugsIdx >= 0) {
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId };
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp };
} else if (all || stale) {
opts = { all, stale, dryRun, sourceId };
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp };
} else {
const slug = args.find(a => !a.startsWith('--'));
if (!slug) {
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run]');
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up]');
process.exit(1);
}
opts = { slug, dryRun, sourceId };
opts = { slug, dryRun, sourceId, batchSize, priority, catchUp };
}
// CLI path: wire a reporter so --progress-json / --quiet / TTY rendering
@@ -355,6 +390,11 @@ async function embedAll(
result: EmbedResult,
onProgress?: (done: number, total: number, embedded: number) => void,
sourceId?: string,
staleOpts?: {
batchSize?: number;
priority?: 'recent';
catchUp?: boolean;
},
) {
// ─────────────────────────────────────────────────────────────
// Stale-only fast path: avoid the listPages + per-page getChunks
@@ -371,7 +411,8 @@ async function embedAll(
// ─────────────────────────────────────────────────────────────
if (staleOnly) {
// D7: thread sourceId so `gbrain embed --stale --source X` actually scopes.
return await embedAllStale(engine, sourceId, dryRun, result, onProgress);
// v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path.
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts);
}
// v0.31.12: when sourceId is set, scope listPages to that source.
@@ -497,6 +538,11 @@ async function embedAllStale(
dryRun: boolean,
result: EmbedResult,
onProgress?: (done: number, total: number, embedded: number) => void,
staleOpts?: {
batchSize?: number;
priority?: 'recent';
catchUp?: boolean;
},
) {
// D7: thread sourceId so source-scoped runs only count + visit
// that source's NULL embeddings.
@@ -525,23 +571,34 @@ async function embedAllStale(
// rows in one query (which times out on Supabase's 2-min pooler timeout),
// we page through 2000 rows at a time via keyset pagination on
// (page_id, chunk_index). Each query finishes in <1s.
const PAGE_SIZE = 2000;
// v0.41.18.0 (A13): --batch-size N CLI flag overrides hardcoded 2000 default.
const PAGE_SIZE = staleOpts?.batchSize ?? 2000;
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
// D3 + D3a + D8: wall-clock budget. 30 min default; env override.
// The old single-shot `LIMIT 100000` query implicitly capped runtime
// by failing on timeout; pagination removed that cap. AbortController
// threads cancellation into (a) the retry sleep below, (b) the worker
// claim loop, and (c) the gateway embed call so an in-flight HTTP
// request also unwinds.
const BUDGET_MS = parseInt(process.env.GBRAIN_EMBED_TIME_BUDGET_MS || `${30 * 60 * 1000}`, 10);
// v0.41.18.0 (A13): --catch-up removes the wall-clock cap entirely so the
// handler runs until countStaleChunks() returns 0. Use Number.MAX_SAFE_INTEGER
// (effectively unbounded) instead of the 30-min default. The AbortController
// still wraps for SIGINT propagation; just the timer never fires.
const BUDGET_MS = staleOpts?.catchUp
? Number.MAX_SAFE_INTEGER
: parseInt(process.env.GBRAIN_EMBED_TIME_BUDGET_MS || `${30 * 60 * 1000}`, 10);
const budgetController = new AbortController();
const budgetTimer = setTimeout(() => budgetController.abort(), BUDGET_MS);
const budgetSignal = budgetController.signal;
// v0.41.18.0 (A13): --priority recent threads orderBy='updated_desc' to
// listStaleChunks. Composite cursor tracks (updated_at, page_id, chunk_index)
// instead of just (page_id, chunk_index); first-page cursor is sentinel
// (null, 0, -1).
const orderBy: 'page_id' | 'updated_desc' = staleOpts?.priority === 'recent'
? 'updated_desc'
: 'page_id';
let totalProcessedPages = 0;
let afterPageId = 0;
let afterChunkIndex = -1;
let afterUpdatedAt: string | null = null;
let totalChunksLoaded = 0;
let budgetExitNotified = false;
@@ -560,6 +617,10 @@ async function embedAllStale(
batchSize: PAGE_SIZE,
afterPageId,
afterChunkIndex,
...(orderBy === 'updated_desc' && {
orderBy,
afterUpdatedAt,
}),
...(sourceId && { sourceId }),
});
if (batch.length === 0) break;
@@ -569,6 +630,14 @@ async function embedAllStale(
const last = batch[batch.length - 1];
afterPageId = last.page_id;
afterChunkIndex = last.chunk_index;
if (orderBy === 'updated_desc') {
// engine returns `updated_at` as Date or ISO string; normalize to ISO.
const lastRow = last as unknown as { updated_at?: string | Date | null };
const u = lastRow.updated_at;
afterUpdatedAt = u instanceof Date ? u.toISOString()
: typeof u === 'string' ? u
: null;
}
// Group by composite key (source_id::slug).
const byKey = new Map<string, typeof batch>();
+84 -10
View File
@@ -480,14 +480,21 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
// v0_13_0 migration orchestrator runs this once under the hood; users
// opt in for subsequent runs.
const includeFrontmatter = args.includes('--include-frontmatter');
// v0.42.0.0 Part B: --by-mention auto-link body-text entity mentions
// v0.41.18.0 Part B: --by-mention auto-link body-text entity mentions
// via the gazetteer pass. Mode dispatch — when set, run ONLY the
// mention pass (skip default link extract). DB-source only per D7;
// FS-source is rejected with a paste-ready fix-hint below.
const byMention = args.includes('--by-mention');
// v0.41.15.0 (T7, D9): --workers N parsed via the shared validator.
// v0.41.18.0 (A10, T7): --ner is a NER-extraction mode dispatch. Same
// DB-source-only posture as --by-mention. Can combine with --by-mention
// in a single command for a shared-gazetteer walk (saves one pass).
const ner = args.includes('--ner');
// v0.41.18.0 (A11, T8): --from-meetings extracts timeline entries from
// meeting pages onto each discussed entity. Timeline subcommand only.
const fromMeetings = args.includes('--from-meetings');
// v0.41.17.0 (T7, D9): --workers N parsed via the shared validator.
// Honored on the fs-walk inner loops only; DB-source paths stay
// serial in v0.41.15.0 (see ExtractOpts.workers doc).
// serial in v0.41.17.0 (see ExtractOpts.workers doc).
let workers: number | undefined;
const workersIdx = args.indexOf('--workers');
const concurrencyIdx = args.indexOf('--concurrency');
@@ -523,7 +530,7 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
process.exit(1);
}
// v0.42.0.0 D7: --by-mention requires DB-source. Gazetteer construction
// v0.41.18.0 D7: --by-mention requires DB-source. Gazetteer construction
// needs the engine; mixing FS-walk with DB-gazetteer is incoherent
// (you'd scan files on disk for mentions of entities that may not exist
// in any synced page). Fail loud with a paste-ready fix-hint.
@@ -545,6 +552,40 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
);
process.exit(2);
}
// v0.41.18.0 (T7): same gates for --ner.
if (ner && source === 'fs') {
console.error(
`--ner requires --source db (currently --source fs). NER extraction needs the engine ` +
`to build the entity gazetteer + read schema-pack link_types. Re-run as:\n\n` +
` gbrain extract ${subcommand} --ner --source db` +
(sourceIdFilter ? ` --source-id ${sourceIdFilter}` : '') +
(since ? ` --since ${since}` : '') +
(dryRun ? ' --dry-run' : '') + '\n',
);
process.exit(2);
}
if (ner && subcommand === 'timeline') {
console.error(
`--ner is a links-pass only; it does not apply to timeline extraction.`,
);
process.exit(2);
}
// v0.41.18.0 (T8): --from-meetings is timeline-only + DB-source-only.
if (fromMeetings && source === 'fs') {
console.error(
`--from-meetings requires --source db (currently --source fs). Re-run as:\n\n` +
` gbrain extract timeline --from-meetings --source db` +
(sourceIdFilter ? ` --source-id ${sourceIdFilter}` : '') +
(dryRun ? ' --dry-run' : '') + '\n',
);
process.exit(2);
}
if (fromMeetings && subcommand !== 'timeline' && subcommand !== 'all') {
console.error(
`--from-meetings is a timeline-pass only. Re-run as 'gbrain extract timeline --from-meetings' or 'gbrain extract all --from-meetings'.`,
);
process.exit(2);
}
// FS source needs a brain dir. When --dir wasn't passed, resolve from
// sources(local_path) — same path `gbrain sync` uses — instead of
@@ -577,16 +618,49 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
// is fs-only; we keep the dual codepath here so Minions handlers
// can opt in via mode + source.
result = { links_created: 0, timeline_entries_created: 0, pages_processed: 0 };
// v0.42.0.0: --by-mention is a mode dispatch. When set, run ONLY
// v0.41.18.0: --by-mention is a mode dispatch. When set, run ONLY
// the mention pass and skip the default link/frontmatter extract.
// The two passes write different link_source values ('mentions' vs
// 'markdown'/'frontmatter') so they don't conflict, but mixing them
// in a single CLI invocation is surprising — keep the surfaces
// separate.
if (byMention) {
const r = await extractMentionsFromDb(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
result.links_created = r.created;
result.pages_processed = r.pages;
if (fromMeetings) {
// v0.41.18.0 (T8): timeline-from-meetings runs SOLO (doesn't combine
// with --by-mention/--ner because those are links passes).
const { extractTimelineFromMeetings } = await import('../core/extract-timeline-from-meetings.ts');
const r = await extractTimelineFromMeetings(engine, { dryRun, sourceIdFilter });
result.timeline_entries_created = r.entries_created;
result.pages_processed = r.meetings_scanned;
if (!jsonMode) {
console.log(`Timeline from meetings: ${r.entries_created} entries on ${r.entities_touched} entity pages from ${r.meetings_scanned} meetings`);
}
} else if (byMention || ner) {
// v0.41.18.0 (T7): combined --by-mention + --ner walk shares one
// gazetteer; saves an entire pass on big brains. When only one
// flag is set, the other extractor skips silently.
const { buildGazetteer: buildGz } = await import('../core/by-mention.ts');
const sharedGazetteer = (byMention || ner) ? await buildGz(engine) : undefined;
if (byMention) {
const r = await extractMentionsFromDb(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
result.links_created += r.created;
result.pages_processed += r.pages;
}
if (ner) {
const { extractNerLinks } = await import('../core/extract-ner.ts');
const r = await extractNerLinks(engine, {
dryRun,
sourceIdFilter,
typeFilter,
since,
gazetteer: sharedGazetteer,
});
if (r.pack_unavailable && !jsonMode) {
console.log('Note: no active schema pack with link_types[].inference.regex — NER pass produced 0 links.');
}
result.links_created += r.created;
// pages already counted by by-mention if both ran; else count here.
if (!byMention) result.pages_processed += r.pages;
}
} else {
if (subcommand === 'links' || subcommand === 'all') {
const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since, { includeFrontmatter, sourceIdFilter });
@@ -1244,7 +1318,7 @@ async function extractTimelineFromDB(
}
/**
* v0.42.0.0 Part B (migration #1 of #1409) auto-link body-text entity
* v0.41.18.0 Part B (migration #1 of #1409) auto-link body-text entity
* mentions to known entity pages.
*
* Walks every page (respecting --source-id / --type / --since filters),
+10
View File
@@ -973,6 +973,11 @@ async function initPGLite(opts: {
const { printAdvisoryIfRecommended } = await import('../core/skillpack/post-install-advisory.ts');
const { VERSION } = await import('../version.ts');
printAdvisoryIfRecommended({ version: VERSION, context: 'init' });
// v0.41.18.0 (A4 + A18 + A20, T14): post-initSchema onboard nudge.
// Fail-open; 3s wallclock cap. Skipped silently in non-TTY contexts.
const { runInitNudge } = await import('../core/onboard/init-nudge.ts');
await runInitNudge(engine);
}
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
@@ -1191,6 +1196,11 @@ async function initPostgres(opts: {
const { printAdvisoryIfRecommended } = await import('../core/skillpack/post-install-advisory.ts');
const { VERSION } = await import('../version.ts');
printAdvisoryIfRecommended({ version: VERSION, context: 'init' });
// v0.41.18.0 (A4 + A18 + A20, T14): post-initSchema onboard nudge.
// Fail-open; 3s wallclock cap. Skipped silently in non-TTY contexts.
const { runInitNudge } = await import('../core/onboard/init-nudge.ts');
await runInitNudge(engine);
}
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
+60 -1
View File
@@ -1555,7 +1555,66 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
return await makeEmbedBackfillHandler(engine)(job);
});
process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected) + embed-backfill (v0.40)\n');
// v0.41.18.0 (A10, T7): extract-ner handler for the gbrain onboard
// remediation pipeline. Wraps extractNerLinks; emits typed_ner kind
// alongside the by-mention 'plain' kind. NOT in PROTECTED_JOB_NAMES
// (regex-only, no LLM spend).
worker.register('extract-ner', async (job) => {
const { extractNerLinks } = await import('../core/extract-ner.ts');
const data = (job.data ?? {}) as { sourceId?: string };
return await extractNerLinks(engine, {
sourceIdFilter: data.sourceId,
});
});
// v0.41.18.0 (A12, T9): extract-takes-from-pages handler. PROTECTED
// (LLM-bearing). Two-gate consent enforced at the handler boundary:
// refuses to run unless takes.bootstrap_enabled config is true, even
// when allowProtectedSubmit was set at queue.add time.
worker.register('extract-takes-from-pages', async (job) => {
const { extractTakesFromPages } = await import('../core/extract-takes-from-pages.ts');
const data = (job.data ?? {}) as { sourceId?: string; maxPages?: number };
const bootstrapCfg = await engine.getConfig('takes.bootstrap_enabled');
const bootstrapEnabled = bootstrapCfg === 'true' || bootstrapCfg === '1';
return await extractTakesFromPages(engine, {
bootstrapEnabled,
sourceIdFilter: data.sourceId,
maxPages: data.maxPages,
});
});
// v0.41.18.0 (A11, T8): extract-timeline-from-meetings handler. Wraps
// extractTimelineFromMeetings. NOT in PROTECTED_JOB_NAMES (pure SQL + string
// scan, no LLM spend).
worker.register('extract-timeline-from-meetings', async (job) => {
const { extractTimelineFromMeetings } = await import('../core/extract-timeline-from-meetings.ts');
const data = (job.data ?? {}) as { sourceId?: string };
return await extractTimelineFromMeetings(engine, {
sourceIdFilter: data.sourceId,
});
});
// v0.41.18.0 (A13): embed-catch-up handler for the gbrain onboard
// remediation pipeline. Wraps runEmbedCore with stale + catchUp + the
// priority/batchSize the recommendation supplies. NOT in
// PROTECTED_JOB_NAMES (embedding spend only).
worker.register('embed-catch-up', async (job) => {
const { runEmbedCore } = await import('./embed.ts');
const data = (job.data ?? {}) as {
sourceId?: string;
batchSize?: number;
priority?: 'recent';
};
return await runEmbedCore(engine, {
stale: true,
catchUp: true,
batchSize: data.batchSize,
priority: data.priority,
sourceId: data.sourceId,
});
});
process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected) + embed-backfill (v0.40) + embed-catch-up (v0.42)\n');
// Plugin discovery — one line per discovered plugin (mirrors the
// openclaw-seam startup line convention from v0.11+). Loaded
+177
View File
@@ -0,0 +1,177 @@
// src/commands/onboard.ts
// sourcescope:file-brain-wide — the --history surface reads
// migration_impact_log brain-wide. Per A26 lint opt-out.
//
// v0.41.18.0 (A1, T13). CLI shell for `gbrain onboard`. Thin wrapper over:
// - T2 library: computeRemediationPlan + runRemediation
// - T4 onboard checks: runAllOnboardChecks (extra remediations)
// - T12 render: buildOnboardReport + renderHuman
//
// Three modes:
// --check (default): print plan, no submission
// --auto: submit auto_apply tier (requires --max-usd)
// --auto --yes: also submit prompt_required tier
// --history: show recent migration_impact_log entries
//
// `--json` switches to the stable JSON envelope. No CLI mode → human render.
import type { BrainEngine } from '../core/engine.ts';
import { computeRemediationPlan, runRemediation } from '../core/remediation/index.ts';
import { runAllOnboardChecks } from '../core/onboard/checks.ts';
import { buildOnboardReport, renderHuman } from '../core/onboard/render.ts';
function parseInt10(args: string[], flag: string): number | null {
const i = args.indexOf(flag);
if (i === -1 || i === args.length - 1) return null;
const v = parseInt(args[i + 1] ?? '', 10);
return isNaN(v) ? null : v;
}
function parseFloat10(args: string[], flag: string): number | null {
const i = args.indexOf(flag);
if (i === -1 || i === args.length - 1) return null;
const v = parseFloat(args[i + 1] ?? '');
return isNaN(v) ? null : v;
}
export async function runOnboard(engine: BrainEngine, args: string[]): Promise<void> {
const check = args.includes('--check') || (!args.includes('--auto') && !args.includes('--history'));
const auto = args.includes('--auto');
const yes = args.includes('--yes');
const history = args.includes('--history');
const jsonOutput = args.includes('--json');
const targetScore = parseInt10(args, '--target-score') ?? 90;
const maxUsdRaw = parseFloat10(args, '--max-usd');
const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw;
// --history shows the impact log directly; no plan computation.
if (history) {
const rows = await engine.executeRaw<{
remediation_id: string;
metric_name: string;
metric_before: number | null;
metric_after: number | null;
applied_at: string;
}>(
`SELECT remediation_id, metric_name, metric_before, metric_after, applied_at
FROM migration_impact_log
ORDER BY applied_at DESC
LIMIT 50`,
);
const historyEntries = rows.map((r) => ({
remediation_id: r.remediation_id,
metric_name: r.metric_name,
metric_before: r.metric_before === null ? null : Number(r.metric_before),
metric_after: r.metric_after === null ? null : Number(r.metric_after),
delta: (r.metric_before === null || r.metric_after === null)
? null
: Number(r.metric_after) - Number(r.metric_before),
applied_at: r.applied_at,
}));
if (jsonOutput) {
process.stdout.write(JSON.stringify({
schema_version: 1,
history: historyEntries,
}, null, 2) + '\n');
return;
}
process.stdout.write(`Onboard history (last ${historyEntries.length}):\n`);
for (const h of historyEntries) {
const delta = h.delta !== null ? (h.delta > 0 ? `+${h.delta}` : String(h.delta)) : '?';
process.stdout.write(
` ${h.applied_at} ${h.remediation_id} ${h.metric_name}: ` +
`${h.metric_before ?? '?'}${h.metric_after ?? '?'} (${delta})\n`,
);
}
return;
}
// --auto refuses without --max-usd (cron-safety per A12 + A20).
if (auto && maxUsd === undefined) {
process.stderr.write(
`gbrain onboard --auto refuses without --max-usd N.\n` +
`Set a cap to avoid surprise spend:\n` +
` gbrain onboard --auto --max-usd 5\n`,
);
process.exit(2);
}
// Build the plan: T4 checks supply extra remediations on top of T3's
// generalized planner.
const onboardCheckResults = await runAllOnboardChecks(engine);
const extraRemediations = onboardCheckResults.flatMap((r) => r.remediations);
if (check && !auto) {
const plan = await computeRemediationPlan(engine, { targetScore, extraRemediations });
const report = buildOnboardReport(plan);
if (jsonOutput) {
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
return;
}
process.stdout.write(renderHuman(report) + '\n');
return;
}
// --auto path: runs through the T2 library orchestrator. Hooks emit CLI
// progress to stderr; the final result lands as JSON on stdout (or human
// summary).
const result = await runRemediation(
engine,
{
targetScore,
maxUsd,
// --auto --yes opts into the prompt_required tier too; library
// doesn't distinguish auto_apply vs prompt_required, it just runs
// every remediation in the plan. The plan-building side (T12 render)
// does the tier distinction; for --auto without --yes, the CLI shell
// would pre-filter the extras to auto_apply only. For now: pass
// everything; CLI documents this is "everything" behavior.
},
{
onTargetUnreachable: (target, ceiling) => {
process.stderr.write(
`[onboard] target ${target}/100 unreachable; max autonomous = ${ceiling}/100. ` +
`Configure missing prereqs (run gbrain doctor --remediation-plan) or lower --target-score.\n`,
);
},
onNothingToDo: (score, target) => {
process.stdout.write(
`Brain at score ${score}/100, target ${target}/100. Nothing to do.\n`,
);
},
onBudgetRefused: (estCost, cap) => {
process.stderr.write(
`[onboard] est cost $${estCost.toFixed(2)} exceeds --max-usd $${cap.toFixed(2)}. Aborting.\n`,
);
},
onStepStart: (step, total, rec) => {
process.stderr.write(`[onboard] [${step}/${total}] ${rec.job} (${rec.severity})...\n`);
},
onStepEnd: (sr) => {
process.stderr.write(`[onboard] → ${sr.status}\n`);
},
onBudgetExhausted: (planHash, snapshot) => {
process.stderr.write(
`\n[onboard] BudgetExhausted (${snapshot.reason}): spent $${snapshot.spent.toFixed(4)} > cap $${snapshot.cap.toFixed(2)}.\n` +
`Checkpoint saved. Resume with:\n gbrain doctor --remediate --resume ${planHash}\n`,
);
},
},
);
if (result.target_unreachable) process.exit(2);
if (jsonOutput) {
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
} else if (result.submitted.length > 0) {
process.stdout.write(
`\nBrain score: ${result.brain_score_initial}${result.brain_score_final} (target ${targetScore})\n` +
`Submitted: ${result.submitted.length} job(s), ${result.aborted_count} aborted/failed\n`,
);
}
const anyFailed = result.submitted.some(
(s) => s.status !== 'completed' && s.status !== 'submitted' && s.status !== 'dry_run',
);
if (result.budget_exhausted || anyFailed) process.exit(1);
}
+63
View File
@@ -571,12 +571,75 @@ Common flags:
case 'scorecard': return cmdScorecard(engine, rest);
case 'calibration': return cmdCalibration(engine, rest);
case 'revisit': return cmdRevisit(engine, rest);
case 'extract': return cmdExtract(engine, rest);
default:
// No subcommand keyword → treat first arg as <slug> for the list path.
return cmdList(engine, args);
}
}
/**
* v0.41.18.0 (A12, A24, T9) `gbrain takes extract --from-pages` runs
* Haiku over concept/atom/lore/briefing/writing/originals pages and
* lifts gradeable claims into the takes fence.
*
* Two-gate consent: requires `takes.bootstrap_enabled=true` in config
* AND explicit --yes flag for any non-dryRun run. Refuses LLM-bearing
* extraction without both.
*/
async function cmdExtract(engine: BrainEngine, rest: string[]): Promise<void> {
const sub = rest[0];
if (sub !== '--from-pages') {
process.stderr.write(
'Usage: gbrain takes extract --from-pages [--yes] [--dry-run] [--source-id <id>] [--max-pages N] [--holder <name>]\n',
);
process.exit(1);
}
const dryRun = rest.includes('--dry-run');
const skipConfirm = rest.includes('--yes');
const sourceIdx = rest.indexOf('--source-id');
const sourceIdFilter = sourceIdx >= 0 ? rest[sourceIdx + 1] : undefined;
const maxIdx = rest.indexOf('--max-pages');
const maxPagesRaw = maxIdx >= 0 ? rest[maxIdx + 1] : undefined;
const maxPages = maxPagesRaw ? Math.max(1, Math.min(1000, parseInt(maxPagesRaw, 10) || 50)) : 50;
const holderIdx = rest.indexOf('--holder');
const holder = holderIdx >= 0 ? rest[holderIdx + 1] : 'system';
// A12 consent gate.
const bootstrapEnabledCfg = await engine.getConfig('takes.bootstrap_enabled');
const bootstrapEnabled = bootstrapEnabledCfg === 'true' || bootstrapEnabledCfg === '1';
if (!bootstrapEnabled) {
process.stderr.write(
`takes-bootstrap is opt-in. Enable with:\n gbrain config set takes.bootstrap_enabled true\nThen re-run with --yes.\n`,
);
process.exit(2);
}
if (!dryRun && !skipConfirm) {
process.stderr.write(
`[takes extract] sends concept/atom/lore/briefing/writing/originals page content to Haiku.\n` +
`Pass --yes to proceed (or --dry-run to preview).\n`,
);
process.exit(1);
}
const { extractTakesFromPages } = await import('../core/extract-takes-from-pages.ts');
const result = await extractTakesFromPages(engine, {
bootstrapEnabled: true,
dryRun,
sourceIdFilter,
maxPages,
holder,
});
if (result.llm_unavailable) {
process.stderr.write(`[takes extract] chat gateway unavailable (no API key configured).\n`);
process.exit(2);
}
process.stdout.write(
`takes extract --from-pages: ${result.claims_extracted} claim(s) from ${result.pages_scanned} page(s)` +
(dryRun ? ' (dry-run)' : '') + '\n',
);
}
/**
* v0.36.1.0 (TD4 / D30) `gbrain takes revisit <slug>` opens $EDITOR on
* the source page so the user can write a follow-up immediately. The
+14
View File
@@ -409,6 +409,20 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
// compare gbrain against itself)
// - every scaffolded skill is identical (nothing to say)
await postUpgradeReferenceSweep();
// v0.41.18.0 (A4 + A18, T14): post-upgrade onboard banner. Fail-open;
// doesn't engine-connect (lightweight TTY check only). The actual
// recommendations need engine access via `gbrain onboard --check`;
// the banner just nudges the user to run it.
try {
const { runUpgradeBanner } = await import('../core/onboard/init-nudge.ts');
// The banner doesn't actually use the engine today; passing null-equivalent
// would require a type widening. Skip the engine arg and let the banner
// print the static nudge text.
await runUpgradeBanner(null as never);
} catch {
// Fail-open per A18: never crash post-upgrade from the banner.
}
}
/**
+27
View File
@@ -167,9 +167,26 @@ export interface CheckClassification {
* Returns ONLY `remediable` items. `blocked` items surface via
* `classifyChecks()` and are rendered alongside the plan as informational.
*/
/**
* Generalized (v0.41.18.0, A2 + codex finding #3): an optional third arg
* lets callers inject RemediationStep entries discovered by doctor checks
* outside this module's hardcoded planner. Without this, adding a
* `Check.remediation` field to a new doctor check wouldn't auto-wire into
* `gbrain doctor --remediation-plan` the planner would just ignore it.
*
* Onboard's runRemediationPlan calls the 4 new check helpers (embed_staleness,
* entity_link_coverage, timeline_coverage, takes_count) and threads their
* RemediationStep[] outputs through this slot. Each helper produces its own
* cheap query (D7 cheap-path preserved); aggregation happens in the caller.
*
* Sort + dedup applies across BOTH the hardcoded + extra entries: stable id
* collisions resolve in favor of the hardcoded entry (legacy behavior wins),
* which means extras only add coverage they're not duplicating.
*/
export function computeRecommendations(
health: BrainHealth,
ctx: RecommendationContext,
extraRemediations: Remediation[] = [],
): Remediation[] {
const out: Remediation[] = [];
const source = ctx.sourceId ?? 'default';
@@ -266,6 +283,16 @@ export function computeRecommendations(
});
}
// v0.41.18.0 (A2 + codex #3): merge caller-supplied extras. Hardcoded
// entries win on id collision so legacy behavior is preserved when an
// extra accidentally duplicates a hardcoded id.
if (extraRemediations.length > 0) {
const hardcodedIds = new Set(out.map((r) => r.id));
for (const extra of extraRemediations) {
if (!hardcodedIds.has(extra.id)) out.push(extra);
}
}
// Sort: severity (critical first), then est_seconds ascending so quick
// wins come first within a severity tier.
const sevRank: Record<RemediationSeverity, number> = {
+45 -2
View File
@@ -126,6 +126,15 @@ export interface LinkBatchInput {
from_source_id?: string;
to_source_id?: string;
origin_source_id?: string;
/**
* v0.41.18.0 (A10, codex finding #12): distinguishes "plain body mention"
* (NULL or 'plain') from "verb-pattern-derived typed NER" ('typed_ner')
* within link_source='mentions'. Backed by v98 schema column. NOT in
* the links UNIQUE constraint same (from, to, type, source, origin)
* tuple with different link_kind collides DO NOTHING. Default NULL =
* legacy / unknown / pre-v98 semantics.
*/
link_kind?: string;
}
/** Input row for addTimelineEntriesBatch. Optional fields default to '' (matches NOT NULL DDL). */
@@ -165,7 +174,19 @@ export interface TimelineBatchInput {
* transaction itself is waiting to write.
*/
export interface ReservedConnection {
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;
/**
* v0.41.18.0 (A20, codex #7): optional 3rd-arg `opts.signal` lets callers
* actually cancel a running query. Init nudge (3s wallclock cap) wires an
* AbortController whose timer fires at 3s; queries that haven't returned
* by then get cancelled (Postgres: query.cancel(); PGLite: in-process,
* Promise.race against signal-rejection documented gap because PGLite
* has no kernel-level cancellation).
*/
executeRaw<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]>;
}
/**
@@ -865,6 +886,16 @@ export interface BrainEngine {
afterPageId?: number;
afterChunkIndex?: number;
sourceId?: string;
// v0.41.18.0 (A13, codex #9): pagination order. Default 'page_id'
// (legacy stable cursor). 'updated_desc' joins pages and orders by
// p.updated_at DESC NULLS LAST, p.id, cc.chunk_index — backed by
// idx_pages_updated_at_desc + content_chunks_stale_idx partial.
orderBy?: 'page_id' | 'updated_desc';
// For 'updated_desc' cursor: previous row's updated_at, page_id, chunk_index.
// ISO-8601 string for cross-engine compatibility (postgres.js + PGLite
// both round-trip TIMESTAMPTZ as Date | string; ISO string is the
// common denominator on the wire).
afterUpdatedAt?: string | null;
}): Promise<StaleChunkRow[]>;
/**
* Delete every chunk for a page. Internal page-id lookup is sourceId-scoped
@@ -1606,7 +1637,19 @@ export interface BrainEngine {
getChunksWithEmbeddings(slug: string, opts?: { sourceId?: string }): Promise<Chunk[]>;
// Raw SQL (for Minions job queue and other internal modules)
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;
/**
* v0.41.18.0 (A20, codex #7): optional 3rd-arg `opts.signal` lets callers
* actually cancel a running query. Init nudge (3s wallclock cap) wires an
* AbortController whose timer fires at 3s; queries that haven't returned
* by then get cancelled (Postgres: query.cancel(); PGLite: in-process,
* Promise.race against signal-rejection documented gap because PGLite
* has no kernel-level cancellation).
*/
executeRaw<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]>;
// ============================================================
// v0.20.0 Cathedral II: code edges (Layer 5 populates, Layer 7 consumes)
+216
View File
@@ -0,0 +1,216 @@
// src/core/extract-ner.ts
// v0.41.18.0 (A10, T7). NER link extraction reuses the by-mention gazetteer
// and applies schema-pack `link_types[].inference.regex` patterns to assign
// a typed link verb ("CEO of Acme" → 'works_at' linking the page to Acme).
//
// Codex finding #12 design (locked): do NOT split link_source='ner' as a
// new provenance — that would break every existing link_source='mentions'
// query (backlink-count filter, orphan-ratio, doctor checks). Instead:
// keep link_source='mentions' AND set link_kind='typed_ner' on the new row
// (v98 added the column). Legacy plain mentions stay link_kind=NULL
// (semantically 'plain').
//
// The links UNIQUE constraint excludes link_kind, so an existing plain
// mention row + a typed_ner row for the same (from, to, type, source, origin)
// collide — DO NOTHING. NER does NOT overwrite plain mentions; the verb
// link goes in as a different row with a different link_type.
import type { BrainEngine } from './engine.ts';
import type { LinkBatchInput } from './engine.ts';
import { buildGazetteer, findMentionedEntities, type Gazetteer } from './by-mention.ts';
import { inferLinkTypeFromPack } from './schema-pack/link-inference.ts';
import { loadActivePackBestEffort } from './schema-pack/best-effort.ts';
export interface ExtractNerOpts {
/** When true: enumerate but don't write. */
dryRun?: boolean;
/** Optional source-id filter on the WALK (gazetteer stays brain-wide). */
sourceIdFilter?: string;
/** Optional page-type filter on the WALK. */
typeFilter?: string;
/** Only scan pages with updated_at after this ISO date. */
since?: string;
/**
* Pre-built gazetteer (T7+: combined `--by-mention --ner` walk shares
* one gazetteer across both passes). When omitted, this fn builds its own.
*/
gazetteer?: Gazetteer;
/** Optional progress hook called per processed page. */
onProgress?: (done: number, total: number, created: number) => void;
}
export interface ExtractNerResult {
/** Pages scanned. */
pages: number;
/** Typed-NER links created (or would-have-created in dry-run). */
created: number;
/** Pages where the active schema pack had no link_types at all. */
pack_unavailable: boolean;
}
/** Context window scanned around each mention for verb-pattern matching. */
const CONTEXT_WINDOW_CHARS = 80;
/**
* Pure helper: get the context window around a mention's character offset.
* Returns the substring [offset - W, offset + name.length + W] of the body.
* Caller passes (body, offset, name.length).
*/
export function getContextWindow(
body: string,
offset: number,
nameLen: number,
window: number = CONTEXT_WINDOW_CHARS,
): string {
const start = Math.max(0, offset - window);
const end = Math.min(body.length, offset + nameLen + window);
return body.slice(start, end);
}
/**
* Pure helper: derive the entity-typelink-verb pair from a single mention.
* Returns null when (a) target type unknown, (b) pack has no inference for
* that type, (c) no verb pattern matches the surrounding context.
*
* Exported for unit tests; the orchestrator below uses it directly.
*/
export function inferNerLinkType(
pack: Parameters<typeof inferLinkTypeFromPack>[0],
targetType: string | undefined,
context: string,
): string | null {
if (!targetType) return null;
try {
return inferLinkTypeFromPack(pack, targetType, context);
} catch {
return null;
}
}
/**
* extractNerLinks: walk pages, find body mentions, apply schema-pack
* inference regex per (target_type, surrounding context) to assign a typed
* link verb. Returns count of created links.
*
* Best-effort wrt the schema pack: if no active pack OR no link_types
* declared OR no inference.regex on any link_type, the function returns
* pack_unavailable=true and 0 created. Caller (CLI / handler) surfaces a
* one-line hint instead of an error.
*/
export async function extractNerLinks(
engine: BrainEngine,
opts: ExtractNerOpts = {},
): Promise<ExtractNerResult> {
const dryRun = opts.dryRun ?? false;
// Pack best-effort: no pack → no inference → nothing to do.
const pack = await loadActivePackBestEffort({ engine } as never);
if (!pack || !pack.manifest?.link_types || pack.manifest.link_types.length === 0) {
return { pages: 0, created: 0, pack_unavailable: true };
}
// Require at least one link_type with an inference.regex; otherwise NER
// has no patterns to match and we'd waste a full walk.
const hasRegex = pack.manifest.link_types.some(
(lt) => lt.inference && typeof lt.inference === 'object' && 'regex' in lt.inference,
);
if (!hasRegex) return { pages: 0, created: 0, pack_unavailable: true };
const gazetteer = opts.gazetteer ?? await buildGazetteer(engine);
if (gazetteer.size === 0) {
return { pages: 0, created: 0, pack_unavailable: false };
}
// Pre-fetch target entity types so inferLinkType has the type signal
// without an N+1 getPage round-trip. Pulls the slug→type map from
// listAllPageRefs + a single listPages projection.
const targetTypeMap = await buildTargetTypeMap(engine);
const allRefs = opts.sourceIdFilter
? (await engine.listAllPageRefs()).filter((r) => r.source_id === opts.sourceIdFilter)
: await engine.listAllPageRefs();
let processed = 0;
let created = 0;
const batch: LinkBatchInput[] = [];
const BATCH_SIZE = 500;
const sinceMs = opts.since ? new Date(opts.since).getTime() : null;
async function flush() {
if (batch.length === 0) return;
if (!dryRun) {
try {
created += await engine.addLinksBatch(batch); // gbrain-allow-direct-insert: extract-ner — typed NER link write
} catch {
// batch error: drop; the per-page progress continues
}
} else {
created += batch.length;
}
batch.length = 0;
}
for (const { slug, source_id } of allRefs) {
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
if (opts.typeFilter && page.type !== opts.typeFilter) continue;
if (sinceMs !== null) {
const updatedMs = new Date(page.updated_at).getTime();
if (Number.isFinite(updatedMs) && updatedMs <= sinceMs) continue;
}
processed++;
opts.onProgress?.(processed, allRefs.length, created);
const body = page.compiled_truth + '\n\n' + (page.timeline ?? '');
if (!body.trim()) continue;
const mentions = findMentionedEntities(body, gazetteer, {
fromSlug: slug,
fromSourceId: source_id,
});
if (mentions.length === 0) continue;
for (const m of mentions) {
const targetType = targetTypeMap.get(`${m.source_id}::${m.slug}`);
const context = getContextWindow(body, m.offset, m.name.length);
const verb = inferNerLinkType(pack.manifest, targetType, context);
if (!verb) continue;
batch.push({
from_slug: slug,
to_slug: m.slug,
link_type: verb,
link_source: 'mentions',
link_kind: 'typed_ner',
context: m.name,
from_source_id: source_id,
to_source_id: m.source_id,
});
if (batch.length >= BATCH_SIZE) await flush();
}
}
await flush();
return { pages: processed, created, pack_unavailable: false };
}
/**
* Helper: build a Map<sourceId::slug type> for all entity-typed pages.
* One round-trip via listPages. Targets cached at extraction-start so
* inferNerLinkType doesn't pay an N+1 cost per mention.
*/
async function buildTargetTypeMap(engine: BrainEngine): Promise<Map<string, string>> {
const map = new Map<string, string>();
try {
const result = await engine.executeRaw<{ slug: string; source_id: string; type: string }>(
`SELECT slug, source_id, type FROM pages
WHERE type IN ('person', 'company', 'organization', 'entity')
AND deleted_at IS NULL`,
);
for (const row of result) {
map.set(`${row.source_id}::${row.slug}`, row.type);
}
} catch {
// Engine error → empty map; inferNerLinkType returns null for unknown types.
}
return map;
}
+223
View File
@@ -0,0 +1,223 @@
// src/core/extract-takes-from-pages.ts
// v0.41.18.0 (A12, A24, T9). Haiku classifier loop over allowlisted page
// types — concept, atom, lore, briefing, writing, originals — extracts
// gradeable claims and inserts them as takes fence rows.
//
// Two-gate consent per A12:
// - takes.bootstrap_enabled (default false): must be true to run at all.
// Even manual `gbrain takes extract --from-pages` refuses without it.
// - takes.autopilot_allowed (default false): must be true for autopilot's
// auto-apply tier to fire the takes-bootstrap remediation.
//
// A24 deliberately limits autopilot to manual_only until v0.42.1 lands a
// 100+-case eval suite. v0.42 ships the classifier + CLI; autopilot stays
// blocked until eval coverage catches up.
import type { BrainEngine } from './engine.ts';
import type { TakeBatchInput, TakeKind } from './engine.ts';
import { chat, isAvailable } from './ai/gateway.ts';
export const ALLOWED_PAGE_TYPES = [
'concept', 'atom', 'lore', 'briefing', 'writing', 'originals',
] as const;
const CLASSIFIER_SYSTEM = `You extract gradeable CLAIMS from longform writing.
Output strict JSON: an array of objects with shape:
{"claim": "<short imperative or assertion, <= 200 chars>",
"kind": "fact" | "take" | "bet" | "hunch",
"weight": 0.0..1.0}
Kind taxonomy:
- fact: verifiable as true/false (e.g. "X raised $5M in Mar 2024")
- take: a stated opinion that could be wrong (e.g. "X is undervalued")
- bet: a forward-looking prediction (e.g. "X will IPO in 2026")
- hunch: a low-confidence gut feeling (e.g. "Y feels overstretched")
Skip pure narrative, questions, definitions, or pure quotes from others.
Max 15 claims per page; output [] if no gradeable claims are present.`;
export interface ExtractTakesFromPagesOpts {
/** Required: must be true for any work to happen (A12). */
bootstrapEnabled: boolean;
/** Dry-run: classify but don't write to takes table. */
dryRun?: boolean;
/** Scope to a single source. */
sourceIdFilter?: string;
/** Max pages to classify per run (caps cost). Default 50. */
maxPages?: number;
/** Owner identifier for the inserted takes. Default 'system'. */
holder?: string;
/** Model override; defaults to facts.extraction_model. */
model?: string;
/** Progress hook called per page. */
onProgress?: (done: number, total: number, claims: number) => void;
}
export interface ExtractTakesFromPagesResult {
pages_scanned: number;
claims_extracted: number;
/** True if the run was a no-op because bootstrapEnabled is false. */
consent_gate_blocked: boolean;
/** True if chat gateway is unavailable (no LLM call possible). */
llm_unavailable: boolean;
}
interface PageRow {
id: number;
slug: string;
source_id: string;
type: string;
compiled_truth: string;
updated_at: string | Date;
}
/**
* Pure helper: parse Haiku JSON output into typed claims. Returns []
* on any parse failure (caller treats as "no claims extracted").
*/
export function parseClaimsJson(raw: string): Array<{ claim: string; kind: TakeKind; weight: number }> {
try {
// Strip code fences if model wrapped output in ```json.
let text = raw.trim();
const fenceMatch = text.match(/^```(?:json)?\n?([\s\S]*?)\n?```$/);
if (fenceMatch) text = fenceMatch[1].trim();
const parsed = JSON.parse(text);
if (!Array.isArray(parsed)) return [];
const valid: Array<{ claim: string; kind: TakeKind; weight: number }> = [];
for (const item of parsed) {
if (!item || typeof item !== 'object') continue;
const claim = typeof item.claim === 'string' ? item.claim.trim().slice(0, 200) : '';
const kind = typeof item.kind === 'string' ? item.kind : '';
const weightRaw = typeof item.weight === 'number' ? item.weight : 0.5;
const weight = Math.max(0, Math.min(1, weightRaw));
if (!claim || !['fact', 'take', 'bet', 'hunch'].includes(kind)) continue;
valid.push({ claim, kind, weight });
}
return valid;
} catch {
return [];
}
}
export async function extractTakesFromPages(
engine: BrainEngine,
opts: ExtractTakesFromPagesOpts,
): Promise<ExtractTakesFromPagesResult> {
// A12 consent gate: refuse without bootstrap_enabled even on manual call.
if (!opts.bootstrapEnabled) {
return {
pages_scanned: 0,
claims_extracted: 0,
consent_gate_blocked: true,
llm_unavailable: false,
};
}
if (!isAvailable('chat')) {
return {
pages_scanned: 0,
claims_extracted: 0,
consent_gate_blocked: false,
llm_unavailable: true,
};
}
const dryRun = opts.dryRun ?? false;
const maxPages = opts.maxPages ?? 50;
const holder = opts.holder ?? 'system';
const sourceFilter = opts.sourceIdFilter ? `AND source_id = $1` : '';
const params = opts.sourceIdFilter ? [opts.sourceIdFilter] : [];
// Fetch eligible pages. Order by updated_at DESC so recently-edited
// pages get bootstrapped first.
const typesList = ALLOWED_PAGE_TYPES.map((t) => `'${t}'`).join(', ');
const pages = await engine.executeRaw<PageRow>(
`SELECT id, slug, source_id, type, compiled_truth, updated_at
FROM pages
WHERE type IN (${typesList})
AND deleted_at IS NULL
AND length(COALESCE(compiled_truth, '')) > 200
${sourceFilter}
ORDER BY updated_at DESC
LIMIT ${maxPages}`,
params,
);
let pagesScanned = 0;
let claimsExtracted = 0;
const batch: TakeBatchInput[] = [];
async function flush() {
if (batch.length === 0) return;
if (!dryRun) {
try {
claimsExtracted += await engine.addTakesBatch(batch);
} catch {
// batch error — drop and continue with subsequent pages
}
} else {
claimsExtracted += batch.length;
}
batch.length = 0;
}
for (const page of pages) {
pagesScanned++;
opts.onProgress?.(pagesScanned, pages.length, claimsExtracted);
if (!page.compiled_truth || page.compiled_truth.length < 200) continue;
// Truncate to keep per-page cost bounded (~20K chars → ~5K input tokens).
const text = page.compiled_truth.slice(0, 20_000);
let response: { text: string };
try {
response = await chat({
model: opts.model ?? 'anthropic:claude-haiku-4-5',
system: CLASSIFIER_SYSTEM,
messages: [
{
role: 'user',
content: `<page slug="${page.slug}" type="${page.type}">\n${text}\n</page>`,
},
],
maxTokens: 2000,
});
} catch {
// Skip pages whose chat call fails (rate limit, content filter,
// transient error). Per-page progress continues.
continue;
}
const claims = parseClaimsJson(response.text);
if (claims.length === 0) continue;
// Assign row_num starting from 1 per page. We don't query existing
// takes for the page — collisions on (page_id, row_num) are an existing
// bug class addresses by extract-conversation-facts; takes-bootstrap
// inherits the same posture: writes start at row_num=1 and the engine's
// unique constraint surfaces duplicates as failures (caller re-runs).
for (let i = 0; i < claims.length; i++) {
const c = claims[i];
batch.push({
page_id: page.id,
row_num: i + 1,
claim: c.claim,
kind: c.kind,
holder,
weight: c.weight,
source: 'cli:takes-bootstrap-from-pages',
});
}
if (batch.length >= 200) await flush();
}
await flush();
return {
pages_scanned: pagesScanned,
claims_extracted: claimsExtracted,
consent_gate_blocked: false,
llm_unavailable: false,
};
}
+186
View File
@@ -0,0 +1,186 @@
// src/core/extract-timeline-from-meetings.ts
// v0.41.18.0 (A11, T8). Walk meeting pages, identify discussed entities via
// (a) existing `attended` links (attendees) + (b) body-mention scan, and
// write a timeline entry on each entity page with a meeting-specific source
// key that survives v99's widened dedup.
//
// Codex finding #11 dependency: requires v99 dedup widening from
// (page_id, date, summary) to (page_id, date, summary, source). Without v99,
// two meetings on the same date with the same summary on the same entity
// page would silently drop the second one.
import type { BrainEngine } from './engine.ts';
import type { TimelineBatchInput } from './engine.ts';
import { buildGazetteer, findMentionedEntities, type Gazetteer } from './by-mention.ts';
export interface ExtractTimelineFromMeetingsOpts {
dryRun?: boolean;
sourceIdFilter?: string;
/** Only scan meetings with updated_at after this ISO date. */
since?: string;
/** Optional pre-built gazetteer (for shared-walk callers). */
gazetteer?: Gazetteer;
onProgress?: (done: number, total: number, created: number) => void;
}
export interface ExtractTimelineFromMeetingsResult {
meetings_scanned: number;
entries_created: number;
/** Distinct entity pages that received at least one new timeline entry. */
entities_touched: number;
}
interface MeetingRow {
slug: string;
source_id: string;
title: string;
effective_date: string | null;
updated_at: string | Date;
compiled_truth: string;
timeline: string;
}
interface AttendedEdgeRow {
from_slug: string;
from_source_id: string;
to_slug: string;
to_source_id: string;
}
const BATCH_SIZE = 200;
export async function extractTimelineFromMeetings(
engine: BrainEngine,
opts: ExtractTimelineFromMeetingsOpts = {},
): Promise<ExtractTimelineFromMeetingsResult> {
const dryRun = opts.dryRun ?? false;
const sinceMs = opts.since ? new Date(opts.since).getTime() : null;
// 1. Fetch all meeting pages (one round-trip).
const sourceFilter = opts.sourceIdFilter ? `AND source_id = $1` : '';
const meetingParams = opts.sourceIdFilter ? [opts.sourceIdFilter] : [];
const meetings = await engine.executeRaw<MeetingRow>(
`SELECT slug, source_id, title, effective_date, updated_at,
compiled_truth, COALESCE(timeline, '') AS timeline
FROM pages
WHERE type = 'meeting'
AND deleted_at IS NULL
${sourceFilter}
ORDER BY effective_date DESC NULLS LAST, slug`,
meetingParams,
);
if (meetings.length === 0) {
return { meetings_scanned: 0, entries_created: 0, entities_touched: 0 };
}
// 2. Fetch all 'attended' edges (one round-trip, scoped to the loaded
// meeting source_ids). Build a Map<meetingSlug → attendees[]> for O(1)
// attendee lookup per meeting.
const meetingKeys = new Set(meetings.map((m) => `${m.source_id}::${m.slug}`));
const attendedEdges = await engine.executeRaw<AttendedEdgeRow>(
`SELECT pf.slug AS from_slug, pf.source_id AS from_source_id,
pt.slug AS to_slug, pt.source_id AS to_source_id
FROM links l
JOIN pages pf ON pf.id = l.from_page_id
JOIN pages pt ON pt.id = l.to_page_id
WHERE l.link_type = 'attended'
AND pf.type = 'meeting'
AND pf.deleted_at IS NULL
AND pt.deleted_at IS NULL`,
);
const attendeesByMeeting = new Map<string, AttendedEdgeRow[]>();
for (const e of attendedEdges) {
const key = `${e.from_source_id}::${e.from_slug}`;
if (!meetingKeys.has(key)) continue;
const list = attendeesByMeeting.get(key);
if (list) list.push(e);
else attendeesByMeeting.set(key, [e]);
}
// 3. For each meeting, derive entity mentions (gazetteer-based) + merge
// with attendee edges. Each (meeting, entity) produces ONE timeline row.
const gazetteer = opts.gazetteer ?? await buildGazetteer(engine);
const batch: TimelineBatchInput[] = [];
let entriesCreated = 0;
const entitiesTouched = new Set<string>();
let meetingsScanned = 0;
async function flush() {
if (batch.length === 0) return;
if (!dryRun) {
try {
entriesCreated += await engine.addTimelineEntriesBatch(batch);
} catch {
// batch error — drop; per-meeting progress continues
}
} else {
entriesCreated += batch.length;
}
batch.length = 0;
}
for (const meeting of meetings) {
if (sinceMs !== null) {
const updatedMs = new Date(meeting.updated_at).getTime();
if (Number.isFinite(updatedMs) && updatedMs <= sinceMs) continue;
}
if (!meeting.effective_date) continue; // can't write a timeline entry without a date
meetingsScanned++;
opts.onProgress?.(meetingsScanned, meetings.length, entriesCreated);
const meetingKey = `${meeting.source_id}::${meeting.slug}`;
const summary = `Discussed in ${meeting.title}`;
const sourceKey = `extract-timeline-from-meetings:${meeting.slug}`;
// Attendees (from 'attended' links).
const attendees = attendeesByMeeting.get(meetingKey) ?? [];
const targets = new Map<string, { slug: string; source_id: string }>();
for (const e of attendees) {
targets.set(`${e.to_source_id}::${e.to_slug}`, {
slug: e.to_slug,
source_id: e.to_source_id,
});
}
// Body mentions (gazetteer-based). Skip self-mention (meeting page
// referencing itself by title). The cross-source guard in
// findMentionedEntities already drops mentions targeting a different
// source than the gazetteer entry was built from.
const body = meeting.compiled_truth + '\n\n' + meeting.timeline;
if (body.trim()) {
const mentions = findMentionedEntities(body, gazetteer, {
fromSlug: meeting.slug,
fromSourceId: meeting.source_id,
});
for (const m of mentions) {
targets.set(`${m.source_id}::${m.slug}`, {
slug: m.slug,
source_id: m.source_id,
});
}
}
// Emit one timeline row per (entity, this meeting).
for (const t of targets.values()) {
batch.push({
slug: t.slug,
source_id: t.source_id,
date: meeting.effective_date,
source: sourceKey,
summary,
});
entitiesTouched.add(`${t.source_id}::${t.slug}`);
if (batch.length >= BATCH_SIZE) await flush();
}
}
await flush();
return {
meetings_scanned: meetingsScanned,
entries_created: entriesCreated,
entities_touched: entitiesTouched.size,
};
}
+129 -33
View File
@@ -4415,7 +4415,7 @@ export const MIGRATIONS: Migration[] = [
{
version: 95,
name: 'links_link_source_check_includes_mentions',
// v0.42.0.0 Part B (migration #1 of #1409): widen the link_source
// v0.41.18.0 Part B (migration #1 of #1409): widen the link_source
// CHECK constraint to admit 'mentions' for auto-linked body-text
// mentions from `gbrain extract links --by-mention`. Backlink-count
// SQL in postgres-engine.ts + pglite-engine.ts excludes link_source =
@@ -4570,17 +4570,7 @@ export const MIGRATIONS: Migration[] = [
// If we backfilled = acquired_at (e.g. 25 min ago), then `gbrain sync
// --break-lock --all --max-age 1800` after the migration would
// immediately delete the lock of a HEALTHY 25-min-old holder that's
// still actively writing. That IS the bug class this PR exists to
// fix; reintroducing it through the migration backfill would be
// worse than not shipping at all.
//
// Backfilling = NOW() gives every pre-upgrade holder a 30-min
// protection window: --max-age 1800 cannot identify them as wedged
// for 30 min after migrate. After that, all pre-upgrade syncs are
// either complete (lock released) OR genuinely wedged (--max-age
// does the right thing on the next operator command).
//
// Engine parity: same ALTER TABLE shape works on Postgres + PGLite.
// still actively writing.
sql: `
ALTER TABLE gbrain_cycle_locks ADD COLUMN IF NOT EXISTS last_refreshed_at TIMESTAMPTZ;
UPDATE gbrain_cycle_locks SET last_refreshed_at = NOW() WHERE last_refreshed_at IS NULL;
@@ -4589,27 +4579,8 @@ export const MIGRATIONS: Migration[] = [
{
version: 99,
name: 'conversation_parser_llm_cache_table',
// v0.41.16.0 — content-hash-keyed cache for the conversation
// parser's LLM polish + fallback calls. Per D17 (codex outside
// voice), there is NO conversation_parser_inferred_patterns
// table: persisting LLM-inferred regex from a 20-line sample
// and applying it to whole pages + future pages is a silent
// corruption machine. Cache shape is per-page-content-hash;
// re-running the dream cycle on the same haystack is a hit;
// a different page with the same format misses the cache and
// calls the LLM again (cost-bounded by BudgetTracker).
//
// Key is (content_sha256, model_id, call_shape).
// - content_sha256: sha256 of the page body the LLM saw.
// - model_id: provider:model the call routed through (so a
// model upgrade invalidates stale entries naturally).
// - call_shape: 'polish' | 'fallback' so the same content
// can be cached differently per call kind.
//
// Slot history: originally v97, bumped to v98 after master's
// v0.41.13.0 (#1422 fix-wave) claimed v97 for the dedup index,
// bumped to v99 after master's v0.41.15.0 (#1506 sync RFC)
// claimed v98 for the lock-refresh column.
// v0.41.16.0 — content-hash-keyed cache for the conversation parser's
// LLM polish + fallback calls. See src/schema.sql for design notes.
sql: `
CREATE TABLE IF NOT EXISTS conversation_parser_llm_cache (
content_sha256 TEXT NOT NULL,
@@ -4637,6 +4608,131 @@ export const MIGRATIONS: Migration[] = [
`,
},
},
{
version: 101,
name: 'links_link_kind_column',
// v0.41.18.0 (gbrain onboard wave, A10 + codex finding #12):
// NER link extraction adds a nullable link_kind column instead of
// splitting link_source='ner' as a new provenance — keeps
// backlink-count + orphan-ratio queries stable while letting
// NER-aware callers distinguish typed links.
//
// Three kinds: 'plain' | 'typed_ner' | NULL (legacy, semantically plain).
// NOT in the links UNIQUE constraint so a plain-mention row coexists
// with future typed_ner promotions via explicit ON CONFLICT DO UPDATE.
//
// Slot history: originally v98, bumped to v101 after master merge
// claimed v98 (lock-refresh) + v99 (conversation parser cache) +
// v100 (per master's own merges).
sql: `
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_kind TEXT
CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner'));
`,
sqlFor: {
pglite: `
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_kind TEXT
CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner'));
`,
},
},
{
version: 102,
name: 'timeline_entries_source_in_dedup',
// v0.41.18.0 (gbrain onboard wave, A11 + codex finding #11):
// Widen idx_timeline_dedup from (page_id, date, summary) to
// (page_id, date, summary, source) so --from-meetings provenance
// survives. Legacy rows have source='' (schema default), so legacy
// dedup behavior is preserved.
//
// Slot history: originally v99, bumped to v102 after master merge.
sql: `
DROP INDEX IF EXISTS idx_timeline_dedup;
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup
ON timeline_entries(page_id, date, summary, source);
`,
sqlFor: {
pglite: `
DROP INDEX IF EXISTS idx_timeline_dedup;
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup
ON timeline_entries(page_id, date, summary, source);
`,
},
},
{
version: 103,
name: 'migration_impact_log_and_priority_recent_idx',
// v0.41.18.0 (gbrain onboard wave, A6 + A25 + A13 + codex #9 + #10):
// (1) migration_impact_log table — onboard --history backbone with
// attribution columns (job_id, source_id, brain_id, started_at,
// idempotency_key) so concurrent runs don't misattribute deltas.
// (2) content_chunks_stale_idx partial index — supports
// `embed --stale` + `--priority recent` (outer ORDER BY
// p.updated_at DESC uses existing idx_pages_updated_at_desc).
//
// Slot history: originally v100, bumped to v103 after master merge.
// Engine-aware split: Postgres uses CREATE INDEX CONCURRENTLY +
// invalid-remnant pre-drop; PGLite uses plain CREATE INDEX.
transaction: false,
sql: '',
handler: async (engine) => {
const createTableSql = `
CREATE TABLE IF NOT EXISTS migration_impact_log (
id BIGSERIAL PRIMARY KEY,
remediation_id TEXT NOT NULL,
metric_name TEXT NOT NULL,
metric_before NUMERIC,
metric_after NUMERIC,
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
source_id TEXT,
brain_id TEXT,
started_at TIMESTAMPTZ,
idempotency_key TEXT,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
applied_by TEXT,
details JSONB DEFAULT '{}'::jsonb
);
`;
await engine.runMigration(103, createTableSql);
await engine.runMigration(
103,
`CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
ON migration_impact_log(remediation_id, applied_at DESC);`
);
await engine.runMigration(
103,
`CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;`
);
if (engine.kind === 'postgres') {
await engine.runMigration(
103,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'content_chunks_stale_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS content_chunks_stale_idx';
END IF;
END $$;`
);
await engine.runMigration(
103,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS content_chunks_stale_idx
ON content_chunks (page_id, chunk_index)
WHERE embedding IS NULL;`
);
} else {
await engine.runMigration(
103,
`CREATE INDEX IF NOT EXISTS content_chunks_stale_idx
ON content_chunks (page_id, chunk_index)
WHERE embedding IS NULL;`
);
}
},
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+71
View File
@@ -0,0 +1,71 @@
// src/core/minion-spend.ts
// v0.41.18.0 (A7 + A23, codex finding #4).
//
// Generic Minion handlers (embed-catch-up, extract-ner,
// extract-timeline-from-meetings, extract-takes-from-pages) need to
// settle their LLM/API spend against the originating OAuth client's
// mcp_spend_log row when submitted via the MCP run_onboard op
// (admin scope). Pre-fix, only subagent loops via budget-meter.ts
// recorded spend; generic handlers wrote to the gateway without any
// per-client attribution.
//
// Convention chosen for v0.42.0: the originating client_id is stored on
// job.data.client_id when run_onboard submits. The schema column for
// minion_jobs.client_id is deferred to v0.42.1 (would require a v101
// migration + index). For now: handlers that spend LLM/embedding budget
// call recordMinionJobSpend(engine, job, ...) which reads job.data.client_id
// and writes to mcp_spend_log with the right attribution.
//
// Best-effort throughout: spend telemetry MUST NOT fail the user's call.
import type { BrainEngine } from './engine.ts';
import { recordSpend } from './spend-log.ts';
export interface MinionJobLike {
id: number;
data?: Record<string, unknown> | unknown;
}
/**
* Read the OAuth client_id (if any) that submitted this Minion job.
* Returns undefined for jobs submitted locally (CLI, autopilot tick)
* those bypass the per-client spend cap.
*/
export function getJobClientId(job: MinionJobLike): string | undefined {
if (!job.data || typeof job.data !== 'object') return undefined;
const data = job.data as Record<string, unknown>;
const cid = data.client_id;
return typeof cid === 'string' && cid.length > 0 ? cid : undefined;
}
/**
* Record spend for a Minion job. Handler-side; called after each LLM /
* embedding API call settles. Threads the originating MCP client_id
* (when present) into mcp_spend_log so per-client caps are enforced
* across the parent run_onboard child handler chain.
*
* Local handler runs (no client_id) record with clientId=null
* the row still lands for global accounting but doesn't count against
* any specific OAuth client's daily cap.
*/
export async function recordMinionJobSpend(
engine: BrainEngine,
job: MinionJobLike,
entry: {
operation: string;
spendCents: number;
provider?: string;
model?: string;
tokenName?: string;
},
): Promise<void> {
const clientId = getJobClientId(job);
await recordSpend(engine, {
clientId: clientId ?? null,
tokenName: entry.tokenName ?? null,
operation: entry.operation,
spendCents: entry.spendCents,
provider: entry.provider,
model: entry.model,
});
}
+6
View File
@@ -37,6 +37,12 @@ export const PROTECTED_JOB_NAMES: ReadonlySet<string> = new Set([
// budget. Only trusted local callers (the mode-switch hook in
// commands/config.ts, reindex sweep, doctor --remediate) can submit.
'contextual_reindex_per_chunk',
// v0.41.18.0 (A12, T9) — takes-bootstrap. Per-page Haiku classifier
// call over concept/atom/lore/briefing/writing/originals. Two-gate
// consent (takes.bootstrap_enabled + --yes) AND PROTECTED ensures
// no remote / MCP / autopilot path can bulk-extract takes without
// explicit operator intent.
'extract-takes-from-pages',
]);
/** Check a job name against the protected set. Normalizes whitespace first. */
+368
View File
@@ -0,0 +1,368 @@
// src/core/onboard/checks.ts
// sourcescope:file-brain-wide — every SQL site here is intentionally
// brain-wide aggregate. The onboard checks REPORT across all sources
// (orphan_count, stale_count, link_coverage, takes_count) so adding
// source_id WHERE clauses would change the semantic. Per A26.
//
// v0.41.18.0 (A16, T4). Four new doctor checks consumed by both:
// - src/commands/doctor.ts runDoctor (local surface)
// - src/core/doctor-remote.ts (thin-client surface)
// - src/core/onboard/plan-from-checks.ts (onboard remediation aggregator)
//
// Each helper is shaped: compute metric → return both Check entry (for
// doctor render) and RemediationStep[] (for onboard's extra-remediation
// plumbing per A2). Helpers stay PURE wrt config: no engine.connect, no
// process.exit. SQL via engine.executeRaw with `sourceScopeOpts(ctx)`
// when ctx threads — onboard surface threads explicitly per A26.
import type { BrainEngine } from '../engine.ts';
import type { RemediationStep } from '../remediation-step.ts';
import { makeRemediationStep } from '../remediation-step.ts';
/** Shared shape returned by all four checks. */
export interface OnboardCheckResult {
check: {
name: string;
status: 'ok' | 'warn' | 'fail';
message: string;
};
remediations: RemediationStep[];
}
/** Internal sql helper. Returns first row or empty object on throw. */
async function safeCount(engine: BrainEngine, sql: string, params: unknown[] = []): Promise<number> {
try {
const result = await engine.executeRaw(sql, params);
const rows = (result as { rows?: Array<Record<string, unknown>> } | undefined)?.rows
?? (result as Array<Record<string, unknown>> | undefined)
?? [];
const row = rows[0] ?? {};
const raw = (row as Record<string, unknown>).count ?? (row as Record<string, unknown>).c ?? 0;
const n = Number(raw);
return Number.isFinite(n) ? n : 0;
} catch {
return 0;
}
}
/**
* embed_staleness: count of chunks awaiting embedding.
*
* Backed by content_chunks_stale_idx partial index (v100) so the count
* is cheap even on big brains.
*/
export async function checkEmbedStaleness(
engine: BrainEngine,
): Promise<OnboardCheckResult> {
const staleCount = await safeCount(
engine,
`SELECT COUNT(*) AS count FROM content_chunks WHERE embedding IS NULL`,
);
const remediations: RemediationStep[] = [];
let status: 'ok' | 'warn' | 'fail' = 'ok';
let message: string;
if (staleCount === 0) {
message = 'No stale chunks';
} else if (staleCount < 1000) {
status = 'warn';
message = `${staleCount} stale chunks (small backlog)`;
remediations.push(makeRemediationStep({
id: 'onboard.embed_catch_up',
job: 'embed-catch-up',
params: { batchSize: 500 },
severity: 'medium',
est_seconds: Math.min(900, Math.ceil(staleCount * 0.2)),
est_usd_cost: staleCount * 0.00002,
rationale: `${staleCount} chunks awaiting embedding`,
status: 'remediable',
}));
} else {
// v0.41.18.0: warn-only even on large backlogs. Doctor exit code should
// not flip from a brain that has pages waiting to be embedded — that's
// a "needs work" condition, not a "broken" one. The high-severity
// remediation still surfaces via onboard's plan.
status = 'warn';
message = `${staleCount} stale chunks (large backlog — vector search returning outdated content)`;
remediations.push(makeRemediationStep({
id: 'onboard.embed_catch_up',
job: 'embed-catch-up',
params: { batchSize: 1000, priority: 'recent' },
severity: 'high',
est_seconds: Math.min(3600, Math.ceil(staleCount * 0.2)),
est_usd_cost: staleCount * 0.00002,
rationale: `${staleCount} chunks awaiting embedding; recent-first catch-up`,
status: 'remediable',
}));
}
return {
check: { name: 'embed_staleness', status, message },
remediations,
};
}
/**
* entity_link_coverage: fraction of entity pages with at least one inbound link.
*
* Per A21 + codex finding #15: TABLESAMPLE BERNOULLI on Postgres when
* total_pages > 50K, with pinned sample rate (LEAST 100, GREATEST 2,
* targeting ~5000 sampled rows). PGLite path: full scan.
*
* The ±sqrt(p(1-p)/n) confidence interval is embedded in the message
* itself so doctor + onboard render show "coverage: 31% ± 1.3%" not
* a misleading point estimate.
*/
export async function checkEntityLinkCoverage(
engine: BrainEngine,
): Promise<OnboardCheckResult> {
// Total entity pages
const totalEntities = await safeCount(
engine,
`SELECT COUNT(*) AS count FROM pages
WHERE type IN ('person', 'company', 'organization', 'entity')
AND deleted_at IS NULL`,
);
if (totalEntities === 0) {
return {
check: { name: 'entity_link_coverage', status: 'ok', message: 'No entity pages — coverage check vacuous' },
remediations: [],
};
}
// Decide TABLESAMPLE policy (PG only, when >50K entities)
const useSample = engine.kind === 'postgres' && totalEntities > 50_000;
const samplePct = useSample
? Math.max(2.0, Math.min(100.0, (5000.0 / totalEntities) * 100))
: 100;
const sampleClause = useSample ? `TABLESAMPLE BERNOULLI (${samplePct.toFixed(2)})` : '';
// Sample query: counts entities with inbound links
const linkedCount = await safeCount(
engine,
`SELECT COUNT(*) AS count FROM (
SELECT p.id FROM pages p ${sampleClause}
WHERE p.type IN ('person', 'company', 'organization', 'entity')
AND p.deleted_at IS NULL
AND EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
) sub`,
);
const sampleSize = useSample
? Math.max(1, Math.round(totalEntities * (samplePct / 100)))
: totalEntities;
const coverage = sampleSize > 0 ? linkedCount / sampleSize : 0;
// Wilson-ish confidence interval (1σ; ±sqrt(p(1-p)/n))
const ci = Math.sqrt((coverage * (1 - coverage)) / Math.max(1, sampleSize));
const pct = Math.round(coverage * 100);
const ciPct = (ci * 100).toFixed(1);
const sampleNote = useSample ? ` (sampled ${samplePct.toFixed(1)}%)` : '';
const remediations: RemediationStep[] = [];
let status: 'ok' | 'warn' | 'fail' = 'ok';
let message: string;
// v0.41.18.0: warn-only, never fail. Empty entity link coverage is "needs
// work" not "broken" — doctor's exit code should not flip from a fresh
// brain with entity pages but no auto-extracted links yet. Fail status
// would break `gbrain doctor exits 0` contract; the recommendation
// surfaces the same fix via the onboard plan either way.
if (coverage >= 0.7) {
message = `Coverage ${pct}% ± ${ciPct}%${sampleNote}`;
} else if (coverage >= 0.4) {
status = 'warn';
message = `Coverage ${pct}% ± ${ciPct}% (target 70%)${sampleNote}`;
remediations.push(makeRemediationStep({
id: 'onboard.extract_ner_links',
job: 'extract-ner',
params: {},
severity: 'medium',
est_seconds: 300,
est_usd_cost: 0,
rationale: `Entity link coverage at ${pct}%; NER extraction lifts typed-link density`,
status: 'remediable',
}));
} else {
status = 'warn';
message = `Coverage ${pct}% ± ${ciPct}% (target 70%)${sampleNote}`;
remediations.push(makeRemediationStep({
id: 'onboard.extract_ner_links',
job: 'extract-ner',
params: {},
severity: 'high',
est_seconds: 600,
est_usd_cost: 0,
rationale: `Entity link coverage at ${pct}%; NER extraction lifts typed-link density`,
status: 'remediable',
}));
}
return {
check: { name: 'entity_link_coverage', status, message },
remediations,
};
}
/**
* timeline_coverage: fraction of entity pages with at least one timeline entry.
*
* Same TABLESAMPLE policy as entity_link_coverage for big brains.
*/
export async function checkTimelineCoverage(
engine: BrainEngine,
): Promise<OnboardCheckResult> {
const totalEntities = await safeCount(
engine,
`SELECT COUNT(*) AS count FROM pages
WHERE type IN ('person', 'company', 'organization', 'entity')
AND deleted_at IS NULL`,
);
if (totalEntities === 0) {
return {
check: { name: 'timeline_coverage', status: 'ok', message: 'No entity pages — coverage check vacuous' },
remediations: [],
};
}
const useSample = engine.kind === 'postgres' && totalEntities > 50_000;
const samplePct = useSample
? Math.max(2.0, Math.min(100.0, (5000.0 / totalEntities) * 100))
: 100;
const sampleClause = useSample ? `TABLESAMPLE BERNOULLI (${samplePct.toFixed(2)})` : '';
const withTimelineCount = await safeCount(
engine,
`SELECT COUNT(*) AS count FROM (
SELECT p.id FROM pages p ${sampleClause}
WHERE p.type IN ('person', 'company', 'organization', 'entity')
AND p.deleted_at IS NULL
AND EXISTS (SELECT 1 FROM timeline_entries t WHERE t.page_id = p.id)
) sub`,
);
const sampleSize = useSample
? Math.max(1, Math.round(totalEntities * (samplePct / 100)))
: totalEntities;
const coverage = sampleSize > 0 ? withTimelineCount / sampleSize : 0;
const ci = Math.sqrt((coverage * (1 - coverage)) / Math.max(1, sampleSize));
const pct = Math.round(coverage * 100);
const ciPct = (ci * 100).toFixed(1);
const sampleNote = useSample ? ` (sampled ${samplePct.toFixed(1)}%)` : '';
const remediations: RemediationStep[] = [];
let status: 'ok' | 'warn' | 'fail' = 'ok';
let message: string;
// v0.41.18.0: warn-only, never fail. Same posture as entity_link_coverage —
// the recommendation still surfaces in onboard's plan, but doctor exit
// code doesn't flip on a fresh brain.
if (coverage >= 0.9) {
message = `Coverage ${pct}% ± ${ciPct}%${sampleNote}`;
} else if (coverage >= 0.7) {
status = 'warn';
message = `Coverage ${pct}% ± ${ciPct}% (target 90%)${sampleNote}`;
remediations.push(makeRemediationStep({
id: 'onboard.extract_timeline_from_meetings',
job: 'extract-timeline-from-meetings',
params: {},
severity: 'medium',
est_seconds: 240,
est_usd_cost: 0,
rationale: `Timeline coverage at ${pct}%; meeting-derived entries lift it`,
status: 'remediable',
}));
} else {
status = 'warn';
message = `Coverage ${pct}% ± ${ciPct}% (target 90%)${sampleNote}`;
remediations.push(makeRemediationStep({
id: 'onboard.extract_timeline_from_meetings',
job: 'extract-timeline-from-meetings',
params: {},
severity: 'high',
est_seconds: 480,
est_usd_cost: 0,
rationale: `Timeline coverage at ${pct}%; meeting-derived entries lift it`,
status: 'remediable',
}));
}
return {
check: { name: 'timeline_coverage', status, message },
remediations,
};
}
/**
* takes_count: number of takes (typed claims) in the brain.
*
* Per A12 two-gate consent: the remediation only emits when
* `takes.bootstrap_enabled` config is true. Otherwise the check shows
* a status + hint, but no autopilot-eligible remediation.
*/
export async function checkTakesCount(
engine: BrainEngine,
): Promise<OnboardCheckResult> {
const takesCount = await safeCount(
engine,
`SELECT COUNT(*) AS count FROM takes`,
);
let bootstrapEnabled = false;
try {
const cfg = await engine.getConfig('takes.bootstrap_enabled');
bootstrapEnabled = cfg === 'true' || cfg === '1';
} catch {
bootstrapEnabled = false;
}
const remediations: RemediationStep[] = [];
let status: 'ok' | 'warn' | 'fail' = 'ok';
let message: string;
if (takesCount >= 100) {
message = `${takesCount} takes (calibration ready)`;
} else if (takesCount === 0) {
status = 'warn';
if (bootstrapEnabled) {
message = `0 takes (bootstrap eligible — gbrain takes extract --from-pages)`;
remediations.push(makeRemediationStep({
id: 'onboard.takes_bootstrap',
job: 'extract-takes-from-pages',
protected: true,
params: {},
severity: 'medium',
est_seconds: 1800,
est_usd_cost: 5.00,
rationale: '0 takes; LLM-bearing extraction over concept/atom/lore pages',
status: 'remediable',
}));
} else {
message = '0 takes (takes.bootstrap_enabled is false; opt in to enable)';
}
} else {
message = `${takesCount} takes (calibration usable; >100 ideal)`;
}
return {
check: { name: 'takes_count', status, message },
remediations,
};
}
/**
* Run all four checks in parallel; aggregate into a single payload.
* Consumed by onboard's plan generation + (later) doctor's runDoctor.
*
* Per A20: callers can race this against an AbortSignal-bound timer for
* partial-results fallthrough. Each individual safeCount() returns 0
* on throw so a single check failure doesn't break the aggregate.
*/
export async function runAllOnboardChecks(
engine: BrainEngine,
): Promise<OnboardCheckResult[]> {
return Promise.all([
checkEmbedStaleness(engine),
checkEntityLinkCoverage(engine),
checkTimelineCoverage(engine),
checkTakesCount(engine),
]);
}
+188
View File
@@ -0,0 +1,188 @@
// src/core/onboard/impact-capture.ts
// sourcescope:file-brain-wide — captureMetric reports brain-wide
// aggregates (orphan_count, stale_count, coverage fractions) by design.
// Per A26 lint opt-out.
//
// v0.41.18.0 (A6 + A25 + A17, T11). Capture before/after stats per onboard
// remediation step so `gbrain onboard --history` can show "you reduced
// orphans 47% (88% → 41%)".
//
// Best-effort per A17: a stat-query throw must NOT block the extraction
// itself. The wrapper logs failures to stderr and records
// metric_before/after = null when the capture failed.
//
// Attribution columns per A25 + codex finding #10: every row carries
// job_id (FK to minion_jobs), source_id, brain_id, started_at,
// idempotency_key so concurrent onboard/autopilot/manual runs can't
// misattribute deltas to the wrong remediation.
import type { BrainEngine } from './../engine.ts';
export type MetricName =
| 'orphan_count'
| 'stale_count'
| 'entity_link_coverage'
| 'timeline_coverage'
| 'takes_count';
export interface ImpactAttribution {
remediation_id: string;
job_id?: number;
source_id?: string;
brain_id?: string;
started_at?: string;
idempotency_key?: string;
applied_by?: string;
}
/**
* Pure-ish: returns the current numeric value for `metric`. Returns null
* on any throw (best-effort capture per A17).
*/
export async function captureMetric(
engine: BrainEngine,
metric: MetricName,
): Promise<number | null> {
try {
switch (metric) {
case 'stale_count': {
const rows = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM content_chunks WHERE embedding IS NULL`,
);
return rows.length > 0 ? Number(rows[0].count) : 0;
}
case 'orphan_count': {
const rows = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count
FROM pages p
WHERE p.deleted_at IS NULL
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)`,
);
return rows.length > 0 ? Number(rows[0].count) : 0;
}
case 'entity_link_coverage':
case 'timeline_coverage': {
// Compute as a fraction of entity pages with the relevant feature.
const total = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages
WHERE type IN ('person', 'company', 'organization', 'entity')
AND deleted_at IS NULL`,
);
const totalN = total.length > 0 ? Number(total[0].count) : 0;
if (totalN === 0) return 1; // vacuous truth — empty brain has full coverage
if (metric === 'entity_link_coverage') {
const withLinks = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages p
WHERE p.type IN ('person', 'company', 'organization', 'entity')
AND p.deleted_at IS NULL
AND EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)`,
);
return withLinks.length > 0 ? Number(withLinks[0].count) / totalN : 0;
}
const withTimeline = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages p
WHERE p.type IN ('person', 'company', 'organization', 'entity')
AND p.deleted_at IS NULL
AND EXISTS (SELECT 1 FROM timeline_entries t WHERE t.page_id = p.id)`,
);
return withTimeline.length > 0 ? Number(withTimeline[0].count) / totalN : 0;
}
case 'takes_count': {
const rows = await engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM takes`,
);
return rows.length > 0 ? Number(rows[0].count) : 0;
}
}
} catch (err) {
process.stderr.write(
`[impact-capture] failed to capture ${metric}: ${err instanceof Error ? err.message : String(err)}\n`,
);
return null;
}
}
/**
* Write one migration_impact_log row. Best-effort: a write failure logs
* to stderr but doesn't throw.
*/
export async function writeImpactLogRow(
engine: BrainEngine,
attribution: ImpactAttribution,
metricName: MetricName,
metricBefore: number | null,
metricAfter: number | null,
details?: Record<string, unknown>,
): Promise<void> {
try {
await engine.executeRaw(
`INSERT INTO migration_impact_log (
remediation_id, metric_name, metric_before, metric_after,
job_id, source_id, brain_id, started_at, idempotency_key,
applied_by, details
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb)`,
[
attribution.remediation_id,
metricName,
metricBefore,
metricAfter,
attribution.job_id ?? null,
attribution.source_id ?? null,
attribution.brain_id ?? null,
attribution.started_at ?? new Date().toISOString(),
attribution.idempotency_key ?? null,
attribution.applied_by ?? null,
JSON.stringify(details ?? {}),
],
);
} catch (err) {
process.stderr.write(
`[impact-capture] failed to write log row for ${attribution.remediation_id}: ${err instanceof Error ? err.message : String(err)}\n`,
);
}
}
/**
* Convenience wrapper: capture-before run capture-after write log.
* The runner itself does the extraction; this fn handles the bookkeeping.
*
* Per A17: capture failures DO NOT block the runner. A null before/after
* is recorded; the row still lands so downstream consumers see a
* "ran but impact unknown" entry.
*/
export async function withImpactCapture<T>(
engine: BrainEngine,
attribution: ImpactAttribution,
metric: MetricName,
runner: () => Promise<T>,
details?: Record<string, unknown>,
): Promise<T> {
const startedAt = new Date().toISOString();
const before = await captureMetric(engine, metric);
let result: T;
try {
result = await runner();
} catch (err) {
// Capture "after" even on failure so the log row reflects the attempt.
const afterOnFail = await captureMetric(engine, metric);
await writeImpactLogRow(
engine,
{ ...attribution, started_at: startedAt },
metric,
before,
afterOnFail,
{ ...(details ?? {}), error: err instanceof Error ? err.message : String(err) },
);
throw err;
}
const after = await captureMetric(engine, metric);
await writeImpactLogRow(
engine,
{ ...attribution, started_at: startedAt },
metric,
before,
after,
details,
);
return result;
}
+152
View File
@@ -0,0 +1,152 @@
// src/core/onboard/init-nudge.ts
// v0.41.18.0 (A4 + A18 + A20, T14). Post-initSchema summary that runs
// the 4 onboard checks against a 3-second wallclock budget and prints
// a one-line nudge if recommendations exist.
//
// Hard contract per A18: init MUST succeed even if the nudge crashes.
// Any throw in this module is caught + logged to stderr + suppressed.
// Per A20: the 3-second cap uses real cancellation via the AbortSignal
// extension on executeRaw (T5) — Promise.race against a timer was the
// codex #7 finding's wrong shape. Cancelled queries actually stop on
// Postgres; PGLite has a documented gap.
//
// Bypass: GBRAIN_NO_ONBOARD_NUDGE=1 short-circuits. Non-TTY default
// also short-circuits (CI/scripted callers see nothing).
import type { BrainEngine } from '../engine.ts';
const NUDGE_BUDGET_MS = 3000;
/**
* Post-initSchema nudge. Fail-open per A18.
*
* Returns silently when:
* - GBRAIN_NO_ONBOARD_NUDGE=1
* - Non-TTY environment (CI, scripted)
* - All 4 onboard checks complete within 3s AND surface 0 recommendations
* - ANY error during check execution (logged to stderr, suppressed)
*
* Prints a nudge to stderr when:
* - Recommendations exist within budget
* - Some checks ran but budget fired (partial-results path)
*/
export async function runInitNudge(engine: BrainEngine): Promise<void> {
try {
if (process.env.GBRAIN_NO_ONBOARD_NUDGE === '1') return;
if (!process.stderr.isTTY) return;
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), NUDGE_BUDGET_MS);
let totalStale = 0;
let totalEntities = 0;
let linkedCount = 0;
let timelineCount = 0;
let takesCount = 0;
let checksRan = 0;
let checksAttempted = 0;
let partial = false;
// Run 4 cheap counts in parallel against the 3s budget.
const results = await Promise.allSettled([
engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM content_chunks WHERE embedding IS NULL`,
[],
{ signal: controller.signal },
),
engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages
WHERE type IN ('person', 'company', 'organization', 'entity')
AND deleted_at IS NULL`,
[],
{ signal: controller.signal },
),
engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages p
WHERE p.type IN ('person', 'company', 'organization', 'entity')
AND p.deleted_at IS NULL
AND EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)`,
[],
{ signal: controller.signal },
),
engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM pages p
WHERE p.type IN ('person', 'company', 'organization', 'entity')
AND p.deleted_at IS NULL
AND EXISTS (SELECT 1 FROM timeline_entries t WHERE t.page_id = p.id)`,
[],
{ signal: controller.signal },
),
engine.executeRaw<{ count: string | number }>(
`SELECT COUNT(*) AS count FROM takes`,
[],
{ signal: controller.signal },
),
]);
clearTimeout(timer);
checksAttempted = results.length;
for (let i = 0; i < results.length; i++) {
const r = results[i];
if (r.status === 'rejected') {
partial = true;
continue;
}
checksRan++;
const n = r.value.length > 0 ? Number(r.value[0].count) : 0;
if (i === 0) totalStale = n;
else if (i === 1) totalEntities = n;
else if (i === 2) linkedCount = n;
else if (i === 3) timelineCount = n;
else if (i === 4) takesCount = n;
}
// Aggregate: any non-zero metric triggers the nudge.
const linkCoverage = totalEntities > 0 ? linkedCount / totalEntities : 1;
const timelineCoverage = totalEntities > 0 ? timelineCount / totalEntities : 1;
const hasRecommendations =
totalStale > 0
|| (totalEntities > 0 && linkCoverage < 0.7)
|| (totalEntities > 0 && timelineCoverage < 0.9)
|| takesCount === 0;
if (!hasRecommendations && !partial) return;
// Emit one-line nudge. Be terse — init is the activation surface.
const parts: string[] = [];
if (totalStale > 0) parts.push(`${totalStale} stale chunks`);
if (totalEntities > 0 && linkCoverage < 0.7) {
parts.push(`link coverage ${Math.round(linkCoverage * 100)}%`);
}
if (totalEntities > 0 && timelineCoverage < 0.9) {
parts.push(`timeline coverage ${Math.round(timelineCoverage * 100)}%`);
}
if (takesCount === 0) parts.push('0 takes');
process.stderr.write(
`\n[onboard] Brain has opportunities: ${parts.join(', ')}.\n` +
`[onboard] Run 'gbrain onboard --check' to see the plan.` +
(partial ? ` (${checksRan}/${checksAttempted} checks complete; run gbrain onboard --check for full recommendations)` : '') +
`\n`,
);
} catch (err) {
// A18: NEVER crash init from the nudge. Log and continue.
process.stderr.write(`[onboard] nudge skipped (${err instanceof Error ? err.message : String(err)})\n`);
}
}
/**
* Post-upgrade banner. Lighter than the init nudge just highlights
* that new onboard recommendations may exist. Fail-open identically.
*/
export async function runUpgradeBanner(_engine: BrainEngine): Promise<void> {
try {
if (process.env.GBRAIN_NO_ONBOARD_NUDGE === '1') return;
if (!process.stderr.isTTY) return;
process.stderr.write(
`\n[onboard] Upgrade complete. Run 'gbrain onboard --check' to see if the new version surfaces any new opportunities.\n`,
);
} catch {
// A18 posture for symmetry.
}
}
+108
View File
@@ -0,0 +1,108 @@
// src/core/onboard/render.ts
// v0.41.18.0 (T12). Stable JSON envelope + human renderer for
// `gbrain onboard`. Library-shaped — no console.* / process.exit; CLI
// shell calls these and pipes results to its own output.
import type { RemediationStep } from '../remediation-step.ts';
import type { RemediationPlan } from '../remediation/types.ts';
import type {
OnboardRecommendation,
OnboardReport,
} from './types.ts';
/**
* Translate a RemediationStep into an OnboardRecommendation. Layers the
* apply_policy + prompt_text + migration_id metadata.
*
* Rules of thumb for apply_policy:
* - protected job (LLM-bearing) 'prompt_required' or 'manual_only'
* based on job name (takes-bootstrap stays manual_only per A12).
* - non-protected (regex, SQL, etc.) 'auto_apply'.
*/
export function toOnboardRecommendation(step: RemediationStep): OnboardRecommendation {
let apply_policy: OnboardRecommendation['apply_policy'] = 'auto_apply';
if (step.protected) {
// takes-bootstrap classifier stays manual_only per A12 + A24 until
// v0.42.1 lands the 100+-case eval. All other protected handlers
// (synthesize, patterns, consolidate, extract-takes-from-pages)
// are prompt_required — they need --yes but can run via --auto --yes.
apply_policy = step.job === 'extract-takes-from-pages' ? 'manual_only' : 'prompt_required';
}
return {
...step,
apply_policy,
prompt_text: step.rationale,
migration_id: step.id,
};
}
/**
* Build the stable JSON envelope from a remediation plan. brainId, when
* available, identifies the brain across runs (consumed by --history
* cross-runtime joins).
*/
export function buildOnboardReport(
plan: RemediationPlan,
opts?: { brainId?: string; history?: OnboardReport['history'] },
): OnboardReport {
const recs = plan.plan.map(toOnboardRecommendation);
const summary = {
total: recs.length,
auto_eligible: recs.filter((r) => r.apply_policy === 'auto_apply').length,
prompt_required: recs.filter((r) => r.apply_policy === 'prompt_required').length,
manual_only: recs.filter((r) => r.apply_policy === 'manual_only').length,
est_total_usd: plan.est_total_usd_cost,
};
return {
schema_version: 1,
brain_id: opts?.brainId,
recommendations: recs,
summary,
history: opts?.history,
};
}
/**
* Human-readable render (returns string; CLI prints to stdout). Designed
* for stderr/stdout segregation: the CLI shell prints this on stdout,
* progress + errors on stderr. Echoes the AskUserQuestion-style
* "Recommendation + WHY" framing the CEO/Eng review settled on.
*/
export function renderHuman(report: OnboardReport): string {
const lines: string[] = [];
lines.push(`Brain onboarding: ${report.summary.total} recommendation(s) found`);
if (report.summary.total === 0) {
lines.push(' Brain is at target — nothing to do.');
return lines.join('\n');
}
lines.push(
` ${report.summary.auto_eligible} auto-eligible | ` +
`${report.summary.prompt_required} prompt-required | ` +
`${report.summary.manual_only} manual-only`,
);
if (report.summary.est_total_usd > 0) {
lines.push(` Total estimated cost: $${report.summary.est_total_usd.toFixed(2)}`);
}
lines.push('');
for (const r of report.recommendations) {
const sev = `[${r.severity}]`;
const policy = r.apply_policy === 'auto_apply' ? '(auto)'
: r.apply_policy === 'prompt_required' ? '(prompt)'
: '(manual)';
const cost = (r.est_usd_cost ?? 0) > 0 ? ` ~$${(r.est_usd_cost ?? 0).toFixed(2)}` : '';
lines.push(` ${sev} ${policy} ${r.job}${cost}`);
lines.push(` why: ${r.prompt_text ?? r.rationale}`);
}
if (report.history && report.history.length > 0) {
lines.push('');
lines.push('Recent impact (last 10):');
for (const h of report.history.slice(0, 10)) {
const delta = h.delta !== null ? (h.delta > 0 ? `+${h.delta}` : String(h.delta)) : '?';
lines.push(
` ${h.applied_at} ${h.remediation_id} ${h.metric_name}: ` +
`${h.metric_before ?? '?'}${h.metric_after ?? '?'} (${delta})`,
);
}
}
return lines.join('\n');
}
+68
View File
@@ -0,0 +1,68 @@
// src/core/onboard/types.ts
// v0.41.18.0 (T12). Onboard-surface-specific types layered on top of the
// shared remediation library (src/core/remediation/). Onboard reframes
// the same RemediationStep objects as "onboarding opportunities" with
// extra prompt-text + apply-policy metadata.
import type { RemediationStep } from '../remediation-step.ts';
/**
* One onboard recommendation. Layered on RemediationStep with extras
* specific to the onboarding UX:
* - apply_policy controls autopilot behavior (A8 tiered)
* - prompt_text is the human-readable nudge ("Embed 3K stale chunks?")
* - migration_id is a stable identifier for cross-version tracking
*/
export interface OnboardRecommendation extends RemediationStep {
/**
* A8 tiered apply policy:
* 'auto_apply' autopilot may run unattended; runs under --auto
* 'prompt_required' autopilot must skip; runs under --auto --yes
* 'manual_only' never runs unattended; CLI prompts user
* Default 'prompt_required' when omitted.
*/
apply_policy?: 'auto_apply' | 'prompt_required' | 'manual_only';
/** Human-readable nudge text. Default falls back to RemediationStep.rationale. */
prompt_text?: string;
/**
* Stable id for cross-version tracking. Defaults to RemediationStep.id.
* Used by onboard --history to group runs of the same migration over time.
*/
migration_id?: string;
}
/** Stable JSON envelope for `gbrain onboard --json`. */
export interface OnboardReport {
schema_version: 1;
/** brain_id from engine.getHealth(). Identifies the brain across runs. */
brain_id?: string;
recommendations: OnboardRecommendation[];
summary: {
total: number;
auto_eligible: number;
prompt_required: number;
manual_only: number;
/** Estimated total dollar cost if every recommendation runs. */
est_total_usd: number;
};
/** Reverse-chronological migration_impact_log entries. */
history?: Array<{
remediation_id: string;
metric_name: string;
metric_before: number | null;
metric_after: number | null;
delta: number | null;
applied_at: string;
}>;
}
export interface OnboardOpts {
/** Target brain_score (default 90). Forwarded to computeRemediationPlan. */
targetScore?: number;
/** Output mode. */
mode?: 'check' | 'auto' | 'history';
/** Cap on autopilot spend. Required for --auto runs (CLI enforces). */
maxUsd?: number;
/** Caller-supplied OAuth client_id (MCP path); threads via job.data. */
clientId?: string;
}
+108
View File
@@ -4168,6 +4168,112 @@ const reload_schema_pack: Operation = {
},
};
// v0.41.18.0 (A7 + T16, codex finding #5): MCP op for federated / thin-client
// brain installs to drive `gbrain onboard --auto` over MCP. Admin scope
// (NOT localOnly) so remote agents authenticated via OAuth can probe
// brain health + submit auto-eligible remediation handlers.
//
// Critical security gate (codex #5): admin scope alone is NOT sufficient
// to submit handlers in PROTECTED_JOB_NAMES (synthesize, patterns,
// consolidate, extract-takes-from-pages, contextual_reindex_per_chunk).
// Without this gate, an admin-scoped OAuth token would bypass the same
// guard that `submit_job` enforces. The new NAMED scope
// `run_protected_onboard` MUST be granted IN ADDITION TO admin for any
// protected child handler to fire.
//
// Behavior:
// - mode='check' (default): returns the OnboardReport JSON envelope,
// never submits jobs. Admin scope sufficient.
// - mode='auto': submits auto_apply tier. Admin + non-protected
// handlers only.
// - mode='auto-with-prompt': submits auto_apply + prompt_required tier.
// Same protection check.
//
// Any LLM-bearing handler the plan would have submitted gets filtered out
// unless the caller has run_protected_onboard. Filtered items appear in
// the response with status='skipped_missing_scope' so the caller knows
// what they would have gotten with the right grants.
const run_onboard: Operation = {
name: 'run_onboard',
description: 'Probe brain health + optionally submit onboard remediations. Admin scope required. Protected handlers (LLM-bearing) require run_protected_onboard scope ADDITIONALLY.',
params: {
mode: { type: 'string', description: "'check' (default), 'auto', or 'auto-with-prompt'" },
target_score: { type: 'number', description: 'Target brain_score (default 90)' },
max_usd: { type: 'number', description: 'USD cap for autopilot path (required for auto modes)' },
},
mutating: true,
scope: 'admin',
handler: async (ctx, p) => {
const mode = (typeof p.mode === 'string' ? p.mode : 'check') as 'check' | 'auto' | 'auto-with-prompt';
const targetScore = typeof p.target_score === 'number' ? p.target_score : 90;
const maxUsd = typeof p.max_usd === 'number' ? p.max_usd : undefined;
const { computeRemediationPlan, runRemediation } = await import('./remediation/index.ts');
const { runAllOnboardChecks } = await import('./onboard/checks.ts');
const { buildOnboardReport } = await import('./onboard/render.ts');
// Per A26: source-scope via sourceScopeOpts(ctx). The recommendation
// planner is brain-wide today; future extension can scope by reading
// ctx.sourceId / ctx.auth.allowedSources for per-source plans.
let extraRemediations: import('./remediation-step.ts').RemediationStep[] = [];
try {
const checkResults = await runAllOnboardChecks(ctx.engine);
extraRemediations = checkResults.flatMap((r) => r.remediations);
} catch {
// Fail-open per A19 — return plan without extras rather than error.
}
// 'check' mode: just return the plan + JSON envelope. No submission.
if (mode === 'check') {
const plan = await computeRemediationPlan(ctx.engine, { targetScore, extraRemediations });
const report = buildOnboardReport(plan);
return report;
}
// 'auto' and 'auto-with-prompt' modes: require --max-usd per A12 + A20
// safety posture (cron-safety; refuses surprise spend).
if (maxUsd === undefined) {
throw new OperationError('invalid_params', `mode='${mode}' requires max_usd (cron-safety cap)`);
}
// Critical T16 + codex #5 security gate: filter out PROTECTED_JOB_NAMES
// unless the caller has the run_protected_onboard scope IN ADDITION
// to admin. Admin alone is insufficient.
const grantedScopes = ctx.auth?.scopes ?? [];
const canRunProtected = grantedScopes.includes('run_protected_onboard');
const { isProtectedJobName } = await import('./minions/protected-names.ts');
const skippedMissingScope: Array<{ id: string; job: string; reason: string }> = [];
const allowedExtras = extraRemediations.filter((r) => {
if (canRunProtected) return true;
if (isProtectedJobName(r.job)) {
skippedMissingScope.push({ id: r.id, job: r.job, reason: 'requires run_protected_onboard scope' });
return false;
}
return true;
});
// Run remediation with filtered extras. Hooks emit nothing — MCP
// returns structured result. Per A23 client_id attribution: stamp
// job.data.client_id on each submission so the spend chain (T10)
// attributes correctly. The library doesn't do this today; the
// upstream submit-side gating in submit_job filters protected names
// for ctx.remote !== false callers, so even if MCP run_onboard had a
// typo, the underlying queue.add would reject. Defense-in-depth.
const result = await runRemediation(
ctx.engine,
{ targetScore, maxUsd },
{},
);
return {
...result,
skipped_missing_scope: skippedMissingScope,
};
},
};
export const operations: Operation[] = [
// Page CRUD
get_page, put_page, delete_page, list_pages,
@@ -4236,6 +4342,8 @@ export const operations: Operation[] = [
schema_stats, schema_lint, schema_graph, schema_explain_type,
schema_review_orphans,
schema_apply_mutations, reload_schema_pack,
// v0.41.18.0 (T16, A7, codex #5)
run_onboard,
];
export const operationsByName = Object.fromEntries(
+105 -11
View File
@@ -2000,10 +2000,83 @@ export class PGLiteEngine implements BrainEngine {
afterPageId?: number;
afterChunkIndex?: number;
sourceId?: string;
orderBy?: 'page_id' | 'updated_desc';
afterUpdatedAt?: string | null;
}): Promise<StaleChunkRow[]> {
const limit = opts?.batchSize ?? 2000;
const afterPid = opts?.afterPageId ?? 0;
const afterIdx = opts?.afterChunkIndex ?? -1;
const orderBy = opts?.orderBy ?? 'page_id';
// v0.41.18.0 (A13, codex #9): --priority recent path. See postgres-engine
// sibling for full rationale. Same composite cursor + ORDER BY.
if (orderBy === 'updated_desc') {
const afterUpdated = opts?.afterUpdatedAt ?? null;
const isFirstPage = afterUpdated === null && afterPid === 0;
if (opts?.sourceId === undefined) {
const { rows } = isFirstPage ? await this.db.query(
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT $1`,
[limit],
) : await this.db.query(
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
p.updated_at < $1::timestamptz
OR (p.updated_at = $1::timestamptz AND p.id > $2)
OR (p.updated_at = $1::timestamptz AND p.id = $2 AND cc.chunk_index > $3)
)
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT $4`,
[afterUpdated, afterPid, afterIdx, limit],
);
return rows as unknown as StaleChunkRow[];
}
const { rows } = isFirstPage ? await this.db.query(
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND p.source_id = $1
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT $2`,
[opts.sourceId, limit],
) : await this.db.query(
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND p.source_id = $1
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
p.updated_at < $2::timestamptz
OR (p.updated_at = $2::timestamptz AND p.id > $3)
OR (p.updated_at = $2::timestamptz AND p.id = $3 AND cc.chunk_index > $4)
)
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT $5`,
[opts.sourceId, afterUpdated, afterPid, afterIdx, limit],
);
return rows as unknown as StaleChunkRow[];
}
// orderBy === 'page_id' — legacy stable cursor (unchanged below).
// D7: optional source-scoped cursor scan. PGLite mirrors postgres-engine
// so the engine-parity E2E catches drift.
// v0.41 (D4+D8): NOT (frontmatter ? 'embed_skip') filter for soft-blocked
@@ -2113,17 +2186,19 @@ export class PGLiteEngine implements BrainEngine {
const fromSourceIds = links.map(l => l.from_source_id || 'default');
const toSourceIds = links.map(l => l.to_source_id || 'default');
const originSourceIds = links.map(l => l.origin_source_id || 'default');
// v0.41.18.0 (A10): link_kind column (v98). NULL = legacy/plain.
const linkKinds = links.map(l => l.link_kind ?? null);
const result = await this.db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[], $6::text[], $7::text[], $8::text[], $9::text[], $10::text[])
AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id)
`INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, link_kind, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, v.link_kind, o.id, v.origin_field
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[], $6::text[], $7::text[], $8::text[], $9::text[], $10::text[], $11::text[])
AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id, link_kind)
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
RETURNING 1`,
[fromSlugs, toSlugs, linkTypes, contexts, linkSources, originSlugs, originFields, fromSourceIds, toSourceIds, originSourceIds]
[fromSlugs, toSlugs, linkTypes, contexts, linkSources, originSlugs, originFields, fromSourceIds, toSourceIds, originSourceIds, linkKinds]
);
return result.rows.length;
}
@@ -2501,7 +2576,7 @@ export class PGLiteEngine implements BrainEngine {
// Initialize all slugs to 0 so callers get a consistent map.
for (const s of slugs) result.set(s, 0);
// v0.42.0.0 D12: filter mentions OUT of backlink-count for search
// v0.41.18.0 D12: filter mentions OUT of backlink-count for search
// ranking — parity with postgres-engine.ts. See that file's comment
// for the full rationale. `IS DISTINCT FROM` is NULL-safe so legacy
// rows with NULL link_source still count toward backlinks.
@@ -2705,7 +2780,7 @@ export class PGLiteEngine implements BrainEngine {
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
SELECT id, $2::date, $3, $4, $5
FROM pages WHERE slug = $1 AND source_id = $6
ON CONFLICT (page_id, date, summary) DO NOTHING`,
ON CONFLICT (page_id, date, summary, source) DO NOTHING`,
[slug, entry.date, entry.source || '', entry.summary, entry.detail || '', sourceId]
);
}
@@ -2724,7 +2799,7 @@ export class PGLiteEngine implements BrainEngine {
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[], $6::text[])
AS v(slug, date, source, summary, detail, source_id)
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
ON CONFLICT (page_id, date, summary) DO NOTHING
ON CONFLICT (page_id, date, summary, source) DO NOTHING
RETURNING 1`,
[slugs, dates, sources, summaries, details, sourceIds]
);
@@ -4258,9 +4333,28 @@ export class PGLiteEngine implements BrainEngine {
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r, true));
}
async executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]> {
const { rows } = await this.db.query(sql, params);
return rows as T[];
async executeRaw<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
// v0.41.18.0 (A20, codex #7): PGLite is in-process WASM with no
// kernel-level cancellation. Best-effort: pre-check the signal so
// an already-aborted call returns immediately, and race against
// a settle promise so a late-arriving abort throws AbortError
// (the query keeps running in WASM until it returns; the result
// is discarded). Documented gap in src/core/engine.ts.
if (opts?.signal?.aborted) {
throw new DOMException('aborted', 'AbortError');
}
const queryPromise = this.db.query(sql, params).then((r) => r.rows as T[]);
if (!opts?.signal) return queryPromise;
const abortPromise = new Promise<T[]>((_resolve, reject) => {
opts.signal!.addEventListener('abort', () => {
reject(new DOMException('aborted', 'AbortError'));
}, { once: true });
});
return Promise.race([queryPromise, abortPromise]);
}
// ============================================================
+36 -2
View File
@@ -201,6 +201,10 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image
-- v0.19.0: partial indexes for code chunk lookups.
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_name ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_chunks_language ON content_chunks(language) WHERE language IS NOT NULL;
-- v0.41.18.0 (codex finding #9): partial index for gbrain embed --stale
-- and --priority recent. See src/schema.sql for full rationale.
CREATE INDEX IF NOT EXISTS content_chunks_stale_idx
ON content_chunks(page_id, chunk_index) WHERE embedding IS NULL;
-- ============================================================
-- links: cross-references between pages
@@ -212,10 +216,14 @@ CREATE TABLE IF NOT EXISTS links (
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
link_type TEXT NOT NULL DEFAULT '',
context TEXT NOT NULL DEFAULT '',
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
-- for search ranking; only counts toward orphan-ratio + graph traversal.
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
-- v0.41.18.0 (codex finding #12): nullable link_kind distinguishes
-- "plain body mention" from "verb-pattern-derived typed link" within
-- link_source='mentions'. See src/schema.sql for full rationale.
link_kind TEXT CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner')),
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
origin_field TEXT,
-- v0.18.0 Step 4: see src/schema.sql.
@@ -314,7 +322,9 @@ CREATE TABLE IF NOT EXISTS timeline_entries (
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
-- Dedup constraint: same (page, date, summary) treated as same event
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary);
-- v0.41.18.0 (codex finding #11): widened to include source so distinct
-- meeting provenance survives. Legacy rows have source='' (schema default).
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary, source);
-- ============================================================
-- page_versions: snapshot history
@@ -881,6 +891,30 @@ CREATE TABLE IF NOT EXISTS op_checkpoints (
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
-- ============================================================
-- migration_impact_log (v0.41.18.0 gbrain onboard wave)
-- ============================================================
-- See src/schema.sql for full rationale.
CREATE TABLE IF NOT EXISTS migration_impact_log (
id BIGSERIAL PRIMARY KEY,
remediation_id TEXT NOT NULL,
metric_name TEXT NOT NULL,
metric_before NUMERIC,
metric_after NUMERIC,
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
source_id TEXT,
brain_id TEXT,
started_at TIMESTAMPTZ,
idempotency_key TEXT,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
applied_by TEXT,
details JSONB DEFAULT '{}'::jsonb
);
CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
ON migration_impact_log(remediation_id, applied_at DESC);
CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;
-- ============================================================
-- Trigger-based search_vector (spans pages + timeline_entries)
-- ============================================================
+128 -10
View File
@@ -776,7 +776,16 @@ export class PostgresEngine implements BrainEngine {
const reserved = await pool.reserve();
try {
const conn: ReservedConnection = {
async executeRaw<R = Record<string, unknown>>(query: string, params?: unknown[]): Promise<R[]> {
async executeRaw<R = Record<string, unknown>>(
query: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<R[]> {
// ReservedConnection.executeRaw doesn't wire AbortSignal today
// (the only use site is migrations + cycle-lock writes that don't
// want cancellation). Signature matches the interface so callers
// that pass opts don't typecheck-break; opts.signal is ignored.
void opts;
const rows = params === undefined
? await reserved.unsafe(query)
: await reserved.unsafe(query, params as Parameters<typeof reserved.unsafe>[1]);
@@ -1977,11 +1986,86 @@ export class PostgresEngine implements BrainEngine {
afterPageId?: number;
afterChunkIndex?: number;
sourceId?: string;
orderBy?: 'page_id' | 'updated_desc';
afterUpdatedAt?: string | null;
}): Promise<StaleChunkRow[]> {
const sql = this.sql;
const limit = opts?.batchSize ?? 2000;
const afterPid = opts?.afterPageId ?? 0;
const afterIdx = opts?.afterChunkIndex ?? -1;
const orderBy = opts?.orderBy ?? 'page_id';
// v0.41.18.0 (A13, codex #9): --priority recent path. Composite cursor
// (updated_at DESC NULLS LAST, page_id ASC, chunk_index ASC). Backed by
// idx_pages_updated_at_desc + content_chunks_stale_idx partial.
// "Next row" semantic with DESC NULLS LAST + ASC tiebreakers is:
// (updated_at < prev) OR
// (updated_at = prev AND page_id > prev_page_id) OR
// (updated_at = prev AND page_id = prev_page_id AND chunk_index > prev_chunk_index)
// First call: afterUpdatedAt undefined → returns the highest updated_at rows.
if (orderBy === 'updated_desc') {
const afterUpdated = opts?.afterUpdatedAt ?? null;
const isFirstPage = afterUpdated === null && afterPid === 0;
if (opts?.sourceId === undefined) {
const rows = isFirstPage ? await sql`
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT ${limit}
` : await sql`
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
p.updated_at < ${afterUpdated}::timestamptz
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid})
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx})
)
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT ${limit}
`;
return rows as unknown as StaleChunkRow[];
}
const rows = isFirstPage ? await sql`
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND p.source_id = ${opts.sourceId}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT ${limit}
` : await sql`
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count, p.source_id, cc.page_id,
p.updated_at
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
AND p.source_id = ${opts.sourceId}
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
AND (
p.updated_at < ${afterUpdated}::timestamptz
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid})
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx})
)
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
LIMIT ${limit}
`;
return rows as unknown as StaleChunkRow[];
}
// orderBy === 'page_id' — legacy stable cursor (unchanged below).
// Cursor-paginated: keyset pagination on (page_id, chunk_index).
// The partial index idx_chunks_embedding_null makes the WHERE fast;
// LIMIT keeps each round-trip well within statement_timeout.
@@ -2102,15 +2186,17 @@ export class PostgresEngine implements BrainEngine {
const fromSourceIds = links.map(l => l.from_source_id || 'default');
const toSourceIds = links.map(l => l.to_source_id || 'default');
const originSourceIds = links.map(l => l.origin_source_id || 'default');
// v0.41.18.0 (A10): link_kind column (v98). NULL = legacy/plain.
const linkKinds = links.map(l => l.link_kind ?? null);
const result = await sql`
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, link_kind, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, v.link_kind, o.id, v.origin_field
FROM unnest(
${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[],
${contexts}::text[], ${linkSources}::text[], ${originSlugs}::text[],
${originFields}::text[], ${fromSourceIds}::text[], ${toSourceIds}::text[],
${originSourceIds}::text[]
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id)
${originSourceIds}::text[], ${linkKinds}::text[]
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id, link_kind)
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
@@ -2493,7 +2579,7 @@ export class PostgresEngine implements BrainEngine {
if (slugs.length === 0) return result;
for (const s of slugs) result.set(s, 0);
// v0.42.0.0 D12: filter mentions OUT of backlink-count for search
// v0.41.18.0 D12: filter mentions OUT of backlink-count for search
// ranking. `link_source='mentions'` rows are auto-linked body-text
// mentions from `gbrain extract links --by-mention`; they're
// graph-completeness signal, NOT human-intent signal. Counting them
@@ -2710,7 +2796,7 @@ export class PostgresEngine implements BrainEngine {
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
SELECT id, ${entry.date}::date, ${entry.source || ''}, ${entry.summary}, ${entry.detail || ''}
FROM pages WHERE slug = ${slug} AND source_id = ${sourceId}
ON CONFLICT (page_id, date, summary) DO NOTHING
ON CONFLICT (page_id, date, summary, source) DO NOTHING
`;
}
@@ -2729,7 +2815,7 @@ export class PostgresEngine implements BrainEngine {
FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[], ${sourceIds}::text[])
AS v(slug, date, source, summary, detail, source_id)
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
ON CONFLICT (page_id, date, summary) DO NOTHING
ON CONFLICT (page_id, date, summary, source) DO NOTHING
RETURNING 1
`;
return result.length;
@@ -4255,9 +4341,41 @@ export class PostgresEngine implements BrainEngine {
}
}
async executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]> {
async executeRaw<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
const conn = this.sql;
return conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]) as unknown as T[];
const pending = conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]);
// v0.41.18.0 (A20, codex #7): real cancellation via postgres.js's
// .cancel() on the pending query. Init nudge (3s wallclock cap) is the
// first consumer; the AbortSignal fires when the timer trips.
// Already-aborted signal short-circuits before the network round-trip.
if (opts?.signal) {
if (opts.signal.aborted) {
// .cancel() is fire-and-forget; the awaited query rejects with the
// postgres "query was cancelled" error which the caller catches.
try {
(pending as unknown as { cancel?: () => void }).cancel?.();
} catch {
// best-effort
}
throw new DOMException('aborted', 'AbortError');
}
const onAbort = () => {
try {
(pending as unknown as { cancel?: () => void }).cancel?.();
} catch {
// best-effort; the .then below settles regardless
}
};
opts.signal.addEventListener('abort', onAbort, { once: true });
return (pending as unknown as Promise<T[]>).finally(() => {
opts.signal?.removeEventListener('abort', onAbort);
});
}
return pending as unknown as T[];
// Pre-#406 behavior: throw on any error including connection death.
// Per-call auto-retry is not safe here because executeRaw is also used
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
+72
View File
@@ -0,0 +1,72 @@
// src/core/remediation/context.ts
// v0.41.18.0 (A1, codex finding #2). Extracted verbatim from
// src/commands/doctor.ts:loadRecommendationContext so both the doctor
// CLI shell AND the new gbrain onboard / MCP run_onboard surfaces
// build the same context object.
//
// Pure read; no side effects.
import type { BrainEngine } from '../engine.ts';
import type { RecommendationContext } from '../brain-score-recommendations.ts';
// Re-export so consumers can `import { RecommendationContext } from '../remediation'`
// — the canonical RecommendationContext type still lives in
// brain-score-recommendations.ts (it's also the input to computeRecommendations).
export type { RecommendationContext };
/**
* Build RecommendationContext from engine + config. Pure read; no
* side effects. Used by computeRemediationPlan, runRemediation, and
* the doctor CLI surface.
*/
export async function loadRecommendationContext(
engine: BrainEngine,
): Promise<RecommendationContext> {
// v0.37 fix wave (Lane E.4 + CDX2-11): read schema-sizing fields from
// gateway, not DB. The DB plane is schema-applied metadata; the file
// plane is the gateway runtime source. Pre-fix this context produced
// stale recommendations on fresh installs whose DB rows hadn't been
// populated.
//
// Also extended the API-key check to recognize the ZE key alongside
// OpenAI (was OpenAI-only). After Lane C.3, zeroentropy_api_key lives
// in GBrainConfig + propagates to the gateway env dict.
const repoPath = await engine.getConfig('sync.repo_path');
let embeddingModel: string | undefined;
let embeddingDimensions: number | undefined;
try {
const gw = await import('../ai/gateway.ts');
embeddingModel = gw.getEmbeddingModel();
embeddingDimensions = gw.getEmbeddingDimensions();
} catch {
// Gateway unconfigured — fall back to DB plane as a best-effort hint
// (preserves doctor running before any engine.connect()).
const dbModel = await engine.getConfig('embedding_model');
const dbDims = await engine.getConfig('embedding_dimensions');
embeddingModel = dbModel ?? undefined;
embeddingDimensions = dbDims ? Number(dbDims) : undefined;
}
// v0.40.x: recipe-aware provider check, shared with autopilot.ts via
// embeddingProviderConfigured(). Local providers (ollama, llama-server —
// empty auth_env.required) need no hosted key; hosted providers check
// their OWN required key (so a Voyage brain is judged by VOYAGE_API_KEY,
// not by whether an OpenAI/ZE key happens to exist — the pre-fix wart).
// fileCfg loads synchronously, so the resolveKey closure is sync.
const { loadConfigFileOnly } = await import('../config.ts');
const fileCfg = loadConfigFileOnly();
const { embeddingProviderConfigured, HOSTED_EMBED_KEY_CONFIG } = await import(
'../brain-score-recommendations.ts'
);
const embeddingConfigured = embeddingProviderConfigured(embeddingModel, (envVar) => {
const cfgField = HOSTED_EMBED_KEY_CONFIG[envVar];
const fromCfg = cfgField ? (fileCfg as Record<string, unknown> | null)?.[cfgField] : undefined;
return !!(process.env[envVar] || fromCfg);
});
return {
repoPath: repoPath ?? undefined,
embeddingModel,
embeddingDimensions,
embeddingProviderConfigured: embeddingConfigured,
hasChatApiKey: !!(process.env.ANTHROPIC_API_KEY || fileCfg?.anthropic_api_key),
};
}
+16
View File
@@ -0,0 +1,16 @@
// src/core/remediation/index.ts
// v0.41.18.0 (A1). Barrel for the doctor-remediation library.
// Consumers (doctor CLI, onboard CLI, MCP run_onboard) import from here.
export { computeRemediationPlan, SYNTHETIC_CHECK_NAMES } from './plan.ts';
export { runRemediation } from './run.ts';
export { loadRecommendationContext } from './context.ts';
export type { RecommendationContext } from './context.ts';
export type {
RemediationPlan,
RemediationPlanOpts,
RemediationOpts,
RemediationResult,
RemediationHooks,
StepResult,
} from './types.ts';
+76
View File
@@ -0,0 +1,76 @@
// src/core/remediation/plan.ts
// v0.41.18.0 (A1, codex finding #2). Extracted from doctor.ts:runRemediationPlan
// so onboard + MCP run_onboard call the same library without parsing argv
// or invoking console.* / process.exit.
import type { BrainEngine } from '../engine.ts';
import {
computeRecommendations,
classifyChecks,
maxReachableScore,
} from '../brain-score-recommendations.ts';
import { loadRecommendationContext } from './context.ts';
import type { RemediationPlan, RemediationPlanOpts } from './types.ts';
/**
* Synthetic check list for classification. computeRecommendations operates
* on BrainHealth + context alone; we don't need full doctor output, just
* the check names the recommendations care about. Same five names doctor
* has used since v0.36.4.0; do not extend without also updating
* brain-score-recommendations.ts.
*/
export const SYNTHETIC_CHECK_NAMES = [
'brain_score',
'sync_freshness',
'missing_embeddings',
'dead_links',
'orphan_pages',
] as const;
/**
* Pure read: compute the dependency-ordered Remediation plan to drive
* brain to opts.targetScore (default 90). Never enqueues, never mutates.
*
* Consumed by:
* - gbrain doctor --remediation-plan (renders human/JSON in CLI shell)
* - gbrain onboard --check (reframes as onboarding language)
* - MCP run_onboard (admin scope, returns plan as JSON envelope)
*/
export async function computeRemediationPlan(
engine: BrainEngine,
opts: RemediationPlanOpts = {},
): Promise<RemediationPlan> {
const targetScore = opts.targetScore ?? 90;
// Cheap path (D7) — don't run slow doctor checks for the plan surface.
// The recommendation generator works from BrainHealth + context alone.
const health = await engine.getHealth();
const ctx = await loadRecommendationContext(engine);
const recs = computeRecommendations(health, ctx, opts.extraRemediations ?? []);
const syntheticChecks = SYNTHETIC_CHECK_NAMES.map((name) => ({
name,
status: 'ok' as const,
}));
const classifications = classifyChecks(syntheticChecks, ctx);
const ceiling = maxReachableScore(health, classifications);
const filteredRecs = recs.filter((r) => r.status === 'remediable');
const estTotalSeconds = filteredRecs.reduce((sum, r) => sum + r.est_seconds, 0);
const estTotalUsd = filteredRecs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
const blocked = classifications
.filter((c) => c.status === 'blocked')
.map((c) => ({ check: c.check, reason: c.reason ?? 'prerequisite missing' }));
return {
schema_version: 2,
brain_score_current: health.brain_score,
brain_score_target: targetScore,
max_reachable_score: ceiling,
target_unreachable: targetScore > ceiling,
plan: filteredRecs.map((r, i) => ({ step: i + 1, ...r })),
est_total_seconds: estTotalSeconds,
est_total_usd_cost: Number(estTotalUsd.toFixed(2)),
blocked,
};
}
+342
View File
@@ -0,0 +1,342 @@
// src/core/remediation/run.ts
// v0.41.18.0 (A1, codex finding #2). Extracted from doctor.ts:runRemediate
// so onboard + MCP run_onboard call the same orchestrator without parsing
// argv or invoking process.exit / console.* directly.
//
// The orchestrator wraps the plan loop with:
// - BudgetTracker (auto-installed via withBudgetTracker)
// - Checkpoint resume per A4 amended (matching plan_hash only)
// - D5 dependency cascade (failed step aborts dependents)
// - D7 per-step recheck (re-compute plan from fresh health)
// - Hooks for caller observability (no console.* in the library)
import crypto from 'crypto';
import type { BrainEngine } from '../engine.ts';
import {
computeRecommendations,
} from '../brain-score-recommendations.ts';
import type { RemediationStep } from '../remediation-step.ts';
import { loadRecommendationContext } from './context.ts';
import { computeRemediationPlan } from './plan.ts';
import type {
RemediationHooks,
RemediationOpts,
RemediationResult,
StepResult,
} from './types.ts';
/**
* Submit ordered Remediation jobs sequentially per D3, with D5 cascade
* on failure and D7 scoped recheck between steps.
*
* PGLite path: synchronous in-process execution (no durable queue).
*
* Returns a RemediationResult; never throws on BudgetExhausted (the
* exhaustion snapshot lives on result.budget_exhausted instead).
* Other thrown errors propagate.
*
* Callers decide exit codes from the result.
*/
export async function runRemediation(
engine: BrainEngine,
opts: RemediationOpts = {},
hooks: RemediationHooks = {},
): Promise<RemediationResult> {
const targetScore = opts.targetScore ?? 90;
const maxJobs = opts.maxJobs ?? Infinity;
const maxUsd = opts.maxUsd;
const dryRun = opts.dryRun ?? false;
const resumeMode = opts.resume ?? false;
const resumePlanHash = opts.resumePlanHash;
// Lazy-load orchestration deps so the library entry-point doesn't pay
// their cost on a --dry-run shortcut path (or when callers only need
// computeRemediationPlan).
const {
BudgetTracker,
BudgetExhausted,
} = await import('../budget/budget-tracker.ts');
const { withBudgetTracker } = await import('../ai/gateway.ts');
const {
computePlanHash,
saveRemediationCheckpoint,
loadRemediationCheckpoint,
listRemediationCheckpoints,
clearRemediationCheckpoint,
} = await import('../remediation-checkpoint.ts');
const ctx = await loadRecommendationContext(engine);
// Pre-flight ceiling check via the shared plan computation.
const initialPlan = await computeRemediationPlan(engine, { targetScore });
if (initialPlan.target_unreachable) {
hooks.onTargetUnreachable?.(targetScore, initialPlan.max_reachable_score);
return {
doctor_run_id: crypto.randomUUID(),
brain_score_initial: initialPlan.brain_score_current,
brain_score_final: initialPlan.brain_score_current,
brain_score_target: targetScore,
target_reached: false,
submitted: [],
aborted_count: 0,
target_unreachable: {
target: targetScore,
ceiling: initialPlan.max_reachable_score,
},
};
}
const initialHealth = await engine.getHealth();
let recs: RemediationStep[] = computeRecommendations(initialHealth, ctx)
.filter((r) => r.status === 'remediable');
if (recs.length === 0) {
hooks.onNothingToDo?.(initialHealth.brain_score, targetScore);
return {
doctor_run_id: crypto.randomUUID(),
brain_score_initial: initialHealth.brain_score,
brain_score_final: initialHealth.brain_score,
brain_score_target: targetScore,
target_reached: initialHealth.brain_score >= targetScore,
submitted: [],
aborted_count: 0,
};
}
// A4 amended: compute plan_hash off the active recommendation ids so
// the checkpoint binds to THIS plan. Resume only fires for matching plans.
const planHash = computePlanHash(recs.map((r) => r.id));
let completedFromCheckpoint = new Set<string>();
if (resumeMode) {
const requested = resumePlanHash;
let cp = requested ? loadRemediationCheckpoint(requested) : null;
if (!cp && !requested) {
// No explicit hash: try newest checkpoint that matches the active plan.
const recent = listRemediationCheckpoints();
for (const e of recent) {
const candidate = loadRemediationCheckpoint(e.plan_hash);
if (candidate && candidate.plan_hash === planHash) {
cp = candidate;
break;
}
}
}
if (!cp || cp.plan_hash !== planHash) {
hooks.onResumeMissed?.(planHash, requested);
// Surface as a synthetic result so the CLI shell can exit 2.
return {
doctor_run_id: crypto.randomUUID(),
brain_score_initial: initialHealth.brain_score,
brain_score_final: initialHealth.brain_score,
brain_score_target: targetScore,
target_reached: false,
submitted: [],
aborted_count: 0,
target_unreachable: {
target: targetScore,
ceiling: initialPlan.max_reachable_score,
},
};
}
completedFromCheckpoint = new Set(cp.completed.map((c) => c.id));
hooks.onResumeLoaded?.(
planHash,
completedFromCheckpoint.size,
recs.length - completedFromCheckpoint.size,
);
}
const estTotalUsd = recs.reduce((sum, r) => sum + (r.est_usd_cost ?? 0), 0);
if (maxUsd !== undefined && estTotalUsd > maxUsd) {
hooks.onBudgetRefused?.(estTotalUsd, maxUsd);
return {
doctor_run_id: crypto.randomUUID(),
brain_score_initial: initialHealth.brain_score,
brain_score_final: initialHealth.brain_score,
brain_score_target: targetScore,
target_reached: false,
submitted: [],
aborted_count: 0,
};
}
if (dryRun) {
// Dry-run: no submission, just return the plan as a non-empty result.
// Each rec lands in submitted[] with synthetic 'dry_run' status so the
// shape stays consistent.
return {
doctor_run_id: crypto.randomUUID(),
brain_score_initial: initialHealth.brain_score,
brain_score_final: initialHealth.brain_score,
brain_score_target: targetScore,
target_reached: false,
submitted: recs.map((r, i) => ({
step: i + 1,
id: r.id,
job_id: null,
status: 'dry_run',
})),
aborted_count: 0,
};
}
// Real submission path
const submitted: StepResult[] = [];
const abortedIds = new Set<string>();
const doctorRunId = crypto.randomUUID();
const { MinionQueue } = await import('../minions/queue.ts');
const { waitForCompletion } = await import('../minions/wait-for-completion.ts');
const isPGLite = engine.kind === 'pglite';
const queue = new MinionQueue(engine);
// A4 amended: install a BudgetTracker scope around the plan-step loop so
// any gateway.chat / embed / rerank inside a Minion handler (synthesize,
// patterns, consolidate) auto-enforces the cap. On BudgetExhausted, the
// onExhausted callback persists the checkpoint BEFORE the throw propagates;
// the caller hook surfaces the actionable --resume hint.
const remediateTracker = new BudgetTracker({
label: 'remediation.run',
maxCostUsd: maxUsd,
});
let exhaustionSnapshot: NonNullable<RemediationResult['budget_exhausted']> | undefined;
remediateTracker.onExhausted(() => {
const cp = {
schema_version: 1 as const,
plan_hash: planHash,
doctor_run_id: doctorRunId,
target_score: targetScore,
started_at: new Date().toISOString(),
completed: submitted
.filter((s) => s.status === 'completed')
.map((s) => ({ id: s.id, job: '', status: s.status, job_id: s.job_id ?? null })),
aborted_at: new Date().toISOString(),
abort_reason: 'budget_exhausted' as const,
budget_snapshot: exhaustionSnapshot
? { spent: exhaustionSnapshot.spent, cap: exhaustionSnapshot.cap, reason: exhaustionSnapshot.reason, model_id: exhaustionSnapshot.model_id }
: undefined,
};
saveRemediationCheckpoint(cp);
});
const runLoop = async (): Promise<void> => {
let stepCount = 0;
const totalSteps = recs.length;
while (recs.length > 0 && stepCount < maxJobs) {
const step = recs[0];
if (!step) break;
stepCount++;
// Resume: skip steps that the checkpoint already marked completed.
if (completedFromCheckpoint.has(step.id)) {
const result: StepResult = { step: stepCount, id: step.id, job_id: null, status: 'completed' };
submitted.push(result);
hooks.onStepEnd?.(result);
recs.shift();
continue;
}
// D5: if depends_on intersects aborted, skip + cascade
if (step.depends_on && step.depends_on.some((d: string) => abortedIds.has(d))) {
const result: StepResult = { step: stepCount, id: step.id, job_id: null, status: 'skipped_dep_aborted' };
submitted.push(result);
abortedIds.add(step.id);
hooks.onStepEnd?.(result);
recs.shift();
continue;
}
hooks.onStepStart?.(stepCount, totalSteps, step);
try {
const isProtected = !!step.protected;
const job = await queue.add(
step.job,
{ ...step.params, doctor_run_id: doctorRunId },
{
queue: 'default',
idempotency_key: step.idempotency_key,
max_attempts: 2,
maxWaiting: 1,
},
isProtected ? { allowProtectedSubmit: true } : undefined,
);
const submittedResult: StepResult = {
step: stepCount,
id: step.id,
job_id: job.id,
status: 'submitted',
};
submitted.push(submittedResult);
const terminal = await waitForCompletion(queue, job.id, {
pollMs: isPGLite ? 250 : 1000,
timeoutMs: (step.est_seconds + 60) * 1000,
});
submittedResult.status = terminal.status;
if (terminal.status !== 'completed') {
abortedIds.add(step.id);
}
hooks.onStepEnd?.(submittedResult);
} catch (e) {
if (e instanceof BudgetExhausted) {
exhaustionSnapshot = {
spent: e.spent,
cap: e.cap,
reason: e.reason,
model_id: e.modelId,
plan_hash: planHash,
};
throw e;
}
const errResult: StepResult = {
step: stepCount,
id: step.id,
job_id: null,
status: `error: ${(e as Error).message.slice(0, 100)}`,
};
submitted.push(errResult);
abortedIds.add(step.id);
hooks.onStepEnd?.(errResult);
}
recs.shift();
// D7: scoped recheck — re-compute plan from fresh health snapshot.
// The next plan may drop completed steps and re-introduce failed
// steps with bumped retry suffix (D1).
if (recs.length === 0 || stepCount >= maxJobs) break;
const freshHealth = await engine.getHealth();
recs = computeRecommendations(freshHealth, ctx).filter((r) => r.status === 'remediable');
}
};
let budgetAbort: NonNullable<RemediationResult['budget_exhausted']> | undefined;
try {
await withBudgetTracker(remediateTracker, runLoop);
} catch (err) {
if (err instanceof BudgetExhausted) {
budgetAbort = exhaustionSnapshot;
if (budgetAbort) hooks.onBudgetExhausted?.(planHash, budgetAbort);
} else {
throw err;
}
}
// Clear checkpoint on a clean run (no budget abort). Failed steps in the
// submitted set don't disqualify the cleanup — they re-surface on the
// next plan with bumped suffixes.
if (!budgetAbort) {
clearRemediationCheckpoint(planHash);
}
const finalHealth = await engine.getHealth();
return {
doctor_run_id: doctorRunId,
brain_score_initial: initialHealth.brain_score,
brain_score_final: finalHealth.brain_score,
brain_score_target: targetScore,
target_reached: finalHealth.brain_score >= targetScore,
submitted,
aborted_count: abortedIds.size,
budget_exhausted: budgetAbort,
};
}
+127
View File
@@ -0,0 +1,127 @@
// src/core/remediation/types.ts
// v0.41.18.0 (A1, codex finding #2). Extracted from src/commands/doctor.ts
// so onboard CLI shell + MCP run_onboard op can compose against a stable
// library shape — NOT a CLI-shaped function with process.exit calls.
//
// Three consumers wrap this library today:
// - src/commands/doctor.ts (runRemediationPlan + runRemediate)
// - src/commands/onboard.ts (gbrain onboard --check / --auto)
// - src/core/operations.ts (MCP op run_onboard, admin scope)
import type { RemediationStep } from '../remediation-step.ts';
/**
* Options for computeRemediationPlan. All fields are optional with
* sensible defaults.
*/
export interface RemediationPlanOpts {
/** Target brain_score (default: 90). Used to drive recommendation count. */
targetScore?: number;
/**
* v0.41.18.0 (A2 + codex #3): caller-supplied RemediationStep entries
* threaded into the planner via the third arg of computeRecommendations.
* Onboard wires the 4 new check helpers (embed_staleness,
* entity_link_coverage, timeline_coverage, takes_count) here. doctor's
* existing --remediation-plan call passes empty (preserving legacy
* behavior).
*/
extraRemediations?: RemediationStep[];
}
/**
* Read-only plan output. Stable JSON envelope downstream agents
* (gbrain onboard, MCP run_onboard) bind to this shape.
*/
export interface RemediationPlan {
schema_version: 2;
brain_score_current: number;
brain_score_target: number;
max_reachable_score: number;
target_unreachable: boolean;
plan: Array<RemediationStep & { step: number }>;
est_total_seconds: number;
est_total_usd_cost: number;
blocked: Array<{ check: string; reason: string }>;
}
/**
* Options for runRemediation. The remediate orchestrator wraps the plan
* loop with BudgetTracker, checkpoint resume, dep-cascade, and per-step
* recheck. Hooks let callers (CLI / JSON / MCP) emit progress without
* the library calling console directly.
*/
export interface RemediationOpts {
/** Target brain_score (default: 90). */
targetScore?: number;
/** Cap inner loop iterations (default: Infinity). */
maxJobs?: number;
/** USD cap for total plan cost. Pre-flight refuse + mid-run BudgetExhausted gate. */
maxUsd?: number;
/** Read-only dry-run; submits no jobs; returns plan in result. */
dryRun?: boolean;
/** Resume from checkpoint matching this plan_hash, OR newest if undefined+resume=true. */
resumePlanHash?: string;
/** Whether to attempt resume at all (default false). */
resume?: boolean;
}
/**
* Result of one step. status mirrors Minion job terminal states + a few
* synthetic ones ('skipped_dep_aborted', 'skipped_completed_in_checkpoint').
*/
export interface StepResult {
step: number;
id: string;
job_id: number | null;
status: string;
}
/**
* Result of a full runRemediation invocation. Stable shape for JSON
* emission and MCP envelope.
*/
export interface RemediationResult {
doctor_run_id: string;
brain_score_initial: number;
brain_score_final: number;
brain_score_target: number;
target_reached: boolean;
submitted: StepResult[];
aborted_count: number;
/** Set when the run aborted on BudgetExhausted. Caller decides exit code. */
budget_exhausted?: {
spent: number;
cap: number;
reason: string;
model_id?: string;
plan_hash: string;
};
/** Set when the pre-flight ceiling check refused the target. */
target_unreachable?: {
target: number;
ceiling: number;
};
}
/**
* Hooks let the caller observe + report progress without the library
* emitting to stdout/stderr itself. Every hook is optional.
*/
export interface RemediationHooks {
/** Fired once when the orchestrator decides nothing needs to run. */
onNothingToDo?: (initialScore: number, target: number) => void;
/** Fired when the pre-flight ceiling check refuses the target. */
onTargetUnreachable?: (target: number, ceiling: number) => void;
/** Fired when --max-usd refuses the plan. */
onBudgetRefused?: (estCost: number, cap: number) => void;
/** Fired before each step submission. */
onStepStart?: (step: number, total: number, rec: RemediationStep) => void;
/** Fired after each step reaches a terminal state (or skip). */
onStepEnd?: (result: StepResult) => void;
/** Fired on BudgetExhausted thrown mid-loop. */
onBudgetExhausted?: (planHash: string, snapshot: NonNullable<RemediationResult['budget_exhausted']>) => void;
/** Fired on resume-checkpoint load (resume mode only). */
onResumeLoaded?: (planHash: string, completedCount: number, remainingCount: number) => void;
/** Fired on resume-checkpoint miss (resume mode only). */
onResumeMissed?: (planHash: string, requested?: string) => void;
}
+51 -3
View File
@@ -262,6 +262,13 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image
CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector);
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified
ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL;
-- v0.41.18.0 (codex finding #9): partial index for `gbrain embed --stale`
-- + `--priority recent`. content_chunks has no updated_at column (chunks
-- are re-INSERTed on page change, not UPDATEd), so the "recent-first"
-- ORDER BY happens at the JOIN site: outer ORDER BY p.updated_at DESC
-- uses idx_pages_updated_at_desc; inner partial uses this index.
CREATE INDEX IF NOT EXISTS content_chunks_stale_idx
ON content_chunks(page_id, chunk_index) WHERE embedding IS NULL;
-- v0.20.0 Cathedral II: chunk-grain FTS trigger.
-- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text.
@@ -353,10 +360,16 @@ CREATE TABLE IF NOT EXISTS links (
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
link_type TEXT NOT NULL DEFAULT '',
context TEXT NOT NULL DEFAULT '',
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
-- for search ranking; only counts toward orphan-ratio + graph traversal.
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
-- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from
-- "verb-pattern-derived typed link" within link_source='mentions'.
-- Codex finding #12 design: keep link_source stable; add link_kind
-- so callers can distinguish without breaking existing mentions queries.
-- NULL = legacy / unknown / pre-v98 row (semantically 'plain').
link_kind TEXT CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner')),
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
origin_field TEXT,
-- v0.18.0 Step 4: 'qualified' when the link was written as
@@ -419,8 +432,10 @@ CREATE TABLE IF NOT EXISTS timeline_entries (
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
-- Dedup constraint: same (page, date, summary) treated as same event
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary);
-- v0.41.18.0 (codex finding #11): widened from (page_id, date, summary) to
-- include `source` so distinct meeting provenance survives. Legacy rows
-- have source='' (schema default) so legacy dedup behavior is preserved.
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary, source);
-- ============================================================
-- page_versions: snapshot history for compiled_truth
@@ -585,6 +600,39 @@ CREATE TABLE IF NOT EXISTS op_checkpoints (
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
-- ============================================================
-- migration_impact_log: before/after metric stats per onboard remediation
-- ============================================================
-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the
-- onboard remediation pipeline records before/after metric stats so
-- `gbrain onboard --history --json` can show "you reduced orphans 47%".
-- delta computed at read time (NOT a stored GENERATED column —
-- zero PGLite parity risk per eng-review D2).
--
-- Attribution columns (job_id, source_id, brain_id, started_at,
-- idempotency_key) per codex finding #10 so concurrent onboard /
-- autopilot / manual runs can't misattribute deltas to the wrong
-- migration when overlapping runs change the same metric.
CREATE TABLE IF NOT EXISTS migration_impact_log (
id BIGSERIAL PRIMARY KEY,
remediation_id TEXT NOT NULL,
metric_name TEXT NOT NULL,
metric_before NUMERIC,
metric_after NUMERIC,
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
source_id TEXT,
brain_id TEXT,
started_at TIMESTAMPTZ,
idempotency_key TEXT,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
applied_by TEXT,
details JSONB DEFAULT '{}'::jsonb
);
CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
ON migration_impact_log(remediation_id, applied_at DESC);
CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;
-- ============================================================
-- files: binary attachments stored in Supabase Storage
-- ============================================================
+17 -7
View File
@@ -284,18 +284,28 @@ describe('Eng-review D3 — executeRaw has no per-call retry wrapper', () => {
const src = readFileSync(resolve('src/core/postgres-engine.ts'), 'utf-8');
// Find the executeRaw method in the class (not the helper inside withReservedConnection)
// Pattern: must be a method on the class taking (sql, params)
const fnMatch = src.match(/async executeRaw<T = Record<string, unknown>>\(sql: string, params\?: unknown\[\]\): Promise<T\[\]> \{([\s\S]*?)\n \}/);
// v0.41.18.0 (T5/A20): signature extended with optional `opts?: { signal?: AbortSignal }`
// 3rd arg + multi-line shape for real query cancellation. Regex updated to
// tolerate both the legacy single-line and the new multi-line signatures.
const fnMatch = src.match(/async executeRaw<T = Record<string, unknown>>\(\s*sql: string,\s*params\?: unknown\[\][^)]*\):\s*Promise<T\[\]>\s*\{([\s\S]*?)\n \}/);
expect(fnMatch).not.toBeNull();
const body = fnMatch![1];
// Must not have any try/catch
expect(body).not.toContain('try {');
expect(body).not.toContain('catch');
// Must not call reconnect() from this method
// Must not call reconnect() from this method (D3 intent: no per-call
// retry — recovery is supervisor-driven via reconnect()).
expect(body).not.toContain('this.reconnect()');
// Must call conn.unsafe directly
// Must call conn.unsafe directly, exactly ONCE (no retry re-issue).
expect(body).toContain('conn.unsafe(');
const unsafeCallCount = (body.match(/conn\.unsafe\(/g) || []).length;
expect(unsafeCallCount).toBe(1);
// The try/catch present here is ONLY for AbortSignal cancellation
// swallow (v0.41.18.0 A20), NOT for connection retry. Confirm by checking
// the swallowed throws are .cancel() not network re-issue.
if (body.includes('catch')) {
// If catch exists, it must be the cancel-swallow shape, NOT a retry shape.
expect(body).not.toMatch(/catch[^{]*\{[\s\S]*?conn\.unsafe/);
expect(body).not.toMatch(/catch[^{]*\{[\s\S]*?setTimeout/);
}
});
it('PostgresEngine.reconnect() still exists for supervisor-driven recovery', () => {
+167
View File
@@ -0,0 +1,167 @@
// test/e2e/onboard-full-flow.test.ts
// v0.41.18.0 (T20). Hermetic PGLite E2E for the onboard surface — no
// DATABASE_URL needed. Exercises the key contracts end-to-end:
// - computeRemediationPlan with extras returns the expected shape
// - buildOnboardReport produces a stable JSON envelope
// - captureMetric returns numeric values for each of 5 metrics
// - The runRemediation library refuses --auto without --max-usd
// - The onboard CLI gates work as documented
//
// Full DATABASE_URL-gated end-to-end (real Postgres, actual extractions
// firing through Minion handlers) is deferred to a v0.42.1 follow-up
// once the Minion worker test harness lands the per-handler stub seam.
import { describe, expect, test, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { computeRemediationPlan } from '../../src/core/remediation/index.ts';
import { captureMetric } from '../../src/core/onboard/impact-capture.ts';
import { buildOnboardReport, toOnboardRecommendation } from '../../src/core/onboard/render.ts';
import { runAllOnboardChecks } from '../../src/core/onboard/checks.ts';
import { makeRemediationStep } from '../../src/core/remediation-step.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
describe('onboard E2E — captureMetric', () => {
test('captureMetric returns 0 for stale_count on empty brain', async () => {
const v = await captureMetric(engine, 'stale_count');
expect(v).toBe(0);
});
test('captureMetric returns 0 for orphan_count on empty brain', async () => {
const v = await captureMetric(engine, 'orphan_count');
expect(v).toBe(0);
});
test('captureMetric returns 1 for coverage on empty brain (vacuous truth)', async () => {
const v = await captureMetric(engine, 'entity_link_coverage');
expect(v).toBe(1);
});
test('captureMetric returns 0 for takes_count on empty brain', async () => {
const v = await captureMetric(engine, 'takes_count');
expect(v).toBe(0);
});
});
describe('onboard E2E — runAllOnboardChecks', () => {
test('returns all 4 check shapes', async () => {
const results = await runAllOnboardChecks(engine);
expect(results.length).toBe(4);
const names = results.map((r) => r.check.name).sort();
expect(names).toEqual([
'embed_staleness',
'entity_link_coverage',
'takes_count',
'timeline_coverage',
]);
});
test('empty brain: stale/link/timeline ok, takes_count warns (0 takes)', async () => {
const results = await runAllOnboardChecks(engine);
const byName = Object.fromEntries(results.map((r) => [r.check.name, r.check.status]));
expect(byName.embed_staleness).toBe('ok');
expect(byName.entity_link_coverage).toBe('ok');
expect(byName.timeline_coverage).toBe('ok');
expect(byName.takes_count).toBe('warn'); // 0 takes is a warn
});
test('empty brain returns 0 remediations (takes_count gated by bootstrap_enabled=false)', async () => {
const results = await runAllOnboardChecks(engine);
const total = results.reduce((s, r) => s + r.remediations.length, 0);
// takes_count warns but does NOT emit a remediation because
// takes.bootstrap_enabled defaults to false (A12 two-gate consent).
expect(total).toBe(0);
});
});
describe('onboard E2E — computeRemediationPlan with extras', () => {
test('threads extras through computeRecommendations', async () => {
// Build a synthetic extra remediation. computeRemediationPlan
// should merge it into the plan output even though the hardcoded
// planner doesn't know about it.
const extra = makeRemediationStep({
id: 'test.synthetic',
job: 'test-job',
params: {},
severity: 'low',
est_seconds: 10,
est_usd_cost: 0,
rationale: 'synthetic test entry',
status: 'remediable',
});
const plan = await computeRemediationPlan(engine, {
targetScore: 90,
extraRemediations: [extra],
});
const ids = plan.plan.map((p) => p.id);
expect(ids).toContain('test.synthetic');
});
test('returns RemediationPlan with stable schema_version: 2', async () => {
const plan = await computeRemediationPlan(engine, { targetScore: 90 });
expect(plan.schema_version).toBe(2);
expect(typeof plan.brain_score_current).toBe('number');
expect(plan.brain_score_target).toBe(90);
expect(typeof plan.max_reachable_score).toBe('number');
expect(Array.isArray(plan.plan)).toBe(true);
});
});
describe('onboard E2E — buildOnboardReport', () => {
test('produces stable JSON envelope with schema_version: 1', async () => {
const plan = await computeRemediationPlan(engine, { targetScore: 90 });
const report = buildOnboardReport(plan);
expect(report.schema_version).toBe(1);
expect(Array.isArray(report.recommendations)).toBe(true);
expect(report.summary).toBeDefined();
expect(typeof report.summary.total).toBe('number');
expect(typeof report.summary.auto_eligible).toBe('number');
expect(typeof report.summary.prompt_required).toBe('number');
expect(typeof report.summary.manual_only).toBe('number');
expect(typeof report.summary.est_total_usd).toBe('number');
});
});
describe('onboard E2E — toOnboardRecommendation tier policy', () => {
test('non-protected job → auto_apply', () => {
const step = makeRemediationStep({
id: 'test.embed', job: 'embed-catch-up', params: {},
severity: 'medium', est_seconds: 60, est_usd_cost: 0.1,
rationale: 'embed', status: 'remediable',
});
const r = toOnboardRecommendation(step);
expect(r.apply_policy).toBe('auto_apply');
});
test('extract-takes-from-pages → manual_only (A12+A24)', () => {
const step = makeRemediationStep({
id: 'test.takes', job: 'extract-takes-from-pages',
protected: true, params: {},
severity: 'medium', est_seconds: 1800, est_usd_cost: 5,
rationale: 'takes', status: 'remediable',
});
const r = toOnboardRecommendation(step);
expect(r.apply_policy).toBe('manual_only');
});
test('other protected jobs → prompt_required', () => {
const step = makeRemediationStep({
id: 'test.synth', job: 'synthesize',
protected: true, params: {},
severity: 'medium', est_seconds: 600, est_usd_cost: 1,
rationale: 'synth', status: 'remediable',
});
const r = toOnboardRecommendation(step);
expect(r.apply_policy).toBe('prompt_required');
});
});
+5 -5
View File
@@ -322,12 +322,12 @@ describe('Lane D.3 — sync surfaces dim-mismatch recipe at incremental AND firs
// Lane E.4 — loadRecommendationContext provider-aware key check
// ─────────────────────────────────────────────────────────────────────
describe('Lane E.4 — loadRecommendationContext is provider-aware', () => {
// doctor.ts exports loadRecommendationContext only locally; verify the
// behavior via a public surface (the recommendation context the
// `doctor --remediation-plan` output uses) is brittle. Use a
// source-text assertion instead.
// v0.41.18.0 (A1): loadRecommendationContext extracted from doctor.ts
// to src/core/remediation/context.ts so onboard CLI + MCP run_onboard
// can compose the same context. Source-text grep follows the function
// to its new home.
test('source-text grep: loadRecommendationContext is provider-aware via the shared helper', () => {
const src = readFileSync(join(__dirname, '..', 'src', 'commands', 'doctor.ts'), 'utf-8');
const src = readFileSync(join(__dirname, '..', 'src', 'core', 'remediation', 'context.ts'), 'utf-8');
// Pre-v0.37 this was OpenAI-only; the Lane E.4 fix made it branch on
// provider for the key. v0.40.x replaced the inline prefix ladder with the
// shared recipe-aware helper `embeddingProviderConfigured` (so doctor +