mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-17 02:12:40 +00:00
Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f60f245512 | ||
|
|
564ffae186 | ||
|
|
428bdc9cd1 | ||
|
|
1d98298a5c | ||
|
|
74f1ba20f1 | ||
|
|
af209a6c61 | ||
|
|
9a59748bb7 | ||
|
|
8b40678e46 |
@@ -88,13 +88,8 @@ jobs:
|
||||
}
|
||||
EOF
|
||||
- name: Run Tier 2 skill tests
|
||||
run: bun test test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
|
||||
run: bun test test/e2e/skills.test.ts
|
||||
env:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
# v0.33.3.0: ZE live API tests skip gracefully when this is unset,
|
||||
# so forks without the secret stay green. The test exercises the
|
||||
# zeroEntropyCompatFetch response-rewriter + URL rewrite + flexible
|
||||
# dim handling + gateway.rerank against the real provider.
|
||||
ZEROENTROPY_API_KEY: ${{ secrets.ZEROENTROPY_API_KEY }}
|
||||
|
||||
@@ -40,9 +40,3 @@ jobs:
|
||||
run: bun run verify
|
||||
- name: Run test shard ${{ matrix.shard }}/4
|
||||
run: scripts/test-shard.sh ${{ matrix.shard }} 4
|
||||
- name: Run *.serial.test.ts at --max-concurrency=1 (shard 1 only)
|
||||
# Serial files share file-wide state (top-level mock.module, module
|
||||
# singletons) that leaks across files in the same bun-test process.
|
||||
# test-shard.sh excludes them; this step runs them at concurrency=1.
|
||||
if: matrix.shard == 1
|
||||
run: bun run test:serial
|
||||
|
||||
@@ -38,6 +38,3 @@ export/
|
||||
# Tier 3 PGLite snapshot fixture (built on demand by build:pglite-snapshot)
|
||||
test/fixtures/pglite-snapshot.tar
|
||||
test/fixtures/pglite-snapshot.version
|
||||
|
||||
# Private brain reports — never check these in (per CLAUDE.md privacy rule)
|
||||
reports/network-intelligence/
|
||||
|
||||
@@ -6,26 +6,10 @@ start here.
|
||||
|
||||
## Install (5 min)
|
||||
|
||||
1. Install gbrain via Bun (the canonical path):
|
||||
```bash
|
||||
curl -fsSL https://bun.sh/install | bash
|
||||
export PATH="$HOME/.bun/bin:$PATH"
|
||||
bun install -g github:garrytan/gbrain
|
||||
```
|
||||
If `bun install -g` aborts or `gbrain doctor` reports `schema_version: 0`,
|
||||
the CLI prints a recovery hint pointing at [#218](https://github.com/garrytan/gbrain/issues/218).
|
||||
Run `gbrain apply-migrations --yes` to recover, or fall back to the
|
||||
deterministic install: `git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain && bun install && bun link`.
|
||||
2. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
1. Clone: `git clone https://github.com/garrytan/gbrain ~/gbrain && cd ~/gbrain`
|
||||
2. Install: `bun install`
|
||||
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
multi-machine sync, init suggests Postgres + pgvector via Supabase.
|
||||
3. **STOP — ask the user about search mode.** `gbrain init` auto-applied a
|
||||
default but printed a 9-cell cost matrix (mode × downstream model)
|
||||
preceded by `[AGENT]` markers. You MUST relay the matrix to the operator
|
||||
and confirm their choice before continuing. Cost spread between corners
|
||||
is 25x — silent acceptance is the wrong default. See
|
||||
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
|
||||
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
|
||||
for existing users (search modes were added in v0.32.3).
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
|
||||
@@ -57,36 +41,13 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
[`docs/mcp/DEPLOY.md`](./docs/mcp/DEPLOY.md).
|
||||
- **Debug:** [`docs/GBRAIN_VERIFY.md`](./docs/GBRAIN_VERIFY.md),
|
||||
[`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`.
|
||||
- **Migrate / upgrade:** `gbrain upgrade` (binary self-update + schema migrations + post-upgrade prompts),
|
||||
[`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
|
||||
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations --yes` (manual schema-only).
|
||||
- **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
|
||||
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`.
|
||||
- **Eval retrieval changes:** capture is off by default. To benchmark a
|
||||
retrieval change against real captured queries, set
|
||||
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
|
||||
and `gbrain eval replay --against base.ndjson`. For public benchmark
|
||||
coverage (LongMemEval, ground-truth scoring), `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.8) runs against an isolated in-memory PGLite
|
||||
per question — your `~/.gbrain` is never opened. Full guide:
|
||||
and `gbrain eval replay --against base.ndjson`. Full guide:
|
||||
[`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
- **Drive the brain to a target health score (v0.36.4.0):** the one-command
|
||||
loop. `gbrain doctor --remediation-plan --json` previews what would be
|
||||
fixed; `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`
|
||||
walks a dependency-ordered plan (sync before extract, embed after
|
||||
consolidate), re-checking score between every step, refusing to spend
|
||||
past the cost cap. Empty brains (no entity pages) or unconfigured embedding
|
||||
keys hit a `max_reachable_score` ceiling and bail with what's missing.
|
||||
Three phase handlers (synthesize / patterns / consolidate) are
|
||||
PROTECTED — only trusted local callers can submit them; MCP cannot.
|
||||
Reference: [`docs/architecture/topologies.md`](./docs/architecture/topologies.md)
|
||||
and the CHANGELOG entry for v0.36.4.0.
|
||||
- **Track a founder/company over time (v0.35.7):** when an entity has
|
||||
typed metric claims in its `## Facts` fence (`metric: mrr`, `value: 50000`,
|
||||
`unit: USD`, `period: monthly` columns), run
|
||||
`gbrain eval trajectory <entity-slug>` for the chronological history
|
||||
with regressions auto-flagged, or `gbrain founder scorecard <entity-slug>`
|
||||
for a four-signal JSON rollup (claim_accuracy / consistency /
|
||||
growth_trajectory / red_flags). MCP op `find_trajectory` exposes the
|
||||
same data — read scope, visibility-filtered for remote callers.
|
||||
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
|
||||
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
|
||||
single-fetch ingestion.
|
||||
|
||||
+1
-5230
File diff suppressed because it is too large
Load Diff
+1
-11
@@ -190,7 +190,7 @@ See `docs/ENGINES.md` for the full guide. In short:
|
||||
3. Run the test suite against your engine
|
||||
4. Document in `docs/`
|
||||
|
||||
The original SQLite engine plan was superseded by PGLite (embedded Postgres 17 via WASM), which uses the same SQL dialect as Postgres and eliminates the need for a separate FTS5/sqlite-vss translation layer. See [`docs/ENGINES.md`](docs/ENGINES.md) for the engine architecture and the rationale.
|
||||
The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`.
|
||||
|
||||
## CONTRIBUTOR_MODE — turn on the dev loop
|
||||
|
||||
@@ -270,16 +270,6 @@ without captured data can still replay), and cost considerations. The
|
||||
NDJSON wire format is documented in
|
||||
[`docs/eval-capture.md`](./docs/eval-capture.md).
|
||||
|
||||
For public benchmark coverage on top of replay, `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.1) runs LongMemEval against gbrain's hybrid
|
||||
retrieval. One in-memory PGLite per question, runtime-enumerated
|
||||
`TRUNCATE` between questions, ground-truth scoring via LongMemEval's
|
||||
published `evaluate_qa.py`. Use it alongside replay when changes affect
|
||||
retrieval quality on long-context conversational data — replay catches
|
||||
regressions on YOUR queries, LongMemEval catches them on a public set the
|
||||
benchmark community already cites. See the "Public benchmarks: LongMemEval"
|
||||
section in [`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
|
||||
## Welcome PRs
|
||||
|
||||
- SQLite engine implementation
|
||||
|
||||
@@ -1,148 +0,0 @@
|
||||
# DESIGN.md
|
||||
|
||||
The design system source of truth for gbrain. Born from the de facto tokens
|
||||
that landed in `admin/src/index.css` during the v0.26.0 admin SPA work and
|
||||
formalized during the v0.36.1.0 Hindsight calibration wave's design review.
|
||||
|
||||
This doc is the calibration target for `/plan-design-review` and `/design-review`.
|
||||
When a question is "does this UI fit the system?", the answer is here.
|
||||
|
||||
## Voice
|
||||
|
||||
GBrain talks like a smart friend who knows your past, not a clinical scoring
|
||||
system. Every user-facing string passes through this filter:
|
||||
|
||||
- Second person, contractions allowed.
|
||||
- Grounded in concrete data the user can verify ("2 of 3 missed" beats
|
||||
"Brier 0.31").
|
||||
- Never preachy. Never "we recommend." Never "according to your data."
|
||||
- Short. Under 25 words for narrative; under one line for status.
|
||||
- Numbers grounded in real outcomes, never abstract metrics without
|
||||
translation.
|
||||
|
||||
Five surfaces use this voice (v0.36.1.0+):
|
||||
`pattern_statement`, `nudge`, `forecast_blurb`, `dashboard_caption`,
|
||||
`morning_pulse`. All five pass through `gateVoice()` in
|
||||
`src/core/calibration/voice-gate.ts` with mode-specific rubrics. A Haiku
|
||||
judge rejects academic-sounding candidates; up to 2 regens; then fall
|
||||
back to a hand-written template from `src/core/calibration/templates.ts`.
|
||||
|
||||
## Color tokens
|
||||
|
||||
CSS variables in `admin/src/index.css`. SVG renderer inlines literals
|
||||
matching these tokens (`src/core/calibration/svg-renderer.ts`).
|
||||
|
||||
| Token | Value | Use |
|
||||
|--------------------|-----------|-------------------------------------------|
|
||||
| `--bg-primary` | `#0a0a0f` | Page background |
|
||||
| `--bg-secondary` | `#14141f` | Sidebar, cards |
|
||||
| `--bg-tertiary` | `#1e1e2e` | Subtle surfaces, borders |
|
||||
| `--text-primary` | `#e0e0e0` | Body text |
|
||||
| `--text-secondary` | `#888` | Headings, labels |
|
||||
| `--text-muted` | `#777` | Tertiary text — TD2 bumped from #555 for WCAG AA contrast (~5.5:1) |
|
||||
| `--accent` | `#3b82f6` | Active states, links, primary CTAs |
|
||||
| `--success` | `#22c55e` | Healthy / ok status |
|
||||
| `--warning` | `#f59e0b` | Doctor warnings |
|
||||
| `--error` | `#ef4444` | Failures, destructive confirmations |
|
||||
|
||||
Dark theme is the only theme. No light mode toggle planned — admin is an
|
||||
operator tool, not a marketing surface. Users live in the terminal with a
|
||||
dark theme already.
|
||||
|
||||
WCAG contrast:
|
||||
- Body text (#e0e0e0 on #0a0a0f) → ~14:1, AAA
|
||||
- Muted text (#777 on #0a0a0f) → ~5.5:1, AA (was 4.0 / fail before TD2)
|
||||
- Accent links (#3b82f6 on #0a0a0f) → ~5.7:1, AA
|
||||
|
||||
## Typography
|
||||
|
||||
| Variable | Value | Use |
|
||||
|--------------------|-----------------------------|---------------------------------|
|
||||
| `--font-sans` | `Inter, system-ui, sans-serif` | UI text, headings, body |
|
||||
| `--font-mono` | `JetBrains Mono, monospace` | Numbers, slugs, code, terminal-ish data |
|
||||
|
||||
Type scale (de facto, not formalized yet):
|
||||
- 18px: sidebar logo / page title
|
||||
- 14px: body
|
||||
- 13px: nav items
|
||||
- 12px: chart captions, secondary labels
|
||||
- 11px: tertiary labels in dense charts
|
||||
|
||||
Numbers in tables and metrics use JetBrains Mono so column alignment is
|
||||
mechanical. Avoid mixing Inter and JetBrains Mono in the same line.
|
||||
|
||||
## Spacing scale
|
||||
|
||||
4 / 8 / 16 / 24 / 32px. Linear-app-style density: 24-32px between major
|
||||
sections, 16px between row groups, 8px within a row. The Calibration tab
|
||||
(approved variant-B mockup) is the canonical example.
|
||||
|
||||
## Layout
|
||||
|
||||
- Sidebar 200px on the left. Active item gets a 3px left-border in `--accent`.
|
||||
- Main content area uses the remaining width.
|
||||
- Max content width: 720px for text-heavy pages (Calibration), 960px for
|
||||
data tables (Request Log).
|
||||
- No 3-column feature grids. No icons in colored circles. No decorative blobs.
|
||||
- Cards earn their existence — heading + content works without a card frame
|
||||
in most cases.
|
||||
|
||||
## Charts
|
||||
|
||||
Server-rendered SVG via `src/core/calibration/svg-renderer.ts`. Pure
|
||||
functions: data → SVG string. No DOM, no React component, no chart library.
|
||||
|
||||
XSS posture: server-side `escapeXml()` on every caller-controlled string.
|
||||
Numeric inputs `.toFixed()`-coerced. Admin SPA renders via
|
||||
`<TrustedSVG>` wrapper with `dangerouslySetInnerHTML`. Endpoint gated by
|
||||
`requireAdmin` middleware.
|
||||
|
||||
Why server-rendered SVG (per D23):
|
||||
- Chart logic stays close to the data math.
|
||||
- Zero new client-side chart-library dep.
|
||||
- SVG is accessible (text labels), scalable, copy-paste-friendly to PR
|
||||
descriptions and docs.
|
||||
- Sets the precedent for future admin charts (contradictions trend, takes
|
||||
scorecard, etc.).
|
||||
|
||||
Four chart renderers in v0.36.1.0:
|
||||
- `renderBrierTrend({ series })` — sparkline + baseline reference at 0.25
|
||||
- `renderDomainBars({ bars })` — horizontal accuracy bars
|
||||
- `renderAbandonedThreadsCard(threads)` — text rows + "revisit now" links
|
||||
- `renderPatternStatementsCard(statements)` — clickable drill-down anchors
|
||||
|
||||
## Interaction patterns
|
||||
|
||||
- Keyboard navigation is REQUIRED for all CLI interaction surfaces. The
|
||||
propose-queue review uses J/K/space/u/q shortcuts (gmail-style).
|
||||
- Loading states: "Loading...". Don't show spinners on sub-200ms operations.
|
||||
- Empty states ARE features: warmth + primary action + context. Cold-brain
|
||||
Calibration page tells the user EXACTLY how to build a profile, not
|
||||
"no data available."
|
||||
- Error states: name what failed + name the next step. Never "an error
|
||||
occurred — please try again."
|
||||
|
||||
## What's NOT here yet (v0.37+ roadmap)
|
||||
|
||||
- Type scale formalization (current values are de facto, not enforced)
|
||||
- Animation tokens (admin SPA has zero animations on purpose; v0.37 may
|
||||
add subtle progress / loading transitions)
|
||||
- Print stylesheet
|
||||
- Light mode (NOT planned — see "Dark theme is the only theme" above)
|
||||
- Component library extraction (the React components live inline in admin/src/pages/;
|
||||
no `<Button>` / `<Card>` abstraction layer yet)
|
||||
|
||||
## How to use this document
|
||||
|
||||
When adding a new UI surface to gbrain:
|
||||
|
||||
1. Pick existing tokens before introducing new ones. New tokens go through
|
||||
`/plan-design-review`.
|
||||
2. Match the voice rules. Run candidates through `gateVoice()` before
|
||||
shipping any user-facing string in the calibration surfaces.
|
||||
3. Match the spacing scale and density. Linear-calm-clarity over
|
||||
dashboard-card-mosaic.
|
||||
4. Match the typography: Inter for UI, JetBrains Mono for numbers.
|
||||
|
||||
When updating this document: it's a living target, not a frozen spec.
|
||||
Major changes go through `/plan-design-review` to keep the system coherent.
|
||||
+19
-124
@@ -16,43 +16,32 @@ If you fetched this file by URL without cloning yet, the companion files live at
|
||||
|
||||
## Step 1: Install GBrain
|
||||
|
||||
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain
|
||||
curl -fsSL https://bun.sh/install | bash
|
||||
export PATH="$HOME/.bun/bin:$PATH"
|
||||
bun install -g github:garrytan/gbrain
|
||||
bun install && bun link
|
||||
```
|
||||
|
||||
Verify: `gbrain --version` should print a version number. If `gbrain` is not found,
|
||||
restart the shell or add the PATH export to the shell profile.
|
||||
|
||||
> **If `bun install -g` aborts or `gbrain doctor` reports `schema_version: 0`** (Bun
|
||||
> occasionally blocks the top-level postinstall hook on global installs, so schema
|
||||
> migrations don't run automatically), the CLI prints a recovery hint pointing at
|
||||
> [#218](https://github.com/garrytan/gbrain/issues/218). Run `gbrain apply-migrations --yes`
|
||||
> to recover. If that doesn't work, fall back to the deterministic install path:
|
||||
>
|
||||
> ```bash
|
||||
> git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain
|
||||
> bun install && bun link
|
||||
> ```
|
||||
> **Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
> postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
> aborts with `Aborted()` when it opens PGLite. Use the `git clone + bun link` path
|
||||
> above. Tracking issue: [#218](https://github.com/garrytan/gbrain/issues/218).
|
||||
|
||||
## Step 2: API Keys
|
||||
|
||||
Ask the user for these. gbrain defaults to the ZeroEntropy embedding + reranker stack
|
||||
(as of v0.36.2.0); OpenAI/Voyage are still supported as fallbacks via `gbrain config
|
||||
set embedding_model <provider:model>`.
|
||||
Ask the user for these:
|
||||
|
||||
```bash
|
||||
export ZEROENTROPY_API_KEY=ze-... # default embedding + reranker (v0.36.2.0+)
|
||||
export OPENAI_API_KEY=sk-... # fallback for vector search; also used for chat models
|
||||
export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality via query expansion
|
||||
export OPENAI_API_KEY=sk-... # required for vector search
|
||||
export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality
|
||||
```
|
||||
|
||||
Save to shell profile or `.env`. Keys are picked up by `gbrain config set` automatically
|
||||
or can be stored in `~/.gbrain/config.json` (file plane). Without any embedding provider,
|
||||
keyword search still works. Without Anthropic, search works but skips query expansion.
|
||||
Save to shell profile or `.env`. Without OpenAI, keyword search still works.
|
||||
Without Anthropic, search works but skips query expansion.
|
||||
|
||||
## Step 3: Create the Brain
|
||||
|
||||
@@ -72,63 +61,6 @@ Read `~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md` and set up the MECE directory
|
||||
structure (people/, companies/, concepts/, etc.) inside the user's brain repo,
|
||||
NOT inside ~/gbrain.
|
||||
|
||||
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
|
||||
|
||||
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
|
||||
tier is Haiku-class or no OpenAI key is configured). The init output included the
|
||||
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
|
||||
default. Stop and ask the operator.
|
||||
|
||||
**Present this matrix verbatim:**
|
||||
|
||||
```
|
||||
Per-query cost @ 10K queries/mo (typical single-user volume):
|
||||
|
||||
Haiku 4.5 Sonnet 4.6 Opus 4.7
|
||||
($1/M) ($3/M) ($5/M)
|
||||
conservative $40/mo $120/mo $200/mo
|
||||
balanced $100/mo $300/mo $500/mo
|
||||
tokenmax $200/mo $600/mo $1,000/mo
|
||||
|
||||
(scales linearly: ×10 for 100K/mo, ÷10 for 1K. 25x corner-to-corner spread.
|
||||
Natural diagonal pairings — cheap/cheap → frontier/frontier — span ~4x.)
|
||||
```
|
||||
|
||||
**Ask the operator (paraphrase if needed):**
|
||||
|
||||
> Your gbrain just installed with search mode `<auto-applied default>`. This is
|
||||
> a one-time setup decision that controls retrieval payload size. Which mode
|
||||
> do you want?
|
||||
>
|
||||
> 1) conservative — tight 4K budget, no LLM expansion, 10 chunks max.
|
||||
> Best for Haiku subagents, cost-sensitive setups, high-volume loops.
|
||||
>
|
||||
> 2) balanced — 12K budget, no expansion, 25 chunks. Sonnet-tier sweet spot.
|
||||
>
|
||||
> 3) tokenmax (recommended default — preserves v0.31.x retrieval shape) —
|
||||
> no budget, LLM expansion ON, 50 chunks. Best for Opus/frontier models.
|
||||
>
|
||||
> Cost depends on BOTH the mode AND the downstream model you run. See the
|
||||
> matrix above for the 9-cell breakdown.
|
||||
|
||||
If the operator picks a non-default mode, run:
|
||||
```bash
|
||||
gbrain config set search.mode <mode>
|
||||
```
|
||||
|
||||
If they pick tokenmax AND want to preserve the literal v0.31.x default
|
||||
(limit=20 instead of tokenmax's 50), also run:
|
||||
```bash
|
||||
gbrain config set search.searchLimit 20
|
||||
```
|
||||
|
||||
Verify the choice with `gbrain search modes` before continuing.
|
||||
|
||||
**Why this matters:** the cost spread between corners of the matrix is 25x.
|
||||
An agent that silently accepts the default and starts running queries against
|
||||
a user who didn't expect tokenmax-class context loads can rack up surprise
|
||||
spend. Confirm before continuing.
|
||||
|
||||
## Step 4: Import and Index
|
||||
|
||||
```bash
|
||||
@@ -163,24 +95,8 @@ and supports `--since YYYY-MM-DD` for incremental runs.
|
||||
|
||||
## Step 5: Load Skills
|
||||
|
||||
If you're running an agent platform (OpenClaw, Hermes, or any repo with a workspace),
|
||||
scaffold the bundled skills into it:
|
||||
|
||||
```bash
|
||||
cd /path/to/agent/workspace
|
||||
gbrain skillpack scaffold --all # copy 43 curated skills + RESOLVER.md
|
||||
```
|
||||
|
||||
Scaffolded skills are first-class files in your repo. Edit freely; re-running scaffold
|
||||
refuses to overwrite anything that exists. Use `gbrain skillpack reference <name>` to
|
||||
diff against gbrain's bundle when you want upstream improvements. (The legacy
|
||||
`gbrain skillpack install` managed-block model was retired in v0.36.0.0 — run
|
||||
`gbrain skillpack migrate-fence` once if upgrading from an older release.)
|
||||
|
||||
Whether you scaffolded or not, read `skills/RESOLVER.md` (in your workspace, or the
|
||||
bundled copy at `~/gbrain/skills/RESOLVER.md` when running from the cloned repo). It's
|
||||
the skill dispatcher — tells you which skill to read for any task. Save this to your
|
||||
memory permanently.
|
||||
Read `~/gbrain/skills/RESOLVER.md`. This is the skill dispatcher. It tells you which
|
||||
skill to read for any task. Save this to your memory permanently.
|
||||
|
||||
The three most important skills to adopt immediately:
|
||||
|
||||
@@ -208,17 +124,14 @@ If skipped, minimal defaults are installed automatically.
|
||||
|
||||
## Step 7: Recurring Jobs
|
||||
|
||||
Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab), or skip the
|
||||
platform glue entirely with `gbrain autopilot --install` (built-in self-maintaining daemon):
|
||||
Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab):
|
||||
|
||||
- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale`
|
||||
— or `gbrain sync --watch` for a continuous loop.
|
||||
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install).
|
||||
- **Dream cycle** (nightly): `gbrain dream` runs the 8-phase overnight maintenance cycle.
|
||||
- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install)
|
||||
- **Dream cycle** (nightly): read `docs/guides/cron-schedule.md` for the full protocol.
|
||||
Entity sweep, citation fixes, memory consolidation, plus (v0.23+) overnight conversation
|
||||
synthesis and cross-session pattern detection. One cron-friendly command. This is what
|
||||
makes the brain compound. Do not skip it. See `docs/guides/cron-schedule.md` for the
|
||||
full protocol.
|
||||
synthesis and cross-session pattern detection. 8 phases, one cron-friendly command. This
|
||||
is what makes the brain compound. Do not skip it.
|
||||
- **Weekly**: `gbrain doctor --json && gbrain embed --stale`
|
||||
|
||||
## Step 8: Integrations
|
||||
@@ -236,18 +149,9 @@ actually works) is the most important.
|
||||
|
||||
## Upgrade
|
||||
|
||||
If you installed via `bun install -g`:
|
||||
|
||||
```bash
|
||||
gbrain upgrade # self-updates the binary, runs schema migrations,
|
||||
# and prints post-upgrade notes for the version range
|
||||
```
|
||||
|
||||
If you installed via `git clone + bun link`:
|
||||
|
||||
```bash
|
||||
cd ~/gbrain && git pull origin master && bun install
|
||||
gbrain apply-migrations --yes # apply schema migrations (idempotent)
|
||||
gbrain init # apply schema migrations (idempotent)
|
||||
gbrain post-upgrade # show migration notes for the version range
|
||||
```
|
||||
|
||||
@@ -255,15 +159,6 @@ Then read `~/gbrain/skills/migrations/v<NEW_VERSION>.md` (and any intermediate
|
||||
versions you skipped) and run any backfill or verification steps it lists. Skipping
|
||||
this is how features ship in the binary but stay dormant in the user's brain.
|
||||
|
||||
**v0.32.3 search modes (one-time upgrade prompt):** if the user's brain was
|
||||
created before v0.32.3, `gbrain post-upgrade` prints a banner including the
|
||||
9-cell cost matrix (mode × downstream model) preceded by `[AGENT]` markers.
|
||||
**Do NOT silently move past the banner.** Present the matrix to the operator
|
||||
verbatim, ask which mode they want (recommended default: `tokenmax` to preserve
|
||||
v0.31.x retrieval shape), then run `gbrain config set search.mode <mode>`. See
|
||||
Step 3.5 above for the full ask-the-user protocol — the upgrade path uses the
|
||||
same matrix and same default.
|
||||
|
||||
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
|
||||
`gbrain extract links --source db && gbrain extract timeline --source db` to
|
||||
backfill the new graph layer (see Step 4.5 above).
|
||||
|
||||
@@ -2,17 +2,13 @@
|
||||
|
||||
Your AI agent is smart but forgetful. GBrain gives it a brain.
|
||||
|
||||
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: **17,888 pages, 4,383 people, 723 companies**, 21 cron jobs running autonomously, built in 12 days. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.
|
||||
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain powering his OpenClaw and Hermes deployments: **17,888 pages, 4,383 people, 723 companies**, 21 cron jobs running autonomously, built in 12 days. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up and the brain is smarter than when you went to bed.
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
**New default in v0.36.2.0: ZeroEntropy** for both embedding (`zembed-1` at 1280d via Matryoshka) and reranker (`zerank-2`). On a real-corpus benchmark vs OpenAI and Voyage: **2.2× faster** (442ms vs OpenAI 973ms), **2.6× cheaper at regular pricing** ($0.05/M vs OpenAI $0.13), wins 11 of 20 queries head-to-head, reshuffles 60% of top-1 results when used as a second-pass reranker. Bring your own key from [zeroentropy.dev](https://dashboard.zeroentropy.dev), or stay on OpenAI/Voyage via `gbrain config set embedding_model <provider:model>` — your choice is sticky.
|
||||
GBrain is those patterns, generalized. 34 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
GBrain is those patterns, generalized. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
**New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself.** One command does the loop you used to run by hand: `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. `gbrain doctor --remediation-plan --json` previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (`reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New `--background` flag on `gbrain embed` submits the job and exits with `job_id=N` for shell composition.
|
||||
|
||||
**New in v0.35.7 — Temporal trajectory + founder scorecard.** Author typed metric assertions in the `## Facts` fence (`mrr=50000`, `arr=2000000`, `team_size=12`) and gbrain stores them as first-class typed columns. `gbrain eval trajectory companies/acme-example` prints the chronological history with regressions auto-flagged inline. `gbrain founder scorecard companies/acme-example` rolls up claim accuracy, consistency, growth direction, and red flags into a stable `schema_version: 1` JSON contract. New MCP op `find_trajectory` exposes the same data to agents (read scope, visibility-filtered for remote callers). The `consolidate` cycle phase now writes `valid_until` on chronologically-superseded facts AND uses semantic upsert on `(page_id, claim, since_date)` — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions).
|
||||
**New in v0.25.0 — BrainBench-Real (session capture, contributor opt-in):** with `GBRAIN_CONTRIBUTOR_MODE=1` set in your shell, every real `query` + `search` call through MCP, CLI, or the subagent tool-bridge gets captured (PII-scrubbed) into an `eval_candidates` table. Snapshot with `gbrain eval export`, replay against your code change with `gbrain eval replay`. Three numbers come back: mean Jaccard@k between captured and current retrieved slugs, top-1 stability, and latency Δ. **Off by default** for production users — no surprise data accumulation. Walkthrough: [docs/eval-bench.md](docs/eval-bench.md). NDJSON wire format: [docs/eval-capture.md](docs/eval-capture.md).
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
@@ -20,124 +16,797 @@ GBrain is those patterns, generalized. Install in 30 minutes. Your agent does th
|
||||
|
||||
## Install
|
||||
|
||||
GBrain runs in three shapes. Pick the one that matches how you use AI agents today.
|
||||
### On an agent platform (recommended)
|
||||
|
||||
### Run with your agent platform
|
||||
GBrain is designed to be installed and operated by an AI agent. If you don't have one running yet:
|
||||
|
||||
Already using [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)? GBrain installs as a skillpack scaffold into your agent's workspace.
|
||||
- **[OpenClaw](https://openclaw.ai)** ... Deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
|
||||
- **[Hermes Agent](https://github.com/NousResearch/hermes-agent)** ... Deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
|
||||
|
||||
Paste this into your agent:
|
||||
|
||||
```
|
||||
Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 34 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
|
||||
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
|
||||
agent operating protocol (install, read order, trust boundary, common tasks). For
|
||||
the full doc map, use `llms.txt` at the same URL root.
|
||||
|
||||
### Standalone CLI (no agent)
|
||||
|
||||
```bash
|
||||
gbrain init --pglite
|
||||
gbrain skillpack scaffold --all # or: scaffold <name> per skill
|
||||
git clone https://github.com/garrytan/gbrain.git && cd gbrain && bun install && bun link
|
||||
gbrain init # local brain, ready in 2 seconds
|
||||
gbrain import ~/notes/ # index your markdown
|
||||
gbrain query "what themes show up across my notes?"
|
||||
```
|
||||
|
||||
That's it. Your agent picks up 43 skills (signal detection, brain-ops, ingest, enrich, citation-fixer, daily-task-manager, cron-scheduler, eval framework, and 35 more). Routing lives in `skills/RESOLVER.md` — the agent reads it once per request, picks the right skill, executes. Scaffolded skills are first-class members of your agent repo — you own them, edit freely; `gbrain skillpack reference <name>` diffs your copy against gbrain's bundle when you want to pull upstream improvements. (The legacy `gbrain skillpack install` managed-block model was retired in v0.36.0.0; run `gbrain skillpack migrate-fence` once if you're upgrading from an older release.)
|
||||
**Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
aborts with `Aborted()` the first time it opens PGLite. Use `git clone + bun install
|
||||
&& bun link` as shown above. See [#218](https://github.com/garrytan/gbrain/issues/218).
|
||||
|
||||
### CLI standalone
|
||||
**Do NOT use `bun add -g gbrain` or `npm install -g gbrain`.** The npm registry
|
||||
has an unrelated package squatting that name (`gbrain@1.3.x`) — you'd silently
|
||||
install the wrong binary and overwrite the canonical one. v0.28.5+ detects this
|
||||
and prints a recovery message on `gbrain upgrade`, but the `git clone + bun link`
|
||||
path above is the only reliable install method until we publish under
|
||||
`@garrytan/gbrain` (tracked v0.29 follow-up). See
|
||||
[#658](https://github.com/garrytan/gbrain/issues/658).
|
||||
|
||||
Use gbrain from any shell, no agent platform required.
|
||||
```
|
||||
3 results (hybrid search, 0.12s):
|
||||
|
||||
1. concepts/do-things-that-dont-scale (score: 0.94)
|
||||
PG's argument that unscalable effort teaches you what users want.
|
||||
[Source: paulgraham.com, 2013-07-01]
|
||||
|
||||
2. originals/founder-mode-observation (score: 0.87)
|
||||
Deep involvement isn't micromanagement if it expands the team's thinking.
|
||||
|
||||
3. concepts/build-something-people-want (score: 0.81)
|
||||
The YC motto. Connected to 12 other brain pages.
|
||||
```
|
||||
|
||||
### MCP server (Claude Code, Cursor, Windsurf)
|
||||
|
||||
GBrain exposes 30+ MCP tools via stdio:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"gbrain": { "command": "gbrain", "args": ["serve"] }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), or your client's MCP config.
|
||||
|
||||
### Remote MCP with OAuth 2.1 (ChatGPT, Claude Desktop, Cowork, Perplexity)
|
||||
|
||||
`gbrain serve --http` starts a production-grade OAuth 2.1 server with an embedded admin dashboard. Zero external infrastructure. Every major AI client connects, every request is scoped, every action is logged.
|
||||
|
||||
```bash
|
||||
bun install -g github:garrytan/gbrain
|
||||
gbrain init --pglite # 2 seconds; no server, no Docker
|
||||
gbrain doctor # verify health
|
||||
# Start the HTTP server (prints admin bootstrap token on first start)
|
||||
gbrain serve --http --port 3131
|
||||
|
||||
# Open the admin dashboard, paste the bootstrap token, register a client
|
||||
open http://localhost:3131/admin
|
||||
|
||||
# Expose publicly (set --public-url so the OAuth issuer matches)
|
||||
ngrok http 3131 --url your-brain.ngrok.app
|
||||
gbrain serve --http --port 3131 --public-url https://your-brain.ngrok.app
|
||||
|
||||
# ChatGPT and other OAuth-aware clients can also connect:
|
||||
claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN"
|
||||
```
|
||||
|
||||
Then point any MCP-aware client (Claude Code, Cursor, Windsurf) at it, or use it from your shell:
|
||||
Register OAuth clients from the `/admin` dashboard — click **Register client**,
|
||||
pick scopes, save the credentials shown once in the reveal modal. Programmatic
|
||||
registration via `oauthProvider.registerClientManual(...)` and the
|
||||
`gbrain auth register-client` CLI are also available.
|
||||
|
||||
- **OAuth 2.1 via the MCP SDK** — client credentials (machine-to-machine: Perplexity, Claude), authorization code + PKCE (browser-based: ChatGPT), refresh token rotation, revocation, protected resource metadata. Optional Dynamic Client Registration behind `--enable-dcr` (DCR redirect_uris must be `https://` or loopback per RFC 6749 §3.1.2.1).
|
||||
- **Scoped operations** — 30 operations tagged `read | write | admin`. `sync_brain` and `file_upload` are `localOnly`, rejected over HTTP.
|
||||
- **React admin dashboard** — 7 screens baked into the binary (~65KB gzip). Live SSE activity feed, agents table, credential reveal, filterable request log, per-client config export.
|
||||
- **Legacy bearer tokens still work** — pre-v0.26 `gbrain auth create` tokens continue to authenticate as `read+write+admin`. v0.22.7's simpler `src/mcp/http-transport.ts` path stays compiled in for backward compat callers; v0.26+ deployments use the OAuth-aware `serve-http.ts`.
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). Hardening defaults, env vars, and threat model: [SECURITY.md](SECURITY.md).
|
||||
|
||||
### Using gbrain with GStack
|
||||
|
||||
If your engineering agent runs on [GStack](https://github.com/garrytan/gstack), point it at gbrain for code lookup instead of grep+read. Cathedral II (v0.21.0) ships call-graph edges and two-pass retrieval — `/investigate`, `/review`, `/plan-eng-review`, and `/office-hours` all benefit when the agent walks the symbol graph instead of scanning files line by line.
|
||||
|
||||
The five magical-moment commands:
|
||||
|
||||
```bash
|
||||
gbrain search "who works at acme AI?"
|
||||
gbrain query "what did bob invest in this quarter?"
|
||||
gbrain graph-query people/garry-tan --depth 2
|
||||
gbrain code-callers searchKeyword # who calls this symbol?
|
||||
gbrain code-callees searchKeyword # what does this symbol call?
|
||||
gbrain code-def BrainEngine # where is X defined?
|
||||
gbrain code-refs BrainEngine # all reference sites
|
||||
gbrain query "how does N+1 handling work" --near-symbol BrainEngine.searchKeyword --walk-depth 2
|
||||
```
|
||||
|
||||
Detailed setup paths (Postgres at scale, Supabase, thin-client mode) live in [`docs/INSTALL.md`](docs/INSTALL.md).
|
||||
All five auto-emit JSON on non-TTY (gh-CLI convention) so a GStack subagent shelling out via bash gets a clean parseable response. Run `gbrain sources add <repo> --strategy code` to index a repo, then your agent's brain-first lookup covers code, not just markdown. ([Cathedral II release notes](CHANGELOG.md#0210---2026-04-25))
|
||||
|
||||
### MCP server (any MCP client)
|
||||
## The 34 Skills
|
||||
|
||||
GBrain ships 34 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task. v0.25.1 added 9 research-flavored skills (`book-mirror` flagship plus 8 pairings); see the new "Research and synthesis" section below.
|
||||
|
||||
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
|
||||
|
||||
### Always-on
|
||||
|
||||
| Skill | What it does |
|
||||
|-------|-------------|
|
||||
| **signal-detector** | Fires on every message. Spawns a cheap model in parallel to capture original thinking and entity mentions. The brain compounds on autopilot. |
|
||||
| **brain-ops** | Brain-first lookup before any external API. The read-enrich-write loop that makes every response smarter. |
|
||||
|
||||
### Content ingestion
|
||||
|
||||
| Skill | What it does |
|
||||
|-------|-------------|
|
||||
| **ingest** | Thin router. Detects input type and delegates to the right ingestion skill. |
|
||||
| **idea-ingest** | Links, articles, tweets become brain pages with analysis, author people pages, and cross-linking. |
|
||||
| **media-ingest** | Video, audio, PDF, books, screenshots, GitHub repos. Transcripts, entity extraction, backlink propagation. |
|
||||
| **meeting-ingestion** | Transcripts become brain pages. Every attendee gets enriched. Every company gets a timeline entry. |
|
||||
| **voice-note-ingest** | Voice notes captured verbatim — exact phrasing preserved, never paraphrased. Routes to originals/concepts/people/companies/ideas/personal/voice-notes based on content. |
|
||||
| **article-enrichment** | Raw article dumps become structured pages with executive summary, verbatim quotes, key insights, and why-it-matters. |
|
||||
|
||||
### Research and synthesis (v0.25.1)
|
||||
|
||||
| Skill | What it does |
|
||||
|-------|-------------|
|
||||
| **book-mirror** | Flagship. Hand the agent a book, get a personalized two-column chapter-by-chapter analysis. Left column preserves the chapter's actual content; right column maps every idea to your life using your words from the brain. ~$6 for a 20-chapter book at Opus. Pairs with `gbrain book-mirror` CLI for the trusted runtime. |
|
||||
| **strategic-reading** | Read a book / article / case study through ONE specific problem-lens. Output: applied playbook with do / avoid / watch-for and short / medium / long-term recommendations. |
|
||||
| **concept-synthesis** | Deduplicate thousands of concept stubs into a tiered intellectual map (T1 Canon to T4 Riff). Trace how ideas evolved across years of notes. |
|
||||
| **perplexity-research** | Brain-augmented web research. Sends brain context to Perplexity so the search focuses on what's NEW vs already-known. Output: Executive Summary + Key New Developments + Confirming Signals + Contradictions or Updates + Recommended Brain Updates + Citations. |
|
||||
| **archive-crawler** | Universal archivist for personal file archives (Dropbox / Backblaze / Gmail-takeout / hard-drive dumps). REFUSES to run unless `archive-crawler.scan_paths:` is set in `gbrain.yml`. Safe-by-default safety fence. |
|
||||
| **academic-verify** | Trace a research claim through publication → methodology → raw data → independent replication. Routes through perplexity-research; produces a verdict (verified / partial / unverifiable / misattributed / retracted). |
|
||||
| **brain-pdf** | Render any brain page to publication-quality PDF via the gstack `make-pdf` binary. Strips frontmatter, sanitizes emoji, applies running headers. |
|
||||
|
||||
### Brain operations
|
||||
|
||||
| Skill | What it does |
|
||||
|-------|-------------|
|
||||
| **enrich** | Tiered enrichment (Tier 1/2/3). Creates and updates person/company pages with compiled truth and timelines. |
|
||||
| **query** | 3-layer search with synthesis and citations. Says "the brain doesn't have info on X" instead of hallucinating. |
|
||||
| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. v0.23 adds the dream cycle's synthesize + patterns phases ... overnight conversation transcripts become reflections, originals, and 25-year patterns. |
|
||||
| **citation-fixer** | Scans pages for missing or malformed citations. Fixes format to match the standard. |
|
||||
| **repo-architecture** | Where new brain files go. Decision protocol: primary subject determines directory, not format. |
|
||||
| **publish** | Share brain pages as password-protected HTML. Zero LLM calls. |
|
||||
| **data-research** | Structured data research with parameterized YAML recipes. Extract investor updates, expenses, company metrics from email. |
|
||||
|
||||
### Operational
|
||||
|
||||
| Skill | What it does |
|
||||
|-------|-------------|
|
||||
| **daily-task-manager** | Task lifecycle with priority levels (P0-P3). Stored as searchable brain pages. |
|
||||
| **daily-task-prep** | Morning prep: calendar lookahead with brain context per attendee, open threads, task review. |
|
||||
| **cron-scheduler** | Schedule staggering (5-min offsets), quiet hours (timezone-aware with wake-up override), idempotency. |
|
||||
| **reports** | Timestamped reports with keyword routing. "What's the latest briefing?" finds it instantly. |
|
||||
| **cross-modal-review** | Quality gate via second model. Refusal routing: if one model refuses, silently switch. |
|
||||
| **webhook-transforms** | External events (SMS, meetings, social mentions) converted into brain pages with entity extraction. |
|
||||
| **testing** | Validates every skill has SKILL.md with frontmatter, manifest coverage, resolver coverage. |
|
||||
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
|
||||
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
|
||||
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
|
||||
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
|
||||
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
|
||||
|
||||
### Identity and setup
|
||||
|
||||
| Skill | What it does |
|
||||
|-------|-------------|
|
||||
| **soul-audit** | 6-phase interview generating SOUL.md (agent identity), USER.md (user profile), ACCESS_POLICY.md (4-tier privacy), HEARTBEAT.md (operational cadence). |
|
||||
| **setup** | Auto-provision PGLite or Supabase. First import. GStack detection. |
|
||||
| **migrate** | Universal migration from Obsidian, Notion, Logseq, markdown, CSV, JSON, Roam. |
|
||||
| **briefing** | Daily briefing with meeting context, active deals, and citation tracking. |
|
||||
|
||||
### Conventions
|
||||
|
||||
Cross-cutting rules in `skills/conventions/`:
|
||||
- **quality.md** ... citations, back-links, notability gate, source attribution
|
||||
- **brain-first.md** ... 5-step lookup before any external API call
|
||||
- **model-routing.md** ... which model for which task
|
||||
- **test-before-bulk.md** ... test 3-5 items before any batch operation
|
||||
- **cross-modal.yaml** ... review pairs and refusal routing chain
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
Signal arrives (meeting, email, tweet, link)
|
||||
-> Signal detector captures ideas + entities (parallel, never blocks)
|
||||
-> Brain-ops: check the brain first (gbrain search, gbrain get)
|
||||
-> Respond with full context
|
||||
-> Write: update brain pages with new information + citations
|
||||
-> Auto-link: typed relationships extracted on every write (zero LLM calls)
|
||||
-> Sync: gbrain indexes changes for next query
|
||||
```
|
||||
|
||||
Every cycle adds knowledge. The agent enriches a person page after a meeting. Next time that person comes up, the agent already has context. The difference compounds daily.
|
||||
|
||||
The system gets smarter on its own. Entity enrichment auto-escalates: a person mentioned once gets a stub page (Tier 3). After 3 mentions across different sources, they get web + social enrichment (Tier 2). After a meeting or 8+ mentions, full pipeline (Tier 1). The brain learns who matters without being told. Deterministic classifiers improve over time via a fail-improve loop that logs every LLM fallback and generates better regex patterns from the failures. `gbrain doctor` shows the trajectory: "intent classifier: 87% deterministic, up from 40% in week 1."
|
||||
|
||||
> "Prep me for my meeting with Jordan in 30 minutes"
|
||||
> ... pulls dossier, shared history, recent activity, open threads
|
||||
|
||||
> "What have I said about the relationship between shame and founder performance?"
|
||||
> ... searches YOUR thinking, not the internet
|
||||
|
||||
## Minions: your sub-agents won't drop work anymore
|
||||
|
||||
A durable, Postgres-native job queue built into the brain. Every long-running agent task is now a job that survives gateway restarts, streams progress, gets paused / resumed / steered mid-flight, and shows up in `gbrain jobs list`. Zero infra beyond your existing brain.
|
||||
|
||||
### The production numbers that matter
|
||||
|
||||
Here's my personal OpenClaw deployment: one Render container. Supabase Postgres holding a 45,000-page brain. 19 cron jobs firing on schedule. Real gateway load from real daily work. The task: pull a month of my social posts from an external API and ingest them end-to-end into the brain as a structured page.
|
||||
|
||||
| | Minions | `sessions_spawn` |
|
||||
|--- |--- |--- |
|
||||
| Wall time | **753ms** | **>10,000ms** (gateway timeout) |
|
||||
| Token cost | **$0.00** | ~$0.03 per run |
|
||||
| Success rate | **100%** | **0%** (couldn't even spawn) |
|
||||
| Memory/job | ~2 MB | ~80 MB |
|
||||
|
||||
Under that 19-cron load, sub-agent spawn couldn't clear the 10-second gateway wall. Minions landed it in under a second for zero tokens. **Scaling:** 19,240 posts across 36 months, single bash loop, ~15 min total, $0.00. Sub-agents: ~9 min best case, ~$1.08 in tokens, ~40% spawn failure. **Lab:** durability ∞ (SIGKILL mid-flight, 10/10 rescued), throughput ~10× faster, fan-out ~21× with no failure wall, memory ~400× less.
|
||||
|
||||
Full benchmarks live in [gbrain-evals](https://github.com/garrytan/gbrain-evals/tree/main/docs/benchmarks).
|
||||
|
||||
### The routing rule
|
||||
|
||||
> **Deterministic** (same input → same steps → same output) → **Minions**
|
||||
> **Judgment** (input requires assessment or decision) → **Sub-agents**
|
||||
|
||||
Pull posts, parse JSON, write a brain page, run a sync — deterministic. $0 tokens, survives restart, millisecond runtime. Triage the inbox, assess meeting priority, decide if a cold email deserves a reply — judgment. What sub-agents are actually good at. `minion_mode: pain_triggered` (the default) automates the routing.
|
||||
|
||||
### What's fixed
|
||||
|
||||
The six daily pains — spawn storms, agents that stop responding, forgotten dispatches, gateway crashes mid-run, runaway grandchildren, debugging soup — all belonged to the "deterministic work through a reasoning model" mistake. Minions fixes them by not making that mistake: `max_children` cap, `timeout_ms` + AbortSignal, `child_done` inbox, full `parent_job_id`/`depth`/transcript per job, Postgres durability with stall detection, cascade cancel via recursive CTE. Plus idempotency keys, attachment validation, `removeOnComplete`, and `gbrain jobs smoke` that proves the install in half a second.
|
||||
|
||||
```bash
|
||||
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
|
||||
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
|
||||
# at /admin, SSE activity feed at /admin/events
|
||||
gbrain jobs smoke # verify install
|
||||
gbrain jobs submit sync --params '{}' # fire a background job
|
||||
gbrain jobs stats # health dashboard
|
||||
gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only)
|
||||
gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`)
|
||||
```
|
||||
|
||||
Per-client guides (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) live under [`docs/mcp/`](docs/mcp/). HTTP server supports DCR-style client registration, scope-gated access (`read`/`write`/`admin`), and built-in rate limiting.
|
||||
`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md).
|
||||
|
||||
## What it does (the loop)
|
||||
Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox.
|
||||
|
||||
```
|
||||
signal → search → respond → write → auto-link → sync
|
||||
(every (brain-first (informed (page + (typed edges (cron
|
||||
message) retrieval) by context) timeline) + backlinks) keeps fresh)
|
||||
**Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now.
|
||||
|
||||
### Health check and self-heal
|
||||
|
||||
Minions is canonical as of v0.11.1 — every `gbrain upgrade` runs the migration automatically (schema → smoke → prefs → host rewrites → env-aware autopilot install). If you ever want to verify manually or wire a cron into your morning briefing:
|
||||
|
||||
```bash
|
||||
gbrain doctor # half-migrated state? prints loud banner + exits non-zero
|
||||
gbrain skillpack-check --quiet # exit 0/1/2 for pipeline gating
|
||||
gbrain skillpack-check | jq # full JSON: {healthy, summary, actions[], doctor, migrations}
|
||||
```
|
||||
|
||||
- **Signal detector** runs on every message your agent receives. Captures ideas, entity mentions, time-sensitive todos, names, links.
|
||||
- **Brain-first lookup** before any external API call. The cheapest, fastest, most personal information source you have.
|
||||
- **Auto-link** fires on every page write. No LLM calls; pure pattern matching on `[[wiki/people/bob]]` style references. New entity → new page stub → graph grows.
|
||||
- **Cron-driven enrichment** runs while you sleep: dedup people pages, fix citations, score salience, find contradictions, prep tomorrow's tasks.
|
||||
If anything's off, `actions[]` tells you the exact command to run. For deeper troubleshooting: [`docs/guides/minions-fix.md`](docs/guides/minions-fix.md).
|
||||
|
||||
The whole loop is described in [`docs/architecture/topologies.md`](docs/architecture/topologies.md) with diagrams.
|
||||
Moving gateway crons to Minions (deterministic scripts, zero LLM tokens per fire): [`docs/guides/minions-shell-jobs.md`](docs/guides/minions-shell-jobs.md).
|
||||
|
||||
## Capabilities
|
||||
## Durable agents: `gbrain agent` (v0.15)
|
||||
|
||||
**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on.
|
||||
Your subagent runs survive crashes now. OpenClaw died mid-run? The worker re-claims on restart and replays from the last committed turn. Fan-out across 50 shards, one shard crashes — the aggregator still claims after every child reaches a terminal state and writes a mixed-outcome summary. Tool calls persist as a two-phase ledger (`pending` → `complete | failed`) so replay is safe by construction, not by hope.
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG.
|
||||
```bash
|
||||
# Submit a single-subagent run
|
||||
gbrain agent run "summarize my last 10 journal pages"
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
# Fan out N prompts across N subagent children + 1 aggregator
|
||||
gbrain agent run "analyze every page" \
|
||||
--fanout-manifest manifests/pages.json \
|
||||
--subagent-def analyzer
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
# Tail a running job (heartbeat per turn + full transcript on completion)
|
||||
gbrain agent logs 1247 --follow --since 5m
|
||||
```
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
Durability is the point: every Anthropic turn commits to `subagent_messages`, every tool call to `subagent_tool_executions`. Worker kills, OpenClaw crashes, timeouts — all resumable. Host repos (your OpenClaw, etc.) ship their own subagent definitions via `GBRAIN_PLUGIN_PATH` + a `gbrain.plugin.json` manifest: see [`docs/guides/plugin-authors.md`](docs/guides/plugin-authors.md). Requires `ANTHROPIC_API_KEY` on the worker.
|
||||
|
||||
**Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle.
|
||||
## Skillify: say "skillify it!" and the bug becomes structurally impossible to repeat
|
||||
|
||||
## Integrations
|
||||
Your OpenClaw hit a new failure. You fix it once in conversation. You say "skillify it!"
|
||||
And now the fix is permanent: a SKILL.md with triggers, a deterministic script with tests, a
|
||||
routing fixture the agent re-evaluates daily, a filing audit that keeps the output from
|
||||
drifting. Ten items. Every one required. The bug can't recur.
|
||||
|
||||
Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in `recipes/` and is discoverable via `gbrain integrations list`.
|
||||
Hermes and similar agent frameworks auto-create skills as a background behavior. Fine until
|
||||
you don't know what the agent shipped. Checklists decay. Tests drift. Resolver entries get
|
||||
stale. Six months later it's an opaque pile nobody has read, nobody has tested, and nobody
|
||||
is sure still works. GBrain ships the same capability except the human stays in the loop
|
||||
and every step is a command you can run.
|
||||
|
||||
- **Voice**: Phone calls create brain pages via Twilio + OpenAI Realtime (or DIY STT+LLM+TTS). Setup recipe: [`recipes/twilio-voice-brain.md`](recipes/twilio-voice-brain.md).
|
||||
- **Email + calendar**: webhook handlers that route to brain signals. [`docs/integrations/meeting-webhooks.md`](docs/integrations/meeting-webhooks.md).
|
||||
- **Embedding providers**: 14 recipes covering OpenAI (default fallback), Voyage, ZeroEntropy (default), Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy. Pricing matrix + decision tree in [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md).
|
||||
- **Credential gateway**: vault-aware secret distribution. [`docs/integrations/credential-gateway.md`](docs/integrations/credential-gateway.md).
|
||||
- **MCP clients**: every major MCP client is supported. [`docs/mcp/`](docs/mcp/) per-client setup.
|
||||
### The four verbs you need (v0.19)
|
||||
|
||||
```bash
|
||||
# 1. Scaffold all 5 stub files for a new skill in one shot.
|
||||
gbrain skillify scaffold webhook-verify \
|
||||
--description "verify ngrok webhooks" \
|
||||
--triggers "verify the webhook,check tunnel" \
|
||||
--writes-pages --writes-to people/,companies/
|
||||
|
||||
# 2. Replace the SKILLIFY_STUB sentinels with real logic + real tests.
|
||||
$EDITOR skills/webhook-verify/scripts/webhook-verify.mjs
|
||||
$EDITOR test/webhook-verify.test.ts
|
||||
|
||||
# 3. Run the 10-item audit: SKILL.md exists, script exists, unit + E2E tests,
|
||||
# LLM evals, resolver entry, trigger eval, check-resolvable gate, brain filing.
|
||||
gbrain skillify check skills/webhook-verify/scripts/webhook-verify.mjs
|
||||
|
||||
# 4. Verify the whole tree: reachability, MECE overlap, DRY, routing gaps,
|
||||
# filing audit, SKILLIFY_STUB sentinels (fails if any skill still has one).
|
||||
gbrain check-resolvable # warnings advisory, errors block
|
||||
gbrain check-resolvable --strict # warnings block too (CI opt-in)
|
||||
```
|
||||
|
||||
Idempotent re-runs. `--force` regenerates stub files but NEVER duplicates a resolver row.
|
||||
Scaffold completes in under 2 seconds. The real work (your rule, your script, your tests)
|
||||
is what you spend time on. Everything else is boilerplate the CLI writes for you.
|
||||
|
||||
### `gbrain routing-eval` — catch the routing gaps your users actually hit
|
||||
|
||||
Drop a `routing-eval.jsonl` fixture next to any skill. Each line is `{intent, expected_skill,
|
||||
ambiguous_with?}`. `gbrain check-resolvable` runs the structural layer by default; `gbrain
|
||||
routing-eval` runs the same structural layer as a dedicated CI verb. The `--llm` flag is
|
||||
accepted as a placeholder for a future LLM tie-break layer; in this release it emits a stderr
|
||||
notice and runs structural only. False positives (wrong skill matched), missed routes (no
|
||||
skill matched), and tautological fixtures (intent copies trigger verbatim) all surface as
|
||||
specific advisories with the exact file:line to fix.
|
||||
|
||||
### Works on your OpenClaw, not just gbrain's repo
|
||||
|
||||
v0.19 teaches `gbrain check-resolvable` to accept `AGENTS.md` as a resolver file alongside
|
||||
`RESOLVER.md`, at either the skills directory OR one level up (OpenClaw-native workspace-root
|
||||
layout). The skill manifest auto-derives from walking `skills/*/SKILL.md` when `manifest.json`
|
||||
is missing. Set `OPENCLAW_WORKSPACE=~/your-openclaw/workspace` and everything just works:
|
||||
|
||||
```bash
|
||||
export OPENCLAW_WORKSPACE=~/your-openclaw/workspace
|
||||
gbrain check-resolvable --verbose
|
||||
# Auto-detects: AGENTS.md at workspace root, 107 skills derived from SKILL.md walk,
|
||||
# 15 unreachable errors surfaced, 108 advisory warnings for overlaps and gaps.
|
||||
```
|
||||
|
||||
First run on a real OpenClaw deployment found 15 unreachable skills out of 102 — about 15%
|
||||
of the tree was dark. The essay's "skills the agent can never reach" footgun, now visible.
|
||||
|
||||
### `gbrain skillpack install` — drop 25 curated skills into your OpenClaw
|
||||
|
||||
The skills gbrain ships are a curated bundle. Install them into your workspace with
|
||||
dependency closure (shared conventions come along), per-file diff protection (your local
|
||||
edits are never clobbered without `--overwrite-local`), a file lock that serializes
|
||||
concurrent installers, and an atomic managed-block update to your AGENTS.md so you can
|
||||
see exactly what gbrain wrote.
|
||||
|
||||
```bash
|
||||
gbrain skillpack list # 25 curated skills
|
||||
gbrain skillpack install brain-ops # one skill + its shared conventions
|
||||
gbrain skillpack install --all # the full bundle
|
||||
gbrain skillpack install brain-ops --dry-run # preview; no writes
|
||||
gbrain skillpack diff brain-ops # compare bundle vs your local copy
|
||||
```
|
||||
|
||||
Re-running is safe. The managed-block markers in your AGENTS.md let `skillpack install`
|
||||
accumulate rows across separate single-skill installs instead of overwriting each other.
|
||||
A receipt comment inside the fence (`<!-- gbrain:skillpack:manifest cumulative-slugs="..." -->`)
|
||||
tracks what gbrain has installed across runs. `install --all` is the only path that prunes;
|
||||
per-skill install never deletes what it didn't install. If you hand-add a row inside the fence,
|
||||
gbrain preserves it on reinstall and emits a stderr notice telling your agent to investigate.
|
||||
|
||||
**Skillify is the piece that makes the skills tree survive six months of compounding work.**
|
||||
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
|
||||
and the anti-patterns it catches.
|
||||
|
||||
## Storage tiering: keep bulk content out of git (v0.22.11)
|
||||
|
||||
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
|
||||
becomes the size driver, declare which directories belong in git and which live in the database only.
|
||||
|
||||
```yaml
|
||||
# gbrain.yml at the brain repo root
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
|
||||
repopulates missing files from the database (container restart, fresh clone, accidental rm).
|
||||
`gbrain storage status` shows the tier breakdown.
|
||||
|
||||
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
|
||||
|
||||
## Getting Data In
|
||||
|
||||
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
|
||||
|
||||
| Recipe | Requires | What It Does |
|
||||
|--------|----------|-------------|
|
||||
| [Public Tunnel](recipes/ngrok-tunnel.md) | — | Fixed URL for MCP + voice (ngrok Hobby $8/mo) |
|
||||
| [Credential Gateway](recipes/credential-gateway.md) | — | Gmail + Calendar access |
|
||||
| [Voice-to-Brain](recipes/twilio-voice-brain.md) | ngrok-tunnel | Phone calls to brain pages (Twilio + OpenAI Realtime) |
|
||||
| [Email-to-Brain](recipes/email-to-brain.md) | credential-gateway | Gmail to entity pages |
|
||||
| [X-to-Brain](recipes/x-to-brain.md) | — | Twitter timeline + mentions + deletions |
|
||||
| [Calendar-to-Brain](recipes/calendar-to-brain.md) | credential-gateway | Google Calendar to searchable daily pages |
|
||||
| [Meeting Sync](recipes/meeting-sync.md) | — | Circleback transcripts to brain pages with attendees |
|
||||
| [Restart Sweep](recipes/restart-sweep.md) | OpenClaw + Telegram | Detect dropped Telegram messages after OpenClaw gateway restarts |
|
||||
|
||||
**Data research recipes** extract structured data from email into tracked brain pages. Built-in recipes for investor updates (MRR, ARR, runway, headcount), expense tracking, and company metrics. Create your own with `gbrain research init`.
|
||||
|
||||
Run `gbrain integrations` to see status.
|
||||
|
||||
## GBrain + GStack
|
||||
|
||||
[GStack](https://github.com/garrytan/gstack) is the engine. GBrain is the mod.
|
||||
|
||||
- **[GStack](https://github.com/garrytan/gstack)** = coding skills (ship, review, QA, investigate, office-hours, retro). 70,000+ stars, 30,000 developers per day. When your agent codes on itself, it uses GStack.
|
||||
- **GBrain** = everything-else skills (brain ops, signal detection, ingestion, enrichment, cron, reports, identity). When your agent remembers, thinks, and operates, it uses GBrain.
|
||||
- **`hosts/gbrain.ts`** = the bridge. Tells GStack's coding skills to check the brain before coding.
|
||||
|
||||
`gbrain init` detects if GStack is installed and reports mod status. If GStack isn't there, it tells you how to get it.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Two engines, one contract.** PGLite (Postgres 17 via WASM, zero-config, default) for personal brains up to ~50K pages. Postgres + pgvector (Supabase or self-hosted) for shared / large / multi-machine deployments. The contract-first `BrainEngine` interface in [`src/core/engine.ts`](src/core/engine.ts) defines ~47 operations both engines implement; CLI and MCP server are generated from one source.
|
||||
```
|
||||
┌──────────────────┐ ┌───────────────┐ ┌──────────────────┐
|
||||
│ Brain Repo │ │ GBrain │ │ AI Agent │
|
||||
│ (git) │ │ (retrieval) │ │ (read/write) │
|
||||
│ │ │ │ │ │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
|
||||
│ = source of │ │ pgvector │ │ define HOW to │
|
||||
│ truth │ │ │ │ use the brain │
|
||||
│ │<───│ hybrid │ │ │
|
||||
│ human can │ │ search │ │ RESOLVER.md │
|
||||
│ always read │ │ (vector + │ │ routes intent │
|
||||
│ & edit │ │ keyword + │ │ to skill │
|
||||
│ │ │ RRF) │ │ │
|
||||
└──────────────────┘ └───────────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
**Brain repo is the system of record.** Your knowledge lives in a regular git repo (your "brain repo") as markdown files. GBrain syncs the repo into Postgres for retrieval; deletes in git become soft-deletes in DB. You can publish public subsets, share team mounts, run thin-client setups pointing at a colleague's brain server. Topologies in [`docs/architecture/topologies.md`](docs/architecture/topologies.md).
|
||||
The repo is the system of record. GBrain is the retrieval layer. The agent reads and writes through both. Human always wins... edit any markdown file and `gbrain sync` picks up the changes.
|
||||
|
||||
**Two organizational axes (brain ⊥ source).** A *brain* is a database (your personal brain, a team mount you joined). A *source* is a repo inside that brain (wiki, gstack, an essay, a knowledge base). Routing lives in `.gbrain-source` dotfiles and resolves via a documented 6-tier precedence chain. Full diagrams in [`docs/architecture/brains-and-sources.md`](docs/architecture/brains-and-sources.md).
|
||||
## The Knowledge Model
|
||||
|
||||
**Why the graph matters.** Vector search returns chunks that are semantically close. The graph returns chunks that are factually connected. Hybrid search pulls from both; auto-linking on every write keeps the graph fresh. Deep dive: [`docs/architecture/RETRIEVAL.md`](docs/architecture/RETRIEVAL.md).
|
||||
Every page follows the compiled truth + timeline pattern:
|
||||
|
||||
```markdown
|
||||
---
|
||||
type: concept
|
||||
title: Do Things That Don't Scale
|
||||
tags: [startups, growth, pg-essay]
|
||||
---
|
||||
|
||||
Paul Graham's argument that startups should do unscalable things early on.
|
||||
The key insight: the unscalable effort teaches you what users actually
|
||||
want, which you can't learn any other way.
|
||||
|
||||
---
|
||||
|
||||
- 2013-07-01: Published on paulgraham.com
|
||||
- 2024-11-15: Referenced in batch W25 kickoff talk
|
||||
```
|
||||
|
||||
Above the `---`: **compiled truth**. Your current best understanding. Gets rewritten when new evidence changes the picture. Below: **timeline**. Append-only evidence trail. Never edited, only added to.
|
||||
|
||||
## Knowledge Graph
|
||||
|
||||
Pages aren't just text. Every mention of a person, company, or concept becomes a typed link in a structured graph. The brain wires itself.
|
||||
|
||||
```
|
||||
Write a meeting page mentioning Alice and Acme AI
|
||||
-> Auto-link extracts entity refs from content (zero LLM calls)
|
||||
-> Infers types: meeting page + person ref => `attended`
|
||||
"CEO of X" pattern => `works_at`
|
||||
"invested in" => `invested_in`
|
||||
"advises", "advisor" => `advises`
|
||||
"founded", "co-founded" => `founded`
|
||||
-> Reconciles stale links: edits remove links no longer in content
|
||||
-> Backlinks rank well-connected entities higher in search
|
||||
```
|
||||
|
||||
```bash
|
||||
gbrain graph-query people/alice --type attended --depth 2
|
||||
# returns who Alice met with, transitively
|
||||
```
|
||||
|
||||
The graph powers questions vector search can't: "who works at Acme AI?", "what has Bob invested in?", "find the connection between Alice and Carol". Backfill an existing brain in one command:
|
||||
|
||||
```bash
|
||||
gbrain extract links --source db # wire up the existing 29K pages
|
||||
gbrain extract timeline --source db # extract dated events from markdown timelines
|
||||
```
|
||||
|
||||
Then ask graph questions or watch the search ranking improve. Benchmarked side-by-side against ripgrep-BM25, vector-only RAG (same embedder), and gbrain-with-graph-disabled: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating hybrid-nograph by **+31.4 points P@5**. Isolate the contribution: v0.11→v0.12 moved the same gbrain codebase from P@5 22.1% → 49.1% on identical inputs, so typed-link extract quality is load-bearing. Full scorecards + reproducible corpus: [gbrain-evals](https://github.com/garrytan/gbrain-evals).
|
||||
|
||||
## Search
|
||||
|
||||
Hybrid search: vector + keyword + RRF fusion + multi-query expansion + 4-layer dedup.
|
||||
|
||||
```
|
||||
Query
|
||||
-> Intent classifier (entity? temporal? event? general?)
|
||||
-> Multi-query expansion (Claude Haiku)
|
||||
-> Vector search (HNSW cosine) + Keyword search (tsvector)
|
||||
-> RRF fusion: score = sum(1/(60 + rank))
|
||||
-> Cosine re-scoring + compiled truth boost
|
||||
-> 4-layer dedup + compiled truth guarantee
|
||||
-> Results
|
||||
```
|
||||
|
||||
Keyword alone misses conceptual matches. Vector alone misses exact phrases. RRF gets both. Search quality is benchmarked and reproducible: `gbrain eval --qrels queries.json` measures P@k, Recall@k, MRR, and nDCG@k. A/B test config changes before deploying them.
|
||||
|
||||
## Why it works: many strategies in concert
|
||||
|
||||
The brain isn't one trick. Every retrieval question goes through ~20 deterministic
|
||||
techniques layered together. No single one is magic; the win comes from stacking
|
||||
them so each layer covers what the others miss.
|
||||
|
||||
```
|
||||
Question
|
||||
│
|
||||
├─ INGESTION (every put_page)
|
||||
│ ├─ Recursive markdown chunking (or semantic / LLM-guided)
|
||||
│ ├─ Embedding cache invalidation on edit
|
||||
│ └─ Idempotent imports (content-hash dedup)
|
||||
│
|
||||
├─ GRAPH EXTRACTION (auto-link post-hook, zero LLM)
|
||||
│ ├─ Entity-ref regex (markdown links + bare slugs)
|
||||
│ ├─ Code-fence stripping (no false-positive slugs in code blocks)
|
||||
│ ├─ Typed inference cascade (FOUNDED → INVESTED → ADVISES → WORKS_AT)
|
||||
│ ├─ Page-role priors (partner-bio language → invested_in)
|
||||
│ ├─ Within-page dedup (same target collapses to one link)
|
||||
│ ├─ Stale-link reconciliation (edits remove dropped refs)
|
||||
│ └─ Multi-type link constraint (same person can works_at AND advises)
|
||||
│
|
||||
├─ SEARCH PIPELINE (every query)
|
||||
│ ├─ Intent classifier (entity / temporal / event / general — auto-routes)
|
||||
│ ├─ Multi-query expansion (Haiku rephrases the question 3 ways)
|
||||
│ ├─ Vector search (HNSW cosine over OpenAI embeddings)
|
||||
│ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery)
|
||||
│ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer)
|
||||
│ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval)
|
||||
│ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both)
|
||||
│ ├─ Cosine re-scoring (re-rank chunks against actual query embedding)
|
||||
│ ├─ Compiled-truth boost (assessments outrank timeline noise)
|
||||
│ ├─ Backlink boost (well-connected entities rank higher)
|
||||
│ └─ Source-aware dedup (one CT chunk per page guaranteed)
|
||||
│
|
||||
├─ GRAPH TRAVERSAL (relational queries)
|
||||
│ ├─ Recursive CTE with cycle prevention (visited-array check)
|
||||
│ ├─ Type-filtered edges (--type works_at, attended, etc.)
|
||||
│ ├─ Direction control (in / out / both)
|
||||
│ └─ Depth-capped (≤10 for remote MCP; DoS prevention)
|
||||
│
|
||||
└─ AGENT WORKFLOW (graph-confident hybrid)
|
||||
├─ Graph-query first (high-precision typed answers)
|
||||
├─ Grep fallback when graph returns nothing
|
||||
└─ Graph hits ranked first in top-K (better P@K and R@K)
|
||||
```
|
||||
|
||||
End-to-end on the BrainBench v1 corpus (240 rich-prose pages, before/after PR #188):
|
||||
|
||||
| Metric | BEFORE PR #188 | AFTER PR #188 | Δ |
|
||||
|-------------------------|----------------|---------------|-------------|
|
||||
| **Precision@5** | 39.2% | **44.7%** | **+5.4 pts**|
|
||||
| **Recall@5** | 83.1% | **94.6%** | **+11.5 pts**|
|
||||
| Correct in top-5 | 217 | 247 | **+30** |
|
||||
| Graph-only F1 (ablation)| 57.8% (grep) | **86.6%** | **+28.8 pts**|
|
||||
|
||||
Plus 5 orthogonal capability checks (identity resolution, temporal queries,
|
||||
performance at 10K-page scale, robustness to malformed input, MCP operation
|
||||
contract). All pass. Full report: [gbrain-evals](https://github.com/garrytan/gbrain-evals).
|
||||
|
||||
The point: each technique handles a class of inputs the others miss. Vector
|
||||
search misses exact slug refs; keyword catches them. Keyword misses conceptual
|
||||
matches; vector catches them. RRF picks the best of both. Compiled-truth boost
|
||||
keeps assessments above timeline noise. Auto-link extraction wires the graph
|
||||
that lets backlink boost rank well-connected entities higher. Graph traversal
|
||||
answers questions search alone can't reach. The agent picks graph-first for
|
||||
precision and falls back to keyword for recall. **All deterministic, all in
|
||||
concert, all measured.**
|
||||
|
||||
## Voice
|
||||
|
||||
Call a phone number. Your AI answers. It knows who's calling, pulls their full context from the brain, and responds like someone who actually knows your world. When the call ends, a brain page appears with the transcript, entity detection, and cross-references.
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/images/voice-client.png" alt="Voice client connected" width="300" />
|
||||
</p>
|
||||
|
||||
> [See it in action](https://x.com/garrytan/status/2043022208512172263)
|
||||
|
||||
The voice recipe ships with GBrain: [Voice-to-Brain](recipes/twilio-voice-brain.md). WebRTC works in a browser tab with zero setup. A real phone number is optional.
|
||||
|
||||
## Engine Architecture
|
||||
|
||||
```
|
||||
CLI / MCP Server
|
||||
(thin wrappers, identical operations)
|
||||
|
|
||||
BrainEngine interface (pluggable)
|
||||
|
|
||||
+--------+--------+
|
||||
| |
|
||||
PGLiteEngine PostgresEngine
|
||||
(default) (Supabase)
|
||||
| |
|
||||
~/.gbrain/ Supabase Pro ($25/mo)
|
||||
brain.pglite Postgres + pgvector
|
||||
embedded PG 17.5
|
||||
|
||||
gbrain migrate --to supabase|pglite
|
||||
(bidirectional migration)
|
||||
```
|
||||
|
||||
PGLite: embedded Postgres, no server, zero config. When your brain outgrows local (1000+ files, multi-device), `gbrain migrate --to supabase` moves everything.
|
||||
|
||||
## File Storage
|
||||
|
||||
Brain repos accumulate binaries. GBrain has a three-stage migration:
|
||||
|
||||
```bash
|
||||
gbrain files mirror <dir> # copy to cloud, local untouched
|
||||
gbrain files redirect <dir> # replace local with .redirect pointers
|
||||
gbrain files clean <dir> # remove pointers, cloud only
|
||||
gbrain files restore <dir> # download everything back (undo)
|
||||
```
|
||||
|
||||
Storage backends: S3-compatible (AWS, R2, MinIO), Supabase Storage, or local.
|
||||
|
||||
## Commands
|
||||
|
||||
```
|
||||
SETUP
|
||||
gbrain init [--supabase|--url] Create brain (PGLite default)
|
||||
gbrain migrate --to supabase|pglite Bidirectional engine migration
|
||||
gbrain upgrade Self-update with feature discovery
|
||||
|
||||
PAGES
|
||||
gbrain get <slug> Read a page (fuzzy slug matching)
|
||||
gbrain put <slug> [< file.md] Write/update (auto-versions)
|
||||
gbrain delete <slug> Delete a page
|
||||
gbrain list [--type T] [--tag T] List with filters
|
||||
|
||||
SEARCH
|
||||
gbrain search <query> Keyword search (tsvector)
|
||||
gbrain query <question> Hybrid search (vector + keyword + RRF)
|
||||
|
||||
IMPORT
|
||||
gbrain import <dir> [--no-embed] [--workers N]
|
||||
Import markdown (idempotent)
|
||||
gbrain sync [--repo <path>] [--workers N]
|
||||
Git-to-brain incremental sync
|
||||
(>100-file diffs auto-parallelize 4 workers on Postgres)
|
||||
gbrain export [--dir ./out/] Export to markdown
|
||||
|
||||
FILES
|
||||
gbrain files list|upload|sync|verify File storage operations
|
||||
|
||||
EMBEDDINGS
|
||||
gbrain embed [<slug>|--all|--stale] Generate/refresh embeddings
|
||||
|
||||
LINKS + GRAPH
|
||||
gbrain link|unlink|backlinks Cross-reference management
|
||||
gbrain extract links|timeline|all Batch backfill from existing pages
|
||||
(--source db|fs, --type, --since, --dry-run)
|
||||
gbrain graph-query <slug> Typed traversal (--type T --depth N
|
||||
--direction in|out|both)
|
||||
|
||||
JOBS (Minions)
|
||||
gbrain jobs submit <name> [--params JSON] [--follow] Submit a background job
|
||||
gbrain jobs list [--status S] [--queue Q] List jobs with filters
|
||||
gbrain jobs get|cancel|retry|delete <id> Manage job lifecycle
|
||||
gbrain jobs prune [--older-than 30d] Clean completed/dead jobs
|
||||
gbrain jobs stats Job health dashboard
|
||||
gbrain jobs smoke One-command health check
|
||||
gbrain jobs work [--queue Q] [--concurrency N] Start worker daemon
|
||||
|
||||
SKILLS (v0.19)
|
||||
gbrain skillify scaffold <name> Create 5 stub files + idempotent resolver row
|
||||
gbrain skillify check [path] 10-item audit of a skill
|
||||
gbrain skillpack list Print the 25 curated skills in the bundle
|
||||
gbrain skillpack install <name> Copy one skill + its shared conventions into target
|
||||
gbrain skillpack install --all Install the full curated bundle
|
||||
gbrain skillpack diff <name> Per-file diff: bundle vs target workspace
|
||||
gbrain check-resolvable [--strict] Resolver audit (reachability, MECE, DRY, routing, filing,
|
||||
SKILLIFY_STUB). Accepts RESOLVER.md OR AGENTS.md.
|
||||
gbrain routing-eval [--llm] [--json] Intent→skill routing accuracy on fixtures
|
||||
|
||||
ADMIN
|
||||
gbrain doctor [--json] [--fast] Health checks (resolver, skills, DB, embeddings)
|
||||
gbrain doctor --fix [--dry-run] Auto-fix DRY violations (delegate inlined rules to conventions)
|
||||
gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only)
|
||||
gbrain stats Brain statistics
|
||||
gbrain serve MCP server (stdio)
|
||||
gbrain serve --http [--port 3131] HTTP MCP server with OAuth 2.1 + admin dashboard
|
||||
[--token-ttl 3600] [--enable-dcr]
|
||||
[--public-url URL] [--log-full-params]
|
||||
gbrain auth create|list|revoke|test Legacy bearer token management
|
||||
gbrain auth register-client <name> Register an OAuth 2.1 client
|
||||
--grant-types client_credentials,authorization_code
|
||||
--scopes "read write admin"
|
||||
gbrain auth revoke-client <client_id> Revoke an OAuth 2.1 client (cascade purges
|
||||
active tokens + auth codes via FK CASCADE)
|
||||
# OAuth 2.1 clients can also be registered from the /admin dashboard or
|
||||
# programmatically via oauthProvider.registerClientManual() for host-repo wrappers.
|
||||
gbrain integrations Integration recipe dashboard
|
||||
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
|
||||
v0.28.2: --url <https://...> registers a federated
|
||||
remote git repo; clone is auto-managed under
|
||||
$GBRAIN_HOME/clones/<id>/ and re-cloned on sync if
|
||||
it goes missing. Also exposed via MCP for remote
|
||||
agent setup (whoami + sources_{add,list,remove,status}).
|
||||
gbrain dream [--dry-run] [--phase N] 8-phase maintenance cycle (lint→backlinks→sync→synthesize
|
||||
→extract→patterns→embed→orphans). v0.23 added synthesize +
|
||||
patterns: transcripts → reflections + cross-session themes.
|
||||
gbrain dream --input <file> Ad-hoc transcript synthesis (implies --phase synthesize)
|
||||
gbrain dream --date YYYY-MM-DD Synthesize a single day; --from/--to for backfill ranges
|
||||
gbrain check-backlinks check|fix Back-link enforcement
|
||||
gbrain lint [--fix] LLM artifact detection
|
||||
gbrain repair-jsonb [--dry-run] Repair v0.12.0 double-encoded JSONB (Postgres)
|
||||
gbrain orphans [--json] [--count] Find pages with zero inbound wikilinks
|
||||
gbrain transcribe <audio> Transcribe audio (Groq Whisper)
|
||||
gbrain research init <name> Scaffold a data-research recipe
|
||||
gbrain research list Show available recipes
|
||||
```
|
||||
|
||||
Run `gbrain --help` for the full reference.
|
||||
|
||||
## Origin Story
|
||||
|
||||
I was setting up my [OpenClaw](https://openclaw.ai) agent and started a markdown brain repo. One page per person, one page per company, compiled truth on top, timeline on the bottom. Within a week: 10,000+ files, 3,000+ people, 13 years of calendar data, 280+ meeting transcripts, 300+ captured ideas.
|
||||
|
||||
The agent runs while I sleep. The dream cycle scans every conversation, enriches missing entities, fixes broken citations, consolidates memory. I wake up and the brain is smarter than when I went to sleep.
|
||||
|
||||
The skills in this repo are those patterns, generalized. What took 11 days to build by hand ships as a mod you install in 30 minutes.
|
||||
|
||||
## Docs
|
||||
|
||||
- [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end
|
||||
- [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory
|
||||
- [`docs/guides/`](docs/guides/) — how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion)
|
||||
- [`docs/integrations/`](docs/integrations/) — connecting external data sources (voice, email, calendar, embedding providers)
|
||||
- [`docs/mcp/`](docs/mcp/) — per-client MCP setup (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork)
|
||||
- [`docs/eval/`](docs/eval/) — eval framework, metric glossary, methodology
|
||||
- [`docs/ethos/`](docs/ethos/) — philosophy (thin harness, fat skills, markdown as recipes, origin story)
|
||||
- [`AGENTS.md`](AGENTS.md) — entry point for non-Claude agents
|
||||
- [`CLAUDE.md`](CLAUDE.md) — entry point for Claude Code (deep operating context)
|
||||
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — contributor guide, test discipline, eval-capture mode
|
||||
- [`SECURITY.md`](SECURITY.md) — OAuth threat model, hardening defaults
|
||||
**For agents:**
|
||||
- **[skills/RESOLVER.md](skills/RESOLVER.md)** ... Start here. The skill dispatcher.
|
||||
- [Individual skill files](skills/) ... 28 standalone instruction sets (25 ship in the curated `gbrain skillpack install` bundle)
|
||||
- [GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md) ... Legacy reference architecture
|
||||
- [Getting Data In](docs/integrations/README.md) ... Integration recipes and data flow
|
||||
- [GBRAIN_VERIFY.md](docs/GBRAIN_VERIFY.md) ... Installation verification
|
||||
|
||||
**For humans:**
|
||||
- [GBRAIN_RECOMMENDED_SCHEMA.md](docs/GBRAIN_RECOMMENDED_SCHEMA.md) ... Brain repo directory structure
|
||||
- [Thin Harness, Fat Skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md) ... Architecture philosophy
|
||||
- [ENGINES.md](docs/ENGINES.md) ... Pluggable engine interface
|
||||
|
||||
**Reference:**
|
||||
- [GBRAIN_V0.md](docs/GBRAIN_V0.md) ... Full product spec
|
||||
- [CHANGELOG.md](CHANGELOG.md) ... Version history
|
||||
|
||||
**Benchmarks:**
|
||||
- [gbrain-evals](https://github.com/garrytan/gbrain-evals) ... BrainBench, the sibling repo that holds the eval harness, corpus, scorecards, and 4-adapter comparisons. Depends on gbrain; not installed alongside gbrain.
|
||||
|
||||
## Contributing
|
||||
|
||||
Run `bun run test` for the fast loop, `bun run verify` for the pre-push gate, `bun run ci:local` to run the full Docker-backed CI stack locally. Detailed test discipline in [`CONTRIBUTING.md`](CONTRIBUTING.md).
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun run test` for the parallel unit-test fast loop (~85s on a Mac dev box, 3700+ tests) or `bun run verify` for the pre-push gate (privacy + jsonb + progress + test-isolation + wasm + admin-build + typecheck). For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
||||
|
||||
Community PRs are batched into release waves rather than merged one-by-one — see the "PR wave workflow" section in [`CLAUDE.md`](CLAUDE.md). Contributor attribution stays attached via `Co-Authored-By:` trailers. We credit every accepted contribution in [`CHANGELOG.md`](CHANGELOG.md).
|
||||
If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in.
|
||||
|
||||
If you find a bug or want a feature: open an issue first. Quick fixes (typo, doc bug, obvious regression) can go straight to a PR. Anything touching schema, retrieval ranking, MCP protocol, or the security boundary needs a design discussion in the issue first.
|
||||
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
||||
|
||||
## License + credit
|
||||
## License
|
||||
|
||||
MIT. Built by Garry Tan to run his OpenClaw and Hermes deployments — the production brain behind his actual AI agents.
|
||||
|
||||
Origin story: [`docs/ethos/ORIGIN.md`](docs/ethos/ORIGIN.md).
|
||||
|
||||
Community PR contributors are credited in `CHANGELOG.md` per release. ZeroEntropy ([@zeroentropy](https://zeroentropy.dev)) for the embedding + reranker stack that became the v0.36.2.0 default. Voyage AI for the asymmetric-encoding recipe template. Ramp Labs for the search quality improvements lineage.
|
||||
MIT
|
||||
|
||||
+3
-15
@@ -68,16 +68,6 @@ The built-in HTTP transport ships with several layers of hardening on by
|
||||
default. All env vars below are optional; the defaults are intentionally
|
||||
conservative.
|
||||
|
||||
### Bind address (v0.34: loopback by default)
|
||||
|
||||
`gbrain serve --http` listens on `127.0.0.1` by default. Personal-laptop
|
||||
installs cannot accidentally publish the brain to the LAN. Self-hosted
|
||||
deployments that need remote access pass `--bind 0.0.0.0` (all
|
||||
interfaces) or `--bind <interface-ip>` (specific NIC). A stderr WARN
|
||||
fires when `--public-url` is set without `--bind` so the operator sees
|
||||
the binding before the first request — common cause of "ngrok forwards
|
||||
to me but the agent can't reach the upstream" misconfigurations.
|
||||
|
||||
### Postgres-only
|
||||
|
||||
`gbrain serve --http` requires a Postgres engine. PGLite is local-only by
|
||||
@@ -135,11 +125,9 @@ GBRAIN_HTTP_TRUST_PROXY=1 gbrain serve --http --port 8787
|
||||
**both** of these are true:
|
||||
|
||||
1. gbrain is reachable only via a trusted reverse proxy (not directly
|
||||
exposed to the internet on the configured port). As of v0.34
|
||||
`gbrain serve --http` binds `127.0.0.1` by default, so the
|
||||
reverse-proxy-only posture is the out-of-the-box shape; only
|
||||
override with `--bind 0.0.0.0` (or a specific interface IP) when
|
||||
gbrain itself needs to accept remote connections directly.
|
||||
exposed to the internet on the configured port). The simplest
|
||||
guarantee is to bind gbrain to `127.0.0.1` or a private interface
|
||||
and have the proxy forward to it.
|
||||
2. The proxy strips any client-supplied `X-Forwarded-For` and `X-Real-IP`
|
||||
headers, then sets them itself. (nginx with `proxy_set_header
|
||||
X-Forwarded-For $remote_addr` does this; Cloudflare and most cloud
|
||||
|
||||
@@ -1,430 +1,5 @@
|
||||
# TODOS
|
||||
|
||||
|
||||
## v0.35.6.0 floor-ratio gate follow-ups (v0.36.x+)
|
||||
|
||||
- [ ] **v0.36.x: Run gbrain-side floor-ratio ablation before flipping any mode-bundle default.** v0.35.6.0 ships the gate default-off (`MODE_BUNDLES[*].floor_ratio = undefined`) because the SkyTwin labeled-retrieval ablation that surfaced the regression isn't reproducible on gbrain's own eval surfaces from outside. Before any mode-bundle default flip, run the gate at `floor_ratio: undefined`, 0.85, 0.90, 0.95 across `gbrain eval longmemeval`, `gbrain eval whoknows`, `gbrain eval suspected-contradictions`, and the BrainBench-Real replay (sibling gbrain-evals repo). Quantify per-mode P@k / R@k / nDCG@k / top-1 stability deltas. Look for: regression on queries that genuinely need the long-tail boost (specific entity lookups, low-frequency topics) vs improvement on queries where weak-overlap pages were leapfrogging. The corpus-level finding determines whether tokenmax (most exposure to the failure mode) should flip first, or whether the gate stays a per-call opt-in indefinitely. Filed during v0.35.6.0 codex outside-voice review.
|
||||
|
||||
- [ ] **v0.36.x: `MODE_BUNDLES.floor_ratio` integration shape — populate after ablation evidence.** v0.35.6.0 leaves `floor_ratio: undefined` in all three bundles deliberately. After the ablation TODO above, set per-mode defaults: probably `tokenmax: 0.85` first (high-context tier, broad searchLimit=50, expansion=on — most exposure to leapfrog), `balanced` second if signal holds, `conservative` only if the ablation shows the gate doesn't hurt on small candidate pools. Update the canonical-bundle tests in `test/search-mode.test.ts` (3 fixtures) when flipping. The KNOBS_HASH_VERSION does NOT need to bump for a default change — the per-bundle default is part of the hash input already.
|
||||
|
||||
- [ ] **v0.36.x: Per-source floor-ratio (federated read).** v0.35.6.0 uses a single global threshold across all sources. Federated-read users (v0.34.1.0+) sharing a query across multiple sources get one floor across the merged result set, which means a high-scoring source can suppress metadata boosts for pages in another source. Codex outside-voice flagged this during v0.35.6.0 review; user explicitly chose the simpler primitive (D9=A). If a federated-read user later reports legitimate per-source winners being suppressed, the fix is a per-source threshold map computed at `runPostFusionStages` entry (one threshold per unique `source_id` in the result set). Plan reference: D9 in `~/.claude/plans/swift-sniffing-nygaard.md`.
|
||||
|
||||
- [ ] **v0.36.x: Reranker top-N expansion when floor-ratio narrows the candidate pool.** Floor-ratio can suppress a legitimate candidate that would have made it to the reranker's top-N. Sanity check after the v0.36 ablation: if tokenmax with `floor_ratio: 0.85` and `reranker_top_n_in: 30` shows the reranker seeing a meaningfully different set than without the gate, consider expanding `reranker_top_n_in` when floor is set (e.g. 30 → 40) so the reranker still has 30 floor-eligible candidates to reorder. Cheap mitigation if the data supports it. Not a blocker.
|
||||
|
||||
|
||||
## dreamy-thompson wave follow-ups (v0.36.x)
|
||||
|
||||
- [ ] **v0.36.x: runThink full rewrite — drop ThinkLLMClient indirection.** v0.36's fix(think) wave landed a gateway-backed adapter at `src/core/think/index.ts:225-251` so `gbrain config set anthropic_api_key` works over MCP stdio (closed #952). The adapter routes through `gateway.chat()` but `runThink` still carries the `ThinkLLMClient` interface as the test seam — it's the last LLM-using path that doesn't use the canonical `__setChatTransportForTests` seam v0.31.12 established for chat/embed. Cleanup: drop `ThinkLLMClient`, drop the `opts.client` injection point, migrate the 12+ existing tests (`test/think-pipeline.serial.test.ts:144,181,222`, `test/think-gateway-adapter.test.ts`, plus 9+ others that stub the interface) to `__setChatTransportForTests`. Pros: codebase consistency, one fewer test-stub pattern, easier to add provider switching for think once it routes through gateway natively. Cons: 12+ test files need migration. Blocked by: v0.36 wave landing on master (so the adapter exists to lean on while migrating tests). Plan reference: D5 + D7 in `~/.claude/plans/ok-i-spun-up-dreamy-thompson.md`.
|
||||
|
||||
- [ ] **v0.36.x: Supabase parity test fixture for `applyForwardReferenceBootstrap`.** v0.36 fixed the underlying bug (bootstrap now uses the DDL connection from `initSchema` so probes run inside the advisory-lock scope) per codex P1 from /ship adversarial review. What remains is the TEST FIXTURE that proves it: the new pre-v18/pre-v34/pre-v60 E2E tests run against local Docker Postgres but not against Supabase-shape pooler topology (transaction pooler + statement_timeout). Real Supabase upgrades have failed multiple times on this exact connection-topology divergence (#699, #820 lineage). Fix: a test fixture that exercises the probe path against deriveDirectUrl + transaction pooler + statement_timeout. Cons: requires Supabase fixture infra OR careful mocking of the connection-selection logic in `db.ts`'s `getDDLConnection` path.
|
||||
|
||||
|
||||
## kinshasa-v3 follow-ups (v0.35.4.0)
|
||||
|
||||
- [ ] **v0.36.x: Fix `supervisor-audit.ts:77` `readSupervisorEvents` to use the dual-week-aware pattern from `stub-guard-audit.ts:readRecentStubGuardEvents`.** The supervisor reader only reads the current ISO-week file, so a 24h sliding window across Monday 00:00 UTC silently loses Sunday's events (they're in last week's file). The new stub-guard reader in v0.35.4.0 fixes this for its own audit log by reading BOTH current and previous week files before timestamp-filtering — the supervisor reader should adopt the same shape. Pin with a unit test that uses a fake-clock fixture set to "Monday 00:01 UTC" with a Sunday 23:55 event in the prior file. Filed during v0.35.4.0 kinshasa-v3 codex outside-voice review.
|
||||
|
||||
- [ ] **v0.36.x: Decommission the stub-guard at `fence-write.ts:190` once the sunset criterion holds.** The guard's purpose is defense-in-depth behind the resolver's prefix-expansion fix. Sunset rule: when `stub_guard_24h` reads <5 hits/week for 3 consecutive weeks across production brains, the prefix-expansion is doing its job and the guard can be removed. The JSDoc names v0.36 as the target — re-check this against actual operator-brain data when planning v0.36.
|
||||
|
||||
- [ ] **v0.36.x: `PREFIX_EXPANSION_DIRS` is hardcoded to `['people', 'companies']` in `src/core/entities/resolve.ts:97`.** New entity directories (funds, advisors, deals, etc.) require a code change to opt in. Consider a config-driven list (`entities.prefix_expansion_dirs: [...]` in `gbrain.yml`) so operators can extend without forking. Filed during v0.35.4.0 plan-eng-review.
|
||||
|
||||
- [ ] **v0.36.x: Sweep the banned private-agent-name references out of `CHANGELOG.md`.** Three pre-existing lines in `CHANGELOG.md` (around lines 2537, 2606, 3304) reference the name that `scripts/check-privacy.sh` enforces against. Pre-existing on master, not introduced by v0.35.4.0; `CHANGELOG.md` is on the script's allow-list so master CI is green, but they still violate the spirit of CLAUDE.md's privacy rule (the allow-list is a meta-documentation exception, not a license to add new references). Replace with `your OpenClaw` or `Garry's OpenClaw` per the script's own suggestion text. Trivial cleanup PR. Filed during v0.35.4.0 privacy audit.
|
||||
|
||||
|
||||
## embed --stale follow-ups (v0.34.4.0)
|
||||
|
||||
- [ ] **v0.35.x: Concurrent NULL→non-NULL upsert race in `embed.ts:429-443` + `postgres-engine.ts:1231`'s `COALESCE(EXCLUDED.embedding, content_chunks.embedding)`.** Two `embed --stale` workers (or `embed --stale` racing with a sync that re-embeds the same chunk) can have the slower writer overwrite the faster one's fresher embedding. Window is small (20 workers, all from the same `listStaleChunks` snapshot) but exists. Tractable fix: a `WHERE content_chunks.embedded_at < EXCLUDED.embedded_at OR content_chunks.embedding IS NULL` predicate on the upsert. Out of scope for v0.34.4.0 because the upsert is not in the diff; pre-existing bug. Filed during v0.34.4.0 codex outside-voice review.
|
||||
|
||||
- [ ] **v0.35.x: New stale rows inserted behind the keyset cursor.** A sync or `gbrain put_page` mid-`embed --stale` creates chunks with `embedding IS NULL` at `(page_id, chunk_index)` already passed by the cursor. Picked up on next run via the partial index; documented limitation. Possible fix: a second pass at end-of-run that does a fresh `countStaleChunks()` and re-enters the loop while count > 0 and budget allows. Filed during v0.34.4.0 codex outside-voice review.
|
||||
|
||||
## MCP fix wave follow-ups (v0.34.1)
|
||||
|
||||
- [ ] **v0.34.x: Source-scope `takes_*` ops (pre-existing leak surfaced during v0.34.1 adversarial review).** `takes_list`, `takes_search`, `takes_scorecard`, `takes_calibration` in `src/core/operations.ts:1248-1335` thread `ctx.takesHoldersAllowList` but never `ctx.sourceId`. An auth'd OAuth client scoped to `source_id='canon-a'` can call `takes_list --page_slug=foo` (slug in `canon-b`) and read takes attached to foreign-source pages. Pre-existing, not introduced by v0.34.1, but the wave was framed as "P0 source-isolation seal on the read path" and `takes_*` surfaces were missed. Fix: extend `TakesListOpts` in `src/core/engine.ts:186` with `sourceId?: string` + `sourceIds?: string[]`; thread `sourceScopeOpts(ctx)` at each op handler; engine `listTakes`/`searchTakes` filter via the `pages` JOIN.
|
||||
|
||||
- [ ] **v0.34.x: Extend `sourceScopeOpts(ctx)` to the 14 read-side ops PR #861 didn't touch.** `get_page`, `get_tags`, `get_links`, `get_backlinks`, `get_timeline`, `list_files`, `get_file`, and the four `takes_*` ops (above) still use the v0.31.8-era `const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {}` pattern. NOT a leak (scalar `ctx.sourceId` IS threaded), but federated_read (#876, `ctx.auth?.allowedSources`) is silently dropped. A "WeCare L3 dept" client gets correct federated results from `search`/`query`/`list_pages`/`traverse_graph`/`find_experts` but only sees its scalar `source_id` for `get_page`/`get_tags`/etc. Fix: route all 14 sites through `sourceScopeOpts(ctx)`.
|
||||
|
||||
- [ ] **v0.34.x: Migration v60 idempotency guard against `--force-retry` race with v64.** `gbrain apply-migrations --force-retry 58` after v64 has already run will re-install the FK with `ON DELETE SET NULL`, silently downgrading the v64 RESTRICT posture. Probability low (operator has to explicitly force-retry 58) but failure mode is invisible. Fix: v60 should probe `pg_constraint.confdeltype` before re-adding and refuse to clobber `'r'` (RESTRICT) with `'n'` (SET NULL).
|
||||
|
||||
- [ ] **v0.34.x: `embedMultimodalOpenAICompat` batching + partial-failure handling.** `src/core/ai/gateway.ts:1180-1255` sends one HTTP request per input. Multi-input callers (10 images) get 10 sequential round-trips with no parallelism; a 401 on input #5 throws and discards inputs #1-#4's already-computed embeddings (wasted spend, no surfacing of the partial array). Voyage's existing path batches. Fix: batch via the provider's `input: [...]` array shape; on partial failure, return successful embeddings + failed-index array.
|
||||
|
||||
- [ ] **v0.34.x: Doctor check `oauth_orphan_source_id`** — surfaces OAuth clients whose source_id was nulled by the v60 D10 silent-widen path (`GBRAIN_ACCEPT_SILENT_WIDEN=1`). Closes the observability gap from v0.34.1's D4 decision. Sibling to the `rls_event_trigger` check pattern in `src/commands/doctor.ts`.
|
||||
|
||||
- [ ] **v0.34.x: `gbrain sources purge` FK error UX.** Post-v0.34, deleting a source is refused if any oauth_client references it (v64 ON DELETE RESTRICT). The CLI currently surfaces the raw Postgres FK violation. Fix: pre-check via `SELECT client_id, client_name FROM oauth_clients WHERE source_id = $1`, print "N OAuth clients reference this source: ... Revoke first via `gbrain auth revoke-client <id>`." Mirrors `assessDestructiveImpact` in destructive-guard.ts (v0.26.5).
|
||||
|
||||
- [ ] **v0.34.x: `hybrid.ts:223` explicit-pick refactor.** The SearchOpts rebuild manually picks fields from HybridSearchOpts. This is the bug shape that caused the original v0.34.1 P0 leak — a new SearchOpts field is silently dropped if not manually added here. The wave added `sourceId` + `sourceIds` to the pick; future fields will keep hitting this footgun. Fix: refactor to spread + TypeScript `Pick<>` helper that narrows HybridSearchOpts → SearchOpts type-safely.
|
||||
|
||||
|
||||
## functional-area-resolver follow-ups (v0.32.3.0)
|
||||
|
||||
- [ ] **v0.33.x: Dogfood `functional-area-resolver` on gbrain's own `skills/RESOLVER.md`** when it crosses ~12KB (currently 8KB). Apply the pattern to the Operational section first (largest). Filed during v0.32.3.0 CEO review.
|
||||
|
||||
- [ ] **v0.33.x: Promote `evals/functional-area-resolver/harness.mjs` to a first-class CLI command** `gbrain routing-eval --ab-compare <variant-dir>`. Removes the one-off harness as maintenance debt; gives every pattern-skill a way to ship its eval. Replaces the placeholder `--llm` flag in `src/core/routing-eval.ts:17-20`. Filed during v0.32.3.0 CEO review.
|
||||
|
||||
- [ ] **v0.33.x: Expand held-out corpus to >=20 fixtures.** The current n=5 saturates at 100% across most cells and can't distinguish "100%" from "95% with one nondeterministic miss." Author independently (don't see variants while authoring). Filed during v0.32.3.0 boil-the-ocean push after codex outside-voice review.
|
||||
|
||||
- [ ] **v0.33.x: Cross-vendor model verification.** Run the harness on Gemini 2.5 Pro and GPT-4o/5 in addition to the three Anthropic models we already covered. Compression gains may not transfer across vendor families (the `(dispatcher for: ...)` clause is interpreted differently by different prompt-tuned models). Wire through the existing gbrain gateway (recipes already exist for both vendors).
|
||||
|
||||
- [ ] **v0.33.x: Per-row description length sweep.** Anthropic's Agent Skills median is ~80 tokens of frontmatter per skill ([Anthropic engineering blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)). Sweep functional-areas at {20, 40, 80, 160} tokens per dispatcher row, eval each. Novel published contribution — no public data exists. ~$5 in API spend. Filed during v0.32.3.0 web research.
|
||||
|
||||
- [ ] **v0.33.x: Structural compression of functional-areas (`(dispatcher for: ...)` → `dispatcher: [...]` YAML form, trim verbose triggers, separate hard gates to sibling file).** Target 13KB → 9-10KB without accuracy regression. Requires another full re-baseline run (~$3 across 3 models) to confirm no regression.
|
||||
|
||||
- [ ] **v0.33.x: Hierarchical compression (area-of-areas).** Two-level: top-level mega-areas (knowledge / ops / comms) pointing to functional-area files loaded lazily. Predicted 13KB → 4-6KB. Risks resolver-of-resolvers-style collapse on the top-level layer. Worth an A/B but its own piece of work. Cross-reference AnyTool ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253)) which formalizes this hierarchy at runtime.
|
||||
|
||||
- [ ] **v0.33.x: Embedding-based area pre-router.** RAG-MCP shape ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1)) — cheap embedding model picks the area; only that area's sub-skills get sent to the LLM. Dramatic per-call payload reduction (~80%). Significant new code surface but big production cost win. Wire through the existing gateway's voyage or openai embedding recipes.
|
||||
|
||||
- [ ] **v0.33.x: Adversarial-intent fixtures.** Intents specifically designed to test dispatcher-vs-subskill behavior on edge cases ("I want to do something brain-related" without specifying what). Targets the prompt-design failure mode (run-1 collapse) that our current 25 fixtures don't surface. ~10-15 fixtures, authored without looking at variant content.
|
||||
|
||||
- [ ] **v0.33.x: Run-2 vs Run-1 prompt-design ablation.** Document the difference between the naive classifier prompt (run-1, every variant 30-60% training) and the dispatcher-aware prompt (run-2+, functional-areas 88-100% training) as a reproducible result. This is the strongest empirical finding from v0.32.3.0 and deserves its own callout in SKILL.md or a sibling METHODOLOGY.md.
|
||||
|
||||
## Embedding-provider follow-ups (v0.32.0)
|
||||
|
||||
- [ ] **v0.32.x: Vertex AI ADC embedding provider (#729 originally).** lucha0404
|
||||
prototyped this with single-source-JSON via `GOOGLE_APPLICATION_CREDENTIALS`.
|
||||
Real ADC is the full chain (metadata server, gcloud creds, service-account
|
||||
JSON). The recipe needs to either use `@ai-sdk/google-vertex` (one new
|
||||
dep, native fit) or implement the chain via Bun.crypto.subtle for RS256
|
||||
JWT signing (zero dep, ~150 lines + RS256 spike). Original Q3 chose
|
||||
zero-dep; revisit the dep budget when scoping.
|
||||
|
||||
- [ ] **v0.32.x: GitHub Copilot embeddings (#691 originally).** tonyxu-io
|
||||
proposed adding Copilot's Metis embedding endpoint as a sidecar recipe.
|
||||
Codex review caught that this is not a recipe-add — it's an outbound OAuth
|
||||
product surface (login flow, browser/device flow, refresh, UX). Needs its
|
||||
own design pass: where does the token live? `~/.gbrain/oauth/copilot.json`
|
||||
mode 0600 was the v0.32 plan; revisit + write `gbrain auth login copilot`.
|
||||
|
||||
- [ ] **v0.32.x: OpenAI Codex OAuth chat provider (#698 originally).** perlantir
|
||||
proposed a chat-only provider that reuses ChatGPT subscription auth instead
|
||||
of API keys. Same OAuth-product-surface argument as #691. Same shared
|
||||
infra: `~/.gbrain/oauth/<provider>.json` + `gbrain auth login <provider>`.
|
||||
Build alongside #691 in one OAuth-subsystem wave.
|
||||
|
||||
- [x] **v0.32.7: CJK PGLite keyword fallback (#765 extracted).** Landed
|
||||
in the CJK fix wave. `hasCJK` + `escapeLikePattern` live in
|
||||
`src/core/cjk.ts`; the CJK branch in `pglite-engine.ts:searchKeyword`
|
||||
uses ILIKE + bigram-frequency-count ranking. Postgres path deferred
|
||||
(see new follow-up below).
|
||||
|
||||
- [ ] **v0.33+: Postgres CJK FTS via pgroonga / zhparser / ngram trigrams.**
|
||||
v0.32.7 only fixed CJK keyword search on PGLite. Multi-tenant Postgres
|
||||
deployments still hit empty results for CJK queries because
|
||||
`to_tsvector('english', ...)` can't segment Chinese / Japanese / Korean.
|
||||
Installing pgroonga or zhparser is an operator decision (extension
|
||||
install permission, multi-tenant rollout), so gbrain can't default it.
|
||||
Plan: doctor advisory pointing at the relevant extension docs;
|
||||
searchKeyword / searchKeywordChunks fall through to PGLite-style ILIKE
|
||||
when the extension isn't installed. Defer until users complain.
|
||||
|
||||
- [ ] **v0.33+: widen CJK ranges to Unicode property escapes.** v0.32.7
|
||||
uses BMP-only ranges (Han `4e00-9fff`, Hiragana `3040-309f`, Katakana
|
||||
`30a0-30ff`, Hangul Syllables `ac00-d7af`). Misses Han Extensions A/B/C,
|
||||
halfwidth katakana, compatibility ideographs, compatibility Jamo, and
|
||||
iteration marks `々` / `〇`. Switch to `\p{Script=Han}` / `\p{Script=Hiragana}` /
|
||||
`\p{Script=Katakana}` / `\p{Script=Hangul}` (TS supports unicode property
|
||||
escapes with the `u` flag). Astral-plane support also requires
|
||||
`Array.from(str)`-style codepoint iteration in the chunker's char-slice
|
||||
fallback (current `String.prototype.slice` splits surrogate pairs).
|
||||
Defer until first user hits the gap.
|
||||
|
||||
- [ ] **v0.33+: `git diff --name-status -z` + NUL framing.** v0.32.7
|
||||
added `core.quotepath=false` which handles non-ASCII paths but doesn't
|
||||
cover tabs, newlines, or quotes in filenames. The `-z` flag with
|
||||
NUL-byte path framing is the robust fix for the whole encoding class.
|
||||
Affects `src/commands/sync.ts:buildDetachedWorkingTreeManifest` +
|
||||
`buildSyncManifest`. Defer until someone files a tab-in-filename issue.
|
||||
|
||||
- [ ] **v0.33+: CJK-aware overlap context in chunker.** v0.32.7
|
||||
`extractTrailingContext` is still whitespace-token-based, so CJK chunks
|
||||
under the maxChars cap have no useful overlap with the previous chunk.
|
||||
Search continuity across chunk boundaries degrades for pure CJK content.
|
||||
The maxChars sliding-window in v0.32.7 IS overlap-protected for the
|
||||
hard-cap path, so this only affects normal-size chunks. Plan: switch
|
||||
`extractTrailingContext` to char-count when `countCJKAwareWords` would
|
||||
have triggered the CJK branch.
|
||||
|
||||
- [ ] **v0.33+: other non-Latin scripts (Thai, Arabic, Cyrillic,
|
||||
Devanagari).** Same five-layer fix pattern as CJK applies: slugify
|
||||
needs the script range, chunker needs density-threshold counting,
|
||||
PGLite keyword fallback would benefit from script-aware tokenization.
|
||||
Defer until first issue.
|
||||
|
||||
- [ ] **v0.33+: embedding pricing refresh mechanism.** v0.32.7 added
|
||||
`src/core/embedding-pricing.ts` as a static lookup table sibling to
|
||||
`anthropic-pricing.ts`. Both drift when providers change rates. Plan:
|
||||
a `gbrain prices refresh` skill that diffs against a published canonical
|
||||
source (OpenAI pricing page, Anthropic pricing page) and proposes an
|
||||
update PR. Or a release-cadence audit checklist item. Today: when the
|
||||
estimate looks off, hand-edit the constants.
|
||||
|
||||
- [ ] **v0.32.x: interactive provider chooser in `gbrain init`.** The full
|
||||
wizard piece of the v0.32 discoverability lane was deferred. Today
|
||||
`gbrain init` (no flags, TTY) silently uses OpenAI default. Plan: hook
|
||||
into `init.ts:resolveAIOptions`, when no `--model` AND TTY AND not
|
||||
`--non-interactive`, call `runExplain([])` (non-JSON path) from
|
||||
`providers.ts:233-350` to print the provider matrix, then prompt with
|
||||
readline (mirror `supabaseWizard()` at `init.ts:108`). Suggest
|
||||
recommended based on env detection. Refuse `user_provided_models`
|
||||
shorthand (already done in v0.32.0). Tests:
|
||||
`test/init-provider-wizard.test.ts` (TTY → prompt fires; non-TTY →
|
||||
falls through; invalid choice → re-prompts).
|
||||
|
||||
- [ ] **v0.32.x: real-credentials per-recipe smoke-test CI matrix.** Codex
|
||||
finding #6 noted that unit tests via `__setEmbedTransportForTests` prove
|
||||
routing but not contract correctness with the actual provider HTTP
|
||||
shape. Provider APIs change quietly (Voyage encoding-format, MiniMax
|
||||
type field, Azure header). One real-call per recipe per month catches
|
||||
drift before users do; <$1/run estimated. Requires API-key budget
|
||||
approval + repo secrets.
|
||||
|
||||
- [ ] **v0.32.x: MiniMax asymmetric retrieval support.** v0.32 ships
|
||||
`embo-01` with `type: 'db'` for both indexing and queries (symmetric
|
||||
retrieval). True asymmetric needs a query/document signal threaded
|
||||
through the embed seam. Worth it for MiniMax users who care about
|
||||
retrieval quality on Chinese content; defer until users complain.
|
||||
|
||||
- [ ] **v0.32.x: un-hardcode the multimodal dispatch at gateway.ts:583.**
|
||||
Currently `recipe.id !== 'voyage'` is hardcoded — harmless until a
|
||||
second multimodal recipe lands. Make it table-driven via
|
||||
`Recipe.touchpoints.embedding.supports_multimodal` +
|
||||
`multimodal_models`. ~10 lines + a contract test.
|
||||
|
||||
## v0.31.2 follow-ups
|
||||
|
||||
### Investigate: `gbrain query <common-keyword>` infinite loop
|
||||
**Priority:** P1
|
||||
**Filed:** 2026-05-08 from v0.31.2 bug report (separate from the sync hang).
|
||||
|
||||
**Evidence:** Two `bun /Users/garrytan/.bun/bin/gbrain query the` processes
|
||||
(PIDs 39429, 46624) on the user's Mac were pegged at 99% CPU for 7
|
||||
straight days before being killed manually. Each used 6+ GB resident
|
||||
memory. Originated from the `algiers-v3` worktree. Not walker-related
|
||||
(query path doesn't traverse files), so the v0.31.2 fix doesn't address
|
||||
it.
|
||||
|
||||
**Likely candidates:**
|
||||
- Query-expansion regex catastrophic backtracking on common single words
|
||||
(`src/core/search/expansion.ts` calls Haiku then post-processes with
|
||||
regex; a one-token query plus an unhelpful expansion could feed a
|
||||
pathological input back into the search pipeline)
|
||||
- Hybrid-search RRF reciprocal-rank-fusion loop iterating over a result
|
||||
set that never shrinks (`src/core/search/hybrid.ts`)
|
||||
- `postgres.js` cursor that never closes when the result set is large
|
||||
(the 6GB RES on `query` smells like accumulated rows in JS memory, not
|
||||
WASM allocation)
|
||||
|
||||
**To reproduce:** create a brain with at least a few thousand pages, run
|
||||
`gbrain query the` and watch CPU + RSS. If it pegs and grows, capture
|
||||
`process.report.getReport()` and a stack trace via `kill -SIGUSR2 <pid>`
|
||||
before killing.
|
||||
|
||||
**Out of scope for v0.31.2** because the user's primary symptom (sync
|
||||
hang) was the higher-evidence bug. Pick this up as v0.31.3 once the
|
||||
sync fix is verified working in production.
|
||||
|
||||
### v0.31.3: PGLite + Postgres E2E for amarillo-shape regression
|
||||
**Priority:** P2
|
||||
**Filed:** 2026-05-08 from v0.31.2 plan (deferred).
|
||||
|
||||
**What:** Plan called for two regression tests pinning the user's exact
|
||||
repro topology: `test/sync-walker-amarillo-shape.test.ts` (PGLite,
|
||||
fast-loop) and `test/e2e/sync-amarillo-shape.test.ts` (real-Postgres,
|
||||
skip-on-no-DB). Unit-level walker + chunker tests landed in v0.31.2
|
||||
(`test/sync-walker-symlink.test.ts` + `test/chunker-timeout.test.ts`),
|
||||
but the engine-integrated regression for the user's exact 1500-file
|
||||
self-symlink topology is still pending. Add when the next sync-related
|
||||
PR is in flight.
|
||||
|
||||
## Thin-client mode follow-ups (v0.31.1, Issue #734)
|
||||
|
||||
- [ ] **v0.31.x: routed-call timing telemetry.** `GBRAIN_TIMING=1` prints
|
||||
`token_mint=Xms http=Yms server=Zms total=Wms` per routed MCP call.
|
||||
Audit log at `~/.gbrain/audit/routed-calls-YYYY-Www.jsonl`. Cherry-pick
|
||||
C from #734 plan; deferred from v0.31.1 to keep scope tight.
|
||||
|
||||
- [ ] **v0.31.2: job-submission routing for `gbrain dream` etc.** Route
|
||||
long-running ops (`dream`, `embed --stale`, `extract`) via `submit_job`
|
||||
+ poll, mirroring the existing `gbrain remote ping` autopilot-cycle
|
||||
pattern. Cherry-pick D from #734 plan. Adds a thin-client async-job
|
||||
render layer (progress events + spinner).
|
||||
|
||||
- [ ] **Per-subcommand thin-client routing for `takes` and `sources`.**
|
||||
CDX-2 audit identified the READ subcommands (`takes_list`, `takes_search`,
|
||||
`sources_list`, `sources_status`) as routable; mutate subcommands edit
|
||||
local files. v0.31.1 refuses both at the top level with hints. Split
|
||||
is a v0.31.x release.
|
||||
|
||||
- [ ] **Privacy decision: lift `localOnly: true` on `get_recent_transcripts`?**
|
||||
Raw chat exports leaving the host is a real tradeoff. Needs explicit
|
||||
per-token scope (`scope: 'transcripts'`) and consent UX. Out of v0.31.1.
|
||||
|
||||
- [ ] **Trust-boundary policy review for remote-caller gates.** Server
|
||||
intentionally disables `think.--save`/`--take` for remote callers
|
||||
(operations.ts:1103-1135) and skips `put_page` auto-link/auto-timeline
|
||||
for remote callers without `trustedWorkspace` (operations.ts:434-451).
|
||||
Subagent-isolation reasons; blocks full thin-client parity. Policy
|
||||
decision, not a routing fix.
|
||||
|
||||
- [ ] **v0.32.0: flip `gbrain auth register-client` default scope from
|
||||
`read` to `read,write,admin`.** Breaking for existing read-only scrapers;
|
||||
ship deprecation warning in v0.31.x. The v0.31.1 `oauth_client_scopes_probe`
|
||||
doctor check surfaces the gap with pinpoint remediation in the meantime.
|
||||
|
||||
- [ ] **v0.31.x: cross-process OAuth token cache at
|
||||
`~/.gbrain/oauth-token-cache.json`.** Cuts ~200ms cold-start cost for
|
||||
shell-loop usage on thin-client installs. Today the in-memory cache is
|
||||
per-process; every `gbrain` invocation pays a fresh token mint.
|
||||
|
||||
- [ ] **v0.31.x: parity test (`test/thin-client-parity.test.ts`).** Plan
|
||||
called for ~400 LOC byte-equal stdout assertions for 12+ ops via an
|
||||
in-process MCP server pointed at the same PGLite as the local-engine
|
||||
path. Harder than expected because it needs MCP server setup that the
|
||||
current test infrastructure doesn't expose. v0.31.1 ships without it;
|
||||
ENG-2's JSON-shape normalization + per-command test coverage is the
|
||||
interim guard.
|
||||
|
||||
## LongMemEval benchmark follow-ups (v0.28.12)
|
||||
|
||||
### Closed: full 500-question 4-adapter run published
|
||||
|
||||
The full 500-question, 4-adapter LongMemEval `_s` benchmark landed in
|
||||
[gbrain-evals#main:ced01f0](https://github.com/garrytan/gbrain-evals/blob/main/docs/benchmarks/2026-05-07-longmemeval-s.md).
|
||||
gbrain-hybrid: 97.60% R@5, beating MemPal raw 96.6% by 1.0pt on the same
|
||||
dataset, K, and n with no LLM in the retrieval loop. Honest null result on
|
||||
query expansion (97.60% with vs without). Closing this entry; remaining
|
||||
follow-ups below.
|
||||
|
||||
### Timeline-aware retrieval signal for temporal-reasoning questions
|
||||
**Priority:** P2
|
||||
|
||||
**What:** gbrain's `links` table + `gbrain extract timeline` already build a
|
||||
graph of dated events. Feed that signal into `searchKeyword` / `searchVector`
|
||||
ranking so questions like "what was the FIRST issue I had after my new
|
||||
car's first service?" get a temporal boost on session ordering.
|
||||
|
||||
**Why:** LongMemEval temporal-reasoning is the only question type where MemPal-raw
|
||||
beats gbrain-hybrid (96.2% vs 94.7%, -1.5pt). Embeddings carry topic
|
||||
similarity; "first" / "before" / "last week" need ordering signal that
|
||||
vector cosine doesn't surface. We have the data infrastructure to fix this
|
||||
(the timeline extraction code), just don't pipe it into search ranking.
|
||||
|
||||
**Pros:** Closes the only categorical loss to MemPal on the public benchmark.
|
||||
Generalizes beyond LongMemEval — every personal-knowledge agent gets
|
||||
temporal questions and most fail them. This is a structural advantage.
|
||||
|
||||
**Cons:** Requires a new SQL ranking factor in `src/core/search/sql-ranking.ts`
|
||||
and signal-extraction work in the query-time path (parsing temporal hints
|
||||
from the question). Maybe ~200 lines + a benchmark line on the gbrain-evals
|
||||
report once it ships.
|
||||
|
||||
**Context:** Per-type breakdown in
|
||||
`gbrain-evals/docs/benchmarks/2026-05-07-longmemeval-s.md` shows we tie
|
||||
or beat MemPal-raw on 5 of 6 types and lose temporal by 1.5pt. Also:
|
||||
`src/core/link-extraction.ts` already extracts dated timeline entries via
|
||||
`parseTimelineEntries`. They land in `timeline_entries` table but aren't
|
||||
used during retrieval ranking.
|
||||
|
||||
**Depends on:** Nothing blocking.
|
||||
|
||||
### Per-question batch consolidation (latency optimization)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** `importFromContent` calls `embedBatch` once per page. Each LongMemEval
|
||||
question imports ~50 sessions = 50 separate API calls. Pre-chunk all sessions
|
||||
for a question, embed in one OpenAI call, then bulk-write.
|
||||
|
||||
**Why:** Drops per-question latency from ~14s to ~3s on a cold cache.
|
||||
Currently the runner ships a 700MB SQLite warm-cache to avoid this; a faster
|
||||
cold path would let CI run the benchmark daily without a fixture.
|
||||
|
||||
**Pros:** Daily benchmark CI gate becomes practical. Cuts cold-cache cost by
|
||||
~10x. Faster iteration when tuning ranking parameters.
|
||||
|
||||
**Cons:** ~80 lines of batch-consolidation code that lives in the runner, not
|
||||
gbrain core. Touches `eval/runner/longmemeval.ts:run()` per-question loop.
|
||||
Less generalizable than the timeline-aware ranker work.
|
||||
|
||||
**Context:** Right now the warm-cache mitigates this in practice (subsequent
|
||||
runs are sub-1-min). The optimization matters only when re-running with a
|
||||
different gbrain version that re-keys the cache.
|
||||
|
||||
**Depends on:** Nothing blocking.
|
||||
|
||||
### LongMemEval `_m` split (200 distractor sessions per haystack)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Run the existing 4-adapter benchmark against the harder `_m` split
|
||||
where each haystack has ~200 distractor sessions instead of ~50.
|
||||
|
||||
**Why:** Pushes retrieval into the regime where gbrain's pipeline either
|
||||
holds up or doesn't. MemPal hasn't published `_m` numbers; we'd have a
|
||||
clean head-to-head once we run it. Also stresses the noise-rejection
|
||||
(source-boost / hard-exclude) layer of gbrain harder than `_s` does.
|
||||
|
||||
**Pros:** Differentiated benchmark line. Forces signal-vs-noise behavior we
|
||||
can't measure on `_s`. Free with our existing runner.
|
||||
|
||||
**Cons:** ~$10-20 in OpenAI embeddings (4x more chunks per question). Cache
|
||||
file grows to ~3GB. ~6-8 hours wall time for the embedding-heavy runs even
|
||||
parallel-3.
|
||||
|
||||
**Depends on:** Nothing blocking. Could ship same shape as `_s` report.
|
||||
|
||||
### Cheaper embedding-model recipe for benchmarks
|
||||
**Priority:** P4
|
||||
|
||||
**What:** Pin `text-embedding-3-small` (or Voyage-3-lite via the v0.27
|
||||
pluggable provider stack) as a benchmark-only embedding model so the
|
||||
cold-cache cost drops 10x. Compare recall against `text-embedding-3-large`
|
||||
and publish the recall-cost tradeoff curve.
|
||||
|
||||
**Why:** "What's the cheapest embedding model that still wins this
|
||||
benchmark?" is a real builder question. We'd publish the answer.
|
||||
|
||||
**Pros:** Useful tradeoff line for users picking gbrain in a cost-sensitive
|
||||
deployment. Validates the v0.27 pluggable-provider work end-to-end.
|
||||
|
||||
**Cons:** Multiple full-benchmark runs ($30+ in API spend) to chart the
|
||||
curve.
|
||||
|
||||
**Depends on:** v0.27 pluggable embedding provider work (already shipped,
|
||||
verify Voyage adapter integration in `src/core/ai/recipes/voyage.ts`).
|
||||
## multimodal embedding follow-ups (v0.28.11 / PR #719)
|
||||
|
||||
### `gbrain doctor`: warn on misconfigured multimodal model
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Add two checks in `src/commands/doctor.ts`. (1) When `embedding_multimodal_model` is set, verify the recipe's required API key is present in the env. (2) When `embedding_multimodal: true` is set but no `embedding_multimodal_model` AND the primary `embedding_model` recipe doesn't declare `supports_multimodal`, surface that gap.
|
||||
|
||||
**Why:** Today these misconfigurations surface only on first image ingest, after the user has already pushed image content into the brain. Doctor catching them at install/upgrade time saves a round of confusion.
|
||||
|
||||
**Pros:** Both checks are read-only and cheap (one env probe + one recipe lookup). Same pattern as existing doctor checks. Surfaces problems before they ship.
|
||||
**Cons:** Doctor's check list grows; needs a `--fast` opt-out path if added to the default scan. ~40 lines.
|
||||
**Context:** PR #719 added the multimodal_model routing key. The recipe-level + model-level validation in `embedMultimodal()` already throws clear errors at runtime, but only when image content hits the gateway. v0.28.x candidate.
|
||||
**Depends on:** None.
|
||||
|
||||
### Reclassify Voyage HTTP 4xx as `AIConfigError` (Codex F2 from PR #719 review)
|
||||
**Priority:** P2
|
||||
|
||||
**What:** `src/core/ai/gateway.ts:626` currently throws `AITransientError` for any non-401/403 4xx response from Voyage's /multimodalembeddings endpoint. Replace with a 4xx-non-429 → `AIConfigError` branch matching `normalizeAIError`'s contract at `src/core/ai/errors.ts:54`.
|
||||
|
||||
**Why:** A config bug (malformed body, unsupported field, model the caller forgot to add to `multimodal_models`) currently presents to the caller as transient and triggers retry storms. PR #719's Change 3 closes the specific wrong-multimodal-model case locally via the `multimodal_models` allow-list, but other 4xx reasons still misclassify.
|
||||
|
||||
**Pros:** Aligns the embedMultimodal error classifier with `normalizeAIError`. Eliminates retry-on-permanent-bug behavior. ~10 lines + 1 test.
|
||||
**Cons:** Changes runtime error class for some failures; existing callers that catch `AITransientError` for these codes now must catch `AIConfigError`. Search before merging.
|
||||
**Context:** Pre-existing in v0.27.1; surfaced because PR #719's new key makes the misclass more reachable. v0.28.x candidate.
|
||||
**Depends on:** None.
|
||||
|
||||
### `gbrain config unset <key>` subcommand (Codex F6 from PR #719 review)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Add `unset` action alongside `show|get|set` in `src/commands/config.ts`. Calls `engine.setConfig(key, '')` (loadConfigWithEngine treats empty string as undefined) so a user who set a key by mistake can clear it. Empty-string write is the minimum-diff implementation; a real DELETE would be cleaner if the engine grows one.
|
||||
|
||||
**Why:** Once a user runs `gbrain config set X val`, there's no normal CLI path to clear it. Empty string is rejected by the current `set` validator (`action === 'set' && key && value` where value is truthy). PR #719 added another DB-merge key (`embedding_multimodal_model`) and surfaces this UX gap.
|
||||
|
||||
**Pros:** Closes a pre-existing UX hole that applies to every DB-merge key (`embedding_multimodal`, `embedding_image_ocr*`, now `embedding_multimodal_model`). Trivial implementation, ~15 lines.
|
||||
**Cons:** Need to decide whether `unset` is a real DELETE (cleaner) or empty-string write (simpler).
|
||||
**Context:** Pre-existing in v0.27.x. Worth doing alongside the doctor checks above so users have a working escape hatch.
|
||||
**Depends on:** None.
|
||||
|
||||
## cross-modal-eval (v0.27.x follow-ups from PR #674 plan)
|
||||
|
||||
### `--budget-usd` hard cap + per-call cost telemetry (T11=B follow-up)
|
||||
@@ -1892,166 +1467,3 @@ doesn't gate on scopes. Adding per-tool scope enforcement would let
|
||||
**Effort estimate:** M (human: ~1 day / CC: ~30 min for the schema-aware gate).
|
||||
**Priority:** P3.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
---
|
||||
|
||||
### `@garrytan/gbrain` scoped-name npm publishing
|
||||
**What:** Publish gbrain to npm under the scoped name `@garrytan/gbrain`
|
||||
instead of the bare `gbrain` name. Provides structural defense against the
|
||||
unrelated `gbrain@1.x` squatter package on npm.
|
||||
|
||||
**Why:** `classifyBunInstall()` at `src/commands/upgrade.ts:395` does a
|
||||
best-effort fingerprint check on `repository.url` + `src/cli.ts` marker, with
|
||||
the comment explicitly accepting that signals are spoofable by a determined
|
||||
squatter. Scoped publishing is the structural answer that closes the loop:
|
||||
`bun add -g @garrytan/gbrain` cannot collide with any non-`@garrytan` package.
|
||||
|
||||
**Pros:** closes the squatter vector; consistent with how high-trust npm
|
||||
packages are published; allows removing `classifyBunInstall`'s spoofable
|
||||
signals later.
|
||||
|
||||
**Cons:** multi-week effort; needs reverse-compatible upgrade path for users
|
||||
on the bare-name install (`bun add -g gbrain` → recovery message pointing
|
||||
at the new scoped name); npm publishing flow changes; CI publish step needs
|
||||
scope-aware tagging.
|
||||
|
||||
**Context:** tracked at `src/commands/upgrade.ts:392-394` since v0.29; reaffirmed
|
||||
during v0.31.8 codex outside-voice review. Issue #658 has the surface-level
|
||||
history.
|
||||
|
||||
**Effort estimate:** L (human: ~1 week / CC: ~half a day for the publishing
|
||||
flow + recovery messaging).
|
||||
**Priority:** P2.
|
||||
**Depends on:** decision on whether to deprecate the bare name or dual-publish
|
||||
during a transition window.
|
||||
|
||||
|
||||
## v0.32.6 follow-ups from PR #880 (gbrain-context post-Codex recalibration)
|
||||
|
||||
These items were demoted from the PR #880 scope because they depend on
|
||||
infrastructure (clock-injection seam, public-API design) that's not in this PR.
|
||||
Filed for a future fix wave.
|
||||
|
||||
### Clock-injection seam in `src/core/context-engine.ts`
|
||||
|
||||
**Status:** Prerequisite for re-promoting perf-budget + snapshot tests.
|
||||
|
||||
**What:** Inject a `now: () => Date` into the engine factory so all `new Date()`
|
||||
call sites (lines 207, 371, and Date.now() at 354) read through one source.
|
||||
~10 lines.
|
||||
|
||||
**Why:** The plan proposed two test infrastructure items (perf budget at p99 <
|
||||
50ms, full-block snapshot for format-drift) that both depend on a stable clock.
|
||||
Without injection, snapshot tests flake on the time field and perf tests
|
||||
double-call `Date` non-deterministically.
|
||||
|
||||
**Effort:** S (CC: ~30 min).
|
||||
|
||||
### Perf-budget assertion (T-NEW2)
|
||||
|
||||
**Depends on:** clock-injection seam above.
|
||||
|
||||
**What:** New test asserting `assemble()` p99 stays under 50ms over 50 warm
|
||||
runs. The headline claim of the engine is "<5ms per turn"; right now nothing
|
||||
ratchets that in.
|
||||
|
||||
**Codex F2 note for the implementation:** Use `Math.floor(50 × 0.95)` (index
|
||||
47) for p95 or the actual sorted-percentile method, NOT `Math.floor(50 ×
|
||||
0.99)` which returns index 49 = the MAX sample and fails on one scheduler
|
||||
pause.
|
||||
|
||||
### Full-block snapshot test (T-NEW3)
|
||||
|
||||
**Depends on:** clock-injection seam above.
|
||||
|
||||
**What:** `expect(result.systemPromptAddition).toMatchSnapshot()` with a
|
||||
deterministic clock + fixture workspace. Pins the wire format so a reorder of
|
||||
fields or rename of `**Location:**` to `**Where:**` is caught.
|
||||
|
||||
### `exports` map entry for `./context-engine` (C-NEW2)
|
||||
|
||||
**Codex F8 note:** Adding `"./context-engine": "./src/core/context-engine.ts"`
|
||||
creates premature public-API obligations around types, lazy SDK loading, `.ts`
|
||||
imports, and engine-version semantics. Plugin loading via
|
||||
`openclaw.extensions` doesn't need it. Revisit when external consumers
|
||||
(gbrain-evals harness, etc) actually need direct engine import.
|
||||
|
||||
### `.ts`-extension import resolution coupling (A3)
|
||||
|
||||
**What:** `src/openclaw-context-engine.ts:25` imports
|
||||
`./core/context-engine.ts` with explicit `.ts` extension. Bun handles natively;
|
||||
standard `tsc` emit + Node ESM require `.js`. If OpenClaw ever transpiles
|
||||
before loading, this breaks.
|
||||
|
||||
**Defer until:** OpenClaw integration fails on this path.
|
||||
|
||||
### Typed `openclaw/plugin-sdk` ambient module shim (A5)
|
||||
|
||||
**What:** Replace `@ts-ignore` at the lazy SDK import in
|
||||
`src/core/context-engine.ts` with `types/openclaw-shim.d.ts` declaring
|
||||
ambient module signatures. ~30 lines. Lets typecheck catch typos and
|
||||
signature changes in the SDK that `@ts-ignore` silences.
|
||||
|
||||
### `loadJsonFile` parse-error warning (C-prior C5)
|
||||
|
||||
**What:** Add `console.warn` on JSON parse failure so the heartbeat cron's
|
||||
mistakes surface in stderr instead of silently degrading to defaults.
|
||||
|
||||
### Fractional-hour timezone offset (C-prior C3)
|
||||
|
||||
**What:** `getTimeInTz` rounds offsets at lines 217-224 (integer
|
||||
`localH - utcH` math). India (UTC+5:30), Nepal (UTC+5:45), Newfoundland
|
||||
(UTC-3:30), Chatham Islands (UTC+12:45) all round to the wrong whole hour
|
||||
in the emitted ISO. `dayOfWeek` and `hour` are correct via `Intl`; only the
|
||||
embedded offset string is wrong. Fix: use `Intl.DateTimeFormat` with
|
||||
`timeZoneName: 'longOffset'`.
|
||||
|
||||
### DST-boundary test (deferred)
|
||||
|
||||
**What:** Lock in `getTimeInTz` behavior across spring-forward / fall-back
|
||||
transitions. Edge case but real if Garry travels during a transition window.
|
||||
|
||||
### Multibyte sanitizer test (deferred)
|
||||
|
||||
**What:** `sanitizeForPrompt(s, 100)` clamps at 100 chars via `.slice(0, 100)`
|
||||
which operates on UTF-16 code units. A surrogate pair could be split mid-pair.
|
||||
Very low likelihood (real attendees are <50 chars) but the test surface is
|
||||
empty.
|
||||
|
||||
### Dynamic airport-tz lookup (Codex parenthetical)
|
||||
|
||||
**What:** `AIRPORT_TZ` as a 30-entry static map is the wrong long-term
|
||||
primitive. Either pull from a small tz library (e.g., `@vvo/tzdb`) keyed on
|
||||
IATA code, or require the heartbeat producer to supply
|
||||
`flights.destinationTimezone` in the JSON shape directly.
|
||||
|
||||
### Workspace contract documentation (DOC1)
|
||||
|
||||
**What:** New `docs/openclaw-context-engine.md` explaining which workspace
|
||||
files the engine reads, their schemas, who's expected to write them, and the
|
||||
atomic-rename concurrency contract. The interface is implicit in the test
|
||||
fixtures today.
|
||||
|
||||
### CLAUDE.md "Key files" annotations (DOC2)
|
||||
|
||||
**What:** Add one-line entries under CLAUDE.md's "Key files" section for
|
||||
`src/core/context-engine.ts` and `src/openclaw-context-engine.ts`. Per
|
||||
project convention for new architectural files.
|
||||
|
||||
### Repo-wide privacy scrub
|
||||
|
||||
**Status:** Out of scope for PR #880 (which scrubbed `test/context-engine.test.ts`
|
||||
and added the new CI guard). The guard surfaced 4 additional pre-existing
|
||||
references in other test files plus ~24 references in non-test files
|
||||
(CHANGELOG entries, docs, skill READMEs). Each entry needs case-by-case
|
||||
judgment.
|
||||
|
||||
**What:** Dedicated pass across:
|
||||
- Non-allowlisted pre-existing test-file matches (extract.test.ts,
|
||||
serve-stdio-lifecycle.test.ts — currently allowlisted as pre-existing
|
||||
but warrant a real scrub).
|
||||
- 24 doc/skill/CHANGELOG matches (most are historical and may not be
|
||||
retroactively rewriteable, but should be triaged).
|
||||
|
||||
**Depends on:** human judgment on which historical CHANGELOG entries to
|
||||
leave intact vs scrub.
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+56
File diff suppressed because one or more lines are too long
Vendored
-56
File diff suppressed because one or more lines are too long
Vendored
+2
-2
@@ -7,8 +7,8 @@
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
|
||||
<script type="module" crossorigin src="/admin/assets/index-CWq369vO.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/admin/assets/index-GxkWX7v3.css">
|
||||
<script type="module" crossorigin src="/admin/assets/index-CDv6_ml5.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/admin/assets/index-BOifXQpQ.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
||||
+2
-6
@@ -3,14 +3,13 @@ import { LoginPage } from './pages/Login';
|
||||
import { DashboardPage } from './pages/Dashboard';
|
||||
import { AgentsPage } from './pages/Agents';
|
||||
import { RequestLogPage } from './pages/RequestLog';
|
||||
import { CalibrationPage } from './pages/Calibration';
|
||||
import { api } from './api';
|
||||
|
||||
type Page = 'login' | 'dashboard' | 'agents' | 'log' | 'calibration';
|
||||
type Page = 'login' | 'dashboard' | 'agents' | 'log';
|
||||
|
||||
function getPage(): Page {
|
||||
const hash = window.location.hash.replace('#', '') || 'dashboard';
|
||||
if (['login', 'dashboard', 'agents', 'log', 'calibration'].includes(hash)) return hash as Page;
|
||||
if (['login', 'dashboard', 'agents', 'log'].includes(hash)) return hash as Page;
|
||||
return 'dashboard';
|
||||
}
|
||||
|
||||
@@ -55,8 +54,6 @@ export function App() {
|
||||
onClick={() => navigate('agents')}>Agents</a>
|
||||
<a className={`nav-item ${page === 'log' ? 'active' : ''}`}
|
||||
onClick={() => navigate('log')}>Request Log</a>
|
||||
<a className={`nav-item ${page === 'calibration' ? 'active' : ''}`}
|
||||
onClick={() => navigate('calibration')}>Calibration</a>
|
||||
</div>
|
||||
<div style={{ marginTop: 'auto', padding: '16px 12px', borderTop: '1px solid var(--border)' }}>
|
||||
<button
|
||||
@@ -81,7 +78,6 @@ export function App() {
|
||||
{page === 'dashboard' && <DashboardPage />}
|
||||
{page === 'agents' && <AgentsPage />}
|
||||
{page === 'log' && <RequestLogPage />}
|
||||
{page === 'calibration' && <CalibrationPage />}
|
||||
</main>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -22,17 +22,6 @@ async function apiFetch(path: string, options?: RequestInit) {
|
||||
return res.json();
|
||||
}
|
||||
|
||||
// v0.36.1.0 (T15 / E6) — SVG fetch (text/plain payload, NOT JSON).
|
||||
async function apiFetchText(path: string) {
|
||||
const res = await fetch(`${BASE}${path}`, { credentials: 'same-origin' });
|
||||
if (res.status === 401) {
|
||||
window.location.hash = '#login';
|
||||
throw new Error('Unauthorized');
|
||||
}
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
return res.text();
|
||||
}
|
||||
|
||||
export const api = {
|
||||
login: (token: string) => apiFetch('/admin/login', { method: 'POST', body: JSON.stringify({ token }) }),
|
||||
signOutEverywhere: () => apiFetch('/admin/api/sign-out-everywhere', { method: 'POST' }),
|
||||
@@ -45,9 +34,4 @@ export const api = {
|
||||
revokeApiKey: (name: string) => apiFetch('/admin/api/api-keys/revoke', { method: 'POST', body: JSON.stringify({ name }) }),
|
||||
updateClientTtl: (clientId: string, tokenTtl: number | null) => apiFetch('/admin/api/update-client-ttl', { method: 'POST', body: JSON.stringify({ clientId, tokenTtl }) }),
|
||||
revokeClient: (clientId: string) => apiFetch('/admin/api/revoke-client', { method: 'POST', body: JSON.stringify({ clientId }) }),
|
||||
// v0.36.1.0 (T15 / E6) — calibration endpoints.
|
||||
calibrationProfile: (holder?: string) =>
|
||||
apiFetch(`/admin/api/calibration/profile${holder ? `?holder=${encodeURIComponent(holder)}` : ''}`),
|
||||
calibrationChart: (type: string, holder?: string) =>
|
||||
apiFetchText(`/admin/api/calibration/charts/${encodeURIComponent(type)}${holder ? `?holder=${encodeURIComponent(holder)}` : ''}`),
|
||||
};
|
||||
|
||||
+1
-4
@@ -4,10 +4,7 @@
|
||||
--bg-tertiary: #1e1e2e;
|
||||
--text-primary: #e0e0e0;
|
||||
--text-secondary: #888;
|
||||
/* v0.36.1.0 TD2 — bumped from #555 (contrast 4.0 on #0a0a0f bg, below WCAG AA
|
||||
4.5 for body text) to #777 (contrast ~5.5, passes AA). Applies globally
|
||||
to Dashboard, Agents, RequestLog, and the new Calibration tab. */
|
||||
--text-muted: #777;
|
||||
--text-muted: #555;
|
||||
--accent: #3b82f6;
|
||||
--success: #22c55e;
|
||||
--warning: #f59e0b;
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
/**
|
||||
* v0.36.1.0 (T15 / E6) — Calibration tab.
|
||||
*
|
||||
* Fetches the active calibration profile + 4 server-rendered SVG charts.
|
||||
* Layout: Linear calm clarity (per D23 mockup variant-B) — single column,
|
||||
* generous whitespace, ONE big sparkline as hero, then patterns, then
|
||||
* domain bars, then abandoned threads.
|
||||
*
|
||||
* Per D23 — SVG markup comes from the server (image/svg+xml endpoint).
|
||||
* Admin SPA renders inside a TrustedSVG wrapper that uses
|
||||
* dangerouslySetInnerHTML. XSS posture: server-side escapeXml() on all
|
||||
* caller-controlled strings + requireAdmin middleware on the endpoint.
|
||||
*/
|
||||
|
||||
import React, { useEffect, useState } from 'react';
|
||||
import { api } from '../api';
|
||||
|
||||
interface CalibrationProfileSummary {
|
||||
holder: string;
|
||||
source_id: string;
|
||||
generated_at: string;
|
||||
published: boolean;
|
||||
total_resolved: number;
|
||||
brier: number | null;
|
||||
accuracy: number | null;
|
||||
partial_rate: number | null;
|
||||
grade_completion: number;
|
||||
pattern_statements: string[];
|
||||
active_bias_tags: string[];
|
||||
voice_gate_passed: boolean;
|
||||
voice_gate_attempts: number;
|
||||
}
|
||||
|
||||
interface ChartSvgProps {
|
||||
type: string;
|
||||
ariaLabel: string;
|
||||
}
|
||||
|
||||
function TrustedSVG({ markup }: { markup: string }) {
|
||||
return (
|
||||
<div
|
||||
style={{ width: '100%', overflow: 'auto' }}
|
||||
// Server-rendered SVG (image/svg+xml) gated by requireAdmin middleware.
|
||||
// All caller-controlled strings pass through escapeXml() server-side.
|
||||
dangerouslySetInnerHTML={{ __html: markup }}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
function ChartSvg({ type, ariaLabel }: ChartSvgProps) {
|
||||
const [markup, setMarkup] = useState<string>('');
|
||||
const [error, setError] = useState<string>('');
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
api
|
||||
.calibrationChart(type)
|
||||
.then(svg => {
|
||||
if (!cancelled) setMarkup(svg);
|
||||
})
|
||||
.catch(err => {
|
||||
if (!cancelled) setError(err.message ?? 'fetch failed');
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [type]);
|
||||
|
||||
if (error) {
|
||||
return (
|
||||
<div style={{ padding: 16, color: 'var(--error)' }} role="alert">
|
||||
{ariaLabel}: {error}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (!markup) {
|
||||
return <div style={{ padding: 16, color: 'var(--text-muted)' }}>{ariaLabel} loading...</div>;
|
||||
}
|
||||
return <TrustedSVG markup={markup} />;
|
||||
}
|
||||
|
||||
export function CalibrationPage() {
|
||||
const [profile, setProfile] = useState<CalibrationProfileSummary | null>(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [error, setError] = useState<string>('');
|
||||
|
||||
useEffect(() => {
|
||||
api
|
||||
.calibrationProfile()
|
||||
.then(p => {
|
||||
setProfile(p);
|
||||
setLoading(false);
|
||||
})
|
||||
.catch(err => {
|
||||
setError(err.message ?? 'fetch failed');
|
||||
setLoading(false);
|
||||
});
|
||||
}, []);
|
||||
|
||||
if (loading) {
|
||||
return <div style={{ padding: 24, color: 'var(--text-secondary)' }}>Loading calibration profile…</div>;
|
||||
}
|
||||
if (error) {
|
||||
return (
|
||||
<div style={{ padding: 24, color: 'var(--error)' }} role="alert">
|
||||
Could not load calibration profile: {error}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (!profile) {
|
||||
return (
|
||||
<div style={{ padding: 24, maxWidth: 700 }}>
|
||||
<h1 style={{ marginBottom: 16 }}>Calibration</h1>
|
||||
<p style={{ color: 'var(--text-secondary)' }}>
|
||||
No calibration profile yet. Builds after 5+ resolved takes.
|
||||
</p>
|
||||
<pre
|
||||
style={{
|
||||
background: 'var(--bg-secondary)',
|
||||
padding: 12,
|
||||
borderRadius: 4,
|
||||
color: 'var(--text-primary)',
|
||||
marginTop: 12,
|
||||
fontFamily: 'var(--font-mono)',
|
||||
}}
|
||||
>
|
||||
gbrain dream --phase calibration_profile
|
||||
</pre>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
const generated = new Date(profile.generated_at);
|
||||
const generatedAgo = Math.floor((Date.now() - generated.getTime()) / (1000 * 60 * 60 * 24));
|
||||
|
||||
return (
|
||||
<div style={{ padding: 32, maxWidth: 720 }}>
|
||||
<h1 style={{ marginBottom: 8 }}>Calibration</h1>
|
||||
<div style={{ color: 'var(--text-muted)', fontSize: 13, marginBottom: 24 }}>
|
||||
Holder: {profile.holder}
|
||||
{' · '}
|
||||
Updated {generatedAgo === 0 ? 'today' : `${generatedAgo}d ago`}
|
||||
{profile.published && ' · published'}
|
||||
{profile.grade_completion < 0.9 && ` · ~${Math.round(profile.grade_completion * 100)}% graded`}
|
||||
{!profile.voice_gate_passed && ' · voice gate fell back to template'}
|
||||
</div>
|
||||
|
||||
<section style={{ marginBottom: 32 }}>
|
||||
<ChartSvg type="brier-trend" ariaLabel="Brier trend" />
|
||||
</section>
|
||||
|
||||
<section style={{ marginBottom: 32 }}>
|
||||
<h2 style={{ fontSize: 14, color: 'var(--text-secondary)', marginBottom: 12, fontWeight: 400 }}>
|
||||
Pattern statements
|
||||
</h2>
|
||||
<ChartSvg type="pattern-statements" ariaLabel="Pattern statements" />
|
||||
</section>
|
||||
|
||||
<section style={{ marginBottom: 32 }}>
|
||||
<ChartSvg type="domain-bars" ariaLabel="Per-domain accuracy" />
|
||||
</section>
|
||||
|
||||
<section style={{ marginBottom: 32 }}>
|
||||
<ChartSvg type="abandoned-threads" ariaLabel="Abandoned threads" />
|
||||
</section>
|
||||
|
||||
{profile.active_bias_tags.length > 0 && (
|
||||
<section style={{ marginBottom: 32, color: 'var(--text-muted)', fontSize: 13 }}>
|
||||
Active bias tags: {profile.active_bias_tags.join(', ')}
|
||||
</section>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -13,18 +13,14 @@
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@dqbd/tiktoken": "^1.0.22",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@jsquash/avif": "^2.1.1",
|
||||
"@jsquash/png": "^3.1.1",
|
||||
"@modelcontextprotocol/sdk": "1.29.0",
|
||||
"ai": "^6.0.168",
|
||||
"cookie-parser": "^1.4.7",
|
||||
"cors": "^2.8.5",
|
||||
"eventsource-parser": "^3.0.8",
|
||||
"exifr": "^7.1.3",
|
||||
"express": "^5.1.0",
|
||||
"express-rate-limit": "^7.5.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"heic-decode": "^2.1.0",
|
||||
"marked": "^18.0.0",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
@@ -149,10 +145,6 @@
|
||||
|
||||
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "^4" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
|
||||
|
||||
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
|
||||
|
||||
"@jsquash/png": ["@jsquash/png@3.1.1", "", {}, "sha512-C10pc+0H6j0h8fENOfnGOvkXCmvpSQTDGlfGd0sHphZhPSGTyLjIrHba0FaZZdsKqA/wlmhYicUHb92vfZphaw=="],
|
||||
|
||||
"@modelcontextprotocol/sdk": ["@modelcontextprotocol/sdk@1.29.0", "", { "dependencies": { "@hono/node-server": "^1.19.9", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "content-type": "^1.0.5", "cors": "^2.8.5", "cross-spawn": "^7.0.5", "eventsource": "^3.0.2", "eventsource-parser": "^3.0.0", "express": "^5.2.1", "express-rate-limit": "^8.2.1", "hono": "^4.11.4", "jose": "^6.1.3", "json-schema-typed": "^8.0.2", "pkce-challenge": "^5.0.0", "raw-body": "^3.0.0", "zod": "^3.25 || ^4.0", "zod-to-json-schema": "^3.25.1" }, "peerDependencies": { "@cfworker/json-schema": "^4.1.1" }, "optionalPeers": ["@cfworker/json-schema"] }, "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ=="],
|
||||
|
||||
"@opentelemetry/api": ["@opentelemetry/api@1.9.0", "", {}, "sha512-3giAOQvZiH5F9bMlMiv8+GSPMeqg0dbaeo58/0SlA9sxSqZhnUtxzX9/2FzyhS9sWQf5S0GJE0AKBrFqjpeYcg=="],
|
||||
@@ -365,8 +357,6 @@
|
||||
|
||||
"eventsource-parser": ["eventsource-parser@3.0.8", "", {}, "sha512-70QWGkr4snxr0OXLRWsFLeRBIRPuQOvt4s8QYjmUlmlkyTZkRqS7EDVRZtzU3TiyDbXSzaOeF0XUKy8PchzukQ=="],
|
||||
|
||||
"exifr": ["exifr@7.1.3", "", {}, "sha512-g/aje2noHivrRSLbAUtBPWFbxKdKhgj/xr1vATDdUXPOFYJlQ62Ft0oy+72V6XLIpDJfHs6gXLbBLAolqOXYRw=="],
|
||||
|
||||
"express": ["express@5.2.1", "", { "dependencies": { "accepts": "^2.0.0", "body-parser": "^2.2.1", "content-disposition": "^1.0.0", "content-type": "^1.0.5", "cookie": "^0.7.1", "cookie-signature": "^1.2.1", "debug": "^4.4.0", "depd": "^2.0.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "finalhandler": "^2.1.0", "fresh": "^2.0.0", "http-errors": "^2.0.0", "merge-descriptors": "^2.0.0", "mime-types": "^3.0.0", "on-finished": "^2.4.1", "once": "^1.4.0", "parseurl": "^1.3.3", "proxy-addr": "^2.0.7", "qs": "^6.14.0", "range-parser": "^1.2.1", "router": "^2.2.0", "send": "^1.1.0", "serve-static": "^2.2.0", "statuses": "^2.0.1", "type-is": "^2.0.1", "vary": "^1.1.2" } }, "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw=="],
|
||||
|
||||
"express-rate-limit": ["express-rate-limit@7.5.1", "", { "peerDependencies": { "express": ">= 4.11" } }, "sha512-7iN8iPMDzOMHPUYllBEsQdWVB6fPDMPqwjBaFrgr4Jgr/+okjvzAy+UHlYYL/Vs0OsOrMkwS6PJDkFlJwoxUnw=="],
|
||||
@@ -409,8 +399,6 @@
|
||||
|
||||
"hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
|
||||
|
||||
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
|
||||
|
||||
"hono": ["hono@4.12.10", "", {}, "sha512-mx/p18PLy5og9ufies2GOSUqep98Td9q4i/EF6X7yJgAiIopxqdfIO3jbqsi3jRgTgw88jMDEzVKi+V2EF+27w=="],
|
||||
|
||||
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
|
||||
@@ -443,8 +431,6 @@
|
||||
|
||||
"kind-of": ["kind-of@6.0.3", "", {}, "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw=="],
|
||||
|
||||
"libheif-js": ["libheif-js@1.19.8", "", {}, "sha512-vQJWusIxO7wavpON1dusciL8Go9jsIQ+EUrckauFYAiSTjcmLAsuJh3SszLpvkwPci3JcL41ek2n+LUZGFpPIQ=="],
|
||||
|
||||
"marked": ["marked@18.0.0", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-2e7Qiv/HJSXj8rDEpgTvGKsP8yYtI9xXHKDnrftrmnrJPaFNM7VRb2YCzWaX4BP1iCJ/XPduzDJZMFoqTCcIMA=="],
|
||||
|
||||
"math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],
|
||||
@@ -549,8 +535,6 @@
|
||||
|
||||
"vary": ["vary@1.1.2", "", {}, "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg=="],
|
||||
|
||||
"wasm-feature-detect": ["wasm-feature-detect@1.8.0", "", {}, "sha512-zksaLKM2fVlnB5jQQDqKXXwYHLQUVH9es+5TOOHwGOVJOCeRBCiPjwSg+3tN2AdTCzjgli4jijCH290kXb/zWQ=="],
|
||||
|
||||
"web-streams-polyfill": ["web-streams-polyfill@4.0.0-beta.3", "", {}, "sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug=="],
|
||||
|
||||
"web-tree-sitter": ["web-tree-sitter@0.22.6", "", {}, "sha512-hS87TH71Zd6mGAmYCvlgxeGDjqd9GTeqXNqTT+u0Gs51uIozNIaaq/kUAbV/Zf56jb2ZOyG8BxZs2GG9wbLi6Q=="],
|
||||
|
||||
@@ -102,16 +102,6 @@ Keeping it running and up to date.
|
||||
| [Upgrades & Auto-Update](guides/upgrades-auto-update.md) | check-update, agent notifications, migration files |
|
||||
| [Live Sync](guides/live-sync.md) | Keep the index current: cron, --watch, webhook approaches |
|
||||
|
||||
## Getting Started
|
||||
|
||||
After setup, the brain is empty. The cold-start skill sequences the highest-leverage
|
||||
data sources to populate it:
|
||||
|
||||
| Guide | What It Covers |
|
||||
|-------|---------------|
|
||||
| [Cold Start](../skills/cold-start/SKILL.md) | Day-one bootstrapping: contacts, calendar, email, conversations, social, archives. Uses ClawVisor for safe credential handling — agents never hold raw API keys. |
|
||||
| [Ask User](../skills/ask-user/SKILL.md) | Choice-gate pattern for human input at decision points. Used by cold-start and other skills. |
|
||||
|
||||
---
|
||||
|
||||
## Appendix: GBrain CLI Quick Reference
|
||||
|
||||
+2
-9
@@ -1,12 +1,5 @@
|
||||
# GBrain v0: Postgres-Native Personal Knowledge Brain
|
||||
|
||||
> **Historical design doc.** This is the original v0 spec from before PGLite landed. Several
|
||||
> forward-looking sections — most notably the SQLite engine plan — were superseded by
|
||||
> PGLite (embedded Postgres via WASM), which uses the same SQL dialect as Postgres and
|
||||
> eliminates the need for a separate FTS5/sqlite-vss translation layer. Kept here for
|
||||
> historical context; see [`ENGINES.md`](ENGINES.md) for the current engine architecture and
|
||||
> the [`CHANGELOG.md`](../CHANGELOG.md) for the actual implementation history.
|
||||
|
||||
## What this is
|
||||
|
||||
GBrain is a compiled intelligence system. Not a note-taking app. Not "chat with your notes."
|
||||
@@ -524,7 +517,7 @@ See `docs/ENGINES.md` for the pluggable engine architecture and future backend p
|
||||
- **Intelligence compiler.** Treat every fact as a first-class claim with source span, entity links, validity window, confidence, and contradiction status. "What changed, why, and what evidence would flip it again?" From Codex review. Builds on compiled truth model.
|
||||
- **Active skills via Trigger.dev.** Application-specific briefings, meeting prep. Belongs in OpenClaw, not generic brain infra.
|
||||
- **Multi-user access.** Supabase RLS + per-user API keys. v0 is single-user.
|
||||
- **SQLite engine.** Superseded by PGLite (embedded Postgres 17 via WASM) before v1. See [`ENGINES.md`](ENGINES.md) for the current engine architecture.
|
||||
- **SQLite engine.** Community PRs welcome. See `docs/SQLITE_ENGINE.md`.
|
||||
- **Docker Compose for self-hosted Postgres.** Community PRs welcome.
|
||||
- **Web UI.** Optional Vercel-hosted dashboard for browsing brain pages.
|
||||
|
||||
@@ -538,7 +531,7 @@ This means:
|
||||
- A future DuckDB engine could implement analytics-heavy workloads
|
||||
- The CLI, MCP server, and library consumers never know which engine runs underneath
|
||||
|
||||
See [`ENGINES.md`](ENGINES.md) for the full interface spec. (The original SQLite engine plan was superseded by PGLite; the contract-first `BrainEngine` interface made that swap clean.)
|
||||
See `docs/ENGINES.md` for the full interface spec and `docs/SQLITE_ENGINE.md` for the SQLite implementation plan.
|
||||
|
||||
## Review history
|
||||
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
# Install
|
||||
|
||||
Three install paths. Pick one. Mix later if needed.
|
||||
|
||||
## 1. Run with an agent platform (recommended)
|
||||
|
||||
Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](https://github.com/garrytan/hermes)?
|
||||
|
||||
```bash
|
||||
bun install -g github:garrytan/gbrain
|
||||
gbrain init --pglite # 2 seconds; no server
|
||||
gbrain skillpack scaffold --all # 43 skills scaffolded into your agent workspace
|
||||
gbrain doctor # green checks all the way down
|
||||
```
|
||||
|
||||
Your agent now reads `skills/RESOLVER.md` once per request, routes intent to the right skill, executes. New entity mentions create new pages. Daily cron runs enrichment overnight.
|
||||
|
||||
Scaffolded skills are first-class files in your agent repo — edit freely. To pull upstream gbrain improvements later, `gbrain skillpack reference <name>` diffs your local copy vs the bundle. The legacy `skillpack install` managed-block model was retired in v0.36.0.0; if you're upgrading from an older release, run `gbrain skillpack migrate-fence` once to strip the legacy fence and keep your existing skill rows.
|
||||
|
||||
To upgrade later: `gbrain upgrade` runs schema migrations + post-upgrade prompts (chunker bumps, the v0.36.2.0 ZeroEntropy switch). Always TTY-only; non-TTY upgrades skip prompts with informational stderr lines.
|
||||
|
||||
## 2. CLI standalone
|
||||
|
||||
No agent platform, just shell + MCP-aware editor.
|
||||
|
||||
```bash
|
||||
bun install -g github:garrytan/gbrain
|
||||
gbrain init --pglite
|
||||
```
|
||||
|
||||
> **If `bun install -g` hits a postinstall error** (Bun blocks postinstall hooks in some environments), the CLI prints a recovery hint pointing at [#218](https://github.com/garrytan/gbrain/issues/218). Run `gbrain doctor` to diagnose, then `gbrain apply-migrations --yes` manually. The deterministic fallback is `git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain && bun install && bun link`.
|
||||
|
||||
The init flow detects your repo size and suggests Supabase for brains > 1000 markdown files. To switch later:
|
||||
|
||||
```bash
|
||||
gbrain migrate --to supabase # PGLite → Postgres
|
||||
gbrain migrate --to pglite # Postgres → PGLite (rare)
|
||||
```
|
||||
|
||||
API keys live in `~/.gbrain/config.json` (file plane) or env vars (`OPENAI_API_KEY`, `ZEROENTROPY_API_KEY`, `VOYAGE_API_KEY`, `ANTHROPIC_API_KEY`). Set via CLI:
|
||||
|
||||
```bash
|
||||
gbrain config set zeroentropy_api_key sk-...
|
||||
gbrain config set anthropic_api_key sk-ant-...
|
||||
```
|
||||
|
||||
Common follow-ups:
|
||||
|
||||
```bash
|
||||
gbrain import ~/my-knowledge # bulk-import a markdown folder
|
||||
gbrain sync --watch # live-sync a git repo (autopilot mode)
|
||||
gbrain autopilot --install # background daemon for nightly enrichment
|
||||
```
|
||||
|
||||
## 3. MCP server (any MCP client)
|
||||
|
||||
```bash
|
||||
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
|
||||
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
|
||||
```
|
||||
|
||||
Per-client setup guides live in [`docs/mcp/`](mcp/):
|
||||
|
||||
- [`docs/mcp/CLAUDE_CODE.md`](mcp/CLAUDE_CODE.md)
|
||||
- [`docs/mcp/CLAUDE_DESKTOP.md`](mcp/CLAUDE_DESKTOP.md)
|
||||
- [`docs/mcp/CHATGPT.md`](mcp/CHATGPT.md)
|
||||
- [`docs/mcp/PERPLEXITY.md`](mcp/PERPLEXITY.md)
|
||||
- [`docs/mcp/DEPLOY.md`](mcp/DEPLOY.md) — production deploy patterns
|
||||
|
||||
The HTTP server ships with an admin SPA at `/admin`, an SSE activity feed at `/admin/events`, DCR-style client registration, scope-gated `read`/`write`/`admin` access, and rate limiting.
|
||||
|
||||
## Thin-client mode
|
||||
|
||||
Connect to someone else's brain without running a local engine:
|
||||
|
||||
```bash
|
||||
gbrain init --mcp-only # configures remote MCP, skips local DB
|
||||
```
|
||||
|
||||
Useful for: team mounts, brain-as-a-service deployments, dev machines without disk space. Most local commands refuse with a paste-ready hint. See [`docs/architecture/topologies.md`](architecture/topologies.md).
|
||||
|
||||
## Verifying the install
|
||||
|
||||
```bash
|
||||
gbrain doctor --json # full health check
|
||||
gbrain models # which AI models are configured for what
|
||||
gbrain models doctor # 1-token probe per configured model
|
||||
```
|
||||
|
||||
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`).
|
||||
@@ -538,75 +538,3 @@ To check what your fork is missing:
|
||||
diff <(grep -A3 "Based on gbrain" ~/<your-fork>/skills/brain-ops/SKILL.md) \
|
||||
<(grep "v[0-9]" ~/gbrain/skills/migrations/ | tail -3)
|
||||
```
|
||||
|
||||
|
||||
## v0.36.5.0 — Free-form secret inheritance for shell jobs calling `gbrain` CLI
|
||||
|
||||
**The change.** Shell-job params get a new `inherit:` field. Pass any
|
||||
snake_case config-key name on it; the worker resolves the value from its
|
||||
`loadConfig()` at child-spawn time and injects it into the child env. Names
|
||||
land in the row; values never persist from `inherit:`. Validation runs
|
||||
**pre-enqueue** in both submit paths (CLI + `submit_job` op), so a malformed
|
||||
payload never lands in `minion_jobs.data`.
|
||||
|
||||
**Why.** Pre-v0.36.5.0, agents that wanted to call `gbrain` from shell jobs
|
||||
had to either write `database_url` to `~/.gbrain/config.json` plaintext or
|
||||
pass `env: { GBRAIN_DATABASE_URL: "..." }` per-job. Both left plaintext
|
||||
secrets somewhere — disk or DB row. `inherit:` keeps names in the row and
|
||||
resolves values at spawn time.
|
||||
|
||||
**What your agent can do.** `inherit:` is free-form. Pass any config-key:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"cmd": "gbrain sync --skip-failed && gbrain embed --stale",
|
||||
"cwd": "/data/gbrain",
|
||||
"inherit": ["database_url", "anthropic_api_key", "voyage_api_key"]
|
||||
}
|
||||
```
|
||||
|
||||
The env-key name in the child is derived by uppercasing the config-key:
|
||||
`database_url` → `GBRAIN_DATABASE_URL`, `anthropic_api_key` →
|
||||
`ANTHROPIC_API_KEY`, `voyage_api_key` → `VOYAGE_API_KEY`, etc. The validator
|
||||
does NOT police which config keys you inherit — the agent is in the same
|
||||
uid as the worker, so it's the agent's call.
|
||||
|
||||
**You can still use `env:`.** v0.36.5.0 does not forbid `env:{ ANYTHING }`.
|
||||
If you have a reason to put a value in the row plaintext (a non-secret
|
||||
correlation token, or a secret you know is OK to persist), pass it via
|
||||
`env:`. Prefer `inherit:` when you want the value out of the row.
|
||||
|
||||
**Worker setup** (one-time, per host):
|
||||
|
||||
- `gbrain config set database_url postgresql://...` (or any other key you
|
||||
want available for inherit)
|
||||
- OR put the key in `~/.gbrain/config.json` directly
|
||||
- OR set `GBRAIN_DATABASE_URL` / `DATABASE_URL` / per-provider env on the
|
||||
worker process
|
||||
|
||||
If the worker can't resolve a requested name, the validator fail-fasts at
|
||||
submit time with `gbrain config set <X>` hint. No more silent "No database
|
||||
URL" failures in child stderr minutes after submission.
|
||||
|
||||
**Also new.** A `gbrain doctor` check `home_dir_in_worktree` warns if
|
||||
`~/.gbrain/` lives inside a git worktree. A retroactive `~/.gbrain/.gitignore`
|
||||
(single line `*`) is now laid down by every `saveConfig()` call AND by
|
||||
`gbrain post-upgrade`, so existing users get coverage without re-running
|
||||
`gbrain init`. Honest scope: the `.gitignore` covers casual `git add` but does
|
||||
NOT cover already-tracked files, screenshots, backups, or `git add -f`.
|
||||
|
||||
**Strategy framing.** For agent-to-gbrain calls, the new canonical guide is
|
||||
`docs/guides/agent-to-gbrain.md`. Two distinct surfaces: HTTP MCP via OAuth
|
||||
for ops with MCP equivalents (`search`, `query`, `put_page`, etc.), and shell
|
||||
job + `inherit:` for `localOnly` admin ops (`sync`, `embed`, `dream`,
|
||||
`doctor`, etc.). Not a fallback hierarchy — pick by op.
|
||||
|
||||
**Errors to handle** (your agent submits shell jobs; surface these clearly):
|
||||
|
||||
| Error | What it means | Agent action |
|
||||
|---|---|---|
|
||||
| `shell: inherit must be an array of config-key names` | `inherit` wasn't an array. | Pass `"inherit": ["database_url", ...]`. |
|
||||
| `shell: inherit entries must be non-empty strings` | Element was empty, non-string, or null. | Use snake_case config-key names. |
|
||||
| `shell: inherit name "<X>" must match [a-z][a-z0-9_]*` | Name failed snake_case regex (uppercase, leading underscore, etc.). | Use the config-key verbatim — `database_url`, not `DATABASE_URL`. |
|
||||
| `shell: inherit requested "<X>" but worker has no <X> configured` | Worker can't resolve the name from its `loadConfig()`. | Run `gbrain config set <X> <value>` on the worker host. |
|
||||
|
||||
|
||||
@@ -1,173 +0,0 @@
|
||||
# ZeroEntropy — zembed-1 + zerank-2
|
||||
|
||||
[ZeroEntropy](https://zeroentropy.dev) ships two specialized small models
|
||||
for retrieval pipelines:
|
||||
|
||||
- **`zembed-1`** — multilingual embedding distilled from zerank-2.
|
||||
Flexible Matryoshka dims (2560/1280/640/320/160/80/40), 32K context,
|
||||
asymmetric `input_type: query|document` encoding. $0.025/1M tokens
|
||||
(sale) / $0.05 regular.
|
||||
- **`zerank-2`** — SOTA multilingual cross-encoder reranker.
|
||||
$0.025/1M tokens (~50% cheaper than Cohere/Voyage rerankers).
|
||||
Plus `zerank-1` and `zerank-1-small` for legacy / open-source needs.
|
||||
|
||||
Both land in gbrain v0.35.0.0 behind the openai-compatible recipe path,
|
||||
alongside OpenAI and Voyage.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Get an API key at
|
||||
[dashboard.zeroentropy.dev](https://dashboard.zeroentropy.dev).
|
||||
2. Export it:
|
||||
```bash
|
||||
export ZEROENTROPY_API_KEY=<your-key>
|
||||
```
|
||||
|
||||
## Embedding switch — zembed-1
|
||||
|
||||
**Important:** `gbrain config set embedding_model …` is NOT a live
|
||||
gateway switch. `embedding_model` and `embedding_dimensions` size the
|
||||
schema and must be stable across engine connects, so they only resolve
|
||||
from the **file plane** (`~/.gbrain/config.json`) and the **env plane**
|
||||
(`GBRAIN_EMBEDDING_MODEL` / `GBRAIN_EMBEDDING_DIMENSIONS`). The DB plane
|
||||
is intentionally ignored for these two keys (same posture as today's
|
||||
Voyage setup).
|
||||
|
||||
### Option A — file plane (recommended for stable installs)
|
||||
|
||||
Edit `~/.gbrain/config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"embedding_model": "zeroentropyai:zembed-1",
|
||||
"embedding_dimensions": 2560
|
||||
}
|
||||
```
|
||||
|
||||
Valid dims: `2560` (default), `1280`, `640`, `320`, `160`, `80`, `40`.
|
||||
Matryoshka-style — smaller trades quality for storage monotonically.
|
||||
Pick the largest that fits your column width.
|
||||
|
||||
### Option B — env plane (CI / Docker)
|
||||
|
||||
```bash
|
||||
export GBRAIN_EMBEDDING_MODEL=zeroentropyai:zembed-1
|
||||
export GBRAIN_EMBEDDING_DIMENSIONS=2560
|
||||
```
|
||||
|
||||
### Re-embed
|
||||
|
||||
Switching embedding models invalidates the vector index. Re-embed:
|
||||
|
||||
```bash
|
||||
gbrain embed --stale --limit 50 # smoke a small batch
|
||||
gbrain embed --stale # full re-embed
|
||||
```
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
gbrain models doctor --json | jq '.probes[] | select(.touchpoint=="embedding_config")'
|
||||
```
|
||||
|
||||
Expected: `status: "ok"`. Invalid dims (e.g. `1024`, `1536`, `3072`)
|
||||
surface as `status: "config"` with a paste-ready
|
||||
`gbrain config set embedding_dimensions <one of 2560|1280|640|320|160|80|40>` fix hint.
|
||||
|
||||
## Reranker switch — zerank-2
|
||||
|
||||
The reranker is the bigger story: gbrain had no cross-encoder reranker
|
||||
stage before v0.35.0.0. It slots between RRF dedup and token-budget
|
||||
enforcement in hybrid search.
|
||||
|
||||
### Default-on with `tokenmax` mode
|
||||
|
||||
`tokenmax` mode now defaults `search.reranker.enabled = true` with
|
||||
`zerank-2`. If you already use `tokenmax` AND have `ZEROENTROPY_API_KEY`
|
||||
set, reranker fires automatically. Without the key, every rerank call
|
||||
fails-open (audit-logged) and search returns RRF order — same UX as
|
||||
before, just with an observable failure surfaced via `gbrain doctor`.
|
||||
|
||||
### Opt-in on `conservative` or `balanced` mode
|
||||
|
||||
```bash
|
||||
gbrain config set search.reranker.enabled true
|
||||
```
|
||||
|
||||
The override sits above the mode-bundle default; opt-out is one flip.
|
||||
|
||||
### Cost anchor
|
||||
|
||||
At 30 candidates × ~400 tokens/chunk × $0.025/1M = **~$0.0003/query**.
|
||||
Rounding error against the `tokenmax + Opus` pairing's ~$700/mo at
|
||||
single-user volume per the CLAUDE.md cost matrix.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
gbrain models doctor --json | jq '.probes[] | select(.touchpoint=="reranker_config")'
|
||||
```
|
||||
|
||||
Two probes run for reranker:
|
||||
- `reranker_config` (zero-network) — validates the model resolves
|
||||
through the recipe registry and is in the touchpoint's allowlist.
|
||||
- A reachability probe sends a minimal `{query: "probe", documents:
|
||||
["probe"]}` rerank to verify auth + URL.
|
||||
|
||||
## Knobs reference
|
||||
|
||||
| Config key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `search.reranker.enabled` | `true` for tokenmax, `false` for others | One-flip opt-in/out |
|
||||
| `search.reranker.model` | `zeroentropyai:zerank-2` | Try `zerank-1` (older SOTA) or `zerank-1-small` (Apache-2.0 open) |
|
||||
| `search.reranker.top_n_in` | `30` | Candidates sent to reranker (caps API spend) |
|
||||
| `search.reranker.top_n_out` | `null` (no truncate) | Truncate reranked output to this many; `null` preserves full length |
|
||||
| `search.reranker.timeout_ms` | `5000` | HTTP timeout; long stalls degrade UX worse than RRF fallback |
|
||||
|
||||
## Failure observability
|
||||
|
||||
Reranker is fail-open by construction: every error class (auth, rate-limit,
|
||||
network, timeout, payload-too-large, unknown) returns the original RRF
|
||||
order unchanged. Failures log to
|
||||
`~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation).
|
||||
|
||||
`gbrain doctor` reads the audit and surfaces:
|
||||
- **auth failures** — any single one warns (config-time problem doctor's
|
||||
own probe should have caught)
|
||||
- **payload-too-large** — any single one warns (workload-mismatch signal)
|
||||
- **transient (network/timeout/rate_limit)** — warns at >=5 in 7 days
|
||||
|
||||
Query text is SHA-256 hashed in the audit; never logged raw.
|
||||
|
||||
## Asymmetric input_type
|
||||
|
||||
ZE zembed-1 (and Voyage v3+) use asymmetric query/document encoding for
|
||||
better retrieval. The gateway's `embedQuery(text)` companion threads
|
||||
`input_type: 'query'`; standard `embed(texts)` defaults to
|
||||
`'document'`. Hybrid search's two query-side embed sites use
|
||||
`embedQuery()` automatically; all ingest paths use `embed()`.
|
||||
|
||||
Symmetric providers (OpenAI text-embedding-3, fixed-dim Voyage models)
|
||||
ignore the field — no behavior change.
|
||||
|
||||
## Cache key versioning
|
||||
|
||||
v0.35.0.0 bumped `KNOBS_HASH_VERSION` 1 → 2 to fold reranker config into
|
||||
the `query_cache.knobs_hash` column. During a rolling deploy:
|
||||
|
||||
- Expect a temporary cache hit-rate dip (~1 hour at default
|
||||
`cache.ttl_seconds = 3600s`)
|
||||
- Hot queries may briefly double their cache row count (one row per
|
||||
version)
|
||||
|
||||
Both clear naturally; no operator action required.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `embedding_config` probe says invalid dim | Defaulting to 1536 (OpenAI default) | Set `embedding_dimensions` to one of 2560/1280/640/320/160/80/40 |
|
||||
| `reranker_config` probe says model not in allowlist | Typo in `search.reranker.model` | Use one of `zerank-2` / `zerank-1` / `zerank-1-small` |
|
||||
| `reranker_health` doctor warns about auth | `ZEROENTROPY_API_KEY` not set or invalid | Re-export the env var; `gbrain models doctor` to verify |
|
||||
| `reranker_health` doctor warns about transient failures | Upstream flake or rate limit | Reranker fails open to RRF; check ZE status page if persistent |
|
||||
| Cache hit rate dipped after upgrade | Expected during rolling deploy | Clears within `cache.ttl_seconds` (default 3600s) |
|
||||
@@ -1,130 +0,0 @@
|
||||
# Why the hybrid + graph stack works
|
||||
|
||||
Vector search alone underdelivers on real personal-knowledge queries. This doc explains why gbrain layers four strategies together and how they compound.
|
||||
|
||||
## The four strategies in concert
|
||||
|
||||
1. **Vector (HNSW on pgvector)** — semantic similarity. Catches "who works on retrieval quality at YC?" → pages mentioning "Garry Tan + retrieval" even when the user never typed "YC".
|
||||
2. **BM25 keyword** — lexical match. Catches names, exact phrases, code identifiers, anything where the user remembers the literal token. Survives the cases where vector search drifts into thematic neighbors.
|
||||
3. **Reciprocal-rank fusion (RRF)** — merges vector + keyword rankings without weighting one over the other globally. Each strategy gets to vote.
|
||||
4. **Knowledge graph traversal** — follows typed edges. Catches "what did Bob invest in this quarter?" by walking `bob ── invested_in ──> company ── dated ──> Q1`. Vector search can't see causal chains; the graph can.
|
||||
|
||||
## Why each one alone fails
|
||||
|
||||
**Vector only.** Returns chunks semantically close to the query. Misses any factual relationship not directly encoded in the embedding. "Companies in Garry's portfolio" returns essays about portfolios, not company pages.
|
||||
|
||||
**Keyword only (ripgrep-style).** Brittle to phrasing. "Who works on retrieval?" misses pages that say "search ranking" instead of "retrieval." Garbage on synonyms, near-misses, or paraphrases.
|
||||
|
||||
**Graph only.** Excellent at "neighbors of Alice" but blind to anything not yet linked. Sparse on fresh pages until backlinks accumulate.
|
||||
|
||||
**Hybrid (vector + keyword + RRF), no graph.** Decent at "what is X?" type queries. Fails on "what is Y's relationship to X?" — those are graph queries and no amount of embedding tuning recovers them.
|
||||
|
||||
## The benchmark
|
||||
|
||||
BrainBench (corpus + harness in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) measures retrieval P@5, R@5, MRR, nDCG@5 on a 240-page Opus-generated rich-prose corpus.
|
||||
|
||||
| Strategy | P@5 | R@5 | Notes |
|
||||
|---|---|---|---|
|
||||
| ripgrep BM25 only | ~18 | ~75 | Lexical-only baseline |
|
||||
| vector-only RAG | ~18 | ~80 | Standard RAG implementation |
|
||||
| gbrain graph-disabled (hybrid + RRF, no graph traversal) | ~18 | ~85 | Hybrid alone |
|
||||
| **gbrain default (full stack)** | **49.1** | **97.9** | Graph + extract-quality lift |
|
||||
|
||||
**+31 P@5 points** from the graph + extract quality work. The graph isn't a marginal feature; it's the load-bearing wall.
|
||||
|
||||
## Auto-link: why zero-LLM-call edge extraction works
|
||||
|
||||
Every `put_page` runs `extractEntityRefs` on the markdown body. It matches:
|
||||
|
||||
- Standard markdown links: `[Garry Tan](wiki/people/garry-tan)`
|
||||
- Obsidian wikilinks: `[[wiki/people/garry-tan|Garry Tan]]`
|
||||
- Typed-link blockquotes: `> **Convention:** see [path](path).`
|
||||
|
||||
Three regexes, zero LLM tokens, single SQL `addLinksBatch` call with `INSERT ... SELECT FROM unnest(...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1`. The graph grows on every write at near-zero cost. On a 17K-page brain, full graph extract completes in seconds.
|
||||
|
||||
Heuristic link-type inference (`attended`, `works_at`, `invested_in`, `founded`, `advises`) fires from surrounding sentence context — also LLM-free. Power users who want richer types add them via the typed-link blockquote convention.
|
||||
|
||||
## ZeroEntropy as reranker: 60% top-1 reshuffle
|
||||
|
||||
v0.36.0.0 ships ZeroEntropy's `zerank-2` as the default reranker (on for the `balanced` mode bundle). On a real-corpus benchmark across 20 queries, zerank-2 reshuffles **60% of top-1 results** after the hybrid + RRF + graph stack. That's the headline number.
|
||||
|
||||
The mechanical reason: hybrid ranking is locally optimal per strategy but globally suboptimal. A cross-encoder reranker reads the query + each candidate document jointly, with full attention. It catches the cases where the vector + keyword + graph signals all agreed on a document that's semantically related but topically wrong.
|
||||
|
||||
The cost: +150ms p50 latency, ~$0.025/M tokens. Disabled with `gbrain config set search.reranker.enabled false`. For agent loops that do downstream LLM work after retrieval, the latency is invisible.
|
||||
|
||||
## Source-aware ranking
|
||||
|
||||
Hybrid search applies a source-factor CASE expression at the SQL layer (lives in `src/core/search/sql-ranking.ts`). Curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `your-openclaw/chat/`, `daily/`, `media/x/`. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`) filter at retrieval, not post-rank.
|
||||
|
||||
The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `SearchOpts.exclude_slug_prefixes`. Temporal queries (`detail: 'high'`) bypass the boost so chat pages re-surface for time-sensitive lookups.
|
||||
|
||||
## Intent-aware query rewriting
|
||||
|
||||
`src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs:
|
||||
|
||||
- **Entity** queries ("who works at X?") apply a higher graph-traversal weight.
|
||||
- **Temporal** queries ("what happened last week?") bypass source-boost so chat/daily pages surface.
|
||||
- **Event** queries ("Acme AI Series A") engage the timeline index.
|
||||
- **General** queries hit the standard hybrid stack.
|
||||
|
||||
The classifier is deterministic (no LLM call). Wrong classification degrades gracefully — the hybrid stack still works without it.
|
||||
|
||||
## Multi-query expansion
|
||||
|
||||
For `detail: 'high'` searches, `src/core/search/expansion.ts` runs a Haiku-class LLM call to produce 2-3 query variants. Each variant runs through the full hybrid stack; results merge via RRF. Catches synonym misses without recall loss.
|
||||
|
||||
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale.
|
||||
|
||||
## Putting it together
|
||||
|
||||
The full pipeline for a `query` op:
|
||||
|
||||
```
|
||||
intent classify
|
||||
│
|
||||
▼
|
||||
expansion (if enabled)
|
||||
│
|
||||
▼
|
||||
hybrid search:
|
||||
├── vector (HNSW on chunk embeddings)
|
||||
├── keyword (BM25 via tsvector)
|
||||
├── source-aware re-rank (CASE in SQL)
|
||||
└── RRF fusion → top 30
|
||||
│
|
||||
▼
|
||||
graph augment (typed-edge traversal from any seed)
|
||||
│
|
||||
▼
|
||||
reranker (zerank-2 cross-encoder, top 30 → reordered)
|
||||
│
|
||||
▼
|
||||
token-budget enforcement (per mode bundle)
|
||||
│
|
||||
▼
|
||||
deduplication (same slug, different chunks → keep best)
|
||||
│
|
||||
▼
|
||||
results
|
||||
```
|
||||
|
||||
Each stage is testable in isolation. Each stage is replaceable. The whole pipeline is < 1ms of orchestration cost; the latency budget goes to the upstream HTTP calls (embedding, rerank) and the index scans.
|
||||
|
||||
## How to verify on your own brain
|
||||
|
||||
```bash
|
||||
# Run the public LongMemEval benchmark
|
||||
gbrain eval longmemeval datasets/longmemeval_s.jsonl
|
||||
|
||||
# Capture your own queries and replay against retrieval changes
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||
# ... use gbrain normally ...
|
||||
gbrain eval export > before.ndjson
|
||||
# ... change something ...
|
||||
gbrain eval replay --against before.ndjson
|
||||
|
||||
# A/B retrieval strategies on a labeled fixture
|
||||
gbrain eval --qrels labels.tsv --config balanced.json
|
||||
```
|
||||
|
||||
Methodology + metric glossary in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](../eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
@@ -1,198 +0,0 @@
|
||||
# System of record
|
||||
|
||||
**The GitHub repo (markdown + frontmatter) is the system of record.
|
||||
The Postgres/PGLite database is a derived cache. We do not back up
|
||||
the database — we rebuild it from the repo.**
|
||||
|
||||
This document is the canonical reference for that contract. Every code
|
||||
path that writes user-knowledge state should match the pattern
|
||||
described here. The CI gate at `scripts/check-system-of-record.sh`
|
||||
enforces it programmatically.
|
||||
|
||||
## Why this matters
|
||||
|
||||
The DB is a derived index over the markdown content. It exists to make
|
||||
search fast, to dedup embedding-similar claims, to materialize the
|
||||
cross-page graph. None of that data is irreplaceable — as long as the
|
||||
markdown is intact, `gbrain sync && gbrain extract all` rebuilds the
|
||||
entire DB from scratch.
|
||||
|
||||
This means:
|
||||
|
||||
- **Disaster recovery is one command.** If your DB volume corrupts, if
|
||||
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
|
||||
a backup. You wipe the DB, re-import from your brain repo, and the
|
||||
derived state regenerates. v0.32.3 ships `gbrain rebuild
|
||||
--confirm-destructive` as the documented one-liner.
|
||||
- **Multi-machine sync is git.** Your brain is a repo. Push from one
|
||||
machine, pull from another, and the second machine's DB rebuilds on
|
||||
its next sync. No "back up the database" step.
|
||||
- **Privacy is in your hands.** Sensitive entity pages can be
|
||||
gitignored (via `gbrain.yml` `db_only` paths or per-page) and they
|
||||
stay on disk but not in git. The fence respects whatever git
|
||||
tracking choice you make at the page level.
|
||||
- **Cross-agent collaboration is possible.** Multiple agents can write
|
||||
to the same brain because the fence is the merge point, not the DB.
|
||||
Git handles concurrent edits the way git handles concurrent edits.
|
||||
|
||||
## The three categories
|
||||
|
||||
Every table in the gbrain schema belongs to exactly one of three
|
||||
categories. The category determines how it gets rebuilt during
|
||||
disaster recovery.
|
||||
|
||||
### FS-canonical (markdown is the source of truth)
|
||||
|
||||
These are user-authored knowledge. The DB row is a derived index over
|
||||
the markdown — wipe the table and `gbrain extract` rebuilds it
|
||||
identically. The CI gate keeps direct DB writes from drifting away
|
||||
from the markdown contract.
|
||||
|
||||
| Category | How it's stored in markdown | Derived DB table | Reconciler |
|
||||
|---|---|---|---|
|
||||
| **Takes** (incl. hunches, bets) | `## Takes` fenced table between `<!--- gbrain:takes:begin -->` / `:end -->` markers | `takes` | `extract takes` |
|
||||
| **Facts** | `## Facts` fenced table between `<!--- gbrain:facts:begin -->` / `:end -->` markers | `facts` | `extract_facts` cycle phase |
|
||||
| **Links** | Inline `[text](slug)` / `[[slug]]` in markdown body + frontmatter `direction: incoming` | `links` | `extract links` |
|
||||
| **Timeline** | `## Timeline` section after `<!-- timeline -->` sentinel | `timeline_entries` | `extract timeline` |
|
||||
| **Tags** | Frontmatter `tags:` YAML array | `tags` | `importFromFile` (reconciles per-page on import) |
|
||||
| **emotional_weight** | Recomputed from takes + tags | `pages.emotional_weight` (signal column) | `recompute_emotional_weight` cycle phase |
|
||||
| **synthesis_evidence** | FK into `takes` rows (`slug#N`) inside synthesis pages | `synthesis_evidence` | `extract takes` (transitively) |
|
||||
|
||||
### Derived from FS but not user-authored
|
||||
|
||||
These hold derived state that's automatically reconstructible from the
|
||||
markdown but not directly authored as markdown by the user. The
|
||||
chunker + embedder rebuild these on import.
|
||||
|
||||
| Table | Source | Notes |
|
||||
|---|---|---|
|
||||
| `pages` | The markdown file as a whole | One row per file; `compiled_truth` + `frontmatter` come from parse |
|
||||
| `content_chunks` | `pages.compiled_truth` after chunker strip | Re-chunked on content_hash change; embedded via configured model |
|
||||
| `page_versions` | Each `pages` UPDATE | Audit history; rebuildable in principle but not in practice |
|
||||
|
||||
### DB-only by design (named exceptions)
|
||||
|
||||
These hold runtime / infrastructure state that's intentionally not in
|
||||
the repo. The architectural rule still holds — these aren't
|
||||
"user knowledge" — but they're DB-only by design.
|
||||
|
||||
| Category | Why it's OK to be DB-only |
|
||||
|---|---|
|
||||
| `raw_data` | Webhook/transcript sidecars; not user-authored knowledge. |
|
||||
| `subagent_messages` / `subagent_tool_executions` / `subagent_rate_leases` | Runtime job state. Replay-only, not persistent knowledge. |
|
||||
| `oauth_clients` / `oauth_tokens` / `access_tokens` | Credentials. Not in source control by definition. |
|
||||
| `mcp_request_log` | Audit trail. Volatile by design. |
|
||||
| `minion_jobs` / `minion_inbox` / `minion_attachments` | Job queue. Restarts re-enqueue or drop. |
|
||||
| `eval_candidates` / `eval_capture_failures` | Contributor-mode dev loop; opt-in capture. |
|
||||
| `dream_verdicts` | Cheap verdict cache. Rebuildable by re-running Haiku. |
|
||||
| `gbrain_cycle_locks` / migration ledger | Infrastructure. |
|
||||
| `config` (some keys) | Site-local routing config (e.g. `sync.repo_path`). |
|
||||
|
||||
A new derived table that holds user-knowledge MUST land FS-first.
|
||||
If you're tempted to add one as "DB-only for now," the structural
|
||||
question is: does it belong in this DB-only-by-design list? If not,
|
||||
it's FS-canonical and needs a fence (or frontmatter field) plus a
|
||||
reconciler.
|
||||
|
||||
## The privacy boundary
|
||||
|
||||
Private knowledge in a fence still lives in the markdown file. If the
|
||||
user commits the page to git, the private data lands in git too. This
|
||||
is the existing operational model — we don't infer git policy.
|
||||
|
||||
For untrusted readers (remote MCP, subagent), the v0.32.2 release ships
|
||||
a 3-layer strip:
|
||||
|
||||
1. **Layer A (chunker):** `src/core/chunkers/recursive.ts` calls
|
||||
`stripFactsFence({keepVisibility: ['world']})` + `stripTakesFence`
|
||||
before chunking. Private fact text never reaches
|
||||
`content_chunks.chunk_text`, embeddings, or search results.
|
||||
2. **Layer B (get_page):** when `ctx.remote === true`, the response
|
||||
body has both fences stripped (private rows from facts; entire
|
||||
takes fence). Local CLI (`ctx.remote === false`) sees the full
|
||||
fence.
|
||||
3. **Layer C (git tracking):** the user decides whether to commit the
|
||||
entity page. `gbrain.yml` `db_only` paths are gitignored
|
||||
automatically; per-page choices via the user's normal git workflow.
|
||||
|
||||
For universally-private entities (a friend's name, an investor's
|
||||
internal notes), mark the entity page's directory as `db_only` in
|
||||
`gbrain.yml`. The file stays on disk but never lands in git.
|
||||
|
||||
## The forget contract
|
||||
|
||||
`gbrain forget <id>` and the MCP `forget_fact` op rewrite the fence
|
||||
row with strikethrough + `valid_until = today` + `context: "forgotten:
|
||||
<reason>"`. The DB's `expired_at = valid_until + now()` derivation
|
||||
reconstructs the forget state on every rebuild because the fence is
|
||||
canonical.
|
||||
|
||||
Strikethrough has two semantics distinguished by context:
|
||||
|
||||
- `~~claim~~` + `context: "superseded by #N"` → row was replaced by
|
||||
a newer row in the same fence
|
||||
- `~~claim~~` + `context: "forgotten: <reason>"` → row was retracted
|
||||
via the forget op
|
||||
|
||||
Both encodings keep the row in the markdown for audit history. To
|
||||
permanently delete a fact, edit the fence directly in markdown and
|
||||
remove the row. The next `extract_facts` cycle wipes the DB row.
|
||||
|
||||
## Disaster recovery
|
||||
|
||||
The promise the rule makes:
|
||||
|
||||
```bash
|
||||
# Snapshot what's there
|
||||
gbrain stats > /tmp/before.txt
|
||||
|
||||
# Wipe and rebuild
|
||||
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
|
||||
# (pages + content_chunks survive
|
||||
# the CASCADE-safe design)
|
||||
# OR manually for v0.32.2:
|
||||
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
|
||||
gbrain sync
|
||||
gbrain extract all
|
||||
|
||||
# Counts match
|
||||
gbrain stats > /tmp/after.txt
|
||||
diff /tmp/before.txt /tmp/after.txt
|
||||
```
|
||||
|
||||
The invariant E2E test at `test/e2e/system-of-record-invariant.test.ts`
|
||||
exercises this exact flow on every CI run.
|
||||
|
||||
## Rule for new code
|
||||
|
||||
When you add a new user-knowledge category:
|
||||
|
||||
1. **Define the markdown shape.** Fence (`<!--- gbrain:NAME:begin
|
||||
--> ... :end -->` table) or frontmatter field.
|
||||
2. **Build a parser** that produces structured data from markdown.
|
||||
See `src/core/fence-shared.ts` for the shared primitives.
|
||||
3. **Build a writer** that round-trips: parse + edit + render produces
|
||||
byte-identical markdown for identical input.
|
||||
4. **Add the engine method** that takes parsed data and stamps a
|
||||
derived table. The method gets an entry in the CI gate's
|
||||
banned-direct-call list.
|
||||
5. **Add a reconciler:** a cycle phase that walks pages, parses the
|
||||
fence, and rebuilds the derived table from scratch. The reconciler
|
||||
is the only legitimate call site for the engine method;
|
||||
`// gbrain-allow-direct-insert: <reason>` annotates it explicitly.
|
||||
6. **Add a round-trip test** in `test/e2e/system-of-record-invariant.test.ts`
|
||||
that proves DELETE + reconcile rebuilds the table byte-identically.
|
||||
|
||||
The CI gate at `scripts/check-system-of-record.sh` fails any PR that
|
||||
adds a new direct call to a derived-table writer outside the
|
||||
reconciler / migration layer without the explicit allow-list comment.
|
||||
|
||||
## Related
|
||||
|
||||
- `~/.claude/plans/system-instruction-you-are-working-expressive-pony.md`
|
||||
— the v0.32.2 design plan (decisions D1-D22 + Q1-Q8, Codex round 1
|
||||
and round 2 finds)
|
||||
- `skills/migrations/v0.32.2.md` — the agent-facing migration guide
|
||||
- `CHANGELOG.md` v0.32.2 entry — the release manifesto
|
||||
- `scripts/check-system-of-record.sh` — the CI gate that enforces
|
||||
the rule
|
||||
@@ -1,367 +0,0 @@
|
||||
# GBrain Deployment Topologies
|
||||
|
||||
GBrain supports three deployment shapes. They compose: a single user can mix
|
||||
all three on the same machine without conflict, because every shape resolves
|
||||
to "which `~/.gbrain/config.json` is active right now?" and `GBRAIN_HOME`
|
||||
controls that selection.
|
||||
|
||||
This page covers the three topologies, when each fits, and concrete setup
|
||||
recipes. Pair this doc with `docs/architecture/brains-and-sources.md` (which
|
||||
covers the in-brain organization axes) — that doc is about WHICH database;
|
||||
this doc is about WHERE that database lives.
|
||||
|
||||
## Quick decision tree
|
||||
|
||||
```
|
||||
"I'm setting up gbrain..."
|
||||
│
|
||||
▼
|
||||
Just for me, on one machine? ─── yes ───▶ Topology 1 (single brain)
|
||||
│
|
||||
no
|
||||
│
|
||||
▼
|
||||
Will a remote machine host the brain
|
||||
while my agent runs locally? ──── yes ───▶ Topology 2 (cross-machine thin client)
|
||||
│
|
||||
no
|
||||
│
|
||||
▼
|
||||
Multiple Conductor worktrees that
|
||||
shouldn't share a code index? ─── yes ───▶ Topology 3 (split-engine)
|
||||
```
|
||||
|
||||
Topologies 2 and 3 stack: a thin-client install can also host per-worktree
|
||||
code engines, and a per-worktree code engine can also point its artifact
|
||||
brain at a remote server.
|
||||
|
||||
## Topology 1 — Single brain (today's default)
|
||||
|
||||
```
|
||||
┌────────────────┐
|
||||
│ one machine │
|
||||
│ ┌──────────┐ │
|
||||
│ │ gbrain │──┼──→ ~/.gbrain/ → PGLite or Supabase
|
||||
│ │ CLI │ │
|
||||
│ └──────────┘ │
|
||||
└────────────────┘
|
||||
```
|
||||
|
||||
What you get: one local DB (PGLite for small brains, Supabase for ~1000+
|
||||
files). All commands work directly against it. `gbrain serve` exposes it
|
||||
to a single agent over MCP.
|
||||
|
||||
When it fits: solo use, single machine, one agent, no Conductor parallelism.
|
||||
This is the default; `gbrain init` (no flags) gives you this.
|
||||
|
||||
Setup:
|
||||
|
||||
```
|
||||
gbrain init # interactive — defaults to PGLite
|
||||
gbrain init --pglite # explicit local
|
||||
gbrain init --supabase # remote Supabase (recommended for 1000+ files)
|
||||
```
|
||||
|
||||
Nothing else here is special. The other two topologies are variations on
|
||||
"who owns the DB" and "how does the agent talk to it."
|
||||
|
||||
## Topology 2 — Cross-machine thin client
|
||||
|
||||
```
|
||||
┌────────────┐ ┌──────────────────┐
|
||||
│ neuromancer│ │ brain-host │
|
||||
│ ┌────────┐ │ HTTP MCP / OAuth │ ┌────────────┐ │
|
||||
│ │ Hermes │─┼───────────────────→│ │ gbrain │──┼──→ Supabase
|
||||
│ │ agent │ │ │ │ serve --http│ │
|
||||
│ └────────┘ │ │ └────────────┘ │
|
||||
│ │ │ (with autopilot)│
|
||||
│ no local │ │ │
|
||||
│ gbrain DB │ │ │
|
||||
└────────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
What you get: the agent on one machine ("neuromancer") consumes a brain
|
||||
hosted on another machine ("brain-host") over HTTP MCP with OAuth. The
|
||||
agent's machine has NO local engine. All queries, searches, embeddings,
|
||||
and indexing happen on the host.
|
||||
|
||||
When it fits:
|
||||
|
||||
- Heavy brain (Supabase + autopilot) lives on a beefy machine; agents
|
||||
elsewhere just consume it.
|
||||
- You want one source of truth across many machines.
|
||||
- Spinning up a parallel local install would create source-ID contention or
|
||||
duplicate work.
|
||||
|
||||
The thin client's `~/.gbrain/config.json` carries a `remote_mcp` field
|
||||
instead of a local DB connection:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"engine": "postgres", // ignored — never used
|
||||
"remote_mcp": {
|
||||
"issuer_url": "https://brain-host.local:3001",
|
||||
"mcp_url": "https://brain-host.local:3001/mcp",
|
||||
"oauth_client_id": "neuromancer-...",
|
||||
"oauth_client_secret": "..." // or set GBRAIN_REMOTE_CLIENT_SECRET
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The CLI dispatch guard refuses any DB-bound command (`sync`, `embed`,
|
||||
`extract`, `migrate`, `apply-migrations`, `repair-jsonb`, `orphans`,
|
||||
`integrity`, `serve`) on a thin-client install with a clear error pointing
|
||||
at the remote host. `gbrain doctor` runs a dedicated thin-client check set
|
||||
(OAuth discovery, token round-trip, MCP smoke).
|
||||
|
||||
### Setup
|
||||
|
||||
**Step 1 — On the host (brain-host):**
|
||||
|
||||
```bash
|
||||
gbrain init --supabase # or --pglite, doesn't matter
|
||||
gbrain serve --http --port 3001 --bind 0.0.0.0 # v0.34: bind explicitly for remote access
|
||||
# (defaults to 127.0.0.1 since v0.34)
|
||||
gbrain auth register-client neuromancer \
|
||||
--grant-types client_credentials \
|
||||
--scopes read,write,admin # admin needed for ping/doctor
|
||||
|
||||
# v0.34: source-scoped client (write to one source, federate reads across
|
||||
# multiple sources). Omit both flags for a v0.33-compatible super-client.
|
||||
gbrain auth register-client neuromancer-dept \
|
||||
--grant-types client_credentials \
|
||||
--scopes read,write \
|
||||
--source dept-x \
|
||||
--federated-read dept-x,shared,parent-canon
|
||||
```
|
||||
|
||||
The `register-client` command prints a `client_id` and `client_secret`.
|
||||
Note both. **Scope must include `admin`** — `submit_job` (used by
|
||||
`gbrain remote ping`) and `run_doctor` (used by `gbrain remote doctor`)
|
||||
both require it.
|
||||
|
||||
**Step 2 — On the thin client (neuromancer):**
|
||||
|
||||
```bash
|
||||
gbrain init --mcp-only \
|
||||
--issuer-url https://brain-host.local:3001 \
|
||||
--mcp-url https://brain-host.local:3001/mcp \
|
||||
--oauth-client-id <id> \
|
||||
--oauth-client-secret <secret>
|
||||
```
|
||||
|
||||
Pre-flight smoke runs three probes (OAuth discovery, token round-trip,
|
||||
MCP initialize). If any fails, init exits with an actionable error. On
|
||||
success, `~/.gbrain/config.json` gets `remote_mcp` set and NO local DB
|
||||
is created.
|
||||
|
||||
**Step 3 — Configure your agent's MCP client.**
|
||||
|
||||
For Claude Desktop / Hermes / openclaw, add a single MCP server entry
|
||||
pointing at the host's `mcp_url` with the bearer token from `register-client`.
|
||||
Example for Claude Desktop's `~/.config/claude/claude_desktop_config.json`:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"mcpServers": {
|
||||
"gbrain": {
|
||||
"type": "url",
|
||||
"url": "https://brain-host.local:3001/mcp",
|
||||
"headers": { "Authorization": "Bearer <client_secret>" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4 — Verify.**
|
||||
|
||||
```bash
|
||||
gbrain doctor # runs thin-client checks (no local DB needed)
|
||||
gbrain remote ping # triggers an autopilot cycle on the host (Tier B)
|
||||
gbrain remote doctor # asks the host to run its own doctor (Tier B)
|
||||
```
|
||||
|
||||
`gbrain sync` and friends will refuse with a clear thin-client error
|
||||
naming the `mcp_url`. That's the correct behavior — those commands need
|
||||
a local engine that doesn't exist here.
|
||||
|
||||
### Re-run guard
|
||||
|
||||
Running `gbrain init` (no flags) on a machine that already has thin-client
|
||||
config set refuses without `--force`. This catches the scripted-setup-loop
|
||||
friction where an orchestrator keeps trying to create a local DB. Use
|
||||
`gbrain init --mcp-only --force` to refresh thin-client config.
|
||||
|
||||
### Storing the OAuth secret
|
||||
|
||||
Three storage paths in priority order:
|
||||
|
||||
1. **`GBRAIN_REMOTE_CLIENT_SECRET` env var** (preferred for headless agents).
|
||||
When set, overrides whatever's in the config file. The init flow doesn't
|
||||
persist a config-file copy when the env var was the source.
|
||||
2. **`~/.gbrain/config.json` with 0600 perms** (default for interactive
|
||||
setup; mirrors how Supabase keys are stored today).
|
||||
3. macOS Keychain integration is on the roadmap; not in v1.
|
||||
|
||||
## Topology 3 — Split-engine, per-worktree code + remote artifacts
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ one machine │
|
||||
│ │
|
||||
│ ┌─ worktree A ──────────────┐ │
|
||||
│ │ GBRAIN_HOME=A/.conductor │ │
|
||||
│ │ gbrain serve --port 3001 │── PGLite (code A) │
|
||||
│ └───────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─ worktree B ──────────────┐ │
|
||||
│ │ GBRAIN_HOME=B/.conductor │ │
|
||||
│ │ gbrain serve --port 3002 │── PGLite (code B) │
|
||||
│ └───────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─ default ~/.gbrain ───────┐ HTTP MCP / OAuth │
|
||||
│ │ gbrain serve --port 3000 │──────────────────────→ remote artifacts
|
||||
│ └───────────────────────────┘ (Supabase / brain-host)
|
||||
│ │
|
||||
│ Agent's MCP config (Hermes / Claude Desktop): │
|
||||
│ mcp__gbrain_code__* → http://localhost:3001 │
|
||||
│ mcp__gbrain_artifacts__* → http://brain-host/mcp │
|
||||
└──────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
What you get: each Conductor worktree has its own per-worktree code index
|
||||
(local PGLite, disposable when the worktree dies). Artifacts (plans,
|
||||
learnings, transcripts) still live in a shared brain that all worktrees
|
||||
can see and write to.
|
||||
|
||||
When it fits:
|
||||
|
||||
- Multiple Conductor worktrees on one machine, all touching the same code
|
||||
repo.
|
||||
- You don't want each worktree's code-import to clobber the others'
|
||||
`last_commit`, source IDs, or symbol tables.
|
||||
- You DO want artifacts (plans, learnings, retros, transcripts) to be
|
||||
visible across worktrees.
|
||||
|
||||
### How it works
|
||||
|
||||
`GBRAIN_HOME` selects which `~/.gbrain` directory is active. Set per worktree:
|
||||
|
||||
```bash
|
||||
export GBRAIN_HOME=/path/to/worktree-A/.conductor/gbrain
|
||||
gbrain init --pglite
|
||||
gbrain serve --http --port 3001
|
||||
```
|
||||
|
||||
Each worktree's `gbrain serve` instance binds its own port and indexes its
|
||||
own DB. Multiple `gbrain serve` processes coexist fine — they're separate
|
||||
OS processes with separate config and separate connection pools.
|
||||
|
||||
The artifact brain runs as a separate `gbrain serve` instance with the
|
||||
default `~/.gbrain` (no GBRAIN_HOME override) — or remote, in which case
|
||||
it's a Topology 2 setup.
|
||||
|
||||
The agent's MCP client config lists multiple servers, each with a unique
|
||||
alias. Tool names are namespaced as `mcp__<alias>__<tool>`, so the agent
|
||||
calls `mcp__gbrain_code__search` for code lookups and `mcp__gbrain_artifacts__search`
|
||||
for artifact lookups.
|
||||
|
||||
### CRITICAL: alias-level routing is manual
|
||||
|
||||
Topology 3 has no smart per-tool routing inside gbrain. The agent picks
|
||||
which brain to query when it picks the alias. **A wrong alias writes (or
|
||||
queries) the wrong brain silently.** This is intentional (explicit beats
|
||||
magic) but real:
|
||||
|
||||
- If the agent calls `mcp__gbrain_artifacts__put_page` with code-shaped
|
||||
content, that page lands in the artifact brain forever.
|
||||
- If the agent calls `mcp__gbrain_code__search` for a question that
|
||||
actually wants artifact context, the search comes back empty.
|
||||
|
||||
Mitigations:
|
||||
|
||||
- Name aliases clearly. `gbrain_code` vs `gbrain_artifacts` is unambiguous;
|
||||
`gbrain` vs `gbrain_local` is not.
|
||||
- Document in your agent's system prompt or rules which alias goes where.
|
||||
Be explicit about "code questions → `gbrain_code`; everything else →
|
||||
`gbrain_artifacts`."
|
||||
- Pair Topology 3 with `gstack`'s per-worktree wiring (which sets the
|
||||
alias names + agent rules consistently across worktrees).
|
||||
|
||||
### Setup (manual; gstack automates this side)
|
||||
|
||||
The gbrain side requires zero new code — `GBRAIN_HOME` and `--port` already
|
||||
exist. Setup looks like:
|
||||
|
||||
```bash
|
||||
# Start the artifact brain (default ~/.gbrain) on port 3000
|
||||
gbrain serve --http --port 3000 &
|
||||
|
||||
# Start a per-worktree code brain on port 3001
|
||||
export GBRAIN_HOME=/path/to/worktree-A/.conductor/gbrain
|
||||
gbrain init --pglite
|
||||
gbrain serve --http --port 3001 &
|
||||
unset GBRAIN_HOME
|
||||
```
|
||||
|
||||
Then configure the agent's MCP config with two entries (different aliases,
|
||||
different ports). For Claude Desktop:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"mcpServers": {
|
||||
"gbrain_artifacts": {
|
||||
"type": "url",
|
||||
"url": "http://localhost:3000/mcp",
|
||||
"headers": { "Authorization": "Bearer <token-A>" }
|
||||
},
|
||||
"gbrain_code": {
|
||||
"type": "url",
|
||||
"url": "http://localhost:3001/mcp",
|
||||
"headers": { "Authorization": "Bearer <token-B>" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The gstack-side wiring (per-worktree home setup, port allocation, automatic
|
||||
MCP config generation, gitignore for the per-worktree DB) is in the gstack
|
||||
repo's setup-gbrain skill — it composes these primitives, gbrain doesn't
|
||||
have to know about Conductor.
|
||||
|
||||
## Combining topologies
|
||||
|
||||
The three shapes compose. A single machine can run:
|
||||
|
||||
- A thin-client default config pointing at a remote artifact brain
|
||||
(Topology 2).
|
||||
- Plus per-worktree code brains under their own `GBRAIN_HOME` (Topology 3).
|
||||
- Each worktree's `gbrain serve` instance is local; the agent's MCP config
|
||||
lists them alongside the remote artifact brain.
|
||||
|
||||
`GBRAIN_HOME` controls which config file is active for any one CLI
|
||||
invocation. `gbrain serve --port` controls which port a server listens on.
|
||||
The agent's MCP client picks the alias and thus the destination per tool
|
||||
call. There's no global gbrain orchestrator that knows about all of them
|
||||
simultaneously — that's by design.
|
||||
|
||||
## When NOT to use these topologies
|
||||
|
||||
- **Don't use Topology 2 if your agent only ever runs on the same machine
|
||||
as the brain.** A local `gbrain` install + `gbrain serve` (stdio) is
|
||||
simpler and faster.
|
||||
- **Don't use Topology 3 if you only have one Conductor worktree at a
|
||||
time.** Per-worktree engines exist to prevent contention; one-at-a-time
|
||||
use has no contention.
|
||||
- **Don't use a `remote_mcp` thin client AND a local engine on the same
|
||||
machine in the same `GBRAIN_HOME`.** The dispatch guard refuses DB-bound
|
||||
commands when `remote_mcp` is set. If you genuinely want both modes on
|
||||
one machine, use `GBRAIN_HOME` to separate them (one home for the thin
|
||||
client, another for the local engine).
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/architecture/brains-and-sources.md` — in-brain organization (brains
|
||||
vs sources axes).
|
||||
- `docs/mcp/CLAUDE_DESKTOP.md` and siblings — per-client MCP setup.
|
||||
- `gbrain init --help` and `gbrain auth --help` for command-level details.
|
||||
@@ -1,166 +0,0 @@
|
||||
# gbrain eval suspected-contradictions (v0.32.6)
|
||||
|
||||
The contradiction probe samples retrieval results, asks an LLM judge whether
|
||||
any pair contradicts on a factual claim relevant to the user's query, and
|
||||
aggregates into a calibrated report. The output is data — the operator
|
||||
decides what to act on. This doc covers the architecture, severity rubric,
|
||||
how to interpret the headline number, and when to act.
|
||||
|
||||
## Why this exists
|
||||
|
||||
gbrain handles contradictions for *curated* pages via compiled-truth-plus-
|
||||
timeline and source-boost: when `companies/acme.md` says MRR is $2M and a
|
||||
chat transcript from 2024 says MRR was $50K, the curated page outranks the
|
||||
chat. `takes.active` filtering hides explicitly-superseded takes. Recency
|
||||
decay biases ranking toward fresher content per source-tier.
|
||||
|
||||
What none of those mechanisms measure: how often do unmarked semantic
|
||||
contradictions actually surface in retrieval? Without a probe, every
|
||||
"should we build the bigger swing (chunk-level `revises` field + ranking
|
||||
change)" decision is vibes. The probe produces evidence.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────┐
|
||||
│ gbrain eval suspected-contradictions │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
┌──────────────────▼───────────────────┐
|
||||
│ For each query: hybridSearch top-K │
|
||||
│ → cross_slug_chunks + intra_page │
|
||||
│ chunk-vs-take pairs │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
┌──────────────────▼───────────────────┐
|
||||
│ Date pre-filter: skip pairs whose │
|
||||
│ dates are >30d apart (Codex fix: │
|
||||
│ same-paragraph-dual-date overrides) │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
┌──────────────────▼───────────────────┐
|
||||
│ Persistent cache lookup │
|
||||
│ (chunk_a_hash, chunk_b_hash, model, │
|
||||
│ prompt_version, truncation_policy) │
|
||||
└────────┬─────────┬────────────────────┘
|
||||
hit│ │miss
|
||||
│ ▼
|
||||
│ ┌─────────────────────────┐
|
||||
│ │ LLM judge call │
|
||||
│ │ → JudgeVerdict │
|
||||
│ │ confidence floor ≥ 0.7 │
|
||||
│ └─────────┬───────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────┐
|
||||
│ Aggregate per-query + global stats │
|
||||
│ Wilson 95% CI on headline % │
|
||||
│ source-tier breakdown │
|
||||
│ hot pages + resolution proposals │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
▼
|
||||
ProbeReport JSON
|
||||
│
|
||||
┌──────────────────┼──────────────────────┬───────────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
doctor (M1) MCP (M3) synthesize (M2) trend (M5)
|
||||
surfaces find_contradictions informational persistent
|
||||
findings op for agents block in prompt tracking
|
||||
```
|
||||
|
||||
## Severity rubric
|
||||
|
||||
The judge assigns severity per finding:
|
||||
|
||||
| Level | Rubric | Example |
|
||||
|---|---|---|
|
||||
| `low` | naming/format differences | "Alice Smith" vs "A. Smith" |
|
||||
| `medium` | factual values that may be stale | revenue figure, headcount, valuation |
|
||||
| `high` | identity / structural claims | founder/CEO/CFO role, company status |
|
||||
|
||||
Doctor sorts findings by severity DESC. The MCP op accepts a severity filter
|
||||
so agents can fetch just the high-priority items.
|
||||
|
||||
## How to interpret the headline number
|
||||
|
||||
The probe outputs `queries_with_contradiction / queries_evaluated` with a
|
||||
Wilson 95% confidence interval:
|
||||
|
||||
```
|
||||
Queries with >=1 contradiction: 12 / 50 (24%) Wilson CI 95%: 14–37%
|
||||
```
|
||||
|
||||
What this says: with 95% confidence, the true rate is between 14% and 37%.
|
||||
The 24% point estimate is the most-likely-value but bounded by sampling
|
||||
noise. **`small_sample_note` fires when n < 30** — at that scale the CI is
|
||||
too wide to act on.
|
||||
|
||||
Decision criteria for the bigger swing (chunk-level `revises` field):
|
||||
|
||||
| Wilson CI lower bound | What it says | Action |
|
||||
|---|---|---|
|
||||
| < 5% | Source-boost + recency-decay + curated pages handle the load | Stop here; this is the right scope |
|
||||
| 5–15% | Real but bounded | Operator decides whether the cost justifies the swing |
|
||||
| > 15% | Real and substantial | Plan the bigger swing in v0.34+ |
|
||||
|
||||
## When to act on findings
|
||||
|
||||
Each finding ships with a `resolution_command` field — paste-ready:
|
||||
|
||||
- `gbrain takes supersede <slug> --row N` — newer take should replace
|
||||
the older chunk text on the same page (intra_page kind).
|
||||
- `gbrain dream --phase synthesize --slug <slug>` — compiled_truth for
|
||||
the curated entity needs an update (cross_slug curated-vs-bulk).
|
||||
- `gbrain takes mark-debate <slug> --row N` — intentional disagreement
|
||||
(e.g., two opinions you want to keep both of).
|
||||
- `# manual review: <a> vs <b>` — judge wasn't sure; operator decides.
|
||||
|
||||
Run `gbrain eval suspected-contradictions review --severity high` to
|
||||
inspect findings without re-running the probe.
|
||||
|
||||
## Cost model
|
||||
|
||||
Default judge is `claude-haiku-4-5` at ~$1/Mtok in, $5/Mtok out. With
|
||||
the v0.32.6 truncation at 1500 chars per pair, ~500 input + 80 output
|
||||
tokens per judge call. Budget cap defaults to $5 in TTY / $1 non-TTY.
|
||||
|
||||
- ~$0.0006 per judge call
|
||||
- ~$0.005 per query (after date pre-filter + cache hits)
|
||||
- ~$0.50 per 100 queries
|
||||
|
||||
The persistent cache means nightly runs against the same query set
|
||||
pay near-zero on re-runs (until you bump PROMPT_VERSION).
|
||||
|
||||
## Trust posture
|
||||
|
||||
- Probe never mutates the brain. Runs only read pages/takes/chunks.
|
||||
Writes go only to `eval_contradictions_runs` and `eval_contradictions_cache`.
|
||||
- MCP `find_contradictions` is read-scope. NOT in the subagent allowlist —
|
||||
user-initiated only, not autonomous-action surface.
|
||||
- Build-fixture script is local-only. The redactor + `isCleanForCommit`
|
||||
gate makes accidental private-data commits hard, but the operator MUST
|
||||
inspect every redaction before commit.
|
||||
|
||||
## See also
|
||||
|
||||
- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`
|
||||
- CHANGELOG: `## [0.32.6]` entry covers the whole release.
|
||||
- Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence
|
||||
+ trend-tracking workflow.
|
||||
- **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a
|
||||
six-member verdict enum (`no_contradiction | contradiction |
|
||||
temporal_supersession | temporal_regression | temporal_evolution |
|
||||
negation_artifact`) and threaded `pages.effective_date` into the judge
|
||||
prompt so the probe stops crying wolf on legitimate change-over-time.
|
||||
v0.35.7 lands the trajectory substrate the probe pointed at:
|
||||
`gbrain eval trajectory <entity>` shows the chronological typed-claim
|
||||
history with regressions flagged inline; `gbrain founder scorecard
|
||||
<entity>` rolls up four signals (accuracy, consistency, growth
|
||||
direction, red flags) into a stable JSON contract. MCP op
|
||||
`find_trajectory` (read scope, visibility-filtered for remote callers)
|
||||
exposes the same data to agents. The probe's `temporal_supersession`
|
||||
verdict and the consolidate phase's `valid_until` writeback both
|
||||
preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant
|
||||
— the probe still emits paste-ready commands, only `consolidate`
|
||||
writes `valid_until` (R1+R8 grep guard pins this).
|
||||
@@ -1,580 +0,0 @@
|
||||
# Embedder Shootout — May 2026 Eval Plan
|
||||
|
||||
**Status:** approved, ready to execute
|
||||
**Owner:** Garry
|
||||
**Plan source:** `~/.claude/plans/system-instruction-you-are-working-linear-origami.md` (review log)
|
||||
**Target wallclock:** ~2 weeks
|
||||
**Target API spend:** ~$525 (hard cap $700)
|
||||
|
||||
## What this is
|
||||
|
||||
A head-to-head A/B/C comparison of three embedding providers under v0.35.0.0's new
|
||||
multi-vendor gateway routing:
|
||||
|
||||
- **OpenAI** `text-embedding-3-large` @ 1536 dims
|
||||
- **Voyage** `voyage-4-large` @ 2048 dims
|
||||
- **ZeroEntropy** `zembed-1` @ 2560 dims (also 1280 in a Matryoshka ablation)
|
||||
|
||||
Each tested with and without the `zerank-2` reranker. Two corpora: public LongMemEval
|
||||
(500q) and BrainBench in-house (145 relational queries + 50 newly-curated Cat 13
|
||||
embedder-sensitive queries).
|
||||
|
||||
The goal: produce a publishable comparison report that answers "which embedder wins,
|
||||
and does zerank-2 carry the win for ZeroEntropy" with bootstrap p-values, suitable
|
||||
for a v0.35.2.0 release-note headline.
|
||||
|
||||
## Why this design
|
||||
|
||||
Locked decisions from the planning review (see plan file + `GSTACK REVIEW REPORT` at
|
||||
the bottom of the linked plan):
|
||||
|
||||
- **Synthetic-only** — LongMemEval (public) + BrainBench (in-house). No `~/.gbrain` data.
|
||||
- **Answer-gen mode** — `gbrain eval longmemeval` runs the default answer-gen path
|
||||
(Anthropic Sonnet), then feeds the resulting hypothesis JSONL to LongMemEval's
|
||||
published `evaluate_qa.py` (OpenAI gpt-4o judge) for real correctness numbers.
|
||||
`--retrieval-only` is NOT used (would produce an attackable headline; the judge
|
||||
expects answer text, not retrieval text).
|
||||
- **`tokenmax` search mode** pinned across all cells (expansion + reranker slot active).
|
||||
- **Serial execution** in one workspace. Clean rate-limit profile; first-contact run on
|
||||
ZE wants debuggable signal.
|
||||
- **7-cell matrix** (no matched-dim cross-vendor row — no shared dim exists across
|
||||
all three vendors; honest framing is "each vendor at marketed sweet spot").
|
||||
|
||||
## Architectural facts that constrain the plan
|
||||
|
||||
- `content_chunks.embedding vector(N)` dim is fixed per brain. Per-question PGLite in
|
||||
LongMemEval makes this free; BrainBench needs separate brain per cell.
|
||||
- pgvector HNSW caps at **2000 dims** (`PGVECTOR_HNSW_VECTOR_MAX_DIMS` in
|
||||
`src/core/vector-index.ts:19`). Voyage 2048 and ZE 2560 fall back to exact vector
|
||||
scan. Helps quality (no HNSW approximation) but adds latency. Footnoted in writeup.
|
||||
- Reranker disable key is **`search.reranker.enabled false`**, NOT `reranker_model none`.
|
||||
`tokenmax` mode defaults reranker=true.
|
||||
- `gbrain/ai/gateway` is NOT exported in v0.35.0.0. PR α exposes it.
|
||||
|
||||
## Matrix
|
||||
|
||||
| Cell | Embedder | Dim | HNSW | Reranker | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| A0 | `openai:text-embedding-3-large` | 1536 | yes | none | OpenAI baseline |
|
||||
| A1 | `openai:text-embedding-3-large` | 1536 | yes | `zerank-2` | mixed-vendor |
|
||||
| B0 | `voyage:voyage-4-large` | 2048 | no (exact) | none | Voyage solo |
|
||||
| B1 | `voyage:voyage-4-large` | 2048 | no (exact) | `zerank-2` | mixed-vendor |
|
||||
| C0 | `zeroentropyai:zembed-1` | 2560 | no (exact) | none | ZE embedder solo |
|
||||
| C1 | `zeroentropyai:zembed-1` | 2560 | no (exact) | `zerank-2` | **ZE full stack** |
|
||||
| C2 | `zeroentropyai:zembed-1` | 1280 | yes | `zerank-2` | ZE-Matryoshka ablation |
|
||||
|
||||
## PR structure — as few as possible
|
||||
|
||||
**PR α — gbrain repo: v0.35.1.0 infra.** All gbrain changes bundled. Lands first.
|
||||
Bisect-friendly commits inside, ship at the very end.
|
||||
|
||||
**PR β — gbrain-evals repo: adapter + smoke + curation + eval receipts + writeup.** The
|
||||
big one. Includes the full eval-run output committed alongside the code that produced
|
||||
it, plus the comparison writeup. Lands when everything is done.
|
||||
|
||||
**PR γ (optional) — gbrain repo: v0.35.2.0 release** that cross-links the gbrain-evals
|
||||
benchmark in CHANGELOG. Small commit; no code changes.
|
||||
|
||||
Total: 2 substantive PRs + 1 optional release commit. **No mid-stream ships.**
|
||||
|
||||
## Conductor sessions
|
||||
|
||||
Each section below is a self-contained brief. Copy-paste into a fresh Conductor session
|
||||
to hand off. Each session ends with a clean deliverable.
|
||||
|
||||
---
|
||||
|
||||
## Session 1 — PR α: gbrain infra (v0.35.1.0)
|
||||
|
||||
**Repo:** `/Users/garrytan/conductor/workspaces/gbrain/<NEW-WORKSPACE>` (fresh from `master`)
|
||||
**Branch:** `garrytan/v0.35.1.0-infra`
|
||||
**Wallclock:** ~2h
|
||||
**API spend:** $0
|
||||
|
||||
### What this session ships
|
||||
Three changes in one PR, bundled so the embedder shootout in gbrain-evals (PR β) has a
|
||||
clean prereq baseline:
|
||||
|
||||
1. Add `voyage:voyage-4-large` ($0.18/M) and `zeroentropyai:zembed-1` ($0.05/M) to the
|
||||
embedding pricing table. Patch the `gbrain models doctor` cost estimator + test.
|
||||
2. Expose `gbrain/ai/gateway` in `package.json` exports map so the gbrain-evals
|
||||
adapters can call `configureGateway({embedding_model, embedding_dimensions, reranker_model})`
|
||||
from outside the gbrain process.
|
||||
3. Add `--resume-from <jsonl>` to `gbrain eval longmemeval` so a mid-run abort
|
||||
(rate-limit, cost-cap, OS interrupt) doesn't lose the cells we already paid for.
|
||||
|
||||
Ships at the end as v0.35.1.0.
|
||||
|
||||
### Prereqs (verify before starting)
|
||||
- On gbrain master at v0.35.0.0 baseline. `cat VERSION` shows `0.35.0.0`.
|
||||
- `bun test` and `bun run verify` both pass on master.
|
||||
|
||||
### Commits (bisect-friendly, one feature per commit)
|
||||
|
||||
```
|
||||
1. feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING
|
||||
- src/core/embedding-pricing.ts: add both entries
|
||||
- test/embedding-pricing.test.ts: pin both with $0.18 and $0.05
|
||||
- Verify: bun test test/embedding-pricing.test.ts
|
||||
|
||||
2. feat(exports): expose gbrain/ai/gateway with canary test
|
||||
- package.json: add "./ai/gateway" to exports map
|
||||
- test/public-exports.test.ts: add canary for configureGateway + embed
|
||||
- scripts/check-exports-count.sh: 17 -> 18
|
||||
- Verify: bun run verify
|
||||
|
||||
3. feat(eval): add --resume-from <jsonl> to longmemeval
|
||||
- src/commands/eval-longmemeval.ts: parse flag, skip questions already in input JSONL
|
||||
- test/eval-longmemeval.test.ts: simulated mid-run abort + resume regression
|
||||
- Verify: bun test test/eval-longmemeval.test.ts
|
||||
|
||||
4. chore: v0.35.1.0
|
||||
- VERSION: 0.35.1.0
|
||||
- package.json: 0.35.1.0
|
||||
- CHANGELOG.md: new entry
|
||||
- bun install (refresh lockfile)
|
||||
```
|
||||
|
||||
### Verify before /ship
|
||||
```bash
|
||||
bun run typecheck
|
||||
bun run verify
|
||||
bun test test/embedding-pricing.test.ts test/public-exports.test.ts test/eval-longmemeval.test.ts
|
||||
```
|
||||
|
||||
### Ship
|
||||
```bash
|
||||
/ship
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- `master` of gbrain at v0.35.1.0
|
||||
- `gbrain/ai/gateway` reachable from external consumers (verified by canary test)
|
||||
- `git tag eval-run-v0.35.1.0-baseline` (annotated, names this exact commit)
|
||||
- `gbrain --version` prints `0.35.1.0`
|
||||
|
||||
### Hand-off to Session 2
|
||||
- gbrain-evals can now `bun update gbrain` to v0.35.1.0
|
||||
- The tag preserves the exact commit for any future reproducibility need
|
||||
|
||||
---
|
||||
|
||||
## Session 2 — PR β setup: gbrain-evals adapter + smoke + subset flag
|
||||
|
||||
**Repo:** `/Users/garrytan/git/gbrain-evals` (or a fresh Conductor workspace cloned from it)
|
||||
**Branch:** `garrytan/embedder-shootout`
|
||||
**Wallclock:** ~3-4h
|
||||
**API spend:** ~$0.10 (smoke verification calls only)
|
||||
|
||||
### What this session ships into PR β (does NOT merge yet)
|
||||
Wire the harness to drive 3 embedding providers via the newly-exposed gbrain gateway:
|
||||
|
||||
1. New typed `EvalAdapterConfig {embedder, dim, reranker?}` passed into each adapter.
|
||||
2. Rewrite `vector.ts` + `hybrid-rrf.ts` to call `configureGateway()` from
|
||||
`gbrain/ai/gateway` instead of the hardcoded `gbrain/embedding` import.
|
||||
3. Critical: hybrid adapter must also route `search.reranker.enabled` (true/false) and
|
||||
`search.mode` (tokenmax) — codex flagged that the existing hybrid never sets these.
|
||||
4. New 3-phase smoke harness: wiring (5 queries × embed roundtrip + dim check) +
|
||||
long-haystack (1 query × 50K-token synthetic haystack) + rerank-payload (1 query
|
||||
× `topNIn=30`). Exit code is the gate.
|
||||
5. New `--include-subset <name>` flag on the BrainBench runner (Cat 13 wiring; subset
|
||||
itself comes in Session 3).
|
||||
|
||||
### Prereqs
|
||||
- Session 1 done. gbrain master at v0.35.1.0.
|
||||
- API keys present: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`,
|
||||
`ZEROENTROPY_API_KEY`. Smoke fails-loud on missing key.
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. chore(deps): bump gbrain pin to v0.35.1.0
|
||||
- package.json + bun.lock
|
||||
- Verify: bun install && bun run typecheck
|
||||
|
||||
2. feat(adapter): typed EvalAdapterConfig + gateway swap
|
||||
- NEW: eval/runner/eval-adapter-config.ts (the type)
|
||||
- eval/runner/adapters/vector.ts: constructor takes EvalAdapterConfig,
|
||||
calls configureGateway({embedding_model, embedding_dimensions})
|
||||
- Drop hardcoded gbrain/embedding import
|
||||
- Verify: existing vector adapter unit tests still pass
|
||||
|
||||
3. feat(adapter): hybrid-rrf wires reranker_enabled + search.mode
|
||||
- eval/runner/adapters/hybrid-rrf.ts: constructor takes EvalAdapterConfig,
|
||||
plumbs search.reranker.enabled + search.mode = tokenmax through
|
||||
- Verify: bun test eval/
|
||||
|
||||
4. feat(smoke): 3-phase smoke harness
|
||||
- NEW: eval/runner/smoke.ts (CLI entry: bun run eval:smoke -- --embedder X --dim Y [--reranker Z])
|
||||
- Phase 1: 5 queries × embed roundtrip, assert vector dim matches config
|
||||
- Phase 2: 1 query × synthetic 50K-token haystack, assert no token-limit error
|
||||
- Phase 3: 1 query × topNIn=30 documents, assert no 5MB payload cap hit
|
||||
- Non-zero exit on any failure
|
||||
- Verify: bun run eval:smoke -- --embedder openai:text-embedding-3-large --dim 1536
|
||||
|
||||
5. feat(runner): --include-subset flag for BrainBench
|
||||
- eval/runner/multi-adapter.ts: parse flag, filter queries by subset tag
|
||||
- Subset itself comes in next commit (Session 3)
|
||||
- Verify: bun run eval:run -- --include-subset cat13-embedder (errors politely because subset file doesn't exist yet)
|
||||
```
|
||||
|
||||
### Smoke verification (run manually before opening PR)
|
||||
```bash
|
||||
bun run eval:smoke -- --embedder openai:text-embedding-3-large --dim 1536
|
||||
bun run eval:smoke -- --embedder voyage:voyage-4-large --dim 2048
|
||||
bun run eval:smoke -- --embedder zeroentropyai:zembed-1 --dim 2560
|
||||
bun run eval:smoke -- --embedder zeroentropyai:zembed-1 --dim 2560 --reranker zeroentropyai:zerank-2
|
||||
```
|
||||
|
||||
All four MUST exit 0. Reports should print the observed vector dim, matching the
|
||||
configured dim.
|
||||
|
||||
### Open PR β
|
||||
```bash
|
||||
gh pr create --base main --title "feat: embedder shootout (adapter + smoke + Cat 13 + eval receipts)" --body "$(cat <<'EOF'
|
||||
## Summary
|
||||
v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support. This PR runs a head-to-head A/B/C comparison across OpenAI, Voyage, and ZeroEntropy under the new gateway routing.
|
||||
|
||||
This first commit batch lands the harness. Cat 13 curation, Phase 1+2 evals, and the
|
||||
writeup follow in subsequent commits to this same PR.
|
||||
|
||||
## Test plan
|
||||
- [x] Adapter unit tests pass
|
||||
- [x] Smoke harness exits 0 against all 3 providers
|
||||
- [ ] Cat 13 subset committed (Session 3)
|
||||
- [ ] LongMemEval x 7 cells run (Session 4)
|
||||
- [ ] BrainBench x 7 cells run (Session 5)
|
||||
- [ ] Writeup committed (Session 5)
|
||||
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- PR β open against gbrain-evals `main`, green CI
|
||||
- Smoke verified against all 3 providers (paste the smoke output in the PR body)
|
||||
- Branch ready for Session 3 (Cat 13 curation)
|
||||
|
||||
### Hand-off to Session 3
|
||||
- Branch `garrytan/embedder-shootout` exists on origin
|
||||
- The `--include-subset cat13-embedder` flag is wired but the subset file doesn't exist
|
||||
yet — that's Session 3
|
||||
|
||||
---
|
||||
|
||||
## Session 3 — PR β: Cat 13 conceptual-recall curation
|
||||
|
||||
**Repo:** `/Users/garrytan/git/gbrain-evals`, branch `garrytan/embedder-shootout` (same as Session 2)
|
||||
**Wallclock:** ~3-4h (heavily user-interactive; AI proposes, you review each)
|
||||
**API spend:** $0
|
||||
|
||||
### What this session ships into PR β
|
||||
Hand-curated 50 embedder-sensitive queries from BrainBench's Cat 13 (conceptual recall)
|
||||
corpus. These are the queries where a graph/keyword adapter would likely miss but a
|
||||
semantic adapter would find.
|
||||
|
||||
Codex flagged the existing 145-query relational corpus as graph/keyword-dominated and
|
||||
weak for embedder claims. Cat 13 is closer to the embedder-sensitive workload but
|
||||
needs hand-selection.
|
||||
|
||||
### Prereqs
|
||||
- Session 2 done. PR β open with adapter + smoke + subset flag.
|
||||
|
||||
### Workflow
|
||||
Interactive: Claude proposes queries in batches of 10, you accept/reject/edit each.
|
||||
|
||||
1. Claude reads the existing Cat 13 raw query pool:
|
||||
```bash
|
||||
ls eval/data/raw/ | grep -i cat13
|
||||
cat eval/data/raw/cat13-*.json | jq '.'
|
||||
```
|
||||
2. Claude proposes 10 candidate queries per batch, each tagged with the inclusion
|
||||
reasoning ("would a graph adapter miss this?")
|
||||
3. User accepts/rejects/edits inline. Target: 50 queries × ~5 batches.
|
||||
4. Claude commits to `eval/data/gold/brainbench-cat13-embedder-subset.json`:
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"subset": "cat13-embedder",
|
||||
"queries": [
|
||||
{
|
||||
"id": "cat13-emb-001",
|
||||
"query": "...",
|
||||
"relevant_chunk_ids": ["..."],
|
||||
"inclusion_reason": "paraphrase relationship; graph adapter wouldn't catch the synonym"
|
||||
}
|
||||
// ... 49 more
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Commit
|
||||
|
||||
```
|
||||
feat(eval): curate Cat 13 conceptual-recall subset (50 embedder-sensitive queries)
|
||||
- NEW: eval/data/gold/brainbench-cat13-embedder-subset.json
|
||||
- Each query tagged with inclusion_reason for future audit
|
||||
```
|
||||
|
||||
### Spot-check before commit
|
||||
- Pick 5 random queries, run them against a hypothetical graph adapter (e.g. grep on
|
||||
the relevant terms) and verify they would NOT surface the right chunk.
|
||||
- Run the same 5 against the existing hybrid adapter and verify they DO.
|
||||
|
||||
### Deliverable
|
||||
- `eval/data/gold/brainbench-cat13-embedder-subset.json` committed to PR β
|
||||
- Exactly 50 queries
|
||||
- Spot-check evidence in the commit message
|
||||
|
||||
### Hand-off to Session 4
|
||||
- PR β now has: adapter + smoke + Cat 13 subset
|
||||
- Ready for the actual eval runs
|
||||
|
||||
---
|
||||
|
||||
## Session 4 — PR β Phase 1: LongMemEval × 7 cells (overnight)
|
||||
|
||||
**Repo:** Same gbrain-evals branch
|
||||
**Wallclock:** ~10.5h (mostly hands-off, kick off and walk away)
|
||||
**API spend:** ~$476 (LongMemEval-heavy; 7 × $68/cell)
|
||||
|
||||
### What this session ships into PR β
|
||||
7 LongMemEval scored receipts (one per matrix cell). Each is a JSONL of 500
|
||||
hypotheses + a JSON file of correctness scores from `evaluate_qa.py`.
|
||||
|
||||
### Prereqs
|
||||
- Sessions 1+2+3 done. PR β has adapter + smoke + Cat 13.
|
||||
- LongMemEval dataset downloaded (gated HuggingFace; one-time setup).
|
||||
- `evaluate_qa.py` checked out somewhere (from
|
||||
https://github.com/xiaowu0162/LongMemEval) with its own venv set up.
|
||||
- API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`,
|
||||
`ZEROENTROPY_API_KEY`.
|
||||
|
||||
### Wrapper script
|
||||
Claude writes `scripts/run-shootout-phase1.sh` in the gbrain-evals branch. Single
|
||||
entry point that loops the 7 cells serially with smoke gating + cost-cap aborts.
|
||||
|
||||
```
|
||||
NEW: scripts/run-shootout-phase1.sh
|
||||
- Per cell: gbrain config set (embedder, dim, reranker, search.reranker.enabled, search.mode=tokenmax)
|
||||
- Per cell: bun run eval:smoke (abort cell on non-zero)
|
||||
- Per cell: gbrain eval longmemeval ... --output results/longmemeval-{cell}.jsonl
|
||||
- Per cell: cost-cap check ($90/cell hard stop)
|
||||
- Per cell: --resume-from existing results/longmemeval-{cell}.jsonl if present
|
||||
- Logs to results/phase1-run-log.txt
|
||||
```
|
||||
|
||||
### Run
|
||||
```bash
|
||||
# Kick off in background; check back in 10-12h
|
||||
bash scripts/run-shootout-phase1.sh 2>&1 | tee results/phase1-run-log.txt &
|
||||
```
|
||||
|
||||
Use `run_in_background: true` if running through Claude. Check back periodically.
|
||||
|
||||
### Scoring (after all 7 cells done)
|
||||
```bash
|
||||
for cell in A0 A1 B0 B1 C0 C1 C2; do
|
||||
python evaluate_qa.py \
|
||||
--input results/longmemeval-${cell}.jsonl \
|
||||
--output results/longmemeval-${cell}-scored.json
|
||||
done
|
||||
```
|
||||
|
||||
Each scored file has correctness %.
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. feat(scripts): Phase 1 LongMemEval wrapper with smoke gating + cost cap
|
||||
- NEW: scripts/run-shootout-phase1.sh
|
||||
|
||||
2. data(phase1): 7 LongMemEval cells (raw hypothesis JSONL)
|
||||
- results/longmemeval-{A0,A1,B0,B1,C0,C1,C2}.jsonl
|
||||
- results/phase1-run-log.txt (run timing + cost ledger)
|
||||
|
||||
3. data(phase1): evaluate_qa.py scoring results
|
||||
- results/longmemeval-{cell}-scored.json × 7
|
||||
```
|
||||
|
||||
### Verify
|
||||
- Each `longmemeval-{cell}.jsonl` has exactly 500 lines
|
||||
- Each `hypothesis` field is non-empty AND is actual answer text (NOT retrieval text)
|
||||
- Each `scored.json` has a `correctness_score` field
|
||||
|
||||
### Deliverable
|
||||
- 7 scored LongMemEval receipts committed to PR β
|
||||
- Real cost ledger committed alongside (compare against estimate)
|
||||
|
||||
### Hand-off to Session 5
|
||||
- Phase 1 done. Phase 2 (BrainBench, ~3.5h) and writeup remaining.
|
||||
|
||||
---
|
||||
|
||||
## Session 5 — PR β Phase 2 + writeup + ship
|
||||
|
||||
**Repo:** Same gbrain-evals branch
|
||||
**Wallclock:** ~7h (3.5h BrainBench + 3h writeup + /ship)
|
||||
**API spend:** ~$56 (BrainBench is cheap)
|
||||
|
||||
### What this session ships into PR β
|
||||
- 7 BrainBench cells (relational corpus + Cat 13 subset)
|
||||
- Final comparison writeup
|
||||
- PR β merged
|
||||
|
||||
### Prereqs
|
||||
- Session 4 done. PR β has Phase 1 receipts.
|
||||
|
||||
### Phase 2 wrapper script
|
||||
```
|
||||
NEW: scripts/run-shootout-phase2.sh
|
||||
- Per cell: configure provider (same as Phase 1)
|
||||
- Per cell: bun run eval:run -- --N 10 --include-subset cat13-embedder
|
||||
--output docs/benchmarks/2026-05-22-{cell}.md
|
||||
- Cost-cap check
|
||||
```
|
||||
|
||||
### Run
|
||||
```bash
|
||||
bash scripts/run-shootout-phase2.sh 2>&1 | tee results/phase2-run-log.txt
|
||||
```
|
||||
|
||||
### Writeup
|
||||
`docs/benchmarks/2026-05-22-embedder-shootout.md`. Structure:
|
||||
|
||||
1. **Headline table** — 7 cells × {LongMemEval correctness %, BrainBench relational MRR + P@5, Cat 13 correctness %, total cost}
|
||||
2. **Two questions answered:**
|
||||
- Which embedder wins solo? (A0 vs B0 vs C0)
|
||||
- Does zerank-2 carry ZE's win? (C0 vs C1 vs A1 vs B1)
|
||||
- Bonus: does dim matter for ZE? (C1 vs C2)
|
||||
3. **Paired-bootstrap p-values** per headline pair (methodology in
|
||||
`gbrain/docs/eval/SEARCH_MODE_METHODOLOGY.md`)
|
||||
4. **HNSW footnote** — Voyage 2048 and ZE 2560 used exact vector scan; OpenAI 1536
|
||||
and ZE 1280 used HNSW. Quality is primary, latency is secondary
|
||||
5. **What this does NOT prove** — synthetic-only, tokenmax-only, no real-brain replay
|
||||
6. **Recommendation:** explicit NON-recommendation to change `gbrain init` default;
|
||||
defer to a v0.36.x evidence pass with real-brain replay data
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. feat(scripts): Phase 2 BrainBench wrapper
|
||||
- NEW: scripts/run-shootout-phase2.sh
|
||||
|
||||
2. data(phase2): 7 BrainBench cells
|
||||
- docs/benchmarks/2026-05-22-{cell}.md × 7
|
||||
|
||||
3. docs(benchmark): embedder shootout comparison writeup
|
||||
- NEW: docs/benchmarks/2026-05-22-embedder-shootout.md
|
||||
- Bootstrap p-values, HNSW footnote, NOT-in-scope section
|
||||
```
|
||||
|
||||
### Ship
|
||||
```bash
|
||||
# Merge PR β to gbrain-evals main
|
||||
gh pr merge --squash --auto
|
||||
# Or non-auto if reviewing one more time:
|
||||
gh pr merge --squash
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- PR β merged to gbrain-evals `main`
|
||||
- Comparison report public at
|
||||
`gbrain-evals/docs/benchmarks/2026-05-22-embedder-shootout.md`
|
||||
|
||||
### Hand-off to Session 6 (optional)
|
||||
- gbrain-evals master has the full data + writeup
|
||||
- Ready for a v0.35.2.0 gbrain release that cross-links it
|
||||
|
||||
---
|
||||
|
||||
## Session 6 (optional) — PR γ: gbrain v0.35.2.0 release
|
||||
|
||||
**Repo:** `/Users/garrytan/conductor/workspaces/gbrain/<NEW-WORKSPACE>` (fresh from master)
|
||||
**Branch:** `garrytan/v0.35.2.0-benchmark-release`
|
||||
**Wallclock:** ~30min
|
||||
**API spend:** $0
|
||||
|
||||
### What this session ships
|
||||
A release-notes-only PR that bumps gbrain to v0.35.2.0 with a CHANGELOG entry
|
||||
cross-linking the embedder shootout benchmark. Optional — could be folded into the
|
||||
next routine release if no rush.
|
||||
|
||||
### Prereqs
|
||||
- Session 5 done. gbrain-evals merged with the comparison writeup.
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. docs(benchmark): mirror embedder shootout summary
|
||||
- NEW: docs/benchmarks/2026-05-22-embedder-shootout.md (slim mirror)
|
||||
- Cross-link to gbrain-evals canonical version
|
||||
|
||||
2. chore: v0.35.2.0
|
||||
- VERSION: 0.35.2.0
|
||||
- package.json: 0.35.2.0
|
||||
- CHANGELOG.md: new entry with the GStack-voice release summary
|
||||
+ "numbers that matter" table from the benchmark
|
||||
```
|
||||
|
||||
### Ship
|
||||
```bash
|
||||
/ship
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- gbrain v0.35.2.0 on master
|
||||
- CHANGELOG entry that drives the release-note headline
|
||||
|
||||
---
|
||||
|
||||
## Cost ledger (revised, post-review)
|
||||
|
||||
| Component | Per cell | × 7 cells |
|
||||
|---|---|---|
|
||||
| LongMemEval embed | <$0.05 | <$0.35 |
|
||||
| LongMemEval Sonnet answer-gen (500q × 2K tokens × $3/M) | $18 | $126 |
|
||||
| LongMemEval gpt-4o judge (500q × $0.10/q) | $50 | $350 |
|
||||
| BrainBench relational embed | $0.05-0.18 | <$1 |
|
||||
| BrainBench Cat 13 answer-gen + judge (50q × $0.14) | $7 | $49 |
|
||||
| Smoke harness (30 calls/cell) | <$0.10 | <$1 |
|
||||
| **Total** | **~$75/cell** | **~$525** |
|
||||
|
||||
**Hard cap: $700.** Per-cell hard cap: $90 (wrapper aborts cell if exceeded; partial
|
||||
JSONL preserved for resume).
|
||||
|
||||
## Failure modes and recovery
|
||||
|
||||
| Failure | Recovery |
|
||||
|---|---|
|
||||
| Voyage/ZE 429 rate-limit mid-cell | `gateway._shrinkState` halves safety_factor and retries. Cell continues. |
|
||||
| ZE 5MB rerank payload cap hit | `applyReranker` fail-opens, returns un-reranked results. Stderr warn. |
|
||||
| Mid-cell OS interrupt / cost-cap abort | Re-run with `gbrain eval longmemeval --resume-from results/longmemeval-{cell}.jsonl`. Picks up where it left off. |
|
||||
| `evaluate_qa.py` auth fail | OPENAI_API_KEY check in wrapper aborts before any spend. |
|
||||
| Adapter typo (bad dim) | `EvalAdapterConfig` runtime assertion at constructor throws AIConfigError. Cell aborts before API call. |
|
||||
|
||||
## NOT in scope (deliberate)
|
||||
|
||||
- **Real `~/.gbrain` replay** — adds 6-12h wallclock + $40-80 embed. Filed as v0.36.x.
|
||||
- **All 3 search modes** — pinned to tokenmax. `conservative` + `balanced` are v0.35.3.0
|
||||
follow-ups if reviewers push back.
|
||||
- **Matched-dim cross-vendor row** — no shared dim exists across all 3 vendors.
|
||||
Permanently out.
|
||||
- **`gbrain eval whoknows` / `cross-modal` / `takes-quality`** — embedding-invariant;
|
||||
rerunning across embedders produces noise.
|
||||
- **`gbrain eval code-retrieval`** — code corpus, separate concern.
|
||||
- **`gbrain eval suspected-contradictions`** — wants a real brain.
|
||||
- **`gbrain init --recommended` default change** — codex correctly flagged the evidence
|
||||
base as insufficient. Defer to v0.36.x with real-brain replay data.
|
||||
|
||||
## What already exists (reused, not rebuilt)
|
||||
|
||||
- `gbrain eval longmemeval` CLI (in-tree, answer-gen mode default)
|
||||
- gbrain-evals BrainBench runner (`eval:run`) — needs adapter parameterization but
|
||||
per-cell test plumbing is reused
|
||||
- Gateway routing for Voyage + ZE (shipped v0.35.0.0)
|
||||
- Reranker pipeline (`src/core/search/rerank.ts`, fail-open)
|
||||
- Pricing table (extended, not rebuilt)
|
||||
- Paired-bootstrap methodology (`docs/eval/SEARCH_MODE_METHODOLOGY.md`)
|
||||
- LongMemEval published `evaluate_qa.py` (invoked externally, not bundled)
|
||||
@@ -1,27 +0,0 @@
|
||||
# Origin story
|
||||
|
||||
GBrain came out of building OpenClaw — Garry's personal AI agent fork. The first version had skills and a brain, but the brain was a flat directory of markdown files. Search was ripgrep. Memory was vibes.
|
||||
|
||||
Two problems surfaced almost immediately.
|
||||
|
||||
First, the agent forgot things between conversations. Every new session re-asked basic questions. Names of people Garry had introduced last week were gone. Decisions made on Tuesday didn't survive to Thursday. The brain existed but the agent couldn't actually use it.
|
||||
|
||||
Second, the agent kept duplicating work. Two different signals about the same company became two different people pages. Three meetings with the same person became three uncorrelated timeline entries. The signal-to-noise ratio decayed in real time.
|
||||
|
||||
GBrain is what you build when you decide both of those are unacceptable.
|
||||
|
||||
The fix wasn't one big idea. It was many small ones layered together:
|
||||
|
||||
- Brain-first lookup before any external API call.
|
||||
- Auto-linking on every page write so the graph grows for free.
|
||||
- Typed edges so "who works at Acme AI?" actually returns something.
|
||||
- Hybrid search because vector alone underdelivers.
|
||||
- Reranker on top because hybrid alone is locally optimal but globally suboptimal.
|
||||
- Nightly cron to dedup, enrich, fix citations, surface contradictions.
|
||||
- An agent that reads `skills/RESOLVER.md` once and knows what to do.
|
||||
|
||||
None of those are novel ideas. The contribution is shipping all of them together, on Postgres + pgvector that runs in WASM (no server), with skills that are markdown (not code), routed by a small text file (not a router LLM).
|
||||
|
||||
The production brain has been running for months now. 17,888 pages. 4,383 people. 723 companies. 21 cron jobs running autonomously. It wakes Garry up smarter than the day before.
|
||||
|
||||
GBrain is what happens when you write the brain you actually wanted to have.
|
||||
@@ -97,14 +97,6 @@ not a baseline comparison. For metric-against-truth eval, use
|
||||
replay tool answers a different question: "did my code change move
|
||||
retrieval, and which queries did it move most?"
|
||||
|
||||
For a third evaluation axis — public benchmark, ground-truth labels, full
|
||||
question-answer pipeline (not just retrieval) — `gbrain eval longmemeval
|
||||
<dataset.jsonl>` (v0.28.8) runs the LongMemEval benchmark against gbrain's
|
||||
hybrid retrieval. Each question gets a clean in-memory PGLite, its haystack
|
||||
imported, the question asked, the hypothesis emitted as JSONL — exactly the
|
||||
shape LongMemEval's `evaluate_qa.py` consumes. Your `~/.gbrain` brain is
|
||||
never opened. See `## Public benchmarks: LongMemEval` below.
|
||||
|
||||
## Best-effort by design
|
||||
|
||||
Replay is not pure. Three things can drift between capture and replay:
|
||||
@@ -230,101 +222,3 @@ Existing `eval_candidates` rows stay until you `gbrain eval prune
|
||||
| `Mean latency Δ: +500ms`, jaccard high | Vector path got slower; check embedding API or HNSW probes |
|
||||
| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields |
|
||||
| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured |
|
||||
|
||||
## Public benchmarks: LongMemEval (v0.28.8)
|
||||
|
||||
`gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval)
|
||||
benchmark directly against gbrain's hybrid retrieval. Different evaluation
|
||||
axis from `eval replay`: public dataset with ground-truth labels, end-to-end
|
||||
question-answer pipeline, hermetic per-question brains.
|
||||
|
||||
```bash
|
||||
# Download the dataset (visit the HF page in a browser; gated/manual download).
|
||||
# Place longmemeval_oracle.json (or _s.json) somewhere local.
|
||||
|
||||
# Retrieval-only (no LLM answer-gen, fastest path, no Anthropic key needed):
|
||||
gbrain eval longmemeval ./longmemeval_oracle.json --limit 50 --retrieval-only \
|
||||
> /tmp/hypothesis.jsonl
|
||||
|
||||
# Full pipeline (Anthropic key required for answer-gen):
|
||||
gbrain eval longmemeval ./longmemeval_oracle.json --limit 50 \
|
||||
> /tmp/hypothesis.jsonl
|
||||
|
||||
# Score with LongMemEval's published evaluate_qa.py (not bundled — needs
|
||||
# OpenAI gpt-4o per their spec):
|
||||
python evaluate_qa.py /tmp/hypothesis.jsonl
|
||||
```
|
||||
|
||||
### Architecture (read this if you're touching the harness)
|
||||
|
||||
- One in-memory PGLite per benchmark run via `createBenchmarkBrain` +
|
||||
`withBenchmarkBrain`. Your `~/.gbrain` is never opened.
|
||||
- Between questions: `TRUNCATE` over runtime-enumerated `pg_tables`, NOT a
|
||||
hardcoded list — schema migrations don't silently leak data across
|
||||
questions. Infrastructure tables (`sources`, `config`,
|
||||
`gbrain_cycle_locks`, `subagent_rate_leases`) are preserved across resets.
|
||||
- Sanitization parity: re-uses `INJECTION_PATTERNS` from
|
||||
`src/core/think/sanitize.ts` so adding a new injection pattern
|
||||
automatically covers takes AND benchmarks. One source of truth.
|
||||
- Retrieved chat content is wrapped in `<chat_session id="..." date="...">`
|
||||
framing; the answer-gen system prompt declares the content UNTRUSTED.
|
||||
Same posture as `<take>` framing.
|
||||
- LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})`.
|
||||
Tests stub the client so the full pipeline runs hermetically without any
|
||||
API key.
|
||||
|
||||
### Flags
|
||||
|
||||
| Flag | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `--limit N` | run all | Cap question count (iterate fast) |
|
||||
| `--retrieval-only` | off | Emit retrieved chunks; no LLM answer-gen |
|
||||
| `--keyword-only` | off | Disable vector path (debug retrieval issues) |
|
||||
| `--expansion` | **off** | Multi-query expansion. Off by default for determinism (no per-query Haiku call). Pass to opt in. |
|
||||
| `--top-k K` | 10 | Retrieval depth |
|
||||
| `--model M` | resolved | Default resolves through `resolveModel()` 6-tier chain (`models.eval.longmemeval` config key) |
|
||||
| `--output FILE` | stdout | Write hypothesis JSONL to file instead of stdout |
|
||||
|
||||
### Numbers
|
||||
|
||||
p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the
|
||||
`test/eval-longmemeval.test.ts` perf gate). Per-question cost well under the
|
||||
500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and
|
||||
LLM latency.
|
||||
|
||||
## Measuring brain consistency over time (v0.32.6)
|
||||
|
||||
`gbrain eval suspected-contradictions` is a complementary measurement
|
||||
instrument: it samples retrieval results for unmarked semantic
|
||||
contradictions (e.g., compiled_truth vs chat content, intra-page chunk
|
||||
vs active take). Where LongMemEval measures retrieval correctness on a
|
||||
fixed labeled set, the contradiction probe measures how often a real
|
||||
brain surfaces conflicting answers.
|
||||
|
||||
### Recommended nightly cadence
|
||||
|
||||
```bash
|
||||
# Once a day, against your top 50 most-frequent queries:
|
||||
gbrain eval suspected-contradictions \
|
||||
--queries-file ~/.gbrain/queries.jsonl \
|
||||
--top-k 5 \
|
||||
--budget-usd 5 \
|
||||
--output ~/.gbrain/probe-runs/$(date +%Y-%m-%d).json
|
||||
```
|
||||
|
||||
Persistent cache (`eval_contradictions_cache`) makes re-runs near-zero
|
||||
cost until you bump `PROMPT_VERSION`. Trend-track via:
|
||||
|
||||
```bash
|
||||
gbrain eval suspected-contradictions trend --days 30
|
||||
```
|
||||
|
||||
The ASCII bar chart shows total flagged per day. Headline % surfaces in
|
||||
`gbrain doctor`'s `contradictions` check with paste-ready resolution
|
||||
commands per high-severity finding.
|
||||
|
||||
### See also
|
||||
|
||||
- `docs/contradictions.md` — architecture, severity rubric, action criteria.
|
||||
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
|
||||
decision criteria gated on Wilson CI lower-bound.
|
||||
|
||||
@@ -1,159 +0,0 @@
|
||||
# `gbrain eval takes-quality` — reproducible cross-modal quality eval
|
||||
|
||||
v0.32+ ships a CI-able quality gate for the takes layer. Three frontier models
|
||||
score a sample of takes against a 5-dimension rubric, the runner aggregates to
|
||||
PASS / FAIL / INCONCLUSIVE, and the receipt persists to `eval_takes_quality_runs`
|
||||
so a follow-up `trend` or `regress` can compare against history.
|
||||
|
||||
This doc is the consumer contract. The sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals)
|
||||
repo and any future CI gate read receipts shaped exactly like the JSON below.
|
||||
Fields are additive-stable at `schema_version: 1`. A breaking shape change
|
||||
bumps the version.
|
||||
|
||||
## Subcommands
|
||||
|
||||
| Command | Brain required? | Exit codes |
|
||||
|---|---|---|
|
||||
| `gbrain eval takes-quality run [flags]` | yes (samples takes) | 0 PASS, 1 FAIL, 2 INCONCLUSIVE |
|
||||
| `gbrain eval takes-quality replay <receipt>` | **no** (disk-only) | 0 PASS, 1 FAIL, 2 INCONCLUSIVE |
|
||||
| `gbrain eval takes-quality trend [flags]` | yes (reads runs table) | 0 |
|
||||
| `gbrain eval takes-quality regress --against <receipt>` | yes | 0 OK, 1 regression |
|
||||
|
||||
`replay` is the only mode that runs without `DATABASE_URL` — it reads the
|
||||
receipt file from disk and re-renders it. The other modes need the brain.
|
||||
|
||||
## `run` flags
|
||||
|
||||
| Flag | Default | Notes |
|
||||
|---|---|---|
|
||||
| `--limit N` | 100 | Random sample of N takes from the brain. |
|
||||
| `--cycles N` | 3 (TTY) / 1 (non-TTY) | Up to N panel calls before giving up; early-stop on PASS or INCONCLUSIVE. |
|
||||
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
|
||||
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
|
||||
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
|
||||
| `--models a,b,c` | `openai:gpt-4o,anthropic:claude-opus-4-7,google:gemini-1.5-pro` | Comma-separated panel. |
|
||||
| `--json` | off | Emit the full receipt to stdout. |
|
||||
|
||||
## Receipt JSON shape (`schema_version: 1`)
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"ts": "2026-05-09T22:00:00.000Z",
|
||||
"rubric_version": "v1.0",
|
||||
"rubric_sha8": "abcd1234",
|
||||
"corpus": {
|
||||
"source": "db",
|
||||
"n_takes": 100,
|
||||
"slug_prefix": null,
|
||||
"corpus_sha8": "abcd1234"
|
||||
},
|
||||
"prompt_sha8": "abcd1234",
|
||||
"models_sha8": "abcd1234",
|
||||
"models": ["openai:gpt-4o", "anthropic:claude-opus-4-7", "google:gemini-1.5-pro"],
|
||||
"cycles_run": 3,
|
||||
"successes_per_cycle": [3, 3, 2],
|
||||
"verdict": "pass",
|
||||
"scores": {
|
||||
"accuracy": { "mean": 7.8, "min": 7, "max": 9, "scores": [9,7,7], "per_model": {...} },
|
||||
"attribution": { "mean": 7.0, "min": 7, "max": 7, "scores": [7,7,7], "per_model": {...} },
|
||||
"weight_calibration": { "mean": 7.5, "min": 7, "max": 8, "scores": [8,7,7], "per_model": {...} },
|
||||
"kind_classification": { "mean": 7.2, "min": 7, "max": 8, "scores": [7,8,7], "per_model": {...} },
|
||||
"signal_density": { "mean": 7.0, "min": 6, "max": 8, "scores": [8,7,6], "per_model": {...} }
|
||||
},
|
||||
"overall_score": 7.3,
|
||||
"cost_usd": 1.85,
|
||||
"improvements": ["..."],
|
||||
"errors": [],
|
||||
"verdictMessage": "PASS: every dim mean >=7 and min >=5 ..."
|
||||
}
|
||||
```
|
||||
|
||||
### Field reference
|
||||
|
||||
- `schema_version` — locks the contract. Adding optional fields is additive
|
||||
and compatible. Renaming, removing, or changing semantics bumps the version.
|
||||
- `rubric_version` + `rubric_sha8` — segregate trend rows by rubric epoch
|
||||
(codex review #3). When the rubric definition changes, both fields update,
|
||||
and trend mode groups runs accordingly so a stricter rubric doesn't
|
||||
silently look like a quality drop.
|
||||
- `corpus.corpus_sha8` — fingerprint over the joined takes-text the judge
|
||||
saw. Determines whether two runs are over the "same" sample.
|
||||
- `models_sha8` — fingerprint over the sorted model id list. Re-ordering
|
||||
models in `--models` doesn't change the sha (sort is stable).
|
||||
- `successes_per_cycle` — count of contributing models per cycle. A model
|
||||
contributes when (a) its JSON parsed AND (b) every declared rubric dim
|
||||
has a finite score (codex review #5 — missing-dim drops the contribution).
|
||||
- `verdict` — `pass` if every dim mean >= 7 AND every dim min across
|
||||
contributing models >= 5; `fail` otherwise; `inconclusive` if fewer than
|
||||
2/3 models contributed complete scores.
|
||||
- `cost_usd` — sum of per-call cost via `pricing.ts`. Unknown models when
|
||||
`--budget-usd` is set produce a `PricingNotFoundError` before any call
|
||||
fires.
|
||||
|
||||
## Receipt persistence
|
||||
|
||||
Receipts persist to **`eval_takes_quality_runs`** (DB-authoritative per
|
||||
codex review #6) AND to disk at `~/.gbrain/eval-receipts/takes-quality-<corpus>-<prompt>-<models>-<rubric>.json`
|
||||
as a best-effort artifact. The DB row carries the full receipt JSON in the
|
||||
`receipt_json` JSONB column, so when the disk artifact is gone, `replay`
|
||||
can still reconstruct via `loadReceiptFromDb` (v0.33+ flag wiring).
|
||||
|
||||
The 4-sha primary key is unique (`UNIQUE` constraint) so re-running an
|
||||
identical eval is `INSERT ... ON CONFLICT DO NOTHING` — idempotent.
|
||||
|
||||
## Trend output
|
||||
|
||||
Plain text (default):
|
||||
|
||||
```
|
||||
ts rubric verdict overall cost corpus
|
||||
─────────────────────────────────────────────────────────────────────────────
|
||||
2026-05-09T22:00:00 v1.0 pass 7.3 $1.85 abcd1234
|
||||
2026-05-08T18:30:00 v1.0 fail 6.8 $1.92 ef567890
|
||||
```
|
||||
|
||||
JSON shape (`--json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"rows": [
|
||||
{ "id": 42, "ts": "...", "rubric_version": "v1.0", "verdict": "pass",
|
||||
"overall_score": 7.3, "cost_usd": 1.85, "corpus_sha8": "abcd1234" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Regress: gating CI on quality
|
||||
|
||||
```bash
|
||||
# Capture a baseline.
|
||||
gbrain eval takes-quality run --limit 100 --json \
|
||||
> .ci/takes-quality-baseline.json
|
||||
|
||||
# Later, after changing the extraction prompt:
|
||||
gbrain eval takes-quality regress --against .ci/takes-quality-baseline.json \
|
||||
--threshold 0.5
|
||||
# exit 0 → no regression past threshold
|
||||
# exit 1 → some dim dropped > 0.5; CI fails
|
||||
```
|
||||
|
||||
The threshold is the per-dim-mean drop counting as regression. Default 0.5.
|
||||
Regress reuses the **same** model panel + slug prefix + source as the prior
|
||||
receipt for an apples-to-apples compare. Diffs in `corpus_sha8` /
|
||||
`prompt_sha8` / `rubric_sha8` are surfaced as informational warnings (the
|
||||
runner doesn't refuse — that's the caller's call).
|
||||
|
||||
## Contract stability
|
||||
|
||||
The shape above is the read contract for downstream consumers. Anything
|
||||
not listed (e.g. internal aggregator state, gateway providerMetadata) is
|
||||
**not** in the receipt and may change without notice.
|
||||
|
||||
When you need to evolve the schema:
|
||||
1. Additive optional field → no version bump; old consumers ignore the
|
||||
new key, new consumers read it.
|
||||
2. Renamed or removed field, or changed semantics → bump
|
||||
`schema_version` to `2`; runner emits both shapes for one release as
|
||||
a deprecation runway.
|
||||
@@ -1,124 +0,0 @@
|
||||
# Evaluation Metric Glossary
|
||||
|
||||
**Auto-generated from `src/core/eval/metric-glossary.ts`. Do not edit by hand.** Run `bun run scripts/generate-metric-glossary.ts` to regenerate.
|
||||
|
||||
Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-English explanation here. Industry terms are preserved verbatim so users searching the literature find what we report.
|
||||
|
||||
## Retrieval Metrics
|
||||
|
||||
### Precision at k (P@k)
|
||||
|
||||
**Key:** `precision@k`
|
||||
|
||||
**Plain English:** Of the top k results the engine returned, what fraction were actually relevant? High precision means few junk results in the top of the list.
|
||||
|
||||
**Range:** 0..1, higher is better. P@10 = 0.7 means 7 of the top 10 results were on-topic.
|
||||
|
||||
### Recall at k (R@k)
|
||||
|
||||
**Key:** `recall@k`
|
||||
|
||||
**Plain English:** Of all the relevant results that exist in the brain, what fraction did the engine find in its top k? High recall means few missed answers.
|
||||
|
||||
**Range:** 0..1, higher is better. R@10 = 0.81 means out of every 100 questions, the right answer was in the top 10 for 81 of them.
|
||||
|
||||
### Mean Reciprocal Rank (MRR)
|
||||
|
||||
**Key:** `mrr`
|
||||
|
||||
**Plain English:** On average, how far down the list is the FIRST relevant result? An MRR of 1.0 means the first hit is always right; an MRR of 0.5 means it's typically at rank 2.
|
||||
|
||||
**Range:** 0..1, higher is better. Computed as the average of 1/rank-of-first-relevant-result across all test queries.
|
||||
|
||||
### Normalized Discounted Cumulative Gain at k (nDCG@k)
|
||||
|
||||
**Key:** `ndcg@k`
|
||||
|
||||
**Plain English:** Like precision@k, but the engine gets MORE credit for putting good results near the top than near rank k. A perfect ordering scores 1.0; a totally random ordering scores near 0.
|
||||
|
||||
**Range:** 0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora.
|
||||
|
||||
## Set-Similarity / Stability Metrics
|
||||
|
||||
### Jaccard similarity at k (set Jaccard @k)
|
||||
|
||||
**Key:** `jaccard@k`
|
||||
|
||||
**Plain English:** How much do two result lists overlap? Compare the top k slugs from the captured baseline against the current run; Jaccard@10 = 1.0 means perfect agreement, 0.0 means zero overlap.
|
||||
|
||||
**Range:** 0..1, higher = more stable. Below 0.5 on a stable corpus means retrieval changed significantly.
|
||||
|
||||
### Top-1 stability rate
|
||||
|
||||
**Key:** `top1_stability`
|
||||
|
||||
**Plain English:** Fraction of queries where the #1 result is the same between two runs. The most aggressive stability check — small ranking shifts that don't change the top answer don't hurt it.
|
||||
|
||||
**Range:** 0..1, higher = more stable. Above 0.85 typically means safe-to-merge for retrieval changes.
|
||||
|
||||
## Statistical-Significance Metrics
|
||||
|
||||
### p-value (paired bootstrap)
|
||||
|
||||
**Key:** `p_value`
|
||||
|
||||
**Plain English:** How likely the observed difference between two modes is just noise. Lower = stronger evidence the difference is real. We compute paired bootstrap with 10,000 resamples and Bonferroni correction across the 12 comparisons (3 modes × 4 metrics).
|
||||
|
||||
**Range:** 0..1, lower = stronger signal. Below 0.05 is the common "statistically significant" threshold; below 0.01 is strong evidence.
|
||||
|
||||
### 95% Confidence Interval (CI)
|
||||
|
||||
**Key:** `confidence_interval`
|
||||
|
||||
**Plain English:** The range we're 95% sure the true value falls inside, given the sample we measured. Narrower CI = more reliable estimate. Computed via bootstrap resampling.
|
||||
|
||||
**Range:** Two-tuple [low, high]. If 0 is inside the CI for a Δ, the difference isn't statistically significant.
|
||||
|
||||
## Operational / Cost Metrics
|
||||
|
||||
### Cache hit rate
|
||||
|
||||
**Key:** `cache_hit_rate`
|
||||
|
||||
**Plain English:** Fraction of searches that reused a recent cached answer instead of running fresh. Higher hit rate = lower latency + lower LLM spend, but stale results may slip through if the threshold is too loose.
|
||||
|
||||
**Range:** 0..1, higher generally better. 0.7-0.9 is the sweet spot for a busy brain; above 0.9 may indicate the similarity threshold is too loose.
|
||||
|
||||
### Average results returned
|
||||
|
||||
**Key:** `avg_results`
|
||||
|
||||
**Plain English:** Mean number of search-result rows the engine returned per call. Should be near the active mode's searchLimit unless the brain is small or the budget is dropping results.
|
||||
|
||||
**Range:** 0..searchLimit. Far below searchLimit suggests budget pressure or sparse retrieval.
|
||||
|
||||
### Average tokens delivered
|
||||
|
||||
**Key:** `avg_tokens`
|
||||
|
||||
**Plain English:** Estimated tokens (chars / 4) in the chunk text returned per search call. The direct measure of how much context an agent loop is paying for each search.
|
||||
|
||||
**Range:** 0..tokenBudget. Approximates OpenAI tiktoken count for English; off by ~5-10% for Anthropic and worse for non-English.
|
||||
|
||||
### Cost per query (USD)
|
||||
|
||||
**Key:** `cost_per_query_usd`
|
||||
|
||||
**Plain English:** Sum of LLM + embedding API charges for one search call. Includes Haiku expansion call (tokenmax mode only) + embedding cost + downstream answer-model cost if measured.
|
||||
|
||||
**Range:** 0..unbounded. Conservative mode is typically <\$0.001 per call; tokenmax with answer-gen can exceed \$0.01.
|
||||
|
||||
### p99 latency (ms)
|
||||
|
||||
**Key:** `p99_latency_ms`
|
||||
|
||||
**Plain English:** 99th percentile wall-clock time per search call. The latency that 1% of users see — long-tail experience, not the average.
|
||||
|
||||
**Range:** 0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.
|
||||
|
||||
---
|
||||
|
||||
## Coverage
|
||||
|
||||
Every metric printed by any `gbrain eval *` or `gbrain search stats` command resolves through `getMetricGloss()` in `src/core/eval/metric-glossary.ts`. Adding a new metric to the glossary REQUIRES updating this doc; the CI guard catches drift.
|
||||
|
||||
@@ -1,285 +0,0 @@
|
||||
# Search Mode Evaluation Methodology
|
||||
|
||||
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
|
||||
|
||||
## 1. What this measures and what it doesn't
|
||||
|
||||
**Measures:** retrieval quality and operational cost on fixed public datasets, under each named search mode, against the same brain content.
|
||||
|
||||
**Does NOT measure:**
|
||||
- Your specific brain content (this is a benchmark, not your bill).
|
||||
- Your specific query distribution.
|
||||
- End-user satisfaction or downstream task success.
|
||||
- Latency under concurrent load.
|
||||
- Production cost (the cost numbers are model-pricing estimates × dataset size, not your actual API spend).
|
||||
|
||||
If you want to know how a mode behaves on YOUR brain, run `gbrain search stats --days 30` after a real usage window, then run `gbrain search tune` for actionable recommendations.
|
||||
|
||||
## 2. Datasets and sizes
|
||||
|
||||
- **LongMemEval** — public split, `n=500` questions. Downloaded from [Hugging Face](https://huggingface.co/datasets/xiaowu0162/longmemeval). The corpus + answer keys are pinned to a specific commit; recorded in every per-run record.
|
||||
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
|
||||
- **BrainBench v1** — `n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
|
||||
|
||||
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
|
||||
|
||||
## 3. Sample selection
|
||||
|
||||
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
|
||||
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
|
||||
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
|
||||
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
|
||||
|
||||
## 4. Run procedure
|
||||
|
||||
The command is the doc. Anyone can reproduce.
|
||||
|
||||
```bash
|
||||
# Setup: in your gbrain working tree, with OPENAI_API_KEY + ANTHROPIC_API_KEY exported.
|
||||
git rev-parse HEAD # record the commit for the methodology footer
|
||||
|
||||
# Sweep all 3 modes × 2 retrieval-focused suites with seed 42.
|
||||
gbrain eval run-all \
|
||||
--modes conservative,balanced,tokenmax \
|
||||
--suites longmemeval,replay \
|
||||
--seed 42 \
|
||||
--limit 500 \
|
||||
--budget-usd-retrieval 5 \
|
||||
--budget-usd-answer 20 \
|
||||
--output docs/eval/results/v0.32.3/
|
||||
|
||||
# Render the comparison.
|
||||
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
|
||||
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
|
||||
```
|
||||
|
||||
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
|
||||
|
||||
## 5. Threats to validity
|
||||
|
||||
Honest list. We name what would let a critic dismiss the numbers.
|
||||
|
||||
- **LongMemEval skews English + technical.** The questions are software-engineering and consumer-product flavored. Performance on a brain rich in non-English / non-technical content (writing, art history, etc.) may differ.
|
||||
- **BrainBench is small** (1240 docs) relative to a production brain (10K-100K pages). Absolute scores aren't predictive of your hit rate; the _delta_ between modes is.
|
||||
- **char/4 token heuristic.** Token-budget enforcement and cost estimates use a character-count / 4 heuristic. Accurate within ~5-10% for English with the OpenAI tiktoken family; off worse for Voyage (we don't use Voyage in chat retrieval, so it doesn't bias the reported numbers, but if you do, your budget caps will be approximate).
|
||||
- **Expansion's quality lift varies by query distribution.** The eval data shows ~97.6% relative quality with LLM expansion vs without (i.e., barely measurable lift) on the LongMemEval corpus. On rarer-entity / longer-tail queries, the lift can be larger. We report the corpus we measured; YMMV.
|
||||
- **Paired bootstrap assumes question-level independence.** Multi-hop questions within the same conversation thread aren't independent; the bootstrap CI is slightly tighter than reality.
|
||||
- **Single brain instance per benchmark.** The benchmark spins up an in-memory PGLite per question. Cache hit rate measured here doesn't reflect a long-running production brain's cache state.
|
||||
|
||||
## 6. Per-question raw outputs
|
||||
|
||||
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
|
||||
|
||||
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
|
||||
|
||||
- **Wins:** the 3 questions where this mode's score exceeded the next-best mode by the largest margin.
|
||||
- **Losses:** the 3 questions where this mode's score fell short of the next-best mode by the largest margin.
|
||||
|
||||
Picked by the score delta, NOT cherry-picked by hand. The README documents the rule so a critic can verify.
|
||||
|
||||
## 7. Pre-registered expectations
|
||||
|
||||
Before running, we expect:
|
||||
|
||||
1. **tokenmax wins Recall@10** by 5-15 percentage points over conservative. LLM expansion + 50-result ceiling helps rare-entity surface forms.
|
||||
2. **conservative wins cost-per-query** by 5-15× over tokenmax. No Haiku expansion + tight 4K budget cap = single-digit-cent queries.
|
||||
3. **balanced lands within 3pp of tokenmax** on Recall@10. Intent weighting (zero-LLM cost) closes most of the expansion gap on common queries.
|
||||
4. **No mode breaks nDCG@10 ≥ 0.65** — the published "ship it" threshold for hybrid retrieval on technical corpora.
|
||||
|
||||
Then we publish whether the data agrees. **If a hypothesis fails, that's documented honestly** in the release README, not buried. Pre-registration is what makes the comparison defensible — without it, a "we expected X and got X" outcome is observation, not prediction.
|
||||
|
||||
## 8. Re-run cadence
|
||||
|
||||
This document + the eval results are regenerated on every release that touches retrieval-affecting code. The `gbrain doctor eval_drift` check surfaces changes to the curated watch-list in `src/core/eval/drift-watch.ts`:
|
||||
|
||||
- `src/core/search/**`
|
||||
- `src/core/embedding.ts`
|
||||
- `src/core/chunkers/**`
|
||||
- `src/core/ai/recipes/anthropic.ts`
|
||||
- `src/core/ai/recipes/openai.ts`
|
||||
- `src/core/operations.ts`
|
||||
|
||||
Additions to the watch-list require a CHANGELOG line.
|
||||
|
||||
## Statistical-significance discipline
|
||||
|
||||
When `gbrain eval compare --md` reports a Δ between two modes, it computes:
|
||||
|
||||
- **Paired bootstrap** with 10,000 resamples per metric. Each resample draws _question-level_ pairs (same question, mode A vs mode B), so question-level variance is differenced out.
|
||||
- **Bonferroni correction** across the 12 comparisons (3 modes × 4 metrics). The reported p-value is the comparison's raw p-value × 12 (clamped at 1.0).
|
||||
- **95% confidence intervals** computed from the bootstrap distribution.
|
||||
|
||||
If the CI for a Δ includes 0 OR the Bonferroni-adjusted p-value exceeds 0.05, the difference is **not** statistically significant. The MD report says "not significant" verbatim.
|
||||
|
||||
## Glossary
|
||||
|
||||
Every metric the report prints has a plain-English entry in `docs/eval/METRIC_GLOSSARY.md`, auto-generated from `src/core/eval/metric-glossary.ts`. The CI guard at `scripts/check-eval-glossary-fresh.sh` regenerates and diffs against the committed file on every test run; a stale doc fails the build.
|
||||
|
||||
## Cost anchors
|
||||
|
||||
The mode-picker prompt at `gbrain init` and the CLAUDE.md `## Search Mode` table both surface these rough cost anchors. Working through the math so they're auditable:
|
||||
|
||||
**Variables:**
|
||||
- `T` = avg tokens per search-result chunk. The recursive chunker targets 300 words / chunk → ~400 tokens (English, OpenAI tiktoken approx).
|
||||
- `N` = chunks delivered per query (capped by the mode's `searchLimit`).
|
||||
- `R` = downstream model input rate. Sonnet 4.6 = \$3/M. Opus 4.7 = \$5/M. Haiku 4.5 = \$1/M.
|
||||
- `Q` = queries per month.
|
||||
|
||||
**Per-query input cost** (downstream agent reads the chunks):
|
||||
|
||||
cost_per_query = T × N × R
|
||||
|
||||
| Mode | T (tokens) | N (chunks) | Sonnet (\$3/M) | Opus (\$5/M) | Haiku (\$1/M) |
|
||||
|---|---|---|---|---|---|
|
||||
| conservative (4K cap, 10 max) | ~400 | 10 (or fewer if budget hits) | \$0.012 | \$0.020 | \$0.004 |
|
||||
| balanced (12K cap, 25 max) | ~400 | ~25 | \$0.030 | \$0.050 | \$0.010 |
|
||||
| tokenmax (no cap, 50 max) | ~400 | ~50 | \$0.060 | \$0.100 | \$0.020 |
|
||||
|
||||
**Monthly cost** (Q × per-query):
|
||||
|
||||
| Mode @ Sonnet | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|
||||
|---|---|---|---|
|
||||
| conservative | \$12 | \$120 | \$1,200 |
|
||||
| balanced | \$30 | \$300 | \$3,000 |
|
||||
| tokenmax | \$60 | \$600 | \$6,000 |
|
||||
|
||||
| Mode @ Opus | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|
||||
|---|---|---|---|
|
||||
| conservative | \$20 | \$200 | \$2,000 |
|
||||
| balanced | \$50 | \$500 | \$5,000 |
|
||||
| tokenmax | \$100 | \$1,000 | \$10,000 |
|
||||
|
||||
**gbrain's own cost** on top:
|
||||
- Query embedding (text-embedding-3-large @ \$0.13/M tokens): ~\$0.00001 per query. Negligible at every scale.
|
||||
- Tokenmax Haiku expansion call (\$1/M input, \$5/M output, ~500 input + 200 output per call): ~\$0.0015 per query, or \$150/mo at 100K queries. Cache hits cut this in half.
|
||||
- Per-page indexing (one-time): bounded by your import volume, not query volume. Not modeled here.
|
||||
|
||||
**Cache hit adjustment.** A warmed brain typically sees 30-50% cache hits on repeat-query traffic. Cache hits skip the downstream input cost entirely (the cached result was already in the agent's context once). So real-world costs run ~50-70% of the table above on a busy brain.
|
||||
|
||||
**Why these numbers DRIFT from your actual bill:**
|
||||
- Your agent's system prompt + reasoning tokens add input that gbrain doesn't see.
|
||||
- Compaction reduces input over a long session.
|
||||
- Most agents make 1-5 searches per turn; cost-per-turn is what bills you, not cost-per-query.
|
||||
- The model price column drifts as providers reprice; pin the rate via `src/core/anthropic-pricing.ts` for a current snapshot.
|
||||
|
||||
The picker copy + CLAUDE.md table are the canonical user-facing source. Update them in lockstep when the underlying chunker size or default `searchLimit` changes.
|
||||
|
||||
## Mode × Model matrix (the 25x spread)
|
||||
|
||||
The per-query math above assumes Sonnet 4.6 downstream. In reality, the
|
||||
downstream model tier is the BIGGER cost lever. Per-query cost at 10K
|
||||
queries/month (typical single-user volume), search payload only (no cache
|
||||
savings):
|
||||
|
||||
| Mode (search tokens) | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
|
||||
fleet); divide by 10 for 1K/mo (light usage).
|
||||
|
||||
**Natural pairings span ~4x** (cheap model + tight mode → frontier model + loose
|
||||
mode). **Mismatches waste capacity:**
|
||||
|
||||
- `tokenmax + Haiku`: Haiku gets 20K of search results stuffed into its
|
||||
context per query. Haiku's reasoning is weaker; more chunks = more noise,
|
||||
not more signal. You pay Haiku rates but get sub-Haiku quality. Wrong
|
||||
direction.
|
||||
- `conservative + Opus`: Opus has 200K context window and can synthesize
|
||||
across many chunks. Capping at 10 chunks / 4K tokens leaves Opus
|
||||
reasoning underfed. You pay Opus rates but get conservative-shape
|
||||
retrieval. Wasted spend.
|
||||
|
||||
**Right-sizing rule:** match the mode's `searchLimit` to the downstream
|
||||
model's "useful context depth":
|
||||
|
||||
- Haiku struggles past ~5-10 chunks of cross-referenced content → conservative
|
||||
- Sonnet handles ~25-40 chunks well → balanced
|
||||
- Opus benefits from 50+ chunks for multi-hop reasoning → tokenmax
|
||||
|
||||
## Realistic-scale anchor (single power-user agent loop)
|
||||
|
||||
The per-query math above is honest but theoretical: it treats each search as an isolated billable event. Real agent loops amortize a lot of context across turns via Anthropic prompt caching. Here's what one heavy power-user loop actually looks like in production, anonymized + scaled so the numbers represent a representative power user rather than any specific deployment.
|
||||
|
||||
**Reference shape — tokenmax in production at a single-user scale:**
|
||||
|
||||
| Quantity | Approximate value |
|
||||
|---|---|
|
||||
| 30-day total agent spend | ~\$700/mo |
|
||||
| 30-day total tokens billed | ~800M |
|
||||
| Turns per month | ~860 (~29/day; one active agent loop) |
|
||||
| Average tokens per turn | ~900K |
|
||||
| Average cost per turn | ~\$0.85 |
|
||||
| Anthropic prompt-cache hit rate | ~88% |
|
||||
|
||||
A "turn" here is one agent loop iteration: read user message, plan, execute tool calls (including gbrain searches), generate response. Each turn typically includes 2-4 gbrain searches.
|
||||
|
||||
**Per-mode scaling from the tokenmax anchor:**
|
||||
|
||||
The cost difference between modes is concentrated in the search-attributable fraction of per-turn cost. System prompt, tool definitions, conversation history, and reasoning tokens don't change with mode — only the chunks gbrain delivers do. Assume 3 searches per turn at the mode's `searchLimit`:
|
||||
|
||||
| Mode | Search tokens/turn | Search cost/turn (at \$3/M effective) | Search-attributable @ 860 turns | Δ vs tokenmax |
|
||||
|---|---|---|---|---|
|
||||
| tokenmax | ~60K (3 × 20K) | ~\$0.18 | ~\$155/mo | — |
|
||||
| balanced | ~30K (3 × 10K) | ~\$0.09 | ~\$77/mo | -\$78 |
|
||||
| conservative | ~12K (3 × 4K) | ~\$0.036 | ~\$31/mo | -\$124 |
|
||||
|
||||
**Implied total agent spend by NATURAL PAIRING** (mode + matched
|
||||
downstream model). Per-turn cost scales with the downstream model's
|
||||
per-token rate, since the cached prefix + uncached portion + reasoning
|
||||
tokens all bill at that rate:
|
||||
|
||||
| Pairing | Per-turn cost | Total @ 860 turns/mo |
|
||||
|---|---|---|
|
||||
| tokenmax + Opus (frontier, max quality) | ~\$0.85 | ~\$700/mo |
|
||||
| balanced + Sonnet (the sweet spot) | ~\$0.50 | ~\$430/mo |
|
||||
| conservative + Haiku (cost-sensitive) | ~\$0.20 | ~\$170/mo |
|
||||
|
||||
**4x spread across natural pairings.** The model tier dominates because
|
||||
the per-token rate applies to the WHOLE per-turn payload (system + tools
|
||||
+ history + reasoning + search), not just gbrain's chunks. Mode choice
|
||||
contributes ~10-20% on top of that base.
|
||||
|
||||
**Mismatched pairings push you off the curve:**
|
||||
|
||||
| Pairing | Per-turn estimate | Total @ 860 turns/mo | Compared to natural |
|
||||
|---|---|---|---|
|
||||
| tokenmax + Haiku | ~\$0.20 | ~\$170/mo | Same cost as conservative+Haiku, worse quality |
|
||||
| conservative + Opus | ~\$0.75 | ~\$640/mo | 92% of tokenmax+Opus spend, conservative-shape retrieval |
|
||||
|
||||
The mismatch math says: a tokenmax+Haiku user pays the same as
|
||||
conservative+Haiku but gets a noisier context (Haiku can't filter signal
|
||||
from 50 chunks). A conservative+Opus user pays nearly the same as
|
||||
tokenmax+Opus but starves Opus on retrieval depth. Both burn budget for
|
||||
no improvement.
|
||||
|
||||
**What this anchor tells us that the per-query math doesn't:**
|
||||
|
||||
1. **At realistic agent-loop scale with disciplined prompt caching, mode choice saves 10-20% of total agent spend** — meaningful, but smaller than the per-query 5x ratio implies. Disciplined prompt-cache layouts blunt the mode delta because most of the per-turn cost is the cached prefix, not the search payload.
|
||||
|
||||
2. **Without that prompt-cache discipline, the per-query framing reasserts itself.** Setups that churn the prompt prefix on every turn (frequent system-prompt edits, untemplated tool defs, no prompt-cache structuring) see search payload contribute a much larger fraction of total cost. Those setups should care about mode choice more, not less.
|
||||
|
||||
3. **The cache hit rate quoted here (~88%) is achievable but not automatic.** It requires structuring the prompt so the cached prefix stays stable across turns: system prompt + tool defs first, history compacted but cache-aware, retrieved chunks appended LAST (where their volatility doesn't invalidate the prefix). Agents that interleave search results inside the cached region pay the prefix-rebuild tax on every turn.
|
||||
|
||||
**Caveats stacked here:**
|
||||
|
||||
- The anchor represents ONE power-user loop. Multi-user fleets aggregate proportionally; the per-user shape doesn't change.
|
||||
- The "3 searches per turn" assumption varies wildly. A code-review agent might issue 10+ searches per turn; a chat-only loop might do 0.
|
||||
- The 88% cache hit rate is the high end of what's achievable. Half that is closer to a default agent without cache-aware prompt layout.
|
||||
- The "Δ vs tokenmax" math assumes the OTHER cost components (system, tools, history, reasoning) stay constant. In practice, conservative's smaller per-turn payload also leaves more room in the context window for history → which can change agent behavior in either direction.
|
||||
|
||||
This anchor + the per-query math both live in this doc on purpose. The per-query framing is what an isolated benchmark would measure (and what `gbrain eval run-all` will produce). The realistic-scale anchor is what an operator actually pays. Both are honest; neither is the whole truth.
|
||||
|
||||
## Reproducibility footer
|
||||
|
||||
Every release that publishes eval numbers includes a footer with:
|
||||
|
||||
- Code commit SHA
|
||||
- Dataset SHA (LongMemEval, BrainBench, Replay)
|
||||
- `--seed N`
|
||||
- Run commands verbatim
|
||||
- API model identifiers used (Anthropic + OpenAI + judge model)
|
||||
|
||||
Without these, the numbers are unfalsifiable. With them, anyone with API keys can re-score.
|
||||
@@ -1,199 +0,0 @@
|
||||
# How a downstream agent should talk to gbrain
|
||||
|
||||
This guide is for authors of downstream agents (hermes, openclaw, future
|
||||
forks) that need to call gbrain operations from their own runtime. Reading
|
||||
this first will save you a debugging cycle: gbrain has **two distinct
|
||||
surfaces**, and which one you pick depends on the operation.
|
||||
|
||||
## The two surfaces
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ gbrain process │
|
||||
│ │
|
||||
Agent (hermes, │ ┌──────────────────┐ ┌────────────────┐ │
|
||||
openclaw, fork) ────┼──▶ MCP ops surface │ │ localOnly │ │
|
||||
│ │ (HTTP + OAuth) │ │ admin ops │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ search, query, │ │ sync, embed, │ │
|
||||
│ │ put_page, │ │ dream, doctor,│ │
|
||||
│ │ get_page, │ │ autopilot, │ │
|
||||
│ │ find_experts, │ │ init, secrets │ │
|
||||
│ │ ... │ │ │ │
|
||||
│ └──────────────────┘ └────────────────┘ │
|
||||
│ ▲ ▲ │
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
│ thin-client OAuth shell-job `inherit:`│
|
||||
│ (preferred for (only path for │
|
||||
│ MCP-equivalent ops) localOnly ops) │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The two surfaces are **not interchangeable**. Pick by op, not by preference.
|
||||
|
||||
## Surface 1 — MCP ops over HTTP (thin-client + OAuth)
|
||||
|
||||
Use for any operation that has an MCP equivalent: `search`, `query`,
|
||||
`put_page`, `get_page`, `find_experts`, `find_orphans`, `find_anomalies`,
|
||||
`get_recent_salience`, `find_trajectory`, and so on. The canonical list is
|
||||
the set of ops in `src/core/operations.ts` whose `localOnly` flag is unset
|
||||
(or `false`).
|
||||
|
||||
### Setup
|
||||
|
||||
The host runs gbrain as a long-lived HTTP server:
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain serve --http --port 3131
|
||||
```
|
||||
|
||||
The agent registers as an OAuth client (one-time):
|
||||
|
||||
```bash
|
||||
gbrain auth register-client hermes \
|
||||
--grant-types client_credentials \
|
||||
--scopes read,write
|
||||
# Prints client_id + client_secret one-time. Store securely.
|
||||
```
|
||||
|
||||
The agent's runtime calls `/mcp` with a bearer token from `client_credentials`
|
||||
grant. Secrets stay in the gbrain serve process; the agent never sees
|
||||
DATABASE_URL or API keys.
|
||||
|
||||
Thin-client mode (`gbrain init --mcp-only`) gives the agent the same
|
||||
client-credentials wiring, plus the `gbrain` CLI itself routes MCP-eligible
|
||||
commands through the configured remote MCP. The agent can call
|
||||
`gbrain search` / `gbrain query` directly and the CLI does the OAuth dance.
|
||||
|
||||
### Why this is preferred for MCP ops
|
||||
|
||||
- Secrets never leave the server process.
|
||||
- OAuth scopes give you `read`, `write`, `admin` separation — agent only gets
|
||||
what it needs.
|
||||
- Source-scoped tokens (`--source dept-x` on `register-client`) confine the
|
||||
agent to a specific source within a federated brain.
|
||||
- One audit surface (`mcp_request_log`) covers every op call uniformly.
|
||||
|
||||
## Surface 2 — localOnly admin ops via shell-job `inherit:`
|
||||
|
||||
Some operations are flagged `localOnly: true` in `src/core/operations.ts` and
|
||||
are **refused** in thin-client mode at `src/cli.ts:isThinClient`. The full
|
||||
list (as of v0.36.5.0) includes:
|
||||
|
||||
- `sync` (filesystem walks need local FS access)
|
||||
- `embed` (orchestrates the embed pipeline)
|
||||
- `extract` (walks markdown files)
|
||||
- `dream` (synthesis cycle)
|
||||
- `doctor` (filesystem hygiene checks)
|
||||
- `autopilot` (background daemon orchestration)
|
||||
- `init` (creates `~/.gbrain/`)
|
||||
- `secrets` (config management)
|
||||
|
||||
For these, the agent cannot route through HTTP MCP. The only path is to run
|
||||
`gbrain` as a CLI subprocess. The recommended pattern is to submit the
|
||||
subprocess as a shell job to the gbrain Minions worker so retry / backoff /
|
||||
DLQ / audit trail all come for free.
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
gbrain jobs submit shell --params '{
|
||||
"cmd": "gbrain sync --skip-failed && gbrain embed --stale",
|
||||
"cwd": "/data/gbrain",
|
||||
"inherit": ["database_url"]
|
||||
}'
|
||||
```
|
||||
|
||||
The `inherit: ["database_url"]` field tells the worker to look up
|
||||
`database_url` from its `loadConfig()` and inject the value into the child
|
||||
env as `GBRAIN_DATABASE_URL`. The DB row in `minion_jobs.data` carries the
|
||||
names only — `inherit: ["database_url"]` — never the value. See
|
||||
[minions-shell-jobs.md#secrets](./minions-shell-jobs.md#secrets) for the
|
||||
full validation rules and error catalog.
|
||||
|
||||
### Why this is preferred over writing secrets into `env:` per-job
|
||||
|
||||
- Pre-v0.36.5.0 callers passed `env: { GBRAIN_DATABASE_URL: "postgresql://..." }`
|
||||
per job. The URL landed plaintext in `minion_jobs.data` and the shell-audit
|
||||
JSONL. Anyone with brain-DB read access (or a brain dump, or a shared brain
|
||||
via mounts) saw the URL. As of v0.36.5.0, this is rejected at pre-enqueue
|
||||
validation. The error message names `inherit: ["database_url"]` as the
|
||||
replacement.
|
||||
|
||||
### Worker setup (one-time, per host)
|
||||
|
||||
The agent's host needs a worker that processes shell jobs:
|
||||
|
||||
```bash
|
||||
# One-shot inline execution (PGLite or Postgres):
|
||||
gbrain jobs submit shell --params '{...}' --follow
|
||||
|
||||
# Persistent worker (Postgres only — PGLite uses --follow inline):
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work
|
||||
```
|
||||
|
||||
`GBRAIN_ALLOW_SHELL_JOBS=1` is the worker-side opt-in. Without it, shell jobs
|
||||
sit in `waiting` indefinitely. Set it on the worker process env (or in your
|
||||
deploy unit / launchd plist), not per-submission — submitter env is a weak
|
||||
proxy for worker env.
|
||||
|
||||
## Decision table
|
||||
|
||||
| Operation | Surface | Why |
|
||||
|---|---|---|
|
||||
| `search` / `query` | HTTP MCP via thin-client | Has MCP op; OAuth-scoped. |
|
||||
| `get_page` / `list_pages` | HTTP MCP | Same. |
|
||||
| `put_page` | HTTP MCP | Same; respects subagent allow-list when applicable. |
|
||||
| `find_experts` / `find_orphans` | HTTP MCP | Same. |
|
||||
| `sync` / `embed` / `extract` | Shell job + `inherit:` | `localOnly: true`. |
|
||||
| `dream` | Shell job + `inherit:` | `localOnly: true`. |
|
||||
| `doctor` | Shell job + `inherit:` (or no inherit if no DB) | `localOnly: true`. |
|
||||
| `autopilot` | Run as a daemon directly on the host | Long-lived, not job-shaped. |
|
||||
| `init` / `secrets` | One-time host setup | Operator action, not agent action. |
|
||||
|
||||
## Recommended patterns
|
||||
|
||||
- **Prefer `inherit:` for secrets you don't want in the row.** Names land in
|
||||
`minion_jobs.data`; values resolve at child-spawn from the worker's config.
|
||||
If a brain DB ever traverses a trust boundary, secrets stay out.
|
||||
- **Free-form names.** `inherit:` accepts any snake_case config-key on your
|
||||
worker — `database_url`, `anthropic_api_key`, `openai_api_key`,
|
||||
`voyage_api_key`, `groq_api_key`, `zeroentropy_api_key`, or any custom
|
||||
field you stuff into `~/.gbrain/config.json`. The agent picks what it
|
||||
needs.
|
||||
- **`env:` still works** for non-secret values, or for cases where you
|
||||
WANT the value in the row (e.g. an opaque correlation token your audit
|
||||
flow needs to read back later). The validator doesn't second-guess you.
|
||||
- **Never try to route a `localOnly` op through thin-client MCP.** It will
|
||||
fail with `localOnly op refused in thin-client mode`. Use shell-job +
|
||||
`inherit:` (for secrets) or `env:` (for non-secrets).
|
||||
|
||||
## Migration: from pre-v0.36.5.0
|
||||
|
||||
If your agent submits shell jobs that pass secrets via `env:`:
|
||||
|
||||
```jsonc
|
||||
// Pre-v0.36.5.0: works but URL persists in minion_jobs.data plaintext.
|
||||
{
|
||||
"cmd": "gbrain sync --skip-failed",
|
||||
"cwd": "/data/gbrain",
|
||||
"env": { "GBRAIN_DATABASE_URL": "postgresql://..." }
|
||||
}
|
||||
```
|
||||
|
||||
Switch to (recommended):
|
||||
|
||||
```jsonc
|
||||
// v0.36.5.0+: name in row, value resolved at child-spawn from worker config.
|
||||
{
|
||||
"cmd": "gbrain sync --skip-failed",
|
||||
"cwd": "/data/gbrain",
|
||||
"inherit": ["database_url"]
|
||||
}
|
||||
```
|
||||
|
||||
Make sure the worker host has `database_url` configured (either via
|
||||
`gbrain config set database_url <value>` or via `GBRAIN_DATABASE_URL` /
|
||||
`DATABASE_URL` env on the worker process). If the worker can't resolve the
|
||||
key, the validator rejects the job at submit time with a paste-ready hint.
|
||||
@@ -46,13 +46,10 @@ pass:
|
||||
**What the env allowlist does AND does not do.** Shell jobs run with a minimal
|
||||
env: `PATH, HOME, USER, LANG, TZ, NODE_ENV`. Your secrets like `OPENAI_API_KEY`
|
||||
and `DATABASE_URL` are NOT passed to the child. You opt-in additional keys per
|
||||
job via `env: { ... }` (non-secret values only — see "Secrets" below) or via
|
||||
`inherit: ["database_url"]` (recommended for secrets — names only in the row,
|
||||
values resolved at child-spawn from `gbrain config set`). This stops accidental
|
||||
`$OPENAI_API_KEY` interpolation in a user-authored script. It does **not**
|
||||
sandbox filesystem reads: a shell script can `cat ~/.env` or any file the
|
||||
worker process can read. The operator picks a safe `cwd`. That is the trust
|
||||
boundary.
|
||||
job via `env: { ... }`. This stops accidental `$OPENAI_API_KEY` interpolation in
|
||||
a user-authored script. It does **not** sandbox filesystem reads: a shell
|
||||
script can `cat ~/.env` or any file the worker process can read. The operator
|
||||
picks a safe `cwd`. That is the trust boundary.
|
||||
|
||||
**Audit trail, not forensic insurance.** Every submission writes a JSONL line
|
||||
to `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override
|
||||
@@ -109,115 +106,6 @@ Note: `--follow` blocks the crontab slot until the job finishes. If 14 shell
|
||||
crons land at the same minute and each takes 30s, they serialize through
|
||||
crontab's spawning limits. Postgres + persistent worker scales better.
|
||||
|
||||
### Calling `gbrain` itself from a shell job — use `inherit:` for DATABASE_URL {#secrets}
|
||||
|
||||
A common pattern is submitting shell jobs that run `gbrain` CLI commands:
|
||||
|
||||
```bash
|
||||
gbrain jobs submit shell --params '{
|
||||
"cmd": "gbrain sync --skip-failed && gbrain embed --stale",
|
||||
"cwd": "/data/gbrain",
|
||||
"inherit": ["database_url"]
|
||||
}'
|
||||
```
|
||||
|
||||
`inherit: ["database_url"]` tells the worker to look up `database_url` from its
|
||||
own `loadConfig()` (file + env merged) and inject the value into the child's
|
||||
env as `GBRAIN_DATABASE_URL`. The job row in `minion_jobs.data` stores
|
||||
`inherit: ["database_url"]` — **names only, never values**. The shell-audit
|
||||
JSONL records the same. Pre-enqueue validation rejects the submission if the
|
||||
worker can't resolve the requested key, with a paste-ready
|
||||
`gbrain config set database_url <value>` hint.
|
||||
|
||||
**Why not just write the URL into `env:` directly?** Pre-v0.36.5.0 callers
|
||||
wrote things like:
|
||||
|
||||
```jsonc
|
||||
// ❌ Deprecated as of v0.36.5.0 — REJECTED at submit time.
|
||||
{
|
||||
"cmd": "gbrain stats",
|
||||
"cwd": "/data/gbrain",
|
||||
"env": { "GBRAIN_DATABASE_URL": "postgresql://..." }
|
||||
}
|
||||
```
|
||||
|
||||
This planted plaintext secrets in `minion_jobs.data` (DB row) and in the
|
||||
shell-audit JSONL. Anyone with read access to the brain DB (or a brain dump,
|
||||
or a shared brain via the mounts feature) saw the URL. v0.36.5.0 doesn't
|
||||
forbid that pattern — the validator trusts the agent — but **prefer
|
||||
`inherit:`** for any secret you want kept out of the row. Names land in the
|
||||
row; values resolve at child-spawn from the worker's config.
|
||||
|
||||
**Scope:** v0.36.5.0 `inherit:` is **free-form**. Pass any snake_case
|
||||
config-key name and the worker resolves the value from `loadConfig()` at
|
||||
child-spawn time:
|
||||
|
||||
- `inherit: ["database_url"]` → child env `GBRAIN_DATABASE_URL`
|
||||
- `inherit: ["anthropic_api_key"]` → child env `ANTHROPIC_API_KEY`
|
||||
- `inherit: ["openai_api_key"]` → child env `OPENAI_API_KEY`
|
||||
- `inherit: ["voyage_api_key"]` → child env `VOYAGE_API_KEY`
|
||||
- `inherit: ["groq_api_key", "zeroentropy_api_key"]` → both injected
|
||||
- Or any arbitrary config-key your worker has (`my_custom_field` →
|
||||
`MY_CUSTOM_FIELD`)
|
||||
|
||||
The env-key name is derived by uppercasing the config-key name. The one
|
||||
override is `database_url` → `GBRAIN_DATABASE_URL` (plain `DATABASE_URL` is
|
||||
ambiguous in most Postgres-app contexts).
|
||||
|
||||
Pre-enqueue validation fail-fasts if the worker can't resolve a requested
|
||||
name. The validator does NOT police which secrets you choose to inherit —
|
||||
the agent submitting the minion is in the same uid as the worker, so it's
|
||||
your call.
|
||||
|
||||
**Output-side leakage (read this).** The `inherit:` allowlist prevents
|
||||
secrets from landing in the JOB ROW INPUT fields (`data.cmd`, `data.argv`,
|
||||
`data.env`). By default it does NOT scrub the OUTPUT fields — if your
|
||||
script prints the secret to stdout or stderr (`echo "$GBRAIN_DATABASE_URL"`,
|
||||
`psql "$GBRAIN_DATABASE_URL"` echoing the URL on error), the value lands
|
||||
plaintext in `result.stdout_tail` / `result.stderr_tail` / `error_text`,
|
||||
and from there into the brain DB row.
|
||||
|
||||
**`redact_secrets: true` opts into output-side scrubbing.** Set it per-job
|
||||
(or pass `--redact-secrets` on the CLI):
|
||||
|
||||
```bash
|
||||
gbrain jobs submit shell --params '{
|
||||
"cmd": "gbrain sync --skip-failed",
|
||||
"cwd": "/data/gbrain",
|
||||
"inherit": ["database_url"],
|
||||
"redact_secrets": true
|
||||
}'
|
||||
|
||||
# Or, equivalently:
|
||||
gbrain jobs submit shell \
|
||||
--params '{"cmd":"gbrain sync --skip-failed","cwd":"/data/gbrain","inherit":["database_url"]}' \
|
||||
--redact-secrets
|
||||
```
|
||||
|
||||
When `redact_secrets: true`, the worker resolves each name in `inherit:` to
|
||||
a value, runs the child, then string-replaces every occurrence of those
|
||||
values in `stdout_tail` / `stderr_tail` (and in the `error_text` derived from
|
||||
`stderr_tail` on non-zero exit) with `<REDACTED:name>` before persistence.
|
||||
Only `inherit:`-resolved values are scrubbed; caller-supplied `env:` values
|
||||
are not (those are the "I'm fine with this in the row" channel by design).
|
||||
|
||||
**Heuristic, not perfect.** The redactor uses literal string-replace. A
|
||||
script that base64-encodes the secret before printing, or that emits it
|
||||
one character at a time, will bypass the scrub. Those are adversarial
|
||||
shapes — the agent + the script are in the same trust domain, so this
|
||||
layer defends against accidental echo (the common case), not deliberate
|
||||
exfiltration.
|
||||
|
||||
**Three rules for shell-job authors who deal with secrets:**
|
||||
|
||||
- **Prefer not to echo secrets at all.** Even with `redact_secrets`, less
|
||||
output means less risk if the redactor ever has an edge-case miss.
|
||||
- **Wrap noisy CLI tools to suppress URLs on error.** `psql --quiet`,
|
||||
`pg_dump --quiet`, or pipe through
|
||||
`2>&1 | sed 's|postgresql://[^@]*@|postgresql://REDACTED@|g'`.
|
||||
- **Inspect with `gbrain jobs get <id>` after a failure** to verify what
|
||||
actually persisted.
|
||||
|
||||
### Submitting with `argv` (no shell interpolation)
|
||||
|
||||
For programmatic callers assembling commands from JSON, use `argv` instead of
|
||||
@@ -273,11 +161,6 @@ gbrain jobs list --status waiting --name shell
|
||||
| `shell: cwd is required and must be an absolute path` | `cwd` must be a string starting with `/`. | Set `cwd` in `--params` to an absolute path. |
|
||||
| `shell: argv must be an array of strings` | `argv` has a non-string entry or isn't an array. | Pass `argv: ["bin","arg1","arg2"]`. |
|
||||
| `shell: env values must all be strings` | `env` has a number/bool/object value. | Stringify: `"env":{"COUNT":"3"}` not `"env":{"COUNT":3}`. |
|
||||
| `shell: inherit must be an array of config-key names` | `inherit` wasn't an array. | Pass `"inherit": ["database_url", ...]`. |
|
||||
| `shell: inherit entries must be non-empty strings` | An element of `inherit` was empty, non-string, or null. | Use snake_case config-key names like `database_url`, `anthropic_api_key`. |
|
||||
| `shell: inherit name "<X>" must match [a-z][a-z0-9_]*` | Name failed snake_case regex (uppercase, leading digit/underscore, special char). | Use the config-key name verbatim — `database_url`, not `DATABASE_URL`. |
|
||||
| `shell: inherit requested "<X>" but worker has no <X> configured` | Worker can't resolve the requested name from `loadConfig()`. | Run `gbrain config set <X> <value>` on the worker host, OR check the config file at `~/.gbrain/config.json`. |
|
||||
| `shell: redact_secrets must be a boolean if set` | Caller passed a non-boolean for `redact_secrets`. | Pass `true` or `false` (or omit). The CLI `--redact-secrets` flag sets it automatically. |
|
||||
| `permission_denied: shell jobs cannot be submitted over MCP` | An MCP client tried to submit a shell job. By design CLI-only. | Submit from CLI or via a trusted operation handler (`ctx.remote === false`). |
|
||||
| `protected job name 'shell' requires CLI or operation-local submitter` | A caller invoked `MinionQueue.add('shell', ...)` without the `trusted` opt-in. | Pass `{ allowProtectedSubmit: true }` as the 4th arg. CLI and `submit_job` do this automatically. |
|
||||
| `aborted: timeout` / `aborted: cancel` / `aborted: shutdown` / `aborted: lock-lost` | The worker's abort signal fired mid-execution. Child got SIGTERM, 5s grace, then SIGKILL. | Expected: timeout / user cancel / deploy restart / stall. Inspect `gbrain jobs get` to see which. |
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
# Skillpacks as scaffolding, not amber
|
||||
|
||||
GBrain v0.33 reshapes `gbrain skillpack` from a package manager into a
|
||||
scaffold + reference library. This guide explains the model and the
|
||||
workflow.
|
||||
|
||||
## Why we changed it
|
||||
|
||||
Pre-v0.33 (the "amber" model):
|
||||
|
||||
- `gbrain skillpack install <name>` copied bundled skills into your
|
||||
workspace AND wrote a managed-block fence into your `RESOLVER.md` /
|
||||
`AGENTS.md` with a `cumulative-slugs="..."` receipt.
|
||||
- Subsequent installs hash-checked every file and refused to overwrite
|
||||
local edits unless you passed `--overwrite-local`.
|
||||
- `gbrain skillpack uninstall` had its own data-loss safeguards (D8
|
||||
receipt gate + D11 content-hash pre-scan) and rebuilt the fence.
|
||||
|
||||
It worked, but it treated personal-AI skills like vendor packages.
|
||||
Users couldn't cleanly fork a skill without the next install fighting
|
||||
them. Every release re-litigated the same managed block. The test
|
||||
surface alone for the managed block was ~1000 lines.
|
||||
|
||||
Skills aren't vendor packages. They're first-class code in your agent
|
||||
repo. You scaffold once, you own them, you fork and edit freely. When
|
||||
gbrain ships a new version, you ask "what changed?" — the agent reads
|
||||
the diff and decides what (if anything) to integrate.
|
||||
|
||||
## The five commands
|
||||
|
||||
### `gbrain skillpack scaffold <name> [--workspace PATH]`
|
||||
|
||||
One-time, additive copy of a bundled skill into your repo. Refuses to
|
||||
overwrite any file that exists. Routing comes from each skill's
|
||||
frontmatter `triggers:` array — gbrain does NOT touch your `RESOLVER.md`
|
||||
or `AGENTS.md` (see "How agents discover scaffolded skills" below).
|
||||
|
||||
```bash
|
||||
cd ~/git/your-agent-repo
|
||||
gbrain skillpack scaffold book-mirror
|
||||
# files in skills/book-mirror/ + (if the skill declares paired source)
|
||||
# src/commands/book-mirror.ts land in your workspace
|
||||
```
|
||||
|
||||
`scaffold --all` copies every bundled skill that's missing. Never
|
||||
prunes.
|
||||
|
||||
If a skill's frontmatter declares paired source files (`sources: [...]`
|
||||
in the SKILL.md YAML head), scaffold copies them too. The partial-state
|
||||
policy handles "skill shipped earlier, gained a paired source later" —
|
||||
scaffold copies the new paired file even when the skill dir already
|
||||
exists.
|
||||
|
||||
### `gbrain skillpack reference <name> [--workspace PATH] [--apply-clean-hunks] [--json]`
|
||||
|
||||
Read-only update lens. Diffs gbrain's bundle against your local copy
|
||||
and emits per-file status (`identical` / `differs` / `missing`) plus
|
||||
unified diffs for any `differs` entries.
|
||||
|
||||
```bash
|
||||
gbrain skillpack reference book-mirror
|
||||
# These files live at <gbrain-path> as reference. Read them and
|
||||
# decide what (if anything) to integrate into your local skills/.
|
||||
# Your local edits are intentional — do not blindly overwrite.
|
||||
#
|
||||
# reference: identical:14 differs:1 missing:0
|
||||
#
|
||||
# differs /your/workspace/skills/book-mirror/SKILL.md
|
||||
# --- a/skills/book-mirror/SKILL.md
|
||||
# +++ b/skills/book-mirror/SKILL.md
|
||||
# @@ -10,3 +10,5 @@
|
||||
# ... unified diff ...
|
||||
```
|
||||
|
||||
`reference --all` sweeps the whole bundle (one-line-per-skill summary).
|
||||
|
||||
`reference <name> --apply-clean-hunks` is the auto-apply path. It
|
||||
parses the diff between gbrain's bundle and your local copy, applies
|
||||
every hunk whose pre-change context matches uniquely. **Two-way merge
|
||||
limitation**: without scaffold-time base tracking (intentionally
|
||||
out-of-scope for v0.33), this cannot distinguish "gbrain changed X"
|
||||
from "you changed X." Applied hunks align everything to gbrain. Use
|
||||
`--dry-run` first to preview, or run plain `reference` to inspect the
|
||||
diff before letting auto-apply touch anything.
|
||||
|
||||
### `gbrain skillpack migrate-fence [--workspace PATH] [--dry-run]`
|
||||
|
||||
One-shot conversion for workspaces on the pre-v0.33 managed-block
|
||||
model. Strips the `<!-- gbrain:skillpack:begin -->` / `end -->`
|
||||
markers and the manifest receipt comment from your resolver file.
|
||||
|
||||
**Preserves every row inside the fence verbatim.** Those rows become
|
||||
user-owned routing the agent can still see during the transition to
|
||||
frontmatter-based discovery.
|
||||
|
||||
```bash
|
||||
cd ~/git/your-agent-repo
|
||||
gbrain skillpack migrate-fence
|
||||
# migrate-fence: fence_stripped
|
||||
# resolver: /your/workspace/skills/RESOLVER.md
|
||||
# fenced slugs: alpha, beta, gamma
|
||||
# already present: alpha, beta
|
||||
# skills copied: gamma (additive — beta and alpha kept their local edits)
|
||||
```
|
||||
|
||||
Idempotent. Re-running after migration finds no fence and exits 0.
|
||||
|
||||
### `gbrain skillpack scrub-legacy-fence-rows [--workspace PATH] [--dry-run]`
|
||||
|
||||
Opt-in cleanup. Once you've confirmed your agent walks frontmatter
|
||||
`triggers:` for routing, this command removes the legacy rows that
|
||||
`migrate-fence` left behind.
|
||||
|
||||
**Two-condition gate** (both must hold for a row to be removed):
|
||||
|
||||
1. `skills/<slug>/` exists on host (it was a real scaffold).
|
||||
2. That skill's frontmatter declares non-empty `triggers:` (proof
|
||||
that frontmatter discovery covers this skill).
|
||||
|
||||
Rows whose slug fails either gate are preserved — user-owned routing
|
||||
the migration shouldn't touch.
|
||||
|
||||
### `gbrain skillpack harvest <slug> --from <host-repo-root> [--no-lint] [--dry-run]`
|
||||
|
||||
Inverse of scaffold: lifts a proven skill from your host repo back
|
||||
into gbrain so other clients can scaffold it. Default behavior:
|
||||
|
||||
- Symlinks in the host skill dir are rejected (canonical-path
|
||||
confinement).
|
||||
- Privacy linter scans the harvested files against
|
||||
`~/.gbrain/harvest-private-patterns.txt` plus built-in defaults
|
||||
(canonical private fork name, common email regex, Slack channel pattern). Any
|
||||
match → rollback (delete the harvested files) and exit non-zero.
|
||||
- `openclaw.plugin.json` updated with the new slug, sorted.
|
||||
- `--no-lint` bypasses the linter (after a manual editorial scrub).
|
||||
|
||||
Use the `skillpack-harvest` skill (its companion editorial workflow)
|
||||
to walk the genericization checklist before running the CLI.
|
||||
|
||||
## How agents discover scaffolded skills
|
||||
|
||||
Routing under the new model lives entirely in each skill's frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: book-mirror
|
||||
triggers:
|
||||
- "personalized version of this book"
|
||||
- "mirror this book"
|
||||
- "two-column book analysis"
|
||||
---
|
||||
```
|
||||
|
||||
Your agent's job at runtime is to walk `skills/*/SKILL.md`, parse the
|
||||
frontmatter, and match the user's intent against every skill's
|
||||
`triggers:` array. When a match scores high enough, invoke that skill.
|
||||
|
||||
This replaces the v0.32 model where `gbrain skillpack install` wrote
|
||||
table rows into your `RESOLVER.md`. Rows are gone (or, for users
|
||||
migrating from the old model, preserved transitionally by
|
||||
`migrate-fence` until they run `scrub-legacy-fence-rows`).
|
||||
|
||||
If you're a downstream agent author updating to this model:
|
||||
|
||||
1. On startup, scan `skills/*/SKILL.md` for frontmatter.
|
||||
2. Build an in-memory routing table from each skill's `triggers:`
|
||||
array.
|
||||
3. On every user message, match against this table — either by
|
||||
substring containment, semantic similarity, or whatever your
|
||||
downstream agent already does for intent classification.
|
||||
|
||||
## Removing a scaffolded skill
|
||||
|
||||
There's no `gbrain skillpack uninstall` command in v0.33. The files
|
||||
in your `skills/<slug>/` are first-class members of your repo —
|
||||
delete them like any other code:
|
||||
|
||||
```bash
|
||||
rm -rf skills/book-mirror
|
||||
# if the skill declared paired source files:
|
||||
rm src/commands/book-mirror.ts
|
||||
# (consult the skill's frontmatter `sources:` array for the full list)
|
||||
|
||||
# if no other scaffolded skill needs them, you can also remove the
|
||||
# shared deps that scaffold drops in:
|
||||
rm skills/_brain-filing-rules.md
|
||||
rm -rf skills/conventions/
|
||||
rm skills/_output-rules.md
|
||||
```
|
||||
|
||||
You own the files. There's no manifest to update, no fence to rebuild.
|
||||
|
||||
## When to use which command (quick decision tree)
|
||||
|
||||
- **New host repo, want a gbrain skill** → `scaffold`
|
||||
- **gbrain shipped a new version, want to see what's changed**
|
||||
→ `reference` (read-only) or `reference --apply-clean-hunks` (auto)
|
||||
- **Upgrading from v0.32 or earlier** → `migrate-fence` (one-shot)
|
||||
- **Cleanup after `migrate-fence`** → `scrub-legacy-fence-rows`
|
||||
- **Lift your fork's skill back into gbrain** → `harvest` + the
|
||||
`skillpack-harvest` editorial skill
|
||||
|
||||
## What about `install` and `uninstall`?
|
||||
|
||||
Both are removed in v0.33. Running either prints an error pointing at
|
||||
the replacement command. No deprecated alias — this is a clean break.
|
||||
If you have existing scripts referencing the old names, update them
|
||||
once and move on.
|
||||
@@ -1,130 +0,0 @@
|
||||
# Embedding providers
|
||||
|
||||
GBrain ships with 14 embedding-provider recipes covering OpenAI, the major hosted alternatives, three local options, and a universal escape hatch (LiteLLM proxy). Run `gbrain providers list` to see the live registry; `gbrain providers explain --json` emits a machine-readable matrix for agents.
|
||||
|
||||
This page is the human-readable counterpart: capability per provider, env-var setup, dimensions, cost, and known constraints.
|
||||
|
||||
## Quick start
|
||||
|
||||
```
|
||||
gbrain providers list # see all providers
|
||||
gbrain providers env <provider-id> # see required env vars
|
||||
gbrain providers test --model openai:text-embedding-3-large # smoke-test
|
||||
gbrain init --pglite --model voyage # use a non-default provider
|
||||
```
|
||||
|
||||
## TL;DR table
|
||||
|
||||
| Provider | env vars | default dims | cost ($/1M tokens) | local? | multimodal? |
|
||||
|---|---|---|---|---|---|
|
||||
| `openai` | `OPENAI_API_KEY` | 1536 | 0.13 | no | no |
|
||||
| `voyage` | `VOYAGE_API_KEY` | 1024 | 0.18 | no | yes (`voyage-multimodal-3`) |
|
||||
| `google` | `GOOGLE_GENERATIVE_AI_API_KEY` | 768 | 0.025 | no | no |
|
||||
| `azure-openai` | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT` | 1536 | 0.13 | no | no |
|
||||
| `minimax` | `MINIMAX_API_KEY` | 1536 | 0.07 | no | no |
|
||||
| `dashscope` | `DASHSCOPE_API_KEY` | 1024 | varies | no | no |
|
||||
| `zhipu` | `ZHIPUAI_API_KEY` | 1024 | varies | no | no |
|
||||
| `ollama` | (none — runs locally) | 768 | 0 | yes | no |
|
||||
| `llama-server` | (none — runs locally) | user-set | 0 | yes | no |
|
||||
| `litellm` | `LITELLM_API_KEY` (optional) | user-set | varies | yes (proxy) | no |
|
||||
| `together` | `TOGETHER_API_KEY` | 768 | varies | no | no |
|
||||
| `anthropic` | (no embedding model — chat only) | — | — | — | — |
|
||||
| `deepseek` | (no embedding model — chat only) | — | — | — | — |
|
||||
| `groq` | (no embedding model — chat only) | — | — | — | — |
|
||||
|
||||
## Decision tree
|
||||
|
||||
- **Cost-sensitive, English-only**: Ollama (free, local) or Voyage (paid, best quality per dollar).
|
||||
- **Quality-first**: Voyage `voyage-4-large` (1024-2048 dims, ~3-4× more dense tokens than OpenAI tiktoken).
|
||||
- **Reranking pair**: Voyage (their reranker `rerank-2.5` pairs cleanly with Voyage embeddings).
|
||||
- **Enterprise compliance**: Azure OpenAI (data residency + private endpoints) or self-hosted via llama-server / Ollama.
|
||||
- **China region**: DashScope (Alibaba) or Zhipu (BigModel). DashScope's international endpoint at `dashscope-intl.aliyuncs.com`; override `provider_base_urls.dashscope` for the China endpoint.
|
||||
- **OSS local, full control**: llama-server (`llama.cpp`) for any GGUF model; Ollama for the curated catalog.
|
||||
- **Anything else**: LiteLLM proxy. Run LiteLLM in front of any provider (Bedrock, Vertex, Cohere, Jina, Fireworks, etc.) and point gbrain at it via `LITELLM_BASE_URL`.
|
||||
|
||||
## Per-provider details
|
||||
|
||||
### OpenAI
|
||||
|
||||
Default. Set `OPENAI_API_KEY`. Models: `text-embedding-3-large` (3072 max, 1536 default), `text-embedding-3-small` (1536). Matryoshka via the `dimensions` field — gbrain pins it from `embedding_dimensions` config so existing 1536-dim brains stay aligned across SDK upgrades.
|
||||
|
||||
### Voyage AI
|
||||
|
||||
Best-in-class quality on the Voyage 4 family (Jan 2026 release). Set `VOYAGE_API_KEY`. Models: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-4-nano`, `voyage-3.5`, `voyage-code-3` (code-tuned), `voyage-finance-2`, `voyage-law-2`, `voyage-multimodal-3` (text + image).
|
||||
|
||||
Voyage 4 family shares an embedding space across all variants, so you can index with `voyage-4-large` and query with `voyage-4-lite` without reindexing. Dims: 256, 512, 1024, 2048. **2048 exceeds pgvector's HNSW cap of 2000** — those brains fall back to exact vector scans (still correct, just slower).
|
||||
|
||||
### Google Gemini
|
||||
|
||||
Set `GOOGLE_GENERATIVE_AI_API_KEY` (the AI Studio public API key). Model: `gemini-embedding-001`. Default 768 dims; Matryoshka up to 3072. Cheap.
|
||||
|
||||
For GCP service-account / Vertex AI auth (production deployments), see the v0.32.x follow-up — Vertex ADC is on the roadmap.
|
||||
|
||||
### Azure OpenAI
|
||||
|
||||
Enterprise OpenAI behind Azure tenancy. Required env: `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` (e.g. `https://my-resource.openai.azure.com`), `AZURE_OPENAI_DEPLOYMENT` (the deployment name from your Azure portal). Optional: `AZURE_OPENAI_API_VERSION` (defaults to `2024-10-21`).
|
||||
|
||||
Unlike vanilla OpenAI, Azure uses `api-key:` header (not `Authorization: Bearer`) and a templated URL with `?api-version=` query param — gbrain handles both via the recipe's resolveAuth + resolveOpenAICompatConfig overrides.
|
||||
|
||||
Models: `text-embedding-3-large`, `text-embedding-3-small`, `text-embedding-ada-002` (your Azure deployment must serve the requested model).
|
||||
|
||||
### MiniMax (海螺AI)
|
||||
|
||||
Set `MINIMAX_API_KEY`. Optional `MINIMAX_GROUP_ID` for org-scoped accounts. Model: `embo-01` (1536 dims).
|
||||
|
||||
MiniMax's API takes a `type: 'db' | 'query'` field for asymmetric retrieval. v0.32 routes everything as `type='db'` (symmetric retrieval — same vector space for indexing and queries). Asymmetric query support is a v0.32.x follow-up.
|
||||
|
||||
### DashScope (Alibaba)
|
||||
|
||||
Set `DASHSCOPE_API_KEY`. International endpoint at `dashscope-intl.aliyuncs.com` by default; override `provider_base_urls.dashscope` for the China endpoint. Models: `text-embedding-v3` (current; Matryoshka 64-1024 dims), `text-embedding-v2`.
|
||||
|
||||
CJK-dominant content tokenizes denser than OpenAI tiktoken; gbrain declares `chars_per_token: 2` so the batch pre-split leaves headroom.
|
||||
|
||||
### Zhipu AI (BigModel)
|
||||
|
||||
Set `ZHIPUAI_API_KEY`. Models: `embedding-3` (current; Matryoshka 256-2048 dims), `embedding-2`. v0.32 default is 1024 (HNSW-compatible). The 2048-dim option works but falls into the exact-scan branch (see Voyage 4 Large note above).
|
||||
|
||||
### Ollama (local)
|
||||
|
||||
No env required — Ollama runs unauthenticated locally. Optional `OLLAMA_BASE_URL` (default `http://localhost:11434/v1`) and `OLLAMA_API_KEY` (for auth-enabled deployments).
|
||||
|
||||
Recipe ships with `nomic-embed-text` (768d, recommended), `mxbai-embed-large` (1024d), `all-minilm` (384d). `gbrain providers test --model ollama:nomic-embed-text` smoke-tests the local install.
|
||||
|
||||
### llama-server (local, llama.cpp)
|
||||
|
||||
`llama.cpp`'s `llama-server --embeddings` endpoint. No env required. Optional `LLAMA_SERVER_BASE_URL` (default `http://localhost:8080/v1`) and `LLAMA_SERVER_API_KEY`.
|
||||
|
||||
User-driven models: launch llama-server with `--model <gguf-path> --embeddings`, then run `gbrain init --embedding-model llama-server:<your-id> --embedding-dimensions <N>`. The recipe refuses the implicit shorthand `--model llama-server` because there's no canonical first model.
|
||||
|
||||
### LiteLLM proxy (universal escape hatch)
|
||||
|
||||
Run [LiteLLM](https://docs.litellm.ai/docs/proxy/quick_start) in front of any provider — Bedrock, Vertex, Cohere, Jina, Fireworks, OctoAI, etc. The proxy normalizes everything to the OpenAI-compatible API; gbrain points at the proxy via `LITELLM_BASE_URL` and proxies the call.
|
||||
|
||||
This is the catch-all for "my provider isn't in the list above." Set up LiteLLM, then `gbrain init --embedding-model litellm:<your-model-id> --embedding-dimensions <N>`.
|
||||
|
||||
## Choosing dimensions
|
||||
|
||||
Three numbers matter:
|
||||
1. **Provider's native dims**: each model has a "true" output dim (e.g. OpenAI `text-embedding-3-large` is 3072 native).
|
||||
2. **Matryoshka reductions**: most modern providers let you request a smaller vector via the `dimensions` field.
|
||||
3. **HNSW cap**: pgvector's HNSW index supports up to 2000 dims. Brains above that fall back to exact vector scans (slower but correct; gbrain handles the SQL automatically via `chunkEmbeddingIndexSql` in `src/core/vector-index.ts`).
|
||||
|
||||
For most users: **stay at 1024 or 1536**. Bigger isn't better below the noise floor; smaller saves disk + RAM with marginal recall loss on Matryoshka providers.
|
||||
|
||||
## My provider isn't listed
|
||||
|
||||
Three options:
|
||||
|
||||
1. **Use LiteLLM proxy** (above) — the universal escape hatch. Works for 100+ providers.
|
||||
2. **Open a feature request** at [github.com/garrytan/gbrain/issues](https://github.com/garrytan/gbrain/issues) with the provider's API docs URL and a setup snippet. Recipes are ~30-40 lines of TypeScript.
|
||||
3. **Submit a recipe**: clone, copy `src/core/ai/recipes/voyage.ts` as the gold-standard openai-compat template, register in `src/core/ai/recipes/index.ts`, add a per-recipe smoke test under `test/ai/recipe-<name>.test.ts`. The recipe contract test (`test/ai/recipes-contract.test.ts`) and IRON RULE regression test pin the structural invariants.
|
||||
|
||||
## Switching providers on an existing brain
|
||||
|
||||
Embedding dimensions are baked into the schema at `gbrain init` time. To change providers post-init, you usually need to re-embed:
|
||||
|
||||
1. Update config: `gbrain config set embedding_model <provider>:<model>` and `embedding_dimensions <N>`.
|
||||
2. Reindex schema if dims changed: `gbrain doctor` will detect the mismatch and print the exact `ALTER TABLE` recipe.
|
||||
3. Re-embed: `gbrain embed --all` (or `--stale` for incremental).
|
||||
|
||||
`gbrain doctor` 8c "alternative_providers" surfaces unconfigured providers whose env is already set — useful when you've configured OpenAI but also have e.g. `VOYAGE_API_KEY` exported and want to know you can switch without extra setup.
|
||||
@@ -1,224 +0,0 @@
|
||||
# Doctor Auto-Heal and Scoring Improvements
|
||||
|
||||
## Summary
|
||||
|
||||
The `gbrain doctor` health score system has several false-positive patterns and missing auto-heal capabilities. After the crash classification fix (shipped in this PR), these are the remaining improvements ranked by impact.
|
||||
|
||||
---
|
||||
|
||||
## 1. Frontmatter severity levels
|
||||
|
||||
### Problem
|
||||
|
||||
`NESTED_QUOTES` warnings dominate the frontmatter check (6,900+ of ~7,100 total issues). These are cosmetic YAML style issues — values like `title: "foo"` where the quotes are technically unnecessary. They don't affect sync, search, embedding, or any functionality.
|
||||
|
||||
By counting them the same as `YAML_PARSE` (actual parse failures) or `MISSING_OPEN` (missing frontmatter delimiters), the frontmatter check is perpetually WARN and the real issues are lost.
|
||||
|
||||
### Evidence
|
||||
|
||||
```
|
||||
frontmatter_integrity: 7131 issues across 3 sources
|
||||
default: 7012 (NESTED_QUOTES=6922, YAML_PARSE=90)
|
||||
media-corpus: 16 (MISSING_OPEN=15, YAML_PARSE=1)
|
||||
zion-brain: 103 (MISSING_OPEN=14, NESTED_QUOTES=89)
|
||||
```
|
||||
|
||||
Only 280 of 7,131 issues are real problems. 96% are cosmetic noise.
|
||||
|
||||
### Proposed Fix
|
||||
|
||||
- Introduce severity levels: `error` (YAML_PARSE, MISSING_OPEN) vs `info` (NESTED_QUOTES)
|
||||
- Doctor WARN/FAIL only on error-level issues
|
||||
- Report info-level in the message text but don't affect check status
|
||||
- Optional `--pedantic` flag includes info-level in status
|
||||
|
||||
### Test Cases
|
||||
|
||||
| Frontmatter issues | Severity breakdown | Expected status |
|
||||
|---|---|---|
|
||||
| 0 issues | n/a | OK |
|
||||
| 50 NESTED_QUOTES only | 0 error, 50 info | OK (with note) |
|
||||
| 3 YAML_PARSE | 3 error | WARN |
|
||||
| 6900 NESTED_QUOTES + 3 YAML_PARSE | 3 error, 6900 info | WARN (mentions 3 errors) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Temporal contradiction awareness
|
||||
|
||||
### Problem
|
||||
|
||||
The contradiction probe flags temporal evolutions as contradictions. Example:
|
||||
|
||||
- Page A (April): "Considering option X"
|
||||
- Page B (May): "Decided on option Y"
|
||||
|
||||
These aren't contradictions — they're the same topic evolving over time. The probe has no time awareness.
|
||||
|
||||
### Evidence
|
||||
|
||||
From a probe run on 50 queries with top-k=15:
|
||||
- 120 contradictions detected (112 high, 8 medium)
|
||||
- After manual review: ~60% were temporal evolutions, not real conflicts
|
||||
- Pages have `effective_date` or `created` timestamps that could disambiguate
|
||||
|
||||
### Proposed Fix
|
||||
|
||||
- Pass `effective_date` / `created` to the judge prompt
|
||||
- Add verdict: `temporal_supersession` (later claim supersedes earlier)
|
||||
- When both pages have dates and claims overlap, bias toward temporal interpretation
|
||||
- Already designed in PR #993
|
||||
|
||||
### Test Cases
|
||||
|
||||
| Page A date | Page A claim | Page B date | Page B claim | Expected verdict |
|
||||
|---|---|---|---|---|
|
||||
| 2026-04 | "Considering X" | 2026-05 | "Chose Y" | temporal_supersession |
|
||||
| 2026-04 | "Revenue is $1M" | 2026-04 | "Revenue is $500K" | contradiction |
|
||||
| null | "X is true" | null | "X is false" | contradiction |
|
||||
| 2025-01 | "CEO of Company" | 2026-01 | "Former CEO" | temporal_supersession |
|
||||
|
||||
---
|
||||
|
||||
## 3. Multi-source drift baseline
|
||||
|
||||
### Problem
|
||||
|
||||
4,791 pages show "multi-source drift" due to a pre-v0.30.3 `putPage` routing bug. These pages exist at the `default` source but should be at a named source. The `sources rehome` command to fix this hasn't shipped yet.
|
||||
|
||||
Every doctor run shows WARN for ~4,800 pages nobody can fix.
|
||||
|
||||
### Proposed Fix
|
||||
|
||||
Allow `doctor.baselines` config to acknowledge known-unfixable counts:
|
||||
|
||||
```yaml
|
||||
doctor:
|
||||
baselines:
|
||||
multi_source_drift: 4800
|
||||
```
|
||||
|
||||
When actual drift ≤ baseline: OK. When drift exceeds baseline: WARN (new drift).
|
||||
|
||||
Store in `.gbrain/doctor-baselines.json` so it works without config too:
|
||||
|
||||
```json
|
||||
{
|
||||
"multi_source_drift": { "count": 4800, "acknowledged_at": "2026-05-15", "reason": "pre-v0.30.3 putPage misroutes" }
|
||||
}
|
||||
```
|
||||
|
||||
### Test Cases
|
||||
|
||||
| Actual drift | Baseline | Expected |
|
||||
|---|---|---|
|
||||
| 4791 | 4800 | OK |
|
||||
| 4900 | 4800 | WARN ("100 new drift beyond baseline") |
|
||||
| 4791 | 0 (no baseline) | WARN (current behavior) |
|
||||
|
||||
---
|
||||
|
||||
## 4. Image assets acknowledgment
|
||||
|
||||
### Problem
|
||||
|
||||
When image files are missing from disk (stored externally, purged from git), the check permanently warns. No way to say "these are intentionally external."
|
||||
|
||||
### Proposed Fix
|
||||
|
||||
- `doctor --acknowledge image_assets` marks current missing count as accepted
|
||||
- Stored in `.gbrain/doctor-baselines.json`
|
||||
- WARN only for NEW missing images beyond acknowledged count
|
||||
- Optional `image_assets.external_storage: true` config to skip disk check entirely
|
||||
|
||||
---
|
||||
|
||||
## 5. Auto-heal mode
|
||||
|
||||
### Problem
|
||||
|
||||
Many doctor warnings have known fixes that are safe to auto-apply:
|
||||
|
||||
| Warning | Auto-fix |
|
||||
|---|---|
|
||||
| Supervisor not running | Start supervisor |
|
||||
| Stale embeddings | Submit `embed --stale` job |
|
||||
| Extract coverage < 70% | Submit `extract all --skip-existing` job |
|
||||
| Stale sync | Submit sync job |
|
||||
| Effective date drift | Run `reindex-frontmatter` |
|
||||
|
||||
### Proposed Fix
|
||||
|
||||
`doctor --auto-heal` mode:
|
||||
|
||||
1. Run all checks
|
||||
2. For fixable WARNs: submit fix as a job (not inline — via job queue)
|
||||
3. Report what was fixed vs needs manual attention
|
||||
4. Idempotent: check queue first, don't submit duplicates
|
||||
5. Safety gate: never auto-heals FAILs, only WARNs
|
||||
|
||||
Config:
|
||||
|
||||
```yaml
|
||||
doctor:
|
||||
autoHeal:
|
||||
enabled: true
|
||||
minInterval: "6h"
|
||||
skip:
|
||||
- image_assets
|
||||
- multi_source_drift
|
||||
```
|
||||
|
||||
### Test Cases
|
||||
|
||||
| Check status | Auto-heal enabled | Job already queued | Expected |
|
||||
|---|---|---|---|
|
||||
| WARN: stale embeds | yes | no | Submit embed job |
|
||||
| WARN: stale embeds | yes | yes | Skip (idempotent) |
|
||||
| FAIL: max_crashes | yes | n/a | Don't auto-fix FAILs |
|
||||
| WARN: stale embeds | no | n/a | Report only |
|
||||
| WARN: image_assets | yes (but skipped) | n/a | Report only |
|
||||
|
||||
---
|
||||
|
||||
## 6. Score delta tracking
|
||||
|
||||
### Problem
|
||||
|
||||
No history — each `doctor` run is a snapshot. Can't tell if score is improving or degrading.
|
||||
|
||||
### Proposed Fix
|
||||
|
||||
- Write each run to `.gbrain/doctor-history.jsonl`:
|
||||
```json
|
||||
{"ts":"2026-05-15T12:00:00Z","score":60,"brain_score":79,"checks":{"supervisor":"ok","embeddings":"ok",...}}
|
||||
```
|
||||
- `doctor --trend` shows last N scores with deltas
|
||||
- `doctor --json` includes `previous_score` and `delta` fields
|
||||
|
||||
---
|
||||
|
||||
## 7. Weighted scoring
|
||||
|
||||
### Problem
|
||||
|
||||
Going from 99% → 100% embed coverage weighs the same as 50% → 51%. But the last percent is the hardest (oversized pages, rate limits).
|
||||
|
||||
### Proposed Fix
|
||||
|
||||
Threshold-based scoring:
|
||||
- 100% = full points
|
||||
- ≥95% = 90% of points
|
||||
- ≥80% = 70% of points
|
||||
- <80% = proportional
|
||||
|
||||
---
|
||||
|
||||
## Priority Order
|
||||
|
||||
1. Frontmatter severity levels (highest noise reduction)
|
||||
2. Temporal contradiction awareness (highest false positive reduction, already designed)
|
||||
3. Auto-heal mode (biggest long-term value)
|
||||
4. Score delta tracking (enables monitoring)
|
||||
5. Multi-source drift baseline (quality of life)
|
||||
6. Image assets acknowledgment (quality of life)
|
||||
7. Weighted scoring (nice to have)
|
||||
@@ -117,24 +117,6 @@ gbrain auth register-client perplexity \
|
||||
--scopes "read write"
|
||||
```
|
||||
|
||||
**v0.34 — source-scoped clients.** Multi-source brains can scope a client's
|
||||
write authority to one source and its read scope to a curated set with the
|
||||
new `--source` and `--federated-read` flags:
|
||||
|
||||
```bash
|
||||
gbrain auth register-client dept-x-agent \
|
||||
--grant-types client_credentials \
|
||||
--scopes "read write" \
|
||||
--source dept-x \
|
||||
--federated-read dept-x,shared,parent-canon
|
||||
```
|
||||
|
||||
`--source` controls the write authority — `put_page` / `add_link` / etc only
|
||||
land in `dept-x`. `--federated-read` controls the read axis independently;
|
||||
queries return rows from any of the listed sources. Omit both flags for the
|
||||
v0.33-compatible super-client shape. Pre-v0.34 clients are backfilled to
|
||||
`source_id='default'` on `gbrain upgrade`.
|
||||
|
||||
Host-repo wrappers can register programmatically:
|
||||
|
||||
```ts
|
||||
@@ -151,18 +133,6 @@ start the server with `--enable-dcr`. DCR is off by default.
|
||||
|
||||
### 3. Expose the server
|
||||
|
||||
**v0.34 — bind explicitly.** `gbrain serve --http` defaults to `127.0.0.1`.
|
||||
To accept connections from the ngrok tunnel (or any non-loopback source),
|
||||
restart with `--bind`:
|
||||
|
||||
```bash
|
||||
gbrain serve --http --port 3131 --bind 0.0.0.0 --public-url https://your-brain.ngrok.app
|
||||
```
|
||||
|
||||
When `--public-url` is set without `--bind`, a stderr WARN fires at
|
||||
startup so the misconfiguration ("the tunnel is up but my agent gets
|
||||
ECONNREFUSED") is loud.
|
||||
|
||||
```bash
|
||||
brew install ngrok
|
||||
ngrok config add-authtoken YOUR_TOKEN
|
||||
|
||||
@@ -1,213 +0,0 @@
|
||||
# Proposal: Temporal Axis for Contradiction Probe
|
||||
|
||||
**Status:** Report / RFC
|
||||
**Date:** 2026-05-14
|
||||
**Context:** A large production run of `gbrain eval suspected-contradictions` surfaced ~115 HIGH findings. Walking through them by hand exposed a structural limitation in the probe.
|
||||
|
||||
## The Problem
|
||||
|
||||
The contradiction probe (`gbrain eval suspected-contradictions`) treats all claims as timeless. When two chunks make conflicting statements, the judge flags a contradiction regardless of whether both statements were true at their respective points in time.
|
||||
|
||||
This worked fine when the brain was mostly static wiki pages. It breaks now that the brain contains:
|
||||
- Conversation transcripts with claims that were true when spoken
|
||||
- Meeting pages capturing what people said on specific dates
|
||||
- Takes that evolve (a founder's ARR claim in January vs. July)
|
||||
- Status records that supersede each other (a state moves from "trial" to "confirmed")
|
||||
|
||||
The probe can't distinguish "this changed" from "this is wrong."
|
||||
|
||||
## Bug-class examples (synthetic placeholders)
|
||||
|
||||
### 1. Temporal Evolution (False Positive)
|
||||
|
||||
```
|
||||
Finding: HIGH
|
||||
A: [daily/transcripts/2026/2026-04-28] "status: trial"
|
||||
B: [meetings/2026-05-07-session] "status: confirmed"
|
||||
Axis: Whether status is trial or confirmed
|
||||
```
|
||||
|
||||
Both are correct as of their respective dates. April 28: trial. May 7: confirmed. The probe flags this because it has no concept of "this claim was valid from X until Y." The May 7 record didn't make the April 28 transcript wrong; it recorded a change.
|
||||
|
||||
### 2. Negation Parsing (False Positive)
|
||||
|
||||
```
|
||||
Finding: HIGH
|
||||
A: [people/alice-example] "person traveled to city-a for alice-example's event — NOT bob-example's event"
|
||||
B: [meetings/2026-05-11-context] mentions of bob-example's event in city-b
|
||||
Axis: Whose event the city-a trip was for
|
||||
```
|
||||
|
||||
The disambiguation fact contains "NOT bob-example's event" as an explicit negation. The judge reads "bob-example's event" as a positive claim and flags it against the alice-example context. The data is correct; the probe can't parse negation.
|
||||
|
||||
### 3. Role Changes (True Positive That Needs Time Awareness)
|
||||
|
||||
```
|
||||
Finding: HIGH
|
||||
A: [sources/notes/2017-03-28] advisor-example: "Partner, venture-firm-a"
|
||||
B: [people/advisor-example] advisor-example: "Senior Policy Advisor, gov-org-b"
|
||||
```
|
||||
|
||||
Both true at their respective times. 2017: partner at venture-firm-a. 2025: gov-org-b advisor. The current probe correctly flags this as a contradiction, but the resolution should be "superseded by time" not "one side is wrong." The 2017 note isn't wrong; it's a historical record.
|
||||
|
||||
## Scenario #1: Founder Tracking (the big one)
|
||||
|
||||
This is the use case that makes a time axis transformative rather than incremental.
|
||||
|
||||
The brain holds hundreds of company pages and thousands of meeting pages. Founders make claims:
|
||||
|
||||
- "We're at $50K MRR" (January OH)
|
||||
- "We hit $200K MRR" (April OH)
|
||||
- "We're at $150K MRR" (July OH — what happened?)
|
||||
|
||||
Today the probe would flag January vs. April as a contradiction. The real signal is April vs. July: **a claimed metric went backwards.** That's not a data quality issue; that's intelligence.
|
||||
|
||||
What a time-aware probe could surface:
|
||||
|
||||
**Claim trajectory tracking:**
|
||||
```
|
||||
Company: Acme Corp
|
||||
2026-01: "$50K MRR" (source: OH transcript)
|
||||
2026-04: "$200K MRR" (source: OH transcript)
|
||||
2026-07: "$150K MRR" (source: OH transcript) ← REGRESSION DETECTED
|
||||
2026-07: "$2M ARR" (source: investor update) ← INCONSISTENT WITH MRR
|
||||
```
|
||||
|
||||
**Prediction vs. outcome:**
|
||||
```
|
||||
Founder: Jane Doe (Acme Corp)
|
||||
2026-01: "We'll hit $1M ARR by June" (source: batch kickoff)
|
||||
2026-06: Actual ARR: $400K (source: investor update)
|
||||
→ Prediction accuracy: 40%
|
||||
→ Pattern: consistently 2-3x optimistic on timeline
|
||||
```
|
||||
|
||||
**Narrative consistency:**
|
||||
```
|
||||
Founder: John Smith (WidgetCo)
|
||||
2026-01: "Our moat is proprietary data" (source: interview)
|
||||
2026-03: "We're pivoting to an API-first model" (source: OH)
|
||||
2026-06: "Our moat is network effects" (source: Demo Day)
|
||||
→ Moat narrative changed 3x in 6 months — flag for review
|
||||
```
|
||||
|
||||
This isn't adversarial. It's the kind of pattern an experienced operator notices intuitively across hundreds of conversations. GBrain can make it systematic.
|
||||
|
||||
## Scenario #2: Event Disambiguation
|
||||
|
||||
Two distinct events within a short window can conflate during ingestion because the probe has no temporal frame to say "event A is a different event from event B."
|
||||
|
||||
Time-aware facts would store (synthetic placeholders):
|
||||
```
|
||||
fact: "alice-example milestone" valid_from: 2026-04-15 valid_until: 2026-04-15
|
||||
fact: "alice-example event in city-a" valid_from: 2026-04-17 valid_until: 2026-04-19
|
||||
fact: "bob-example milestone" valid_from: 2026-05-04 valid_until: 2026-05-04
|
||||
fact: "bob-example event in city-b" valid_from: 2026-05-12 valid_until: 2026-05-12
|
||||
```
|
||||
|
||||
The probe should recognize these as two distinct events with non-overlapping time windows, not as contradictions about "whose event."
|
||||
|
||||
## Scenario #3: Role and Status Changes
|
||||
|
||||
People change roles. Companies change status. The brain records history. Synthetic examples representative of the cases observed in production:
|
||||
|
||||
- advisor-example: venture-firm-a partner (2019) → gov-org-b advisor (2025)
|
||||
- investor-example: fund-a partner → fund-b CEO (2023)
|
||||
- agent-fork: provider restriction event (2026-04-04) ≠ shutdown
|
||||
- fund-c: "interesting fund" (early) → "declined" (later) → "losing confidence" (latest)
|
||||
|
||||
All of these are correct historical records. The probe should classify them as **temporal supersession** rather than **contradiction.**
|
||||
|
||||
## Scenario #4: Decision Tracking
|
||||
|
||||
Multi-step decisions that supersede earlier framings example (synthetic):
|
||||
```
|
||||
2026-04-24: "status: trial" (initial framing)
|
||||
2026-04-25: "status: in progress" (confirmed, no longer "trial")
|
||||
2026-05-07: "status: finalized" (session record)
|
||||
2026-05-11: follow-up actions taken
|
||||
```
|
||||
|
||||
Each step supersedes the previous. A time-aware probe would show the **evolution chain** rather than flagging each pair as a contradiction.
|
||||
|
||||
## What Exists Today
|
||||
|
||||
The probe already has some temporal infrastructure:
|
||||
|
||||
1. **`date-filter.ts`** — `shouldSkipForDateMismatch()` pre-filters pairs, but only checks whether dates are "too far apart" (a coarse heuristic). It doesn't reason about which claim is newer or whether one supersedes the other.
|
||||
|
||||
2. **`auto-supersession.ts`** — proposes resolution commands, checks `since_date` on takes. But this is post-hoc (after the judge flags a contradiction). The judge itself doesn't see dates.
|
||||
|
||||
3. **Facts table** has `valid_from` and `valid_until` columns. These exist but are sparsely populated and not used by the probe.
|
||||
|
||||
4. **Takes table** has `since_date`. Also sparsely populated.
|
||||
|
||||
## What Would Need to Change
|
||||
|
||||
### Phase 1: Judge prompt enhancement (smallest change, biggest impact)
|
||||
|
||||
Pass the source dates to the judge. The current judge prompt shows two text chunks and asks "are these contradictory?" If it also showed:
|
||||
|
||||
```
|
||||
Statement A (from: 2026-04-28):
|
||||
"status: trial"
|
||||
|
||||
Statement B (from: 2026-05-07):
|
||||
"status: confirmed"
|
||||
```
|
||||
|
||||
The judge could output a `temporal_supersession` verdict instead of `contradiction`. New verdict taxonomy:
|
||||
|
||||
- `no_contradiction` — statements are compatible
|
||||
- `contradiction` — genuinely conflicting claims at the same point in time
|
||||
- `temporal_supersession` — newer claim updates/replaces older claim (not an error)
|
||||
- `temporal_regression` — a metric or status went backwards (potential signal)
|
||||
- `temporal_evolution` — legitimate change over time, neither supersession nor regression
|
||||
- `negation_artifact` — one side contains an explicit negation the judge misread
|
||||
|
||||
### Phase 2: Claim trajectory view (new command)
|
||||
|
||||
```bash
|
||||
gbrain eval trajectory "Acme Corp MRR"
|
||||
gbrain eval trajectory "advisor-example role"
|
||||
gbrain eval trajectory "deal-x status"
|
||||
```
|
||||
|
||||
Pull all time-stamped claims about an entity+attribute, sort chronologically, detect:
|
||||
- Regressions (metric went down)
|
||||
- Contradictions within the same time window
|
||||
- Prediction vs. outcome gaps
|
||||
- Narrative drift (moat story changed 3x)
|
||||
|
||||
### Phase 3: Automatic `valid_from`/`valid_until` population
|
||||
|
||||
During `extract_facts`, infer temporal bounds from source context:
|
||||
- Meeting page dated 2026-04-28 → claims valid_from 2026-04-28
|
||||
- Takes from transcripts → valid_from = transcript date
|
||||
- Imported notes → valid_from = note date
|
||||
- Entity pages with no date → valid_from = page created date (weakest signal)
|
||||
|
||||
### Phase 4: Founder scorecard
|
||||
|
||||
For founders specifically, a temporal probe could generate:
|
||||
- **Claim accuracy score** — what they predicted vs. what happened
|
||||
- **Consistency score** — how stable their narrative is over time
|
||||
- **Growth trajectory** — whether the numbers are actually moving
|
||||
- **Red flag detector** — metrics going backwards, story changing, timeline slipping
|
||||
|
||||
## Recommendation
|
||||
|
||||
Start with Phase 1. The judge prompt change is small. It immediately eliminates the temporal false positives (which were a majority of the residual HIGH findings in the production audit) and gives the probe a new vocabulary for time-aware reasoning.
|
||||
|
||||
Phase 2 (trajectory view) is the one that would change how operators use the brain for founder evaluation. Worth scoping as a standalone feature.
|
||||
|
||||
Phases 3–4 are downstream and can wait.
|
||||
|
||||
## Appendix: Production probe stats (2026-05-14)
|
||||
|
||||
- ~107K pages, ~257K chunks
|
||||
- Previous run: ~115 HIGH findings across 50 queries
|
||||
- After manual resolution: ~25 residual findings
|
||||
- Of those ~25: roughly two-thirds temporal false positives, the remainder probe artifacts (self-contradiction, negation parsing)
|
||||
- 0 genuine data contradictions remained on the queries tested
|
||||
- Fresh targeted probe on a representative entity-role query: 0 contradictions (was 14+ before fixes)
|
||||
@@ -1,93 +0,0 @@
|
||||
# Takes vs Facts — Architectural Distinction
|
||||
|
||||
gbrain has two epistemological storage layers that serve different purposes.
|
||||
**Never conflate them.**
|
||||
|
||||
## Takes (cold storage — `takes` table)
|
||||
|
||||
The epistemological layer. WHO believes WHAT, with confidence weight and time.
|
||||
|
||||
- **Source:** Extracted from brain pages (markdown) by LLM analysis
|
||||
- **Scope:** Multi-holder — captures beliefs from *any* speaker, not just the brain owner
|
||||
- **Kinds:** `take` (opinion), `fact` (verifiable), `bet` (prediction), `hunch` (intuition)
|
||||
- **Lifecycle:** Cold storage, retrospective. Updated when pages change or re-extraction runs.
|
||||
- **Scale:** 100K+ rows across thousands of holders in a mature brain
|
||||
|
||||
**Example takes:**
|
||||
- `holder=people/garry-tan kind=bet` "AI will replace 50% of coding by 2030" (w=0.75)
|
||||
- `holder=people/jared-friedman kind=take` "Momo has strong retention" (w=0.80)
|
||||
- `holder=world kind=fact` "Clipboard raised $100M Series C" (w=1.0)
|
||||
- `holder=brain kind=hunch` "Garry has a hero/rescuer pattern" (w=0.70)
|
||||
|
||||
**Query surface:** `gbrain takes list`, `gbrain takes search`, `gbrain think`
|
||||
|
||||
## Facts (hot memory — `facts` table, v0.31)
|
||||
|
||||
Personal knowledge from the brain owner's conversations. Real-time capture.
|
||||
|
||||
- **Source:** Extracted per-turn from conversation by the facts hook (Haiku)
|
||||
- **Scope:** Single-user — only the brain owner's stated knowledge
|
||||
- **Kinds:** `event`, `preference`, `commitment`, `belief`, `fact`
|
||||
- **Lifecycle:** Hot storage, real-time. Captured as conversations happen.
|
||||
- **Bridge:** Dream cycle `consolidate` phase promotes hot facts → cold takes nightly
|
||||
|
||||
**Example facts:**
|
||||
- `kind=event` "I have a meeting with Brian tomorrow"
|
||||
- `kind=preference` "I don't drink coffee"
|
||||
- `kind=commitment` "We decided on nesting custody"
|
||||
- `kind=belief` "I think the market is overheated"
|
||||
|
||||
**Query surface:** `gbrain recall`, MCP `_meta.brain_hot_memory`
|
||||
|
||||
## The Category Error
|
||||
|
||||
**Never dump takes into the facts table.** Takes include other people's attributed
|
||||
beliefs (Jared's assessment of a company, PG's view on schools, a founder's
|
||||
revenue claims). These are NOT the brain owner's personal facts.
|
||||
|
||||
**Never dump facts into the takes table without transformation.** Facts are
|
||||
scoped to what the owner said in conversation. They become takes only through
|
||||
the dream cycle's consolidate phase, which adds proper attribution, deduplication,
|
||||
and temporal reasoning.
|
||||
|
||||
## The Bridge
|
||||
|
||||
The dream cycle's `consolidate` phase (v0.31) is the one-way bridge:
|
||||
|
||||
```
|
||||
hot facts → [dream consolidate] → cold takes
|
||||
```
|
||||
|
||||
Facts flow in ONE direction. The consolidate phase:
|
||||
1. Groups related facts by entity
|
||||
2. Deduplicates against existing takes
|
||||
3. Promotes durable facts to takes with proper holder/weight
|
||||
4. Marks consolidated facts with `consolidated_at` + `consolidated_into`
|
||||
|
||||
## Production Extraction Data (2026-05-10)
|
||||
|
||||
First full takes extraction run on a ~100K-page brain:
|
||||
- **Model:** Azure GPT-5.5 (ties Opus quality at 1/8th cost — $0.033 vs $0.260/page)
|
||||
- **Result:** 100,720 takes from 28,256 on-disk pages, $361.49, 83 errors (0.3%)
|
||||
- **Breakdown:** 70,960 takes / 24,342 facts / 2,875 bets / 2,649 hunches
|
||||
- **Holders:** 6,239 unique holders
|
||||
- **Cross-modal eval:** 6.8/10 overall (GPT-5.5 + Opus 4.6 scored independently)
|
||||
|
||||
### Eval Dimensions
|
||||
|
||||
| Dimension | Score | Notes |
|
||||
|-----------|-------|-------|
|
||||
| Accuracy | 7.5 | Claims faithfully represent sources |
|
||||
| Attribution | 6.5 | Holder/subject confusion was #1 issue |
|
||||
| Weight calibration | 7.0 | Good range usage, some false precision |
|
||||
| Kind classification | 6.5 | Occasional fact/take misclassification |
|
||||
| Signal density | 6.5 | Some trivial extractions pass through |
|
||||
|
||||
### Key Learnings for Extraction Prompts
|
||||
|
||||
1. **Holder ≠ subject.** "Garry has a hero/rescuer pattern" → holder=brain, NOT people/garry-tan
|
||||
2. **Atomic claims.** Split compound claims into separate rows
|
||||
3. **Amplification ≠ endorsement.** Retweet-only → max weight 0.55
|
||||
4. **Self-reported ≠ verified.** "Reports 7 figures" → holder=person, weight=0.75, NOT world/1.0
|
||||
5. **No false precision.** Use 0.05 increments (0.35, 0.55, 0.75), not 0.74 or 0.82
|
||||
6. **"So what" test.** Skip Twitter handles, follower counts, obvious metadata
|
||||
@@ -1,2 +0,0 @@
|
||||
# Per-run output JSONLs land here; only baseline-runs/<date>-<model>.jsonl is canonical.
|
||||
run-*.jsonl
|
||||
@@ -1,189 +0,0 @@
|
||||
# functional-area-resolver A/B eval
|
||||
|
||||
Maintainer-side eval evidence for the `functional-area-resolver` skill. Lives
|
||||
outside `skills/` deliberately — the skillpack bundler walks `skills/<skill>/`
|
||||
recursively, so an eval surface in there would ship to every downstream
|
||||
`gbrain skillpack install`. This directory is NOT bundled. The pattern (in
|
||||
SKILL.md) ships everywhere; the eval evidence stays in the gbrain repo where
|
||||
maintainers can re-baseline.
|
||||
|
||||
## What this proves
|
||||
|
||||
Three resolver shapes tested across three Anthropic frontier models. The
|
||||
pattern in `skills/functional-area-resolver/SKILL.md` (functional-area
|
||||
dispatchers with `(dispatcher for: ...)` clauses) **beats the verbose
|
||||
bullet-list baseline by +13 to +17pp on training while shipping at 48% the
|
||||
size**, and **catastrophically beats compression without the dispatcher
|
||||
clause** on Sonnet (100% vs 41.7% training, lenient).
|
||||
|
||||
## Methodology
|
||||
|
||||
### Variants
|
||||
|
||||
- `variants/baseline.md` — the verbose 270-row bullet-list shape extracted
|
||||
from a real production AGENTS.md at git commit `93848ff3b^` (pre-compression
|
||||
state), with owner PII scrubbed. ~25KB.
|
||||
- `variants/functional-areas.md` — the dispatcher pattern at git commit
|
||||
`93848ff3b` (the commit titled "AGENTS.md: functional-area resolver —
|
||||
25KB→13KB, 100% routing accuracy"). ~13KB.
|
||||
- `variants/resolver-of-resolvers.md` — derived mechanically from
|
||||
functional-areas by stripping `(dispatcher for: ...)` clauses. The ablation
|
||||
case: same structure, no sub-skill visibility. ~10KB.
|
||||
|
||||
### Corpora
|
||||
|
||||
- `fixtures.jsonl` — 20 hand-authored training fixtures used to develop the
|
||||
variants. Headline accuracy on training is informative but not the claim
|
||||
(same-author overfitting risk).
|
||||
- `fixtures-held-out.jsonl` — 5 fixtures authored BEFORE the variants and
|
||||
not adjusted afterward. Held-out is the canonical claim, but small n means
|
||||
it saturates near 100% for most cells.
|
||||
|
||||
### Scoring
|
||||
|
||||
Every output row carries two scores:
|
||||
|
||||
- **STRICT** (`correct`) — predicted slug equals expected exactly.
|
||||
- **LENIENT** (`correct_lenient`) — predicted is in the same dispatcher area
|
||||
as expected per the variant's `(dispatcher for: ...)` clauses. For variants
|
||||
without dispatcher clauses (baseline, resolver-of-resolvers), LENIENT
|
||||
collapses to STRICT.
|
||||
|
||||
Both matter:
|
||||
- STRICT measures "does the LLM return the exact slug?"
|
||||
- LENIENT measures "does the LLM land in the right area, even if it picks a
|
||||
more-specific sub-skill?" This reflects production agent behavior — landing
|
||||
in `gmail` for an email intent succeeds even if the resolver wrote
|
||||
`executive-assistant`.
|
||||
|
||||
### Repeats + statistics
|
||||
|
||||
- n=3 seeded repeats per (fixture, variant, model).
|
||||
- 95% confidence interval via t-distribution across the 3 seeded means
|
||||
(t-critical=4.303 for df=2).
|
||||
- Models: `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`.
|
||||
|
||||
### Receipt format
|
||||
|
||||
Each run writes one JSONL with:
|
||||
- Header row: `{kind:'receipt', model, prompt_template_hash, fixtures_hash,
|
||||
fixtures_held_out_hash, harness_sha, ts, cmd_args}` — binds the run to a
|
||||
specific harness version and inputs so re-runs are auditable.
|
||||
- One row per (fixture × variant × seed): full row schema in `harness-runner.ts`.
|
||||
|
||||
Baseline receipts committed in `baseline-runs/` after the v0.32.3.0
|
||||
re-baseline.
|
||||
|
||||
## Results (2026-05-11)
|
||||
|
||||
Training corpus (n=20, 3 seeds, LENIENT scoring):
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 | Size |
|
||||
|---|---|---|---|---|
|
||||
| baseline | 81.7% ± 7.2% | 86.7% ± 7.2% | 73.3% ± 7.2% | 25KB |
|
||||
| **functional-areas** | **98.3% ± 7.2%** | **100% ± 0%** | **88.3% ± 7.2%** | **13KB** |
|
||||
| resolver-of-resolvers | 63.3% ± 14.3% | 41.7% ± 7.2% | 65.0% ± 12.4% | 10KB |
|
||||
|
||||
Held-out corpus (n=5, 3 seeds, LENIENT scoring):
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 |
|
||||
|---|---|---|---|
|
||||
| baseline | 100% ± 0% | 100% ± 0% | 100% ± 0% |
|
||||
| **functional-areas** | **100% ± 0%** | **100% ± 0%** | **100% ± 0%** |
|
||||
| resolver-of-resolvers | 100% ± 0% | **73.3% ± 28.7%** | 100% ± 0% |
|
||||
|
||||
Strict numbers and the per-fixture failure traces are in the receipts.
|
||||
|
||||
## How to reproduce
|
||||
|
||||
From the gbrain repo root with `ANTHROPIC_API_KEY` set:
|
||||
|
||||
```bash
|
||||
cd evals/functional-area-resolver
|
||||
|
||||
# Smoke test (1 call, ~$0.01)
|
||||
node harness.mjs --limit 1 --yes
|
||||
|
||||
# Full run on Opus 4.7 (225 calls, ~$1.70)
|
||||
node harness.mjs --model opus --parallel 3 --yes
|
||||
|
||||
# Cross-model
|
||||
node harness.mjs --model sonnet --parallel 3 --yes # ~$1.00
|
||||
node harness.mjs --model haiku --parallel 3 --yes # ~$0.30
|
||||
|
||||
# Re-score an existing run without spending more API budget
|
||||
node rescore.mjs baseline-runs/2026-05-11-opus-4-7.jsonl
|
||||
|
||||
# Unit tests (no API key required)
|
||||
bun test harness-runner.test.ts
|
||||
```
|
||||
|
||||
The harness routes through gbrain's gateway, so it inherits gbrain's auth,
|
||||
rate-lease, and cost-meter behavior. Without `ANTHROPIC_API_KEY` it exits with
|
||||
a clear error.
|
||||
|
||||
## Important caveat: the prompt is load-bearing
|
||||
|
||||
The harness uses a dispatcher-aware prompt (see
|
||||
`harness-runner.ts:PROMPT_TEMPLATE`) that explicitly tells the LLM:
|
||||
|
||||
> Some entries are functional-area dispatchers shaped like:
|
||||
> "**Area name**: triggers... → `dispatcher-skill` (dispatcher for: subskill-a, subskill-b, ...)"
|
||||
> When the user's intent matches an area, RETURN THE MOST-SPECIFIC SUB-SKILL
|
||||
> from that area's "dispatcher for" list, not the dispatcher itself.
|
||||
|
||||
**Without this instruction, every compression variant collapses to ~30-60%
|
||||
on training.** A naive "return the skill slug" prompt makes the LLM pick the
|
||||
area lead instead of drilling into the dispatcher list. This was the failure
|
||||
mode in run-1 (synthetic variants + naive prompt) before the real-variants +
|
||||
dispatcher-aware-prompt re-baseline.
|
||||
|
||||
If you adopt the pattern in your own agent, the SKILL.md guidance applies
|
||||
to your harness prompt. Lift the PROMPT_TEMPLATE from this harness or write
|
||||
your own instruction explaining the dispatcher list.
|
||||
|
||||
## Limitations and v0.33.x follow-ups
|
||||
|
||||
1. Held-out corpus is small (n=5). Saturated at 100% across most cells. Grow
|
||||
to >=20 in v0.33.x.
|
||||
2. Single vendor (Anthropic). Cross-vendor (Gemini, GPT) is v0.33.x.
|
||||
3. No description-length sweep yet. Anthropic Agent Skills median is ~80
|
||||
tokens of frontmatter; we haven't measured the per-row description length
|
||||
sweet spot. v0.33.x.
|
||||
4. Same-author training corpus + variants. Held-out mitigates partially.
|
||||
5. No adversarial fixtures (e.g., "I want to do something brain-related"
|
||||
without specifying what). v0.33.x.
|
||||
|
||||
See `TODOS.md` for the full list.
|
||||
|
||||
## Prior art
|
||||
|
||||
This eval implements a **static-prompt analog** of hierarchical agent routing,
|
||||
a 2024-2025 research direction. The published hierarchical schemes resolve
|
||||
the hierarchy at runtime via a second LLM call; this skill inlines the
|
||||
hierarchy into a single-LLM-pass dispatcher list.
|
||||
|
||||
- AnyTool ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253)) — meta-agent → category → tool hierarchy, +35.4pp over flat retrieval at 16K APIs.
|
||||
- RAG-MCP ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1)) — embedding-based pre-retrieval, 49.2% token reduction at 3.2× accuracy gain.
|
||||
- Anthropic Agent Skills ([engineering blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)) — progressive disclosure (~80-token frontmatter loaded at startup; body loaded on match).
|
||||
|
||||
## File listing
|
||||
|
||||
```
|
||||
evals/functional-area-resolver/
|
||||
├── README.md # this file
|
||||
├── fixtures.jsonl # 20 training fixtures
|
||||
├── fixtures-held-out.jsonl # 5 held-out blind fixtures
|
||||
├── variants/
|
||||
│ ├── baseline.md # 25KB, PII-scrubbed from production
|
||||
│ ├── functional-areas.md # 13KB, PII-scrubbed from production
|
||||
│ └── resolver-of-resolvers.md # 10KB, derived ablation
|
||||
├── harness.mjs # thin Node CLI shim
|
||||
├── harness-runner.ts # TS runner via gbrain gateway
|
||||
├── harness-runner.test.ts # 45 unit tests (no API key)
|
||||
├── rescore.mjs # zero-cost lenient re-score
|
||||
└── baseline-runs/
|
||||
├── 2026-05-11-opus-4-7.jsonl # 225-row Opus baseline
|
||||
├── 2026-05-11-sonnet-4-6.jsonl # 225-row Sonnet baseline
|
||||
└── 2026-05-11-haiku-4-5.jsonl # 225-row Haiku baseline
|
||||
```
|
||||
@@ -1,226 +0,0 @@
|
||||
{"kind":"receipt","model":"anthropic:claude-haiku-4-5-20251001","prompt_template_hash":"17340040af579ca1","fixtures_hash":"feccc99122ea86d5","fixtures_held_out_hash":"5d6256cc9dced124","harness_sha":"fcc395282a92f2b047d4407f2b5a891c069adaac","ts":"2026-05-12T02:51:49.980Z","cmd_args":["--model","haiku","--parallel","3","--yes","--output","run-haiku-4-5.jsonl"]}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":5,"latency_ms":718,"ts":"2026-05-12T02:51:50.698Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":5,"latency_ms":1569,"ts":"2026-05-12T02:51:51.549Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":5,"latency_ms":1163,"ts":"2026-05-12T02:51:51.143Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":739,"ts":"2026-05-12T02:51:52.288Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":602,"ts":"2026-05-12T02:51:52.151Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":738,"ts":"2026-05-12T02:51:52.288Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":882,"ts":"2026-05-12T02:51:53.170Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":706,"ts":"2026-05-12T02:51:52.994Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":706,"ts":"2026-05-12T02:51:52.994Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":706,"ts":"2026-05-12T02:51:53.877Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":1239,"ts":"2026-05-12T02:51:54.410Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":886,"ts":"2026-05-12T02:51:54.057Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":921,"ts":"2026-05-12T02:51:55.331Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":705,"ts":"2026-05-12T02:51:55.115Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":797,"ts":"2026-05-12T02:51:55.207Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":718,"ts":"2026-05-12T02:51:56.050Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":663,"ts":"2026-05-12T02:51:55.995Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":782,"ts":"2026-05-12T02:51:56.114Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":721,"ts":"2026-05-12T02:51:56.835Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":679,"ts":"2026-05-12T02:51:56.793Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":646,"ts":"2026-05-12T02:51:56.760Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":667,"ts":"2026-05-12T02:51:57.502Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":649,"ts":"2026-05-12T02:51:57.484Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":1016,"ts":"2026-05-12T02:51:57.851Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":651,"ts":"2026-05-12T02:51:58.503Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":653,"ts":"2026-05-12T02:51:58.504Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":859,"ts":"2026-05-12T02:51:58.710Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":9,"latency_ms":631,"ts":"2026-05-12T02:51:59.341Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":9,"latency_ms":1021,"ts":"2026-05-12T02:51:59.731Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":9,"latency_ms":682,"ts":"2026-05-12T02:51:59.392Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":7,"latency_ms":642,"ts":"2026-05-12T02:52:00.373Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":7,"latency_ms":687,"ts":"2026-05-12T02:52:00.418Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":7,"latency_ms":795,"ts":"2026-05-12T02:52:00.526Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":728,"ts":"2026-05-12T02:52:01.254Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":662,"ts":"2026-05-12T02:52:01.188Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":906,"ts":"2026-05-12T02:52:01.432Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":624,"ts":"2026-05-12T02:52:02.056Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":682,"ts":"2026-05-12T02:52:02.114Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":672,"ts":"2026-05-12T02:52:02.104Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":9,"latency_ms":711,"ts":"2026-05-12T02:52:02.825Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":9,"latency_ms":671,"ts":"2026-05-12T02:52:02.785Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":9,"latency_ms":751,"ts":"2026-05-12T02:52:02.865Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":6,"latency_ms":670,"ts":"2026-05-12T02:52:03.535Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":6,"latency_ms":649,"ts":"2026-05-12T02:52:03.514Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":6,"latency_ms":827,"ts":"2026-05-12T02:52:03.692Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":4,"latency_ms":682,"ts":"2026-05-12T02:52:04.374Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":4,"latency_ms":760,"ts":"2026-05-12T02:52:04.452Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":4,"latency_ms":691,"ts":"2026-05-12T02:52:04.383Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":9,"latency_ms":642,"ts":"2026-05-12T02:52:05.094Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":9,"latency_ms":698,"ts":"2026-05-12T02:52:05.150Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":9,"latency_ms":632,"ts":"2026-05-12T02:52:05.084Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":7,"latency_ms":665,"ts":"2026-05-12T02:52:05.815Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":7,"latency_ms":850,"ts":"2026-05-12T02:52:06.000Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":7,"latency_ms":636,"ts":"2026-05-12T02:52:05.786Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":5,"latency_ms":949,"ts":"2026-05-12T02:52:06.950Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":5,"latency_ms":803,"ts":"2026-05-12T02:52:06.804Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":5,"latency_ms":747,"ts":"2026-05-12T02:52:06.748Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":1,"predicted":"calendar-event-create","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":779,"ts":"2026-05-12T02:52:07.729Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":6,"latency_ms":779,"ts":"2026-05-12T02:52:07.729Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":6,"latency_ms":1134,"ts":"2026-05-12T02:52:08.084Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":5,"latency_ms":706,"ts":"2026-05-12T02:52:08.790Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":5,"latency_ms":1453,"ts":"2026-05-12T02:52:09.537Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":5,"latency_ms":882,"ts":"2026-05-12T02:52:08.966Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":6,"latency_ms":810,"ts":"2026-05-12T02:52:10.347Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":6,"latency_ms":1777,"ts":"2026-05-12T02:52:11.314Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":6,"latency_ms":842,"ts":"2026-05-12T02:52:10.379Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":668,"ts":"2026-05-12T02:52:11.982Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":624,"ts":"2026-05-12T02:52:11.938Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":712,"ts":"2026-05-12T02:52:12.026Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":1403,"ts":"2026-05-12T02:52:13.429Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":680,"ts":"2026-05-12T02:52:12.706Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":750,"ts":"2026-05-12T02:52:12.776Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":1018,"ts":"2026-05-12T02:52:14.447Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":4118,"ts":"2026-05-12T02:52:17.547Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":673,"ts":"2026-05-12T02:52:14.102Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":5,"latency_ms":741,"ts":"2026-05-12T02:52:18.288Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":5,"latency_ms":580,"ts":"2026-05-12T02:52:18.127Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":5,"latency_ms":575,"ts":"2026-05-12T02:52:18.122Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"data-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":6,"latency_ms":573,"ts":"2026-05-12T02:52:18.861Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":9,"latency_ms":579,"ts":"2026-05-12T02:52:18.867Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":9,"latency_ms":579,"ts":"2026-05-12T02:52:18.867Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":598,"ts":"2026-05-12T02:52:19.465Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":556,"ts":"2026-05-12T02:52:19.423Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":561,"ts":"2026-05-12T02:52:19.428Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":631,"ts":"2026-05-12T02:52:20.096Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":602,"ts":"2026-05-12T02:52:20.067Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":610,"ts":"2026-05-12T02:52:20.075Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":571,"ts":"2026-05-12T02:52:20.667Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":696,"ts":"2026-05-12T02:52:20.792Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":628,"ts":"2026-05-12T02:52:20.724Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":743,"ts":"2026-05-12T02:52:21.535Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":612,"ts":"2026-05-12T02:52:21.404Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":630,"ts":"2026-05-12T02:52:21.422Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-mirror-synthesis","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":8,"latency_ms":1639,"ts":"2026-05-12T02:52:23.174Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"book-mirror-synthesis","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":8,"latency_ms":585,"ts":"2026-05-12T02:52:22.120Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"book-mirror-synthesis","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":8,"latency_ms":656,"ts":"2026-05-12T02:52:22.191Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":6,"latency_ms":673,"ts":"2026-05-12T02:52:23.847Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":6,"latency_ms":673,"ts":"2026-05-12T02:52:23.847Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":6,"latency_ms":529,"ts":"2026-05-12T02:52:23.703Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":704,"ts":"2026-05-12T02:52:24.551Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":535,"ts":"2026-05-12T02:52:24.382Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":705,"ts":"2026-05-12T02:52:24.552Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":9,"latency_ms":707,"ts":"2026-05-12T02:52:25.259Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":9,"latency_ms":707,"ts":"2026-05-12T02:52:25.259Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":9,"latency_ms":707,"ts":"2026-05-12T02:52:25.259Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":7,"latency_ms":815,"ts":"2026-05-12T02:52:26.074Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":7,"latency_ms":697,"ts":"2026-05-12T02:52:25.956Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":7,"latency_ms":689,"ts":"2026-05-12T02:52:25.948Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":587,"ts":"2026-05-12T02:52:26.661Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":572,"ts":"2026-05-12T02:52:26.646Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":1134,"ts":"2026-05-12T02:52:27.208Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"transcript-save","expected":"meeting-ingestion","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":733,"ts":"2026-05-12T02:52:27.941Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"transcript-save","expected":"meeting-ingestion","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":623,"ts":"2026-05-12T02:52:27.831Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":7,"latency_ms":553,"ts":"2026-05-12T02:52:27.761Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":9,"latency_ms":1752,"ts":"2026-05-12T02:52:29.693Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":9,"latency_ms":929,"ts":"2026-05-12T02:52:28.870Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":9,"latency_ms":637,"ts":"2026-05-12T02:52:28.578Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":6,"latency_ms":582,"ts":"2026-05-12T02:52:30.275Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":6,"latency_ms":547,"ts":"2026-05-12T02:52:30.241Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":6,"latency_ms":882,"ts":"2026-05-12T02:52:30.575Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":4,"latency_ms":569,"ts":"2026-05-12T02:52:31.144Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":4,"latency_ms":587,"ts":"2026-05-12T02:52:31.162Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":4,"latency_ms":619,"ts":"2026-05-12T02:52:31.194Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"exa","expected":"perplexity-research","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":5,"latency_ms":712,"ts":"2026-05-12T02:52:31.907Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":9,"latency_ms":558,"ts":"2026-05-12T02:52:31.753Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"exa","expected":"perplexity-research","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":5,"latency_ms":537,"ts":"2026-05-12T02:52:31.732Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":7,"latency_ms":706,"ts":"2026-05-12T02:52:32.613Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":7,"latency_ms":743,"ts":"2026-05-12T02:52:32.650Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":7,"latency_ms":4370,"ts":"2026-05-12T02:52:36.277Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":5,"latency_ms":587,"ts":"2026-05-12T02:52:36.864Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":5,"latency_ms":624,"ts":"2026-05-12T02:52:36.901Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":5,"latency_ms":634,"ts":"2026-05-12T02:52:36.911Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":6,"latency_ms":577,"ts":"2026-05-12T02:52:37.488Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":6,"latency_ms":587,"ts":"2026-05-12T02:52:37.498Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":6,"latency_ms":1335,"ts":"2026-05-12T02:52:38.246Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":5,"latency_ms":1277,"ts":"2026-05-12T02:52:39.523Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":5,"latency_ms":560,"ts":"2026-05-12T02:52:38.806Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":5,"latency_ms":735,"ts":"2026-05-12T02:52:38.981Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":6,"latency_ms":666,"ts":"2026-05-12T02:52:40.189Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":6,"latency_ms":666,"ts":"2026-05-12T02:52:40.189Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":6,"latency_ms":666,"ts":"2026-05-12T02:52:40.189Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":8,"latency_ms":668,"ts":"2026-05-12T02:52:40.857Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":8,"latency_ms":689,"ts":"2026-05-12T02:52:40.879Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":8,"latency_ms":555,"ts":"2026-05-12T02:52:40.745Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":730,"ts":"2026-05-12T02:52:41.609Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":729,"ts":"2026-05-12T02:52:41.609Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":603,"ts":"2026-05-12T02:52:41.483Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":5,"latency_ms":699,"ts":"2026-05-12T02:52:42.308Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":5,"latency_ms":567,"ts":"2026-05-12T02:52:42.176Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":5,"latency_ms":1623,"ts":"2026-05-12T02:52:43.232Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":547,"ts":"2026-05-12T02:52:43.779Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":552,"ts":"2026-05-12T02:52:43.784Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":660,"ts":"2026-05-12T02:52:43.892Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":6,"latency_ms":575,"ts":"2026-05-12T02:52:44.467Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":6,"latency_ms":673,"ts":"2026-05-12T02:52:44.565Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":6,"latency_ms":606,"ts":"2026-05-12T02:52:44.498Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":825,"ts":"2026-05-12T02:52:45.390Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":554,"ts":"2026-05-12T02:52:45.119Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":606,"ts":"2026-05-12T02:52:45.171Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":588,"ts":"2026-05-12T02:52:45.979Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":873,"ts":"2026-05-12T02:52:46.264Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":586,"ts":"2026-05-12T02:52:45.977Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":579,"ts":"2026-05-12T02:52:46.843Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":556,"ts":"2026-05-12T02:52:46.820Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":675,"ts":"2026-05-12T02:52:46.939Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":7,"latency_ms":564,"ts":"2026-05-12T02:52:47.503Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":7,"latency_ms":901,"ts":"2026-05-12T02:52:47.840Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":7,"latency_ms":606,"ts":"2026-05-12T02:52:47.545Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":726,"ts":"2026-05-12T02:52:48.566Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":778,"ts":"2026-05-12T02:52:48.618Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":563,"ts":"2026-05-12T02:52:48.403Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":551,"ts":"2026-05-12T02:52:49.169Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":568,"ts":"2026-05-12T02:52:49.186Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":799,"ts":"2026-05-12T02:52:49.417Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":567,"ts":"2026-05-12T02:52:49.984Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":1347,"ts":"2026-05-12T02:52:50.764Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":662,"ts":"2026-05-12T02:52:50.079Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":572,"ts":"2026-05-12T02:52:51.336Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":587,"ts":"2026-05-12T02:52:51.351Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":578,"ts":"2026-05-12T02:52:51.342Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":624,"ts":"2026-05-12T02:52:51.975Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":560,"ts":"2026-05-12T02:52:51.911Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":615,"ts":"2026-05-12T02:52:51.966Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":4740,"ts":"2026-05-12T02:52:56.715Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":722,"ts":"2026-05-12T02:52:52.698Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":815,"ts":"2026-05-12T02:52:52.791Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":5,"latency_ms":577,"ts":"2026-05-12T02:52:57.292Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":5,"latency_ms":762,"ts":"2026-05-12T02:52:57.477Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":7,"latency_ms":583,"ts":"2026-05-12T02:52:57.298Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":9,"latency_ms":555,"ts":"2026-05-12T02:52:58.032Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":9,"latency_ms":705,"ts":"2026-05-12T02:52:58.182Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":9,"latency_ms":587,"ts":"2026-05-12T02:52:58.064Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":599,"ts":"2026-05-12T02:52:58.781Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":577,"ts":"2026-05-12T02:52:58.759Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":793,"ts":"2026-05-12T02:52:58.975Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":836,"ts":"2026-05-12T02:52:59.811Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":2306,"ts":"2026-05-12T02:53:01.281Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":1092,"ts":"2026-05-12T02:53:00.067Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":9,"latency_ms":568,"ts":"2026-05-12T02:53:01.849Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":9,"latency_ms":628,"ts":"2026-05-12T02:53:01.909Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":9,"latency_ms":593,"ts":"2026-05-12T02:53:01.874Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":7,"latency_ms":626,"ts":"2026-05-12T02:53:02.535Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":7,"latency_ms":641,"ts":"2026-05-12T02:53:02.550Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":7,"latency_ms":925,"ts":"2026-05-12T02:53:02.834Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":5,"latency_ms":615,"ts":"2026-05-12T02:53:03.449Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":5,"latency_ms":706,"ts":"2026-05-12T02:53:03.540Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":5,"latency_ms":723,"ts":"2026-05-12T02:53:03.557Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":878,"ts":"2026-05-12T02:53:04.435Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":619,"ts":"2026-05-12T02:53:04.176Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":606,"ts":"2026-05-12T02:53:04.163Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":917,"ts":"2026-05-12T02:53:05.352Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":542,"ts":"2026-05-12T02:53:04.977Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":603,"ts":"2026-05-12T02:53:05.038Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":6,"latency_ms":1164,"ts":"2026-05-12T02:53:06.516Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":6,"latency_ms":629,"ts":"2026-05-12T02:53:05.981Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":6,"latency_ms":609,"ts":"2026-05-12T02:53:05.961Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":655,"ts":"2026-05-12T02:53:07.171Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":649,"ts":"2026-05-12T02:53:07.165Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":691,"ts":"2026-05-12T02:53:07.207Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":550,"ts":"2026-05-12T02:53:07.757Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":577,"ts":"2026-05-12T02:53:07.784Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":1182,"ts":"2026-05-12T02:53:08.389Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":667,"ts":"2026-05-12T02:53:09.056Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":843,"ts":"2026-05-12T02:53:09.232Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":667,"ts":"2026-05-12T02:53:09.056Z"}
|
||||
@@ -1,226 +0,0 @@
|
||||
{"kind":"receipt","model":"anthropic:claude-opus-4-7","prompt_template_hash":"17340040af579ca1","fixtures_hash":"feccc99122ea86d5","fixtures_held_out_hash":"5d6256cc9dced124","harness_sha":"ca99fbfeb5f304e1e237eebd11ce0196ea8a9b18","ts":"2026-05-12T03:16:08.329Z","cmd_args":["--model","opus","--parallel","3","--yes","--output","baseline-runs/2026-05-11-opus-4-7.jsonl"]}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":7,"latency_ms":1844,"ts":"2026-05-12T03:16:10.173Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":7,"latency_ms":1703,"ts":"2026-05-12T03:16:10.032Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":7,"latency_ms":1672,"ts":"2026-05-12T03:16:10.001Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":1,"predicted":"entity-detector","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10940,"output_tokens":9,"latency_ms":4547,"ts":"2026-05-12T03:16:14.720Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10940,"output_tokens":7,"latency_ms":1730,"ts":"2026-05-12T03:16:11.903Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10940,"output_tokens":7,"latency_ms":1766,"ts":"2026-05-12T03:16:11.939Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":10,"latency_ms":2035,"ts":"2026-05-12T03:16:16.755Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":10,"latency_ms":2849,"ts":"2026-05-12T03:16:17.569Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":10,"latency_ms":2096,"ts":"2026-05-12T03:16:16.816Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":9,"latency_ms":1646,"ts":"2026-05-12T03:16:19.215Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":9,"latency_ms":1847,"ts":"2026-05-12T03:16:19.416Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":9,"latency_ms":1512,"ts":"2026-05-12T03:16:19.081Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":10,"latency_ms":1488,"ts":"2026-05-12T03:16:20.904Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":10,"latency_ms":1498,"ts":"2026-05-12T03:16:20.914Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":10,"latency_ms":1584,"ts":"2026-05-12T03:16:21.000Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":10,"latency_ms":2227,"ts":"2026-05-12T03:16:23.227Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":10,"latency_ms":1817,"ts":"2026-05-12T03:16:22.817Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":10,"latency_ms":1446,"ts":"2026-05-12T03:16:22.446Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":9,"latency_ms":2117,"ts":"2026-05-12T03:16:25.345Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":9,"latency_ms":1587,"ts":"2026-05-12T03:16:24.816Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":9,"latency_ms":1587,"ts":"2026-05-12T03:16:24.816Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":10,"latency_ms":1653,"ts":"2026-05-12T03:16:26.998Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":10,"latency_ms":2538,"ts":"2026-05-12T03:16:27.883Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":10,"latency_ms":1921,"ts":"2026-05-12T03:16:27.266Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":11,"latency_ms":1432,"ts":"2026-05-12T03:16:29.315Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":11,"latency_ms":1619,"ts":"2026-05-12T03:16:29.502Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":11,"latency_ms":1910,"ts":"2026-05-12T03:16:29.793Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":13,"latency_ms":1896,"ts":"2026-05-12T03:16:31.689Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":13,"latency_ms":1678,"ts":"2026-05-12T03:16:31.471Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":13,"latency_ms":2108,"ts":"2026-05-12T03:16:31.901Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":9,"latency_ms":1722,"ts":"2026-05-12T03:16:33.623Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":9,"latency_ms":2109,"ts":"2026-05-12T03:16:34.010Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":9,"latency_ms":4048,"ts":"2026-05-12T03:16:35.949Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1622,"ts":"2026-05-12T03:16:37.572Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1522,"ts":"2026-05-12T03:16:37.472Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":2637,"ts":"2026-05-12T03:16:38.587Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":12,"latency_ms":1752,"ts":"2026-05-12T03:16:40.339Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":12,"latency_ms":1603,"ts":"2026-05-12T03:16:40.190Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":12,"latency_ms":1575,"ts":"2026-05-12T03:16:40.162Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":12,"latency_ms":2603,"ts":"2026-05-12T03:16:42.942Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":12,"latency_ms":1578,"ts":"2026-05-12T03:16:41.917Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":12,"latency_ms":1692,"ts":"2026-05-12T03:16:42.031Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1674,"ts":"2026-05-12T03:16:44.616Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1955,"ts":"2026-05-12T03:16:44.897Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":2103,"ts":"2026-05-12T03:16:45.045Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":7,"latency_ms":2048,"ts":"2026-05-12T03:16:47.093Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":7,"latency_ms":2522,"ts":"2026-05-12T03:16:47.567Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":10,"latency_ms":1825,"ts":"2026-05-12T03:16:46.870Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":10,"latency_ms":1734,"ts":"2026-05-12T03:16:49.301Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":10,"latency_ms":1666,"ts":"2026-05-12T03:16:49.234Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":10,"latency_ms":1694,"ts":"2026-05-12T03:16:49.261Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":9,"latency_ms":1531,"ts":"2026-05-12T03:16:50.832Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":9,"latency_ms":1615,"ts":"2026-05-12T03:16:50.916Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":9,"latency_ms":1503,"ts":"2026-05-12T03:16:50.804Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":7,"latency_ms":1960,"ts":"2026-05-12T03:16:52.876Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":7,"latency_ms":1469,"ts":"2026-05-12T03:16:52.385Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":7,"latency_ms":1686,"ts":"2026-05-12T03:16:52.602Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":3911,"ts":"2026-05-12T03:16:56.787Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":1684,"ts":"2026-05-12T03:16:54.560Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":2066,"ts":"2026-05-12T03:16:54.942Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10950,"output_tokens":7,"latency_ms":1576,"ts":"2026-05-12T03:16:58.363Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10950,"output_tokens":7,"latency_ms":1634,"ts":"2026-05-12T03:16:58.421Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10950,"output_tokens":7,"latency_ms":2666,"ts":"2026-05-12T03:16:59.453Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":9,"latency_ms":1933,"ts":"2026-05-12T03:17:01.386Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":9,"latency_ms":2022,"ts":"2026-05-12T03:17:01.475Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":9,"latency_ms":1881,"ts":"2026-05-12T03:17:01.334Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":2322,"ts":"2026-05-12T03:17:03.797Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":1639,"ts":"2026-05-12T03:17:03.114Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":1854,"ts":"2026-05-12T03:17:03.329Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":9,"latency_ms":1694,"ts":"2026-05-12T03:17:05.491Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":9,"latency_ms":1621,"ts":"2026-05-12T03:17:05.418Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":9,"latency_ms":1493,"ts":"2026-05-12T03:17:05.292Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":9,"latency_ms":1662,"ts":"2026-05-12T03:17:07.153Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":9,"latency_ms":1736,"ts":"2026-05-12T03:17:07.227Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":9,"latency_ms":1538,"ts":"2026-05-12T03:17:07.030Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":7,"latency_ms":1569,"ts":"2026-05-12T03:17:08.796Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":7,"latency_ms":1569,"ts":"2026-05-12T03:17:08.796Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":7,"latency_ms":1746,"ts":"2026-05-12T03:17:08.973Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5997,"output_tokens":5,"latency_ms":1428,"ts":"2026-05-12T03:17:10.401Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5997,"output_tokens":5,"latency_ms":1414,"ts":"2026-05-12T03:17:10.387Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5997,"output_tokens":5,"latency_ms":1903,"ts":"2026-05-12T03:17:10.876Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":10,"latency_ms":1603,"ts":"2026-05-12T03:17:12.479Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":10,"latency_ms":1551,"ts":"2026-05-12T03:17:12.428Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":10,"latency_ms":1738,"ts":"2026-05-12T03:17:12.615Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":9,"latency_ms":4130,"ts":"2026-05-12T03:17:16.745Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":9,"latency_ms":1735,"ts":"2026-05-12T03:17:14.351Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":9,"latency_ms":1704,"ts":"2026-05-12T03:17:14.320Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":10,"latency_ms":3997,"ts":"2026-05-12T03:17:20.742Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":10,"latency_ms":1578,"ts":"2026-05-12T03:17:18.323Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":10,"latency_ms":1617,"ts":"2026-05-12T03:17:18.362Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":10,"latency_ms":1656,"ts":"2026-05-12T03:17:22.398Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":10,"latency_ms":1652,"ts":"2026-05-12T03:17:22.394Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":10,"latency_ms":1575,"ts":"2026-05-12T03:17:22.317Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":9,"latency_ms":2173,"ts":"2026-05-12T03:17:24.571Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":9,"latency_ms":1848,"ts":"2026-05-12T03:17:24.246Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":9,"latency_ms":1678,"ts":"2026-05-12T03:17:24.076Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":10,"latency_ms":1362,"ts":"2026-05-12T03:17:25.933Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":10,"latency_ms":1747,"ts":"2026-05-12T03:17:26.318Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":10,"latency_ms":1747,"ts":"2026-05-12T03:17:26.318Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":11,"latency_ms":2525,"ts":"2026-05-12T03:17:28.843Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":11,"latency_ms":1404,"ts":"2026-05-12T03:17:27.722Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":11,"latency_ms":1603,"ts":"2026-05-12T03:17:27.921Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":13,"latency_ms":3273,"ts":"2026-05-12T03:17:32.116Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":13,"latency_ms":2071,"ts":"2026-05-12T03:17:30.914Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":13,"latency_ms":1820,"ts":"2026-05-12T03:17:30.663Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":10,"latency_ms":1443,"ts":"2026-05-12T03:17:33.559Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":10,"latency_ms":1549,"ts":"2026-05-12T03:17:33.665Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":10,"latency_ms":1563,"ts":"2026-05-12T03:17:33.679Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1824,"ts":"2026-05-12T03:17:35.503Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1533,"ts":"2026-05-12T03:17:35.212Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1365,"ts":"2026-05-12T03:17:35.044Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":12,"latency_ms":1511,"ts":"2026-05-12T03:17:37.014Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":12,"latency_ms":1880,"ts":"2026-05-12T03:17:37.383Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":12,"latency_ms":1602,"ts":"2026-05-12T03:17:37.105Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":12,"latency_ms":1496,"ts":"2026-05-12T03:17:38.879Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":12,"latency_ms":1470,"ts":"2026-05-12T03:17:38.853Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":12,"latency_ms":2355,"ts":"2026-05-12T03:17:39.738Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1703,"ts":"2026-05-12T03:17:41.441Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1598,"ts":"2026-05-12T03:17:41.337Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1574,"ts":"2026-05-12T03:17:41.313Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":7,"latency_ms":1674,"ts":"2026-05-12T03:17:43.115Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":7,"latency_ms":1755,"ts":"2026-05-12T03:17:43.197Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":7,"latency_ms":1830,"ts":"2026-05-12T03:17:43.271Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":10,"latency_ms":1478,"ts":"2026-05-12T03:17:44.750Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":10,"latency_ms":2431,"ts":"2026-05-12T03:17:45.702Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"exa","expected":"perplexity-research","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":6,"latency_ms":1496,"ts":"2026-05-12T03:17:44.767Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":9,"latency_ms":1883,"ts":"2026-05-12T03:17:47.585Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":9,"latency_ms":1445,"ts":"2026-05-12T03:17:47.147Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":9,"latency_ms":1597,"ts":"2026-05-12T03:17:47.299Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":7,"latency_ms":1448,"ts":"2026-05-12T03:17:49.033Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":7,"latency_ms":2841,"ts":"2026-05-12T03:17:50.426Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":7,"latency_ms":1414,"ts":"2026-05-12T03:17:48.999Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1393,"ts":"2026-05-12T03:17:51.819Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1478,"ts":"2026-05-12T03:17:51.904Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1625,"ts":"2026-05-12T03:17:52.051Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6007,"output_tokens":7,"latency_ms":1694,"ts":"2026-05-12T03:17:53.745Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6007,"output_tokens":7,"latency_ms":1694,"ts":"2026-05-12T03:17:53.745Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6007,"output_tokens":7,"latency_ms":1720,"ts":"2026-05-12T03:17:53.771Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":9,"latency_ms":1740,"ts":"2026-05-12T03:17:55.511Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":9,"latency_ms":1920,"ts":"2026-05-12T03:17:55.691Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":9,"latency_ms":1563,"ts":"2026-05-12T03:17:55.334Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":2563,"ts":"2026-05-12T03:17:58.255Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1528,"ts":"2026-05-12T03:17:57.220Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1528,"ts":"2026-05-12T03:17:57.220Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":9,"latency_ms":1585,"ts":"2026-05-12T03:17:59.840Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"google-contacts","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":7,"latency_ms":1570,"ts":"2026-05-12T03:17:59.825Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"google-contacts","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":7,"latency_ms":1909,"ts":"2026-05-12T03:18:00.164Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":9,"latency_ms":1376,"ts":"2026-05-12T03:18:01.540Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":9,"latency_ms":1389,"ts":"2026-05-12T03:18:01.553Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":9,"latency_ms":1398,"ts":"2026-05-12T03:18:01.562Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":9,"latency_ms":1527,"ts":"2026-05-12T03:18:03.089Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":9,"latency_ms":1573,"ts":"2026-05-12T03:18:03.135Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":9,"latency_ms":1516,"ts":"2026-05-12T03:18:03.078Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4724,"output_tokens":10,"latency_ms":1606,"ts":"2026-05-12T03:18:04.741Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4724,"output_tokens":10,"latency_ms":1689,"ts":"2026-05-12T03:18:04.824Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4724,"output_tokens":10,"latency_ms":1682,"ts":"2026-05-12T03:18:04.817Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1670,"ts":"2026-05-12T03:18:06.494Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":2416,"ts":"2026-05-12T03:18:07.240Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1489,"ts":"2026-05-12T03:18:06.313Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":9,"latency_ms":3205,"ts":"2026-05-12T03:18:10.445Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":9,"latency_ms":4901,"ts":"2026-05-12T03:18:12.141Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":9,"latency_ms":4556,"ts":"2026-05-12T03:18:11.796Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":10,"latency_ms":1779,"ts":"2026-05-12T03:18:13.921Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":10,"latency_ms":1782,"ts":"2026-05-12T03:18:13.924Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":10,"latency_ms":2264,"ts":"2026-05-12T03:18:14.406Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":10,"latency_ms":1905,"ts":"2026-05-12T03:18:16.311Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":10,"latency_ms":1512,"ts":"2026-05-12T03:18:15.918Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":10,"latency_ms":1535,"ts":"2026-05-12T03:18:15.941Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":9,"latency_ms":1430,"ts":"2026-05-12T03:18:17.741Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":9,"latency_ms":1933,"ts":"2026-05-12T03:18:18.244Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":9,"latency_ms":1902,"ts":"2026-05-12T03:18:18.213Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":10,"latency_ms":1602,"ts":"2026-05-12T03:18:19.846Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":10,"latency_ms":1606,"ts":"2026-05-12T03:18:19.850Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":10,"latency_ms":1786,"ts":"2026-05-12T03:18:20.030Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"concept-synthesis","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":9,"latency_ms":1583,"ts":"2026-05-12T03:18:21.613Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"concept-synthesis","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":9,"latency_ms":1412,"ts":"2026-05-12T03:18:21.442Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":11,"latency_ms":1521,"ts":"2026-05-12T03:18:21.551Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1538,"ts":"2026-05-12T03:18:23.151Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1534,"ts":"2026-05-12T03:18:23.148Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1375,"ts":"2026-05-12T03:18:22.989Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":7,"latency_ms":2125,"ts":"2026-05-12T03:18:25.276Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":7,"latency_ms":2337,"ts":"2026-05-12T03:18:25.488Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":7,"latency_ms":1945,"ts":"2026-05-12T03:18:25.096Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1453,"ts":"2026-05-12T03:18:26.941Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1398,"ts":"2026-05-12T03:18:26.886Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":7,"latency_ms":1295,"ts":"2026-05-12T03:18:26.783Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":12,"latency_ms":1249,"ts":"2026-05-12T03:18:28.190Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":12,"latency_ms":1502,"ts":"2026-05-12T03:18:28.443Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":12,"latency_ms":1721,"ts":"2026-05-12T03:18:28.662Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":12,"latency_ms":1577,"ts":"2026-05-12T03:18:30.240Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":12,"latency_ms":1894,"ts":"2026-05-12T03:18:30.557Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":12,"latency_ms":1350,"ts":"2026-05-12T03:18:30.013Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1974,"ts":"2026-05-12T03:18:32.531Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1650,"ts":"2026-05-12T03:18:32.207Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":5071,"ts":"2026-05-12T03:18:35.628Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":10,"latency_ms":1363,"ts":"2026-05-12T03:18:36.991Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":10,"latency_ms":1978,"ts":"2026-05-12T03:18:37.606Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":10,"latency_ms":1567,"ts":"2026-05-12T03:18:37.195Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":10,"latency_ms":1639,"ts":"2026-05-12T03:18:39.245Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":10,"latency_ms":1780,"ts":"2026-05-12T03:18:39.386Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":10,"latency_ms":2166,"ts":"2026-05-12T03:18:39.772Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":9,"latency_ms":1785,"ts":"2026-05-12T03:18:41.557Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":9,"latency_ms":1546,"ts":"2026-05-12T03:18:41.318Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":9,"latency_ms":2157,"ts":"2026-05-12T03:18:41.929Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":7,"latency_ms":1536,"ts":"2026-05-12T03:18:43.465Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":7,"latency_ms":1624,"ts":"2026-05-12T03:18:43.553Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":7,"latency_ms":1452,"ts":"2026-05-12T03:18:43.381Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1862,"ts":"2026-05-12T03:18:45.415Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1922,"ts":"2026-05-12T03:18:45.475Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1589,"ts":"2026-05-12T03:18:45.142Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4734,"output_tokens":7,"latency_ms":3982,"ts":"2026-05-12T03:18:49.458Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4734,"output_tokens":7,"latency_ms":1555,"ts":"2026-05-12T03:18:47.030Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4734,"output_tokens":7,"latency_ms":1221,"ts":"2026-05-12T03:18:46.696Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":9,"latency_ms":1319,"ts":"2026-05-12T03:18:50.777Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":9,"latency_ms":1587,"ts":"2026-05-12T03:18:51.045Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":9,"latency_ms":1399,"ts":"2026-05-12T03:18:50.857Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1862,"ts":"2026-05-12T03:18:52.907Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1773,"ts":"2026-05-12T03:18:52.818Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1525,"ts":"2026-05-12T03:18:52.570Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":9,"latency_ms":9556,"ts":"2026-05-12T03:19:02.463Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":9,"latency_ms":2096,"ts":"2026-05-12T03:18:55.003Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":9,"latency_ms":1919,"ts":"2026-05-12T03:18:54.826Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":9,"latency_ms":1639,"ts":"2026-05-12T03:19:04.102Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":9,"latency_ms":1731,"ts":"2026-05-12T03:19:04.194Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":9,"latency_ms":1741,"ts":"2026-05-12T03:19:04.204Z"}
|
||||
@@ -1,226 +0,0 @@
|
||||
{"kind":"receipt","model":"anthropic:claude-sonnet-4-6","prompt_template_hash":"17340040af579ca1","fixtures_hash":"feccc99122ea86d5","fixtures_held_out_hash":"5d6256cc9dced124","harness_sha":"fcc395282a92f2b047d4407f2b5a891c069adaac","ts":"2026-05-12T02:49:32.050Z","cmd_args":["--model","sonnet","--parallel","3","--yes","--output","run-sonnet-4-6.jsonl"]}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":5,"latency_ms":2307,"ts":"2026-05-12T02:49:34.357Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":5,"latency_ms":1033,"ts":"2026-05-12T02:49:33.083Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":5,"latency_ms":1682,"ts":"2026-05-12T02:49:33.732Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":1,"predicted":"gbrain","expected":"gbrain","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":2141,"ts":"2026-05-12T02:49:36.498Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":2,"predicted":"gbrain","expected":"gbrain","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1435,"ts":"2026-05-12T02:49:35.792Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":3,"predicted":"gbrain","expected":"gbrain","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1121,"ts":"2026-05-12T02:49:35.478Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1034,"ts":"2026-05-12T02:49:37.532Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1285,"ts":"2026-05-12T02:49:37.783Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1537,"ts":"2026-05-12T02:49:38.035Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1052,"ts":"2026-05-12T02:49:39.087Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1069,"ts":"2026-05-12T02:49:39.104Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1260,"ts":"2026-05-12T02:49:39.295Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1413,"ts":"2026-05-12T02:49:40.708Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1400,"ts":"2026-05-12T02:49:40.695Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1369,"ts":"2026-05-12T02:49:40.664Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1072,"ts":"2026-05-12T02:49:41.780Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":2004,"ts":"2026-05-12T02:49:42.712Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1221,"ts":"2026-05-12T02:49:41.929Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1108,"ts":"2026-05-12T02:49:43.820Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1380,"ts":"2026-05-12T02:49:44.092Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1629,"ts":"2026-05-12T02:49:44.341Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1204,"ts":"2026-05-12T02:49:45.545Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1128,"ts":"2026-05-12T02:49:45.469Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1106,"ts":"2026-05-12T02:49:45.447Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1241,"ts":"2026-05-12T02:49:46.786Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":948,"ts":"2026-05-12T02:49:46.493Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1028,"ts":"2026-05-12T02:49:46.573Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":9,"latency_ms":1723,"ts":"2026-05-12T02:49:48.509Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":9,"latency_ms":1592,"ts":"2026-05-12T02:49:48.378Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":9,"latency_ms":1293,"ts":"2026-05-12T02:49:48.079Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":7,"latency_ms":1376,"ts":"2026-05-12T02:49:49.885Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":7,"latency_ms":1691,"ts":"2026-05-12T02:49:50.201Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":3,"predicted":"idea-ingest","expected":"idea-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":7,"latency_ms":1512,"ts":"2026-05-12T02:49:50.021Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1593,"ts":"2026-05-12T02:49:51.794Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1593,"ts":"2026-05-12T02:49:51.794Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1593,"ts":"2026-05-12T02:49:51.794Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1522,"ts":"2026-05-12T02:49:53.316Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1395,"ts":"2026-05-12T02:49:53.189Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1030,"ts":"2026-05-12T02:49:52.824Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":9,"latency_ms":1670,"ts":"2026-05-12T02:49:54.987Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":9,"latency_ms":1536,"ts":"2026-05-12T02:49:54.853Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":9,"latency_ms":1319,"ts":"2026-05-12T02:49:54.636Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":6,"latency_ms":1031,"ts":"2026-05-12T02:49:56.018Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":6,"latency_ms":947,"ts":"2026-05-12T02:49:55.934Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":6,"latency_ms":973,"ts":"2026-05-12T02:49:55.960Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":4,"latency_ms":1539,"ts":"2026-05-12T02:49:57.557Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":4,"latency_ms":1660,"ts":"2026-05-12T02:49:57.678Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":4,"latency_ms":1550,"ts":"2026-05-12T02:49:57.568Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":9,"latency_ms":1276,"ts":"2026-05-12T02:49:58.954Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":9,"latency_ms":1273,"ts":"2026-05-12T02:49:58.951Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":9,"latency_ms":1624,"ts":"2026-05-12T02:49:59.302Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":7,"latency_ms":1264,"ts":"2026-05-12T02:50:00.566Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":7,"latency_ms":1518,"ts":"2026-05-12T02:50:00.820Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":7,"latency_ms":1537,"ts":"2026-05-12T02:50:00.839Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":5,"latency_ms":1296,"ts":"2026-05-12T02:50:02.135Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":5,"latency_ms":1216,"ts":"2026-05-12T02:50:02.055Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":5,"latency_ms":1509,"ts":"2026-05-12T02:50:02.348Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1240,"ts":"2026-05-12T02:50:03.588Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1908,"ts":"2026-05-12T02:50:04.256Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1303,"ts":"2026-05-12T02:50:03.651Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":5,"latency_ms":1022,"ts":"2026-05-12T02:50:05.278Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":5,"latency_ms":1553,"ts":"2026-05-12T02:50:05.809Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":5,"latency_ms":1199,"ts":"2026-05-12T02:50:05.455Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":6,"latency_ms":1439,"ts":"2026-05-12T02:50:07.248Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":6,"latency_ms":1062,"ts":"2026-05-12T02:50:06.871Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":6,"latency_ms":1062,"ts":"2026-05-12T02:50:06.871Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1078,"ts":"2026-05-12T02:50:08.326Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":961,"ts":"2026-05-12T02:50:08.209Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1078,"ts":"2026-05-12T02:50:08.326Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1042,"ts":"2026-05-12T02:50:09.368Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1183,"ts":"2026-05-12T02:50:09.509Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1052,"ts":"2026-05-12T02:50:09.378Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1246,"ts":"2026-05-12T02:50:10.755Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1596,"ts":"2026-05-12T02:50:11.105Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1708,"ts":"2026-05-12T02:50:11.217Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":5,"latency_ms":1686,"ts":"2026-05-12T02:50:12.903Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":5,"latency_ms":1085,"ts":"2026-05-12T02:50:12.302Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":5,"latency_ms":1242,"ts":"2026-05-12T02:50:12.459Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":4,"latency_ms":1209,"ts":"2026-05-12T02:50:14.112Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":4,"latency_ms":1372,"ts":"2026-05-12T02:50:14.275Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":4,"latency_ms":992,"ts":"2026-05-12T02:50:13.895Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1214,"ts":"2026-05-12T02:50:15.489Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1055,"ts":"2026-05-12T02:50:15.330Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1263,"ts":"2026-05-12T02:50:15.538Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1007,"ts":"2026-05-12T02:50:16.545Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1241,"ts":"2026-05-12T02:50:16.779Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1596,"ts":"2026-05-12T02:50:17.134Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"freshness-monitor","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1887,"ts":"2026-05-12T02:50:19.021Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"freshness-monitor","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1330,"ts":"2026-05-12T02:50:18.464Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"benchmark-gbrain","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1777,"ts":"2026-05-12T02:50:18.911Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1016,"ts":"2026-05-12T02:50:20.037Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1700,"ts":"2026-05-12T02:50:20.721Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1524,"ts":"2026-05-12T02:50:20.545Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":5907,"ts":"2026-05-12T02:50:26.628Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1594,"ts":"2026-05-12T02:50:22.315Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1251,"ts":"2026-05-12T02:50:21.972Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1258,"ts":"2026-05-12T02:50:27.886Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1228,"ts":"2026-05-12T02:50:27.856Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1253,"ts":"2026-05-12T02:50:27.881Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1218,"ts":"2026-05-12T02:50:29.105Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":941,"ts":"2026-05-12T02:50:28.828Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":978,"ts":"2026-05-12T02:50:28.865Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":9,"latency_ms":1408,"ts":"2026-05-12T02:50:30.513Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":9,"latency_ms":1391,"ts":"2026-05-12T02:50:30.496Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":9,"latency_ms":1480,"ts":"2026-05-12T02:50:30.585Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":7,"latency_ms":937,"ts":"2026-05-12T02:50:31.522Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":7,"latency_ms":869,"ts":"2026-05-12T02:50:31.454Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":7,"latency_ms":1021,"ts":"2026-05-12T02:50:31.606Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":871,"ts":"2026-05-12T02:50:32.477Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1044,"ts":"2026-05-12T02:50:32.650Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1108,"ts":"2026-05-12T02:50:32.714Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":982,"ts":"2026-05-12T02:50:33.696Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1058,"ts":"2026-05-12T02:50:33.772Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":957,"ts":"2026-05-12T02:50:33.671Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":9,"latency_ms":984,"ts":"2026-05-12T02:50:34.756Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":9,"latency_ms":1450,"ts":"2026-05-12T02:50:35.222Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":9,"latency_ms":1341,"ts":"2026-05-12T02:50:35.113Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":6,"latency_ms":1502,"ts":"2026-05-12T02:50:36.724Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":6,"latency_ms":1326,"ts":"2026-05-12T02:50:36.548Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":6,"latency_ms":1326,"ts":"2026-05-12T02:50:36.548Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":4,"latency_ms":1060,"ts":"2026-05-12T02:50:37.784Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":4,"latency_ms":1600,"ts":"2026-05-12T02:50:38.324Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":4,"latency_ms":1415,"ts":"2026-05-12T02:50:38.139Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":9,"latency_ms":1062,"ts":"2026-05-12T02:50:39.386Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":9,"latency_ms":1062,"ts":"2026-05-12T02:50:39.386Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":9,"latency_ms":971,"ts":"2026-05-12T02:50:39.295Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":7,"latency_ms":1634,"ts":"2026-05-12T02:50:41.020Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":7,"latency_ms":999,"ts":"2026-05-12T02:50:40.386Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":7,"latency_ms":890,"ts":"2026-05-12T02:50:40.277Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":5,"latency_ms":1141,"ts":"2026-05-12T02:50:42.161Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":5,"latency_ms":938,"ts":"2026-05-12T02:50:41.958Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":5,"latency_ms":878,"ts":"2026-05-12T02:50:41.898Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1185,"ts":"2026-05-12T02:50:43.347Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":990,"ts":"2026-05-12T02:50:43.151Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":954,"ts":"2026-05-12T02:50:43.115Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":5,"latency_ms":1043,"ts":"2026-05-12T02:50:44.390Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":5,"latency_ms":1011,"ts":"2026-05-12T02:50:44.358Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":5,"latency_ms":922,"ts":"2026-05-12T02:50:44.269Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":6,"latency_ms":1193,"ts":"2026-05-12T02:50:45.583Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":6,"latency_ms":1196,"ts":"2026-05-12T02:50:45.586Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":6,"latency_ms":5248,"ts":"2026-05-12T02:50:49.638Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1257,"ts":"2026-05-12T02:50:50.895Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1487,"ts":"2026-05-12T02:50:51.125Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1100,"ts":"2026-05-12T02:50:50.738Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1413,"ts":"2026-05-12T02:50:52.538Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1236,"ts":"2026-05-12T02:50:52.361Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1590,"ts":"2026-05-12T02:50:52.715Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":5,"latency_ms":1452,"ts":"2026-05-12T02:50:54.167Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":5,"latency_ms":1202,"ts":"2026-05-12T02:50:53.917Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":5,"latency_ms":1452,"ts":"2026-05-12T02:50:54.167Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1003,"ts":"2026-05-12T02:50:55.170Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":987,"ts":"2026-05-12T02:50:55.154Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1296,"ts":"2026-05-12T02:50:55.463Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":6,"latency_ms":1360,"ts":"2026-05-12T02:50:56.824Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":6,"latency_ms":1029,"ts":"2026-05-12T02:50:56.493Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":6,"latency_ms":2308,"ts":"2026-05-12T02:50:57.772Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-pdf-auto","expected":"brain-pdf","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":8,"latency_ms":2125,"ts":"2026-05-12T02:50:59.897Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf-auto","expected":"brain-pdf","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":8,"latency_ms":1538,"ts":"2026-05-12T02:50:59.310Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-pdf-auto","expected":"brain-pdf","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":8,"latency_ms":1188,"ts":"2026-05-12T02:50:58.960Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":826,"ts":"2026-05-12T02:51:00.723Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1031,"ts":"2026-05-12T02:51:00.928Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":949,"ts":"2026-05-12T02:51:00.846Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1494,"ts":"2026-05-12T02:51:02.422Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1311,"ts":"2026-05-12T02:51:02.239Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1253,"ts":"2026-05-12T02:51:02.181Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":7,"latency_ms":1047,"ts":"2026-05-12T02:51:03.469Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":7,"latency_ms":1143,"ts":"2026-05-12T02:51:03.565Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":7,"latency_ms":919,"ts":"2026-05-12T02:51:03.341Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1301,"ts":"2026-05-12T02:51:04.866Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1159,"ts":"2026-05-12T02:51:04.724Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1417,"ts":"2026-05-12T02:51:04.982Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1209,"ts":"2026-05-12T02:51:06.191Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1609,"ts":"2026-05-12T02:51:06.591Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1432,"ts":"2026-05-12T02:51:06.414Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1765,"ts":"2026-05-12T02:51:08.356Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":3599,"ts":"2026-05-12T02:51:10.190Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1060,"ts":"2026-05-12T02:51:07.651Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1213,"ts":"2026-05-12T02:51:11.403Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":878,"ts":"2026-05-12T02:51:11.068Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1040,"ts":"2026-05-12T02:51:11.230Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":1542,"ts":"2026-05-12T02:51:12.945Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":930,"ts":"2026-05-12T02:51:12.333Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":971,"ts":"2026-05-12T02:51:12.374Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1203,"ts":"2026-05-12T02:51:14.148Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1513,"ts":"2026-05-12T02:51:14.458Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1342,"ts":"2026-05-12T02:51:14.287Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":5,"latency_ms":4435,"ts":"2026-05-12T02:51:18.893Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":5,"latency_ms":1355,"ts":"2026-05-12T02:51:15.813Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":5,"latency_ms":978,"ts":"2026-05-12T02:51:15.436Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"voice-note-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":5,"latency_ms":1026,"ts":"2026-05-12T02:51:19.919Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"voice-note-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":5,"latency_ms":1323,"ts":"2026-05-12T02:51:20.216Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"voice-note-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":5,"latency_ms":1372,"ts":"2026-05-12T02:51:20.265Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1295,"ts":"2026-05-12T02:51:21.560Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":2501,"ts":"2026-05-12T02:51:22.766Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1110,"ts":"2026-05-12T02:51:21.375Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1067,"ts":"2026-05-12T02:51:23.833Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1059,"ts":"2026-05-12T02:51:23.825Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1237,"ts":"2026-05-12T02:51:24.003Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":9,"latency_ms":1122,"ts":"2026-05-12T02:51:25.125Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":9,"latency_ms":1537,"ts":"2026-05-12T02:51:25.540Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":9,"latency_ms":1755,"ts":"2026-05-12T02:51:25.758Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":7,"latency_ms":900,"ts":"2026-05-12T02:51:26.658Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":7,"latency_ms":852,"ts":"2026-05-12T02:51:26.610Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":7,"latency_ms":1438,"ts":"2026-05-12T02:51:27.196Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":5,"latency_ms":1174,"ts":"2026-05-12T02:51:28.370Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":5,"latency_ms":1347,"ts":"2026-05-12T02:51:28.543Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":5,"latency_ms":1098,"ts":"2026-05-12T02:51:28.294Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":1455,"ts":"2026-05-12T02:51:29.998Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":990,"ts":"2026-05-12T02:51:29.533Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":1801,"ts":"2026-05-12T02:51:30.344Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":2027,"ts":"2026-05-12T02:51:32.371Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":1009,"ts":"2026-05-12T02:51:31.353Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":950,"ts":"2026-05-12T02:51:31.294Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"acp-coding","expected":"skill-creator","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":7,"latency_ms":1008,"ts":"2026-05-12T02:51:33.379Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"acp-coding","expected":"skill-creator","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":7,"latency_ms":931,"ts":"2026-05-12T02:51:33.302Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"acp-coding","expected":"skill-creator","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":7,"latency_ms":1036,"ts":"2026-05-12T02:51:33.407Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-prep","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":901,"ts":"2026-05-12T02:51:34.308Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":892,"ts":"2026-05-12T02:51:34.299Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":1016,"ts":"2026-05-12T02:51:34.423Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":879,"ts":"2026-05-12T02:51:35.302Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":948,"ts":"2026-05-12T02:51:35.371Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":930,"ts":"2026-05-12T02:51:35.353Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1022,"ts":"2026-05-12T02:51:36.393Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1406,"ts":"2026-05-12T02:51:36.777Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":907,"ts":"2026-05-12T02:51:36.278Z"}
|
||||
@@ -1,8 +0,0 @@
|
||||
// 5 held-out blind fixtures. Authored before the variant resolvers were
|
||||
// fully reviewed; target skills present in both real variants.
|
||||
// Held-out accuracy is the headline claim in skills/functional-area-resolver/SKILL.md.
|
||||
{"intent":"Skillify the JSON parsing helper I wrote last week","expected_skill":"skillify"}
|
||||
{"intent":"Create a new skill for cataloging books I've finished","expected_skill":"skill-creator"}
|
||||
{"intent":"Build me a daily prep summary for tomorrow","expected_skill":"daily-task-prep"}
|
||||
{"intent":"Pull the contact details for Maria from my address book","expected_skill":"google-contacts"}
|
||||
{"intent":"Run a healthcheck on my services","expected_skill":"healthcheck"}
|
||||
@@ -1,24 +0,0 @@
|
||||
// 20 training fixtures for the functional-area-resolver A/B eval.
|
||||
// Each line: {"intent": "<user phrasing>", "expected_skill": "<skill slug>"}
|
||||
// Target skills are present in BOTH variants (verified against the
|
||||
// real production AGENTS.md at git commit 93848ff3b^ and 93848ff3b).
|
||||
{"intent":"Create a person page for John Smith and enrich it from his GitHub","expected_skill":"enrich"}
|
||||
{"intent":"What do we know about Stripe","expected_skill":"gbrain"}
|
||||
{"intent":"Make a PDF from my brain page on dispatcher patterns","expected_skill":"brain-pdf"}
|
||||
{"intent":"Publish this brain page as a shareable link","expected_skill":"brain-publish"}
|
||||
{"intent":"Run brain integrity — what's lost in my archive","expected_skill":"brain-librarian"}
|
||||
{"intent":"Fix the broken citations on this page","expected_skill":"citation-fixer"}
|
||||
{"intent":"Make a personalized version of Atomic Habits with my brain context","expected_skill":"book-mirror"}
|
||||
{"intent":"Read Thinking Fast and Slow through the lens of my product work","expected_skill":"strategic-reading"}
|
||||
{"intent":"Synthesize my concepts about resolver design and routing","expected_skill":"concept-synthesis"}
|
||||
{"intent":"Crawl my dropbox archive for old notes I should pull in","expected_skill":"archive-crawler"}
|
||||
{"intent":"Ingest this article from The Atlantic into my brain","expected_skill":"idea-ingest"}
|
||||
{"intent":"Process this YouTube video into the brain","expected_skill":"media-ingest"}
|
||||
{"intent":"I have a meeting transcript to file from this morning","expected_skill":"meeting-ingestion"}
|
||||
{"intent":"Save this voice memo and transcribe it","expected_skill":"voice-note-ingest"}
|
||||
{"intent":"What's on my calendar tomorrow","expected_skill":"google-calendar"}
|
||||
{"intent":"Draft a reply email to Sarah","expected_skill":"executive-assistant"}
|
||||
{"intent":"Research what's new about WebGPU adoption","expected_skill":"perplexity-research"}
|
||||
{"intent":"Pull my recent X posts and ingest them","expected_skill":"x-ingest"}
|
||||
{"intent":"Check me into the coffee shop I'm at","expected_skill":"checkin"}
|
||||
{"intent":"Add a task for tomorrow's meeting prep","expected_skill":"daily-task-manager"}
|
||||
@@ -1,302 +0,0 @@
|
||||
/**
|
||||
* Unit tests for the functional-area-resolver A/B eval harness.
|
||||
* Run with: bun test evals/functional-area-resolver/harness-runner.test.ts
|
||||
*
|
||||
* Covers every pure function so contributors can debug without spending
|
||||
* money on every iteration. main() smoke test is omitted in this slice
|
||||
* (it would require mocking gateway transport + filesystem; the harness's
|
||||
* --limit 1 mode is a sufficient real smoke check at ~$0.01 per run).
|
||||
*/
|
||||
|
||||
import { test, expect } from 'bun:test';
|
||||
import {
|
||||
parseFixtures,
|
||||
buildPrompt,
|
||||
parseModelResponse,
|
||||
scoreFixture,
|
||||
scoreFixtureLenient,
|
||||
parseDispatcherLists,
|
||||
meanAndCI95,
|
||||
estimateCost,
|
||||
hashContent,
|
||||
parseArgs,
|
||||
resolveModel,
|
||||
PROMPT_TEMPLATE,
|
||||
MODEL_ID,
|
||||
MODEL_ALIASES,
|
||||
} from './harness-runner.ts';
|
||||
|
||||
test('parseFixtures: parses valid JSONL', () => {
|
||||
const raw = `{"intent":"foo","expected_skill":"bar"}\n{"intent":"baz","expected_skill":"qux"}\n`;
|
||||
const out = parseFixtures(raw);
|
||||
expect(out).toEqual([
|
||||
{ intent: 'foo', expected_skill: 'bar' },
|
||||
{ intent: 'baz', expected_skill: 'qux' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseFixtures: skips // comments and blank lines', () => {
|
||||
const raw = `// header comment\n{"intent":"a","expected_skill":"b"}\n\n// another comment\n{"intent":"c","expected_skill":"d"}\n`;
|
||||
const out = parseFixtures(raw);
|
||||
expect(out).toHaveLength(2);
|
||||
expect(out[0].intent).toBe('a');
|
||||
});
|
||||
|
||||
test('parseFixtures: throws on missing required fields', () => {
|
||||
expect(() => parseFixtures(`{"intent":"foo"}\n`)).toThrow(/missing required fields/);
|
||||
});
|
||||
|
||||
test('parseFixtures: throws on invalid JSON', () => {
|
||||
expect(() => parseFixtures(`{not json}\n`)).toThrow(/Bad fixture JSON/);
|
||||
});
|
||||
|
||||
test('buildPrompt: injects variant content and intent', () => {
|
||||
const prompt = buildPrompt('RESOLVER X', 'INTENT Y');
|
||||
expect(prompt).toContain('RESOLVER X');
|
||||
expect(prompt).toContain('INTENT Y');
|
||||
expect(prompt).not.toContain('<<<RESOLVER_CONTENT>>>');
|
||||
expect(prompt).not.toContain('<<<INTENT>>>');
|
||||
});
|
||||
|
||||
test('parseModelResponse: bare slug', () => {
|
||||
expect(parseModelResponse('enrich')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: strips fenced output', () => {
|
||||
expect(parseModelResponse('```\nenrich\n```')).toBe('enrich');
|
||||
expect(parseModelResponse('```text\nenrich\n```')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: extracts from JSON object', () => {
|
||||
expect(parseModelResponse('{"skill": "book-mirror"}')).toBe('book-mirror');
|
||||
expect(parseModelResponse('{"skill_slug": "query"}')).toBe('query');
|
||||
});
|
||||
|
||||
test('parseModelResponse: strips quotes and backticks', () => {
|
||||
expect(parseModelResponse('"enrich"')).toBe('enrich');
|
||||
expect(parseModelResponse('`enrich`')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: picks first slug-shaped token if model prefaces with prose', () => {
|
||||
expect(parseModelResponse('The skill is enrich.')).toBe('the'); // first token wins; documents permissive matcher
|
||||
expect(parseModelResponse('enrich is the answer')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: lowercases output', () => {
|
||||
expect(parseModelResponse('ENRICH')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('scoreFixture: exact match returns 1', () => {
|
||||
expect(scoreFixture('enrich', 'enrich')).toBe(1);
|
||||
});
|
||||
|
||||
test('scoreFixture: mismatch returns 0', () => {
|
||||
expect(scoreFixture('enrich', 'query')).toBe(0);
|
||||
});
|
||||
|
||||
test('scoreFixture: case-sensitive at this layer (caller lowercases via parseModelResponse)', () => {
|
||||
expect(scoreFixture('Enrich', 'enrich')).toBe(0);
|
||||
});
|
||||
|
||||
test('meanAndCI95: empty array returns zeros', () => {
|
||||
expect(meanAndCI95([])).toEqual({ mean: 0, halfWidthCI: 0 });
|
||||
});
|
||||
|
||||
test('meanAndCI95: single value returns mean with zero CI', () => {
|
||||
expect(meanAndCI95([0.95])).toEqual({ mean: 0.95, halfWidthCI: 0 });
|
||||
});
|
||||
|
||||
test('meanAndCI95: three equal values returns mean with zero CI', () => {
|
||||
const r = meanAndCI95([1, 1, 1]);
|
||||
expect(r.mean).toBe(1);
|
||||
expect(r.halfWidthCI).toBe(0);
|
||||
});
|
||||
|
||||
test('meanAndCI95: three different values returns plausible CI', () => {
|
||||
const r = meanAndCI95([0.8, 0.9, 1.0]);
|
||||
expect(r.mean).toBeCloseTo(0.9, 5);
|
||||
expect(r.halfWidthCI).toBeGreaterThan(0);
|
||||
expect(r.halfWidthCI).toBeLessThan(0.5);
|
||||
});
|
||||
|
||||
test('estimateCost: uses Opus 4.7 pricing by default', () => {
|
||||
const cost = estimateCost(100, 'claude-opus-4-7', 1000, 50);
|
||||
// 100 calls * 1000 input tokens = 100K input → $0.50 at $5/MTok
|
||||
// 100 calls * 50 output tokens = 5K output → $0.125 at $25/MTok
|
||||
expect(cost).toBeCloseTo(0.625, 2);
|
||||
});
|
||||
|
||||
test('estimateCost: Sonnet pricing differs from Opus', () => {
|
||||
const opus = estimateCost(100, 'claude-opus-4-7', 1000, 50);
|
||||
const sonnet = estimateCost(100, 'claude-sonnet-4-6', 1000, 50);
|
||||
const haiku = estimateCost(100, 'claude-haiku-4-5-20251001', 1000, 50);
|
||||
expect(sonnet).toBeLessThan(opus);
|
||||
expect(haiku).toBeLessThan(sonnet);
|
||||
});
|
||||
|
||||
test('estimateCost: zero calls returns zero', () => {
|
||||
expect(estimateCost(0)).toBe(0);
|
||||
});
|
||||
|
||||
test('estimateCost: unknown model returns zero', () => {
|
||||
expect(estimateCost(100, 'unknown-model')).toBe(0);
|
||||
});
|
||||
|
||||
test('hashContent: produces stable 16-char hex prefix', () => {
|
||||
const h1 = hashContent('hello world');
|
||||
const h2 = hashContent('hello world');
|
||||
expect(h1).toBe(h2);
|
||||
expect(h1).toHaveLength(16);
|
||||
expect(h1).toMatch(/^[0-9a-f]+$/);
|
||||
});
|
||||
|
||||
test('hashContent: different inputs produce different hashes', () => {
|
||||
expect(hashContent('a')).not.toBe(hashContent('b'));
|
||||
});
|
||||
|
||||
test('parseArgs: defaults are sensible', () => {
|
||||
expect(parseArgs([])).toEqual({
|
||||
limit: null,
|
||||
parallel: 1,
|
||||
output: null,
|
||||
help: false,
|
||||
yes: false,
|
||||
model: MODEL_ID,
|
||||
variantsDir: 'variants',
|
||||
variantFiles: null,
|
||||
});
|
||||
});
|
||||
|
||||
test('parseArgs: --model alias', () => {
|
||||
expect(parseArgs(['--model', 'sonnet']).model).toBe('sonnet');
|
||||
expect(parseArgs(['--model', 'anthropic:claude-haiku-4-5-20251001']).model).toBe('anthropic:claude-haiku-4-5-20251001');
|
||||
});
|
||||
|
||||
test('parseArgs: --variants comma-list', () => {
|
||||
expect(parseArgs(['--variants', 'a,b,c']).variantFiles).toEqual(['a', 'b', 'c']);
|
||||
});
|
||||
|
||||
test('parseArgs: --variants-dir', () => {
|
||||
expect(parseArgs(['--variants-dir', 'variants-sweep']).variantsDir).toBe('variants-sweep');
|
||||
});
|
||||
|
||||
test('resolveModel: aliases', () => {
|
||||
expect(resolveModel('opus')).toEqual({ full: 'anthropic:claude-opus-4-7', bare: 'claude-opus-4-7' });
|
||||
expect(resolveModel('sonnet')).toEqual({ full: 'anthropic:claude-sonnet-4-6', bare: 'claude-sonnet-4-6' });
|
||||
expect(resolveModel('haiku').full).toBe(MODEL_ALIASES.haiku);
|
||||
});
|
||||
|
||||
test('resolveModel: passthrough for full id', () => {
|
||||
expect(resolveModel('anthropic:claude-opus-4-7').bare).toBe('claude-opus-4-7');
|
||||
expect(resolveModel('anthropic:claude-something-future').bare).toBe('claude-something-future');
|
||||
});
|
||||
|
||||
test('resolveModel: non-anthropic provider passes through unchanged', () => {
|
||||
expect(resolveModel('openai:gpt-4o')).toEqual({ full: 'openai:gpt-4o', bare: 'openai:gpt-4o' });
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: extracts dispatcher → sub-skills', () => {
|
||||
const variant = `
|
||||
- **Brain**: foo bar → \`brain-ops\` (dispatcher for: enrich, query, citation-fixer)
|
||||
- **Comms**: email → \`exec-assist\` (dispatcher for: gmail, slack)
|
||||
- Bare row → \`bare-skill\`
|
||||
`;
|
||||
const m = parseDispatcherLists(variant);
|
||||
expect(m.size).toBe(2);
|
||||
expect(m.get('brain-ops')).toEqual(new Set(['brain-ops', 'enrich', 'query', 'citation-fixer']));
|
||||
expect(m.get('exec-assist')).toEqual(new Set(['exec-assist', 'gmail', 'slack']));
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: accepts ASCII -> arrow (SKILL.md template format)', () => {
|
||||
// Codex review P2-2: SKILL.md Step 4 documents the template with `->`,
|
||||
// but the production variants use Unicode `→`. The regex must match
|
||||
// both or downstream users following the template silently fall through
|
||||
// to strict-only scoring.
|
||||
const variant = `
|
||||
- **Brain**: foo bar -> \`brain-ops\` (dispatcher for: enrich, query)
|
||||
- **Comms**: email -> \`exec-assist\` (dispatcher for: gmail)
|
||||
`;
|
||||
const m = parseDispatcherLists(variant);
|
||||
expect(m.size).toBe(2);
|
||||
expect(m.get('brain-ops')).toEqual(new Set(['brain-ops', 'enrich', 'query']));
|
||||
expect(m.get('exec-assist')).toEqual(new Set(['exec-assist', 'gmail']));
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: mixed Unicode + ASCII arrows in same file', () => {
|
||||
// A real-world fork could migrate gradually; harness must handle both.
|
||||
const variant = `
|
||||
- **Brain**: foo → \`brain-ops\` (dispatcher for: enrich, query)
|
||||
- **Comms**: email -> \`exec-assist\` (dispatcher for: gmail, slack)
|
||||
`;
|
||||
const m = parseDispatcherLists(variant);
|
||||
expect(m.size).toBe(2);
|
||||
expect(m.get('brain-ops')?.has('enrich')).toBe(true);
|
||||
expect(m.get('exec-assist')?.has('gmail')).toBe(true);
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: zero dispatchers when no clauses present', () => {
|
||||
const variant = `
|
||||
- Row 1 → \`alpha\`
|
||||
- Row 2 → \`beta\`
|
||||
`;
|
||||
expect(parseDispatcherLists(variant).size).toBe(0);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: exact match = 1', () => {
|
||||
expect(scoreFixtureLenient('enrich', 'enrich', new Map())).toBe(1);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: same-area sub-skill = 1', () => {
|
||||
const lists = new Map([['brain-ops', new Set(['brain-ops', 'enrich', 'query'])]]);
|
||||
expect(scoreFixtureLenient('enrich', 'query', lists)).toBe(1);
|
||||
expect(scoreFixtureLenient('brain-ops', 'enrich', lists)).toBe(1);
|
||||
expect(scoreFixtureLenient('enrich', 'brain-ops', lists)).toBe(1);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: cross-area = 0', () => {
|
||||
const lists = new Map([
|
||||
['brain-ops', new Set(['brain-ops', 'enrich'])],
|
||||
['comms', new Set(['comms', 'gmail'])],
|
||||
]);
|
||||
expect(scoreFixtureLenient('enrich', 'gmail', lists)).toBe(0);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: no dispatcher map = falls back to strict', () => {
|
||||
expect(scoreFixtureLenient('foo', 'bar', new Map())).toBe(0);
|
||||
});
|
||||
|
||||
test('parseArgs: --limit', () => {
|
||||
expect(parseArgs(['--limit', '5']).limit).toBe(5);
|
||||
});
|
||||
|
||||
test('parseArgs: --limit rejects non-positive', () => {
|
||||
expect(() => parseArgs(['--limit', '0'])).toThrow();
|
||||
expect(() => parseArgs(['--limit', '-3'])).toThrow();
|
||||
expect(() => parseArgs(['--limit', 'foo'])).toThrow();
|
||||
});
|
||||
|
||||
test('parseArgs: --parallel', () => {
|
||||
expect(parseArgs(['--parallel', '4']).parallel).toBe(4);
|
||||
});
|
||||
|
||||
test('parseArgs: --output', () => {
|
||||
expect(parseArgs(['--output', '/tmp/x.jsonl']).output).toBe('/tmp/x.jsonl');
|
||||
});
|
||||
|
||||
test('parseArgs: --help and --yes', () => {
|
||||
expect(parseArgs(['--help']).help).toBe(true);
|
||||
expect(parseArgs(['--yes']).yes).toBe(true);
|
||||
});
|
||||
|
||||
test('parseArgs: rejects unknown flags', () => {
|
||||
expect(() => parseArgs(['--bogus'])).toThrow(/Unknown flag/);
|
||||
});
|
||||
|
||||
test('MODEL_ID is pinned to Opus 4.7', () => {
|
||||
expect(MODEL_ID).toBe('anthropic:claude-opus-4-7');
|
||||
});
|
||||
|
||||
test('PROMPT_TEMPLATE contains both placeholders', () => {
|
||||
expect(PROMPT_TEMPLATE).toContain('<<<RESOLVER_CONTENT>>>');
|
||||
expect(PROMPT_TEMPLATE).toContain('<<<INTENT>>>');
|
||||
});
|
||||
@@ -1,599 +0,0 @@
|
||||
/**
|
||||
* functional-area-resolver A/B eval runner.
|
||||
*
|
||||
* Reads three variant resolver files + two fixture corpora, runs each
|
||||
* (fixture, variant, seed in {1,2,3}) through Anthropic Opus 4.7 via
|
||||
* gbrain's gateway, scores the response, writes one JSONL row per call,
|
||||
* computes per-variant accuracy mean + 95% CI, prints a summary table.
|
||||
*
|
||||
* Receipts bind (model, prompt_template_hash, fixtures_hash, ts, seed)
|
||||
* so re-runs are auditable. Output JSONL begins with a receipt header.
|
||||
*
|
||||
* Pinned to anthropic:claude-opus-4-7. Update MODEL_ID and re-baseline
|
||||
* when Anthropic ships a new Opus generation. Cost: ~$1.70 per full run
|
||||
* (225 calls × ~$0.0076 each at $5/$25 per MTok input/output).
|
||||
*
|
||||
* Lives outside `skills/` deliberately — the skillpack bundler walks
|
||||
* `skills/<skill>/` recursively, so an eval surface in there would ship
|
||||
* to every downstream install. Importing `src/core/ai/gateway.ts` is
|
||||
* legitimate from this location because the eval is gbrain-repo-only.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { execSync } from 'node:child_process';
|
||||
|
||||
import { configureGateway, chat } from '../../src/core/ai/gateway.ts';
|
||||
import { loadConfig } from '../../src/core/config.ts';
|
||||
import { ANTHROPIC_PRICING } from '../../src/core/anthropic-pricing.ts';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = resolve(__dirname, '..', '..');
|
||||
|
||||
// Default model — pinned so the canonical baseline-runs/<date>-opus-4-7.jsonl
|
||||
// stays reproducible. Override with --model for cross-model eval (T3a).
|
||||
export const MODEL_ID = 'anthropic:claude-opus-4-7';
|
||||
|
||||
export const MODEL_ALIASES: Record<string, string> = {
|
||||
opus: 'anthropic:claude-opus-4-7',
|
||||
sonnet: 'anthropic:claude-sonnet-4-6',
|
||||
haiku: 'anthropic:claude-haiku-4-5-20251001',
|
||||
};
|
||||
|
||||
export function resolveModel(spec: string): { full: string; bare: string } {
|
||||
const full = MODEL_ALIASES[spec] ?? spec;
|
||||
const bare = full.startsWith('anthropic:') ? full.slice('anthropic:'.length) : full;
|
||||
return { full, bare };
|
||||
}
|
||||
|
||||
const VARIANT_NAMES = ['baseline', 'functional-areas', 'resolver-of-resolvers'] as const;
|
||||
type VariantName = (typeof VARIANT_NAMES)[number];
|
||||
|
||||
const SEEDS = [1, 2, 3] as const;
|
||||
|
||||
export interface Fixture {
|
||||
intent: string;
|
||||
expected_skill: string;
|
||||
}
|
||||
|
||||
export interface RunRow {
|
||||
kind: 'run';
|
||||
fixture_id: number;
|
||||
corpus: 'training' | 'held_out';
|
||||
variant: VariantName;
|
||||
seed: number;
|
||||
predicted: string;
|
||||
expected: string;
|
||||
/** Strict score: predicted exactly equals expected. */
|
||||
correct: 0 | 1;
|
||||
/** Lenient score: predicted is in the same dispatcher area as expected (T1a). */
|
||||
correct_lenient: 0 | 1;
|
||||
model: string;
|
||||
input_tokens: number;
|
||||
output_tokens: number;
|
||||
latency_ms: number;
|
||||
ts: string;
|
||||
}
|
||||
|
||||
export interface ReceiptRow {
|
||||
kind: 'receipt';
|
||||
model: string;
|
||||
prompt_template_hash: string;
|
||||
fixtures_hash: string;
|
||||
fixtures_held_out_hash: string;
|
||||
/** Git sha of the harness at run time (T4). Detect stale numbers when harness changes. */
|
||||
harness_sha: string | null;
|
||||
ts: string;
|
||||
cmd_args: string[];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pure functions (testable without API key)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export const PROMPT_TEMPLATE = `You are a routing classifier for a skill-based agent. Given the resolver below and the user's intent, return the single most-specific skill slug that should handle the intent.
|
||||
|
||||
Rules:
|
||||
- Return ONLY a slug. No explanation, no quotes, no markdown — just the slug.
|
||||
- Some entries are functional-area dispatchers shaped like:
|
||||
"**Area name**: triggers... → \`dispatcher-skill\` (dispatcher for: subskill-a, subskill-b, subskill-c, ...)"
|
||||
When the user's intent matches an area, RETURN THE MOST-SPECIFIC SUB-SKILL from that area's "dispatcher for" list, not the dispatcher itself. The dispatcher slug is only correct when no listed sub-skill is more specific to the intent.
|
||||
- If a row has no dispatcher list, return its slug directly.
|
||||
|
||||
RESOLVER:
|
||||
<<<RESOLVER_CONTENT>>>
|
||||
|
||||
USER INTENT: <<<INTENT>>>
|
||||
|
||||
SKILL SLUG:`;
|
||||
|
||||
export function parseFixtures(rawJsonl: string): Fixture[] {
|
||||
const out: Fixture[] = [];
|
||||
const lines = rawJsonl.split('\n');
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (trimmed.length === 0) continue;
|
||||
if (trimmed.startsWith('//')) continue;
|
||||
let obj: any;
|
||||
try {
|
||||
obj = JSON.parse(trimmed);
|
||||
} catch (err) {
|
||||
throw new Error(`Bad fixture JSON: ${trimmed.slice(0, 80)} — ${(err as Error).message}`);
|
||||
}
|
||||
if (typeof obj.intent !== 'string' || typeof obj.expected_skill !== 'string') {
|
||||
throw new Error(`Fixture missing required fields: ${trimmed.slice(0, 80)}`);
|
||||
}
|
||||
out.push({ intent: obj.intent, expected_skill: obj.expected_skill });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function loadVariant(path: string): string {
|
||||
return readFileSync(path, 'utf8');
|
||||
}
|
||||
|
||||
export function buildPrompt(variantContent: string, intent: string): string {
|
||||
return PROMPT_TEMPLATE.replace('<<<RESOLVER_CONTENT>>>', variantContent).replace('<<<INTENT>>>', intent);
|
||||
}
|
||||
|
||||
export function parseModelResponse(raw: string): string {
|
||||
// The model may return: bare slug, fenced slug, quoted slug, JSON-wrapped
|
||||
// slug, or slug with a leading explanation. We strip the obvious wrappers
|
||||
// and take the first line that looks like a slug.
|
||||
let s = raw.trim();
|
||||
// Strip ```...``` fences
|
||||
s = s.replace(/^```[a-zA-Z]*\n?/, '').replace(/\n?```\s*$/, '').trim();
|
||||
// If the response is JSON like {"skill": "foo"}, extract.
|
||||
if (s.startsWith('{')) {
|
||||
try {
|
||||
const obj = JSON.parse(s);
|
||||
if (typeof obj.skill === 'string') return obj.skill.trim().toLowerCase();
|
||||
if (typeof obj.skill_slug === 'string') return obj.skill_slug.trim().toLowerCase();
|
||||
if (typeof obj.expected_skill === 'string') return obj.expected_skill.trim().toLowerCase();
|
||||
} catch {}
|
||||
}
|
||||
// Strip surrounding quotes and backticks
|
||||
s = s.replace(/^[`"']|[`"']$/g, '').trim();
|
||||
// Take first non-empty line
|
||||
const firstLine = s.split(/\r?\n/).map(l => l.trim()).find(l => l.length > 0) ?? '';
|
||||
// If it starts with a prose preamble, look for a slug-shaped token
|
||||
const slugMatch = firstLine.match(/[a-z][a-z0-9-]+/i);
|
||||
return (slugMatch ? slugMatch[0] : firstLine).toLowerCase();
|
||||
}
|
||||
|
||||
export function scoreFixture(predicted: string, expected: string): 0 | 1 {
|
||||
return predicted === expected ? 1 : 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse every "...→ `dispatcher-slug` (dispatcher for: a, b, c, ...)" line
|
||||
* out of a variant resolver. Returns a map: dispatcher_slug → set of sub-skill
|
||||
* slugs reachable through it. Also includes the dispatcher_slug itself in
|
||||
* the set so it's a self-member.
|
||||
*
|
||||
* Variant shapes:
|
||||
* - functional-areas.md: "→ `brain-ops` (dispatcher for: enrich, query, ...)"
|
||||
* - resolver-of-resolvers.md: "→ `brain-ops`" (no dispatcher clause; returns {})
|
||||
* - baseline.md: per-skill rows (each row's slug becomes its own area)
|
||||
*
|
||||
* Used by lenientScore: a predicted slug counts as "same area as expected"
|
||||
* if both belong to the same dispatcher's reachable set, OR predicted is the
|
||||
* dispatcher and expected is a sub-skill (or vice versa).
|
||||
*/
|
||||
export function parseDispatcherLists(variantContent: string): Map<string, Set<string>> {
|
||||
const out = new Map<string, Set<string>>();
|
||||
// Match both Unicode `→` (used in the real production AGENTS.md the variants
|
||||
// came from) AND ASCII `->` (what SKILL.md's template emits when a user
|
||||
// follows the documented instructions). Codex review P2-2: without ASCII
|
||||
// support, downstream-authored resolvers silently fall through to strict
|
||||
// scoring even though SKILL.md tells the user the template uses `->`.
|
||||
const re = /(?:→|->)\s*`([a-z][a-z0-9-]*)`\s*\(dispatcher for:\s*([^)]+)\)/g;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = re.exec(variantContent)) !== null) {
|
||||
const dispatcher = m[1];
|
||||
const subSkills = m[2].split(',').map(s => s.trim()).filter(s => /^[a-z][a-z0-9-]*$/.test(s));
|
||||
const set = new Set<string>([dispatcher, ...subSkills]);
|
||||
out.set(dispatcher, set);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lenient scoring: predicted is correct if (predicted == expected) OR
|
||||
* (both predicted and expected are in the same dispatcher's reachable set
|
||||
* per the variant). This is the T1a re-scoring that surfaces "the LLM
|
||||
* picked a legitimate sub-skill, just not the one my fixture named."
|
||||
*
|
||||
* For variants with no dispatcher clauses (baseline, resolver-of-resolvers),
|
||||
* lenient collapses to strict.
|
||||
*/
|
||||
export function scoreFixtureLenient(
|
||||
predicted: string,
|
||||
expected: string,
|
||||
dispatcherLists: Map<string, Set<string>>,
|
||||
): 0 | 1 {
|
||||
if (predicted === expected) return 1;
|
||||
for (const set of dispatcherLists.values()) {
|
||||
if (set.has(predicted) && set.has(expected)) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** Capture the harness git sha so receipts can detect stale numbers. */
|
||||
export function getHarnessSha(): string | null {
|
||||
try {
|
||||
const sha = execSync('git rev-parse HEAD', { cwd: __dirname, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
return sha.length === 40 ? sha : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Mean and 95% CI via t-distribution (n=3, df=2, t-critical ≈ 4.303).
|
||||
* For n=3 with df=2 the 95% two-tailed t-critical is 4.303 per standard
|
||||
* tables. Returns the half-width of the CI (mean ± halfWidth).
|
||||
*/
|
||||
export function meanAndCI95(values: number[]): { mean: number; halfWidthCI: number } {
|
||||
if (values.length === 0) return { mean: 0, halfWidthCI: 0 };
|
||||
const mean = values.reduce((a, b) => a + b, 0) / values.length;
|
||||
if (values.length === 1) return { mean, halfWidthCI: 0 };
|
||||
const variance = values.reduce((acc, v) => acc + (v - mean) ** 2, 0) / (values.length - 1);
|
||||
const stdErr = Math.sqrt(variance / values.length);
|
||||
const tCrit = values.length === 3 ? 4.303 : values.length === 2 ? 12.706 : 1.96;
|
||||
return { mean, halfWidthCI: tCrit * stdErr };
|
||||
}
|
||||
|
||||
export function estimateCost(
|
||||
numCalls: number,
|
||||
modelBare: string = 'claude-opus-4-7',
|
||||
inputTokensPerCall = 1000,
|
||||
outputTokensPerCall = 50,
|
||||
): number {
|
||||
const pricing = ANTHROPIC_PRICING[modelBare];
|
||||
if (!pricing) return 0;
|
||||
const input = (numCalls * inputTokensPerCall) / 1_000_000;
|
||||
const output = (numCalls * outputTokensPerCall) / 1_000_000;
|
||||
return input * pricing.input + output * pricing.output;
|
||||
}
|
||||
|
||||
export function hashContent(content: string): string {
|
||||
return createHash('sha256').update(content).digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
export function writeJsonl(rows: (RunRow | ReceiptRow)[], outputPath: string): void {
|
||||
const dir = dirname(outputPath);
|
||||
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
|
||||
const lines = rows.map(r => JSON.stringify(r)).join('\n') + '\n';
|
||||
writeFileSync(outputPath, lines, 'utf8');
|
||||
}
|
||||
|
||||
export interface ParsedArgs {
|
||||
limit: number | null;
|
||||
parallel: number;
|
||||
output: string | null;
|
||||
help: boolean;
|
||||
yes: boolean;
|
||||
/** Model alias ('opus','sonnet','haiku') or full provider:model id. */
|
||||
model: string;
|
||||
/** Variants directory (default ./variants). */
|
||||
variantsDir: string;
|
||||
/** Custom variant glob (overrides default 3 variants); used by description-length sweep. */
|
||||
variantFiles: string[] | null;
|
||||
}
|
||||
|
||||
export function parseArgs(argv: string[]): ParsedArgs {
|
||||
const out: ParsedArgs = {
|
||||
limit: null, parallel: 1, output: null, help: false, yes: false,
|
||||
model: MODEL_ID, variantsDir: 'variants', variantFiles: null,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--help' || a === '-h') out.help = true;
|
||||
else if (a === '--yes' || a === '-y') out.yes = true;
|
||||
else if (a === '--limit') {
|
||||
const v = parseInt(argv[++i], 10);
|
||||
if (!Number.isFinite(v) || v < 1) throw new Error(`--limit must be a positive integer`);
|
||||
out.limit = v;
|
||||
} else if (a === '--parallel') {
|
||||
const v = parseInt(argv[++i], 10);
|
||||
if (!Number.isFinite(v) || v < 1) throw new Error(`--parallel must be a positive integer`);
|
||||
out.parallel = v;
|
||||
} else if (a === '--output') {
|
||||
out.output = argv[++i];
|
||||
} else if (a === '--model') {
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error(`--model requires a value (alias or provider:model)`);
|
||||
out.model = v;
|
||||
} else if (a === '--variants-dir') {
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error(`--variants-dir requires a path`);
|
||||
out.variantsDir = v;
|
||||
} else if (a === '--variants') {
|
||||
// Comma-separated list of variant file basenames (without .md). Used by sweep.
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error(`--variants requires a comma-separated list`);
|
||||
out.variantFiles = v.split(',').map(s => s.trim()).filter(Boolean);
|
||||
} else if (a.startsWith('--')) {
|
||||
throw new Error(`Unknown flag: ${a}`);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Gateway wrapper (mockable via __setChatTransportForTests)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function callModel(prompt: string, modelFull: string): Promise<{ text: string; input_tokens: number; output_tokens: number; latency_ms: number }> {
|
||||
const t0 = Date.now();
|
||||
const result = await chat({
|
||||
model: modelFull,
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
maxTokens: 64,
|
||||
});
|
||||
return {
|
||||
text: result.text,
|
||||
input_tokens: result.usage.input_tokens,
|
||||
output_tokens: result.usage.output_tokens,
|
||||
latency_ms: Date.now() - t0,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const HELP = `functional-area-resolver A/B eval harness
|
||||
|
||||
Usage:
|
||||
bun run harness-runner.ts [flags]
|
||||
node harness.mjs [flags] # CLI shim
|
||||
|
||||
Flags:
|
||||
--limit N Run only the first N (fixture × variant × seed) tuples
|
||||
--parallel N Run N tuples in parallel (default 1; gateway rate-lease bound)
|
||||
--output PATH Write JSONL to PATH (default: ./run-<ISO-ts>.jsonl)
|
||||
--model SPEC Model alias (opus|sonnet|haiku) or full provider:model id
|
||||
Default: opus (anthropic:claude-opus-4-7)
|
||||
--variants-dir PATH Override variants directory (default: ./variants)
|
||||
--variants A,B,C Comma-separated variant basenames (default: all 3 in variants-dir)
|
||||
Useful for description-length sweep where you have 4+ variants.
|
||||
--yes Skip the cost-estimate confirmation prompt
|
||||
--help Print this help
|
||||
|
||||
Cost rough estimates (75 calls/variant × num-variants × 3 seeds):
|
||||
Opus: ~$1.70 per 225-call run (1 model × 3 variants × 25 fixtures × 3 seeds)
|
||||
Sonnet: ~$1.02 per 225-call run
|
||||
Haiku: ~$0.34 per 225-call run
|
||||
|
||||
Output JSONL has each row scored TWICE: 'correct' (strict, predicted==expected)
|
||||
and 'correct_lenient' (predicted and expected are in the same dispatcher area).
|
||||
Summary reports both.
|
||||
`;
|
||||
|
||||
async function maybePromptCost(numCalls: number, modelFull: string, autoConfirm: boolean): Promise<boolean> {
|
||||
const { bare } = resolveModel(modelFull);
|
||||
const cost = estimateCost(numCalls, bare);
|
||||
process.stderr.write(`Estimated cost: ~$${cost.toFixed(2)} for ${numCalls} LLM calls via ${modelFull}.\n`);
|
||||
if (autoConfirm) return true;
|
||||
if (!process.stdin.isTTY) {
|
||||
process.stderr.write('Non-TTY context; pass --yes to confirm.\n');
|
||||
return false;
|
||||
}
|
||||
process.stderr.write('Press Enter to continue or Ctrl-C to abort. ');
|
||||
return await new Promise(resolve => {
|
||||
process.stdin.once('data', () => resolve(true));
|
||||
process.stdin.once('end', () => resolve(false));
|
||||
});
|
||||
}
|
||||
|
||||
export async function main(argv: string[]): Promise<number> {
|
||||
let args: ParsedArgs;
|
||||
try {
|
||||
args = parseArgs(argv);
|
||||
} catch (err) {
|
||||
process.stderr.write(`Error: ${(err as Error).message}\n\n${HELP}`);
|
||||
return 2;
|
||||
}
|
||||
|
||||
if (args.help) {
|
||||
process.stdout.write(HELP);
|
||||
return 0;
|
||||
}
|
||||
|
||||
const { full: modelFull, bare: modelBare } = resolveModel(args.model);
|
||||
|
||||
// Self-configure the gateway (matches src/commands/eval-cross-modal.ts:195-220).
|
||||
const config = loadConfig();
|
||||
configureGateway({
|
||||
embedding_model: config?.embedding_model,
|
||||
embedding_dimensions: config?.embedding_dimensions,
|
||||
expansion_model: config?.expansion_model,
|
||||
chat_model: config?.chat_model ?? modelFull,
|
||||
chat_fallback_chain: config?.chat_fallback_chain,
|
||||
base_urls: config?.provider_base_urls,
|
||||
env: { ...process.env } as Record<string, string>,
|
||||
});
|
||||
|
||||
// Provider-aware auth check (codex review P2-3). The CLI advertises full
|
||||
// provider:model support and the test suite covers `openai:gpt-4o`, so the
|
||||
// env-var gate must match the provider that will actually be called.
|
||||
// Unknown providers fall through to the gateway, which will raise a clear
|
||||
// recipe-specific error if any required env var is missing.
|
||||
const REQUIRED_ENV_BY_PROVIDER: Record<string, string> = {
|
||||
anthropic: 'ANTHROPIC_API_KEY',
|
||||
openai: 'OPENAI_API_KEY',
|
||||
google: 'GOOGLE_GENERATIVE_AI_API_KEY',
|
||||
groq: 'GROQ_API_KEY',
|
||||
voyage: 'VOYAGE_API_KEY',
|
||||
together: 'TOGETHER_API_KEY',
|
||||
deepseek: 'DEEPSEEK_API_KEY',
|
||||
minimax: 'MINIMAX_API_KEY',
|
||||
dashscope: 'DASHSCOPE_API_KEY',
|
||||
zhipu: 'ZHIPUAI_API_KEY',
|
||||
};
|
||||
const providerId = modelFull.includes(':') ? modelFull.split(':', 1)[0] : 'anthropic';
|
||||
const requiredEnv = REQUIRED_ENV_BY_PROVIDER[providerId];
|
||||
if (requiredEnv && !process.env[requiredEnv]) {
|
||||
process.stderr.write(`Error: ${requiredEnv} is not set. The harness needs it to reach ${modelFull}.\n`);
|
||||
return 2;
|
||||
}
|
||||
|
||||
// Load fixtures + variants.
|
||||
const evalsDir = __dirname;
|
||||
const fixturesTraining = parseFixtures(readFileSync(join(evalsDir, 'fixtures.jsonl'), 'utf8'));
|
||||
const fixturesHeldOut = parseFixtures(readFileSync(join(evalsDir, 'fixtures-held-out.jsonl'), 'utf8'));
|
||||
|
||||
// Dynamic variants: --variants overrides the default 3, --variants-dir overrides location.
|
||||
const variantsAbsDir = resolve(evalsDir, args.variantsDir);
|
||||
const variantBasenames = args.variantFiles
|
||||
?? (VARIANT_NAMES as readonly string[]).map(n => n);
|
||||
const variants: Record<string, string> = {};
|
||||
const dispatcherListsByVariant: Record<string, Map<string, Set<string>>> = {};
|
||||
for (const name of variantBasenames) {
|
||||
const content = loadVariant(join(variantsAbsDir, `${name}.md`));
|
||||
variants[name] = content;
|
||||
dispatcherListsByVariant[name] = parseDispatcherLists(content);
|
||||
}
|
||||
|
||||
// Build the (fixture × variant × seed) tuple list.
|
||||
type Tuple = { fixture: Fixture; corpus: 'training' | 'held_out'; fixture_id: number; variant: string; seed: number };
|
||||
const tuples: Tuple[] = [];
|
||||
for (const variant of variantBasenames) {
|
||||
fixturesTraining.forEach((f, i) => {
|
||||
for (const seed of SEEDS) tuples.push({ fixture: f, corpus: 'training', fixture_id: i, variant, seed });
|
||||
});
|
||||
fixturesHeldOut.forEach((f, i) => {
|
||||
for (const seed of SEEDS) tuples.push({ fixture: f, corpus: 'held_out', fixture_id: i, variant, seed });
|
||||
});
|
||||
}
|
||||
const totalCalls = args.limit ? Math.min(args.limit, tuples.length) : tuples.length;
|
||||
const workQueue = tuples.slice(0, totalCalls);
|
||||
|
||||
// Cost-estimate prompt (skipped for tiny --limit runs to keep dev iteration fast).
|
||||
if (totalCalls >= 20) {
|
||||
const proceed = await maybePromptCost(totalCalls, modelFull, args.yes);
|
||||
if (!proceed) {
|
||||
process.stderr.write('Aborted.\n');
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Compute receipt header.
|
||||
const fixturesHash = hashContent(readFileSync(join(evalsDir, 'fixtures.jsonl'), 'utf8'));
|
||||
const fixturesHeldOutHash = hashContent(readFileSync(join(evalsDir, 'fixtures-held-out.jsonl'), 'utf8'));
|
||||
const promptTemplateHash = hashContent(PROMPT_TEMPLATE);
|
||||
const harnessSha = getHarnessSha();
|
||||
const tsStart = new Date().toISOString();
|
||||
const receipt: ReceiptRow = {
|
||||
kind: 'receipt',
|
||||
model: modelFull,
|
||||
prompt_template_hash: promptTemplateHash,
|
||||
fixtures_hash: fixturesHash,
|
||||
fixtures_held_out_hash: fixturesHeldOutHash,
|
||||
harness_sha: harnessSha,
|
||||
ts: tsStart,
|
||||
cmd_args: argv,
|
||||
};
|
||||
|
||||
// Output path.
|
||||
const outputPath = args.output ?? join(evalsDir, `run-${tsStart.replace(/[:.]/g, '-')}.jsonl`);
|
||||
process.stderr.write(`Writing receipt + ${totalCalls} runs to ${outputPath}\n`);
|
||||
|
||||
const rows: (RunRow | ReceiptRow)[] = [receipt];
|
||||
|
||||
// Sequential or simple bounded-parallel execution.
|
||||
let completed = 0;
|
||||
async function processTuple(t: Tuple): Promise<RunRow> {
|
||||
const prompt = buildPrompt(variants[t.variant], t.fixture.intent);
|
||||
const { text, input_tokens, output_tokens, latency_ms } = await callModel(prompt, modelFull);
|
||||
const predicted = parseModelResponse(text);
|
||||
const correct = scoreFixture(predicted, t.fixture.expected_skill);
|
||||
const correct_lenient = scoreFixtureLenient(
|
||||
predicted,
|
||||
t.fixture.expected_skill,
|
||||
dispatcherListsByVariant[t.variant] ?? new Map(),
|
||||
);
|
||||
const row: RunRow = {
|
||||
kind: 'run',
|
||||
fixture_id: t.fixture_id,
|
||||
corpus: t.corpus,
|
||||
variant: t.variant as VariantName,
|
||||
seed: t.seed,
|
||||
predicted,
|
||||
expected: t.fixture.expected_skill,
|
||||
correct,
|
||||
correct_lenient,
|
||||
model: modelFull,
|
||||
input_tokens,
|
||||
output_tokens,
|
||||
latency_ms,
|
||||
ts: new Date().toISOString(),
|
||||
};
|
||||
completed++;
|
||||
if (completed % 10 === 0 || completed === totalCalls) {
|
||||
process.stderr.write(` ${completed}/${totalCalls} done\n`);
|
||||
}
|
||||
return row;
|
||||
}
|
||||
|
||||
// Bounded parallel: chunk into args.parallel-sized batches.
|
||||
for (let i = 0; i < workQueue.length; i += args.parallel) {
|
||||
const batch = workQueue.slice(i, i + args.parallel);
|
||||
const results = await Promise.all(batch.map(processTuple));
|
||||
rows.push(...results);
|
||||
}
|
||||
|
||||
// Write JSONL.
|
||||
writeJsonl(rows, outputPath);
|
||||
|
||||
// Compute per-variant accuracy. Both strict + lenient. Held-out is the
|
||||
// headline; training is reported separately.
|
||||
const runRows = rows.filter((r): r is RunRow => r.kind === 'run');
|
||||
type CorpusKey = 'training' | 'held_out';
|
||||
type Acc = { training: number[]; held_out: number[] };
|
||||
const strictSummary: Record<string, Acc> = {};
|
||||
const lenientSummary: Record<string, Acc> = {};
|
||||
for (const variant of variantBasenames) {
|
||||
strictSummary[variant] = { training: [], held_out: [] };
|
||||
lenientSummary[variant] = { training: [], held_out: [] };
|
||||
for (const corpus of ['training', 'held_out'] as const) {
|
||||
for (const seed of SEEDS) {
|
||||
const subset = runRows.filter(r => r.variant === variant && r.corpus === corpus && r.seed === seed);
|
||||
if (subset.length === 0) continue;
|
||||
strictSummary[variant][corpus].push(subset.reduce((a, r) => a + r.correct, 0) / subset.length);
|
||||
lenientSummary[variant][corpus].push(subset.reduce((a, r) => a + r.correct_lenient, 0) / subset.length);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Print summary.
|
||||
const fmt = (vals: number[]) => {
|
||||
if (vals.length === 0) return '—';
|
||||
const { mean, halfWidthCI } = meanAndCI95(vals);
|
||||
return `${(mean * 100).toFixed(1)}% ± ${(halfWidthCI * 100).toFixed(1)}%`;
|
||||
};
|
||||
|
||||
process.stderr.write(`\n=== A/B Eval Summary (model: ${modelFull}) ===\n`);
|
||||
process.stderr.write(' | STRICT scoring | LENIENT (same-area)\n');
|
||||
process.stderr.write('Variant | Held-out | Training | Held-out | Training\n');
|
||||
process.stderr.write('------------------------------|------------------------|------------------------|----------------------|----------------------\n');
|
||||
for (const variant of variantBasenames) {
|
||||
process.stderr.write(
|
||||
`${variant.padEnd(30)}| ${fmt(strictSummary[variant].held_out).padEnd(22)} | ${fmt(strictSummary[variant].training).padEnd(22)} | ${fmt(lenientSummary[variant].held_out).padEnd(20)} | ${fmt(lenientSummary[variant].training)}\n`,
|
||||
);
|
||||
}
|
||||
process.stderr.write('\nLENIENT counts a prediction as correct if it shares a dispatcher area with the expected target.\n');
|
||||
process.stderr.write('For variants without "(dispatcher for: ...)" clauses (baseline, resolver-of-resolvers), LENIENT == STRICT.\n');
|
||||
process.stderr.write('\nReceipt + runs written to: ' + outputPath + '\n');
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Bun entrypoint: run main when invoked as a script.
|
||||
if (import.meta.main) {
|
||||
main(process.argv.slice(2)).then(code => process.exit(code));
|
||||
}
|
||||
@@ -1,59 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Thin CLI shim for the functional-area-resolver A/B eval harness.
|
||||
*
|
||||
* Spawns the TypeScript runner via `bun` because the runner imports
|
||||
* gbrain's gateway from `src/core/ai/gateway.ts` directly. The runner
|
||||
* does the actual work; this file exists so users can invoke `node
|
||||
* harness.mjs` without remembering the bun incantation.
|
||||
*
|
||||
* If `bun` isn't on PATH (or this script is invoked outside the gbrain
|
||||
* repo), exit 2 with a clear message — the harness is a gbrain-side
|
||||
* proof-of-pattern, not a portable tool.
|
||||
*/
|
||||
|
||||
import { spawnSync, execFileSync } from 'node:child_process';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import { existsSync } from 'node:fs';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const runnerPath = resolve(__dirname, 'harness-runner.ts');
|
||||
const gatewayPath = resolve(__dirname, '..', '..', 'src', 'core', 'ai', 'gateway.ts');
|
||||
|
||||
function fail(message, code = 2) {
|
||||
process.stderr.write(message + '\n');
|
||||
process.exit(code);
|
||||
}
|
||||
|
||||
// Missing-binary fallback (F-E2): we need `bun` AND we need to be in
|
||||
// the gbrain repo so the runner can import the gateway.
|
||||
try {
|
||||
execFileSync('which', ['bun'], { stdio: 'ignore' });
|
||||
} catch {
|
||||
fail(
|
||||
'harness.mjs: `bun` is not on PATH.\n' +
|
||||
'This harness is a gbrain-maintainer-side tool — run it from a\n' +
|
||||
'gbrain repo checkout with `bun` installed (https://bun.sh).',
|
||||
);
|
||||
}
|
||||
|
||||
if (!existsSync(gatewayPath)) {
|
||||
fail(
|
||||
`harness.mjs: cannot find gbrain gateway at ${gatewayPath}.\n` +
|
||||
'This harness is the gbrain-side A/B eval surface. Run it from a\n' +
|
||||
'gbrain repo checkout, not from an installed skillpack.',
|
||||
);
|
||||
}
|
||||
|
||||
if (!existsSync(runnerPath)) {
|
||||
fail(`harness.mjs: runner missing at ${runnerPath}`);
|
||||
}
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const result = spawnSync('bun', ['run', runnerPath, ...args], {
|
||||
stdio: 'inherit',
|
||||
cwd: __dirname,
|
||||
});
|
||||
|
||||
process.exit(result.status ?? 1);
|
||||
@@ -1,121 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Re-score an existing run-*.jsonl (or baseline-runs/*.jsonl) with the lenient
|
||||
* dispatcher-area scoring rule, without re-running any LLM calls.
|
||||
*
|
||||
* Usage: node rescore.mjs <run-file.jsonl>
|
||||
*
|
||||
* Reads the receipt header to identify which variants were used, loads them
|
||||
* from ./variants/<name>.md, parses their (dispatcher for: ...) clauses, then
|
||||
* applies scoreFixtureLenient to every row. Prints a STRICT vs LENIENT
|
||||
* accuracy table without mutating the file.
|
||||
*
|
||||
* This is T1a from the v0.32.3.0 boil-the-ocean push.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
function parseDispatcherLists(variantContent) {
|
||||
const out = new Map();
|
||||
const re = /→\s*`([a-z][a-z0-9-]*)`\s*\(dispatcher for:\s*([^)]+)\)/g;
|
||||
let m;
|
||||
while ((m = re.exec(variantContent)) !== null) {
|
||||
const dispatcher = m[1];
|
||||
const subSkills = m[2].split(',').map(s => s.trim()).filter(s => /^[a-z][a-z0-9-]*$/.test(s));
|
||||
out.set(dispatcher, new Set([dispatcher, ...subSkills]));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function lenientScore(predicted, expected, dispatcherLists) {
|
||||
if (predicted === expected) return 1;
|
||||
for (const set of dispatcherLists.values()) {
|
||||
if (set.has(predicted) && set.has(expected)) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function meanAndCI(values) {
|
||||
if (values.length === 0) return { mean: 0, ci: 0 };
|
||||
const mean = values.reduce((a, b) => a + b, 0) / values.length;
|
||||
if (values.length === 1) return { mean, ci: 0 };
|
||||
const variance = values.reduce((acc, v) => acc + (v - mean) ** 2, 0) / (values.length - 1);
|
||||
const stdErr = Math.sqrt(variance / values.length);
|
||||
const tCrit = values.length === 3 ? 4.303 : values.length === 2 ? 12.706 : 1.96;
|
||||
return { mean, ci: tCrit * stdErr };
|
||||
}
|
||||
|
||||
function fmt(vals) {
|
||||
if (vals.length === 0) return '—';
|
||||
const { mean, ci } = meanAndCI(vals);
|
||||
return `${(mean * 100).toFixed(1)}% ± ${(ci * 100).toFixed(1)}%`;
|
||||
}
|
||||
|
||||
const runFile = process.argv[2];
|
||||
if (!runFile) {
|
||||
console.error('Usage: node rescore.mjs <run-file.jsonl>');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const absRun = resolve(process.cwd(), runFile);
|
||||
if (!existsSync(absRun)) {
|
||||
console.error(`File not found: ${absRun}`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const lines = readFileSync(absRun, 'utf8').split('\n').filter(l => l.trim().length > 0);
|
||||
const rows = lines.map(l => JSON.parse(l));
|
||||
|
||||
const receipt = rows.find(r => r.kind === 'receipt');
|
||||
const runRows = rows.filter(r => r.kind === 'run');
|
||||
|
||||
console.error(`Re-scoring ${runRows.length} rows from ${absRun}`);
|
||||
console.error(`Receipt: model=${receipt?.model ?? '?'} fixtures_hash=${receipt?.fixtures_hash ?? '?'} ts=${receipt?.ts ?? '?'}`);
|
||||
|
||||
// Identify variants and load them
|
||||
const variantsUsed = [...new Set(runRows.map(r => r.variant))];
|
||||
const variantsDir = join(__dirname, 'variants');
|
||||
const dispatcherLists = {};
|
||||
for (const v of variantsUsed) {
|
||||
const path = join(variantsDir, `${v}.md`);
|
||||
if (!existsSync(path)) {
|
||||
console.error(`Warning: variant file missing for "${v}" at ${path} — lenient score will collapse to strict for this variant.`);
|
||||
dispatcherLists[v] = new Map();
|
||||
continue;
|
||||
}
|
||||
dispatcherLists[v] = parseDispatcherLists(readFileSync(path, 'utf8'));
|
||||
}
|
||||
|
||||
const SEEDS = [1, 2, 3];
|
||||
|
||||
const strictSummary = {};
|
||||
const lenientSummary = {};
|
||||
for (const v of variantsUsed) {
|
||||
strictSummary[v] = { training: [], held_out: [] };
|
||||
lenientSummary[v] = { training: [], held_out: [] };
|
||||
for (const corpus of ['training', 'held_out']) {
|
||||
for (const seed of SEEDS) {
|
||||
const subset = runRows.filter(r => r.variant === v && r.corpus === corpus && r.seed === seed);
|
||||
if (subset.length === 0) continue;
|
||||
strictSummary[v][corpus].push(subset.reduce((a, r) => a + r.correct, 0) / subset.length);
|
||||
const lenientHits = subset.reduce((a, r) => a + lenientScore(r.predicted, r.expected, dispatcherLists[v]), 0);
|
||||
lenientSummary[v][corpus].push(lenientHits / subset.length);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n=== Re-scored from ${runFile} ===\n`);
|
||||
console.log(' | STRICT scoring | LENIENT (same-area)');
|
||||
console.log('Variant | Held-out | Training | Held-out | Training');
|
||||
console.log('------------------------------|------------------------|------------------------|----------------------|----------------------');
|
||||
for (const v of variantsUsed) {
|
||||
console.log(
|
||||
`${v.padEnd(30)}| ${fmt(strictSummary[v].held_out).padEnd(22)} | ${fmt(strictSummary[v].training).padEnd(22)} | ${fmt(lenientSummary[v].held_out).padEnd(20)} | ${fmt(lenientSummary[v].training)}`,
|
||||
);
|
||||
}
|
||||
console.log('\nLENIENT counts a prediction correct if it shares a dispatcher area with expected.');
|
||||
console.log('For variants without "(dispatcher for: ...)" clauses, LENIENT == STRICT.');
|
||||
@@ -1,380 +0,0 @@
|
||||
<!-- A/B EVAL FIXTURE — synthetic resolver shape, do not invoke from agent context. -->
|
||||
<!-- Variant: BASELINE — 270-row bullet-list shape. Extracted from a production AGENTS.md at the pre-compression state; owner PII scrubbed. ~25KB. -->
|
||||
|
||||
# AGENTS.md
|
||||
|
||||
This folder is home. Treat it that way.
|
||||
|
||||
## Hard Gates (NEVER VIOLATE)
|
||||
|
||||
⛔ **RUNTIME CONTEXT > PROJECT DOCS.** When the OpenClaw runtime context block (Group Chat Context, Inbound Context, capabilities) contradicts a project doc rule, the runtime wins. The runtime knows the actual channel state for THIS turn; project docs are stale by definition. The 2026-05-06 silent-drop recurrence happened because I trusted a wrong HEARTBEAT rule over the correct runtime warning. Don't do that again.
|
||||
|
||||
⛔ **NEVER RESTART GATEWAY.** Tell the owner. He does it himself. No exceptions.
|
||||
|
||||
⛔ **BRAIN-FIRST STORAGE.** ALL valuable outputs → `/your/brain/path/` or Supabase IMMEDIATELY. Use `/your/tmp` for scratch (not `/tmp`). `/tmp` hard limit: 2GB. See `skills/conventions/brain-first.md`.
|
||||
|
||||
⛔ **DATA LOSS GATE.** Before ANY bulk delete: read `skills/data-loss-gate/SKILL.md`, present confirmation card, wait for "yes."
|
||||
|
||||
⛔ **NO WIKILINKS.** Standard markdown links only: `[Name](path)`. Never `[[wikilinks]]`.
|
||||
|
||||
⛔ **GBRAIN MASTER READ-ONLY.** Never push to master on <owner>/gbrain. Never merge PRs. Branch → push → PR only. See `skills/github-agents/SKILL.md`.
|
||||
|
||||
⛔ **PUBLIC REPO GUARD.** Before ANY public GitHub interaction: read `skills/public-repo-guard/SKILL.md`. Run PII scanner on ALL content.
|
||||
|
||||
⚡ **MINIONS OVER SUB-AGENTS.** Use gbrain Minions (shell jobs) for batch/deterministic work. Sub-agents only when LLM reasoning is required mid-task. Always set `--timeout-ms 900000` for long jobs.
|
||||
|
||||
## Gate -1 — Acknowledge Immediately
|
||||
|
||||
For any request taking >5 sec: send a one-line ack with rough time estimate FIRST, then start tools. Never go silent into a tool chain. Calibration: lookup ~10s, multi-tool ~30-60s, transcription ~2-3min, sub-agent ~1-3min, heavy batch ~3-5min, browser ~2-5min. Overestimate slightly.
|
||||
|
||||
For tasks >1 min: spawn a progress-update subagent (one-liner every 30-60s with concrete progress %). Critical in group topics with no typing indicator.
|
||||
|
||||
## Gate 0 — Access Control
|
||||
|
||||
On EVERY inbound message, check `sender_id` FIRST.
|
||||
- **the owner (<OWNER_ID_A> or <OWNER_ID_B>):** Proceed. Full access.
|
||||
- **Known non-the owner:** Read `skills/multi-user/SKILL.md` immediately. It governs everything.
|
||||
- **Unknown sender:** "This is a private agent." → notify the owner → stop.
|
||||
|
||||
## Gate 0.5 — Critical Life Events
|
||||
|
||||
If the owner mentions a **death, funeral, birth, hospitalization, emergency, diagnosis, accident, divorce, or arrest** — IMMEDIATELY write to BOTH `MEMORY.md` AND `memory/YYYY-MM-DD.md`. Priority 0. No deferral.
|
||||
|
||||
## Gate 1 — Signal Detection (the owner only)
|
||||
|
||||
Every the owner message: scan for entity mentions (people, companies, deals, YC batches). For each: search brain, load context, update if stale. Read `skills/entity-detector/ENTITY-DETECTION.md` for the full protocol.
|
||||
|
||||
**Brain-First Content Resolution (MANDATORY):** When the owner references ANY content — article, essay, concept, tweet, meeting, book, person, company — by name or description, search gbrain FIRST. Never ask "which article?" or "can you share the link?" The brain has 100K pages. Search it. Only ask the owner if gbrain + memory + web all fail.
|
||||
|
||||
## Gate 2 — Session Startup
|
||||
|
||||
Before first substantive reply:
|
||||
1. Read `ops/tasks.md` for task state
|
||||
2. Read `memory/heartbeat-state.json` for location, blockers, last checks
|
||||
3. Read relevant `memory/YYYY-MM-DD.md` for recent context
|
||||
4. Check calendar if time-sensitive
|
||||
|
||||
**Brain link rule:** Every brain path in output MUST be a clickable GitHub URL: `[name](https://github.com/<owner>/brain/blob/main/path.md)`. Never bare paths. Never invented URLs. `<owner>.github.io/brain/` does NOT exist.
|
||||
|
||||
**After every brain write:** `bash scripts/brain-commit-link.sh "<message>"`. Always absolute paths for brain writes (`/your/brain/path/...`).
|
||||
|
||||
**Repo dev:** `/your/gbrain`, `/your/gstack`, `/your/brain/path` are PRODUCTION READ-ONLY for code changes. All dev work → `/your/git-projects/<repo>-<feature>/`. See `skills/repo-dev/SKILL.md`.
|
||||
|
||||
## Gate 3 — Outbound Link Gate
|
||||
|
||||
Before EVERY reply containing a brain reference:
|
||||
1. Path must be absolute GitHub URL
|
||||
2. Commit must be pushed (not just local)
|
||||
3. Use `brain-commit-link.sh` output for the URL
|
||||
4. Never invent URLs. Never use `<owner>.github.io`.
|
||||
|
||||
## Skill Resolver
|
||||
|
||||
Read the skill file before acting. If two could match, read both. Non-the owner senders: only WORK/FAMILY-accessible skills.
|
||||
|
||||
### Always-on (every message)
|
||||
- Gate -1: any request taking >5 sec → `acknowledge`
|
||||
- Gate 0: sender_id != the owner → `multi-user`
|
||||
- Gate 1: the owner messages only → `entity-detector`
|
||||
- Non-the owner user shares info about themselves/work/vendors → `group-chat-intel`
|
||||
- Any brain read/write/lookup/citation → `brain-ops`
|
||||
- Any brain page write OR chat reply mentioning a repo/project → `brain-link-refs`
|
||||
- Any outbound reply to the owner that references a brain page or workspace file → `brain-link-report`
|
||||
- Any outbound report/alert with external links (oppo alerts → `report-quality-gate`
|
||||
- Any outbound reply in a multi-user group (floor scope < FULL) that references... → `brain-pdf-auto`
|
||||
- Any time-sensitive claim: "in N minutes" → `context-now`
|
||||
- the owner corrects a behavior, output, or decision → `correction-pipeline`
|
||||
- Presenting choices with inline buttons, user decision gate, button callback → `ask-user`
|
||||
|
||||
### Political donations
|
||||
- Donation tracking → `political-donations`
|
||||
|
||||
### Brain operations
|
||||
- Creating a new file - where does it go? → `repo-architecture`
|
||||
- Brain directory structure, "where is X in the brain", schema, filing rules → `/your/brain/path/README.md (directory tree + key locations table) + /your/brain/path/schema.md (conventions)`
|
||||
- Storing/retrieving binary files (images, PDFs, audio, video) → `Read brain/STORAGE.md - .redirect.yaml pointers + Supabase Storage`
|
||||
- Creating/enriching a person or company page → `enrich`
|
||||
- Resolving X handle stubs to real people ("who is @handle" → `x-handle-enrich`
|
||||
- Scoring/rating a person, rationalizing scores, "what score is X" → `person-score`
|
||||
- Unknown sender emails the owner → `cold-email-lookup`
|
||||
- Pitch deck, data room, financial model shared → `diligence`
|
||||
- Fix broken citations in brain pages → `citation-fixer`
|
||||
- Publish/share a brain page as link → `brain-publish`
|
||||
- Generate PDF from brain page, "brain pdf", "send me the pdf", … → `brain-pdf`
|
||||
- Generate PDF from any non-brain content: reports → `pdf-generation`
|
||||
- Read a book/article through lens of a specific problem, "read this through the lens", "extract a playbook", "what can I learn" → `strategic-reading`
|
||||
- Personalized book analysis, "book mirror", "apply this book", … → `book-mirror`
|
||||
- Deep-retrieval book mirror, "extreme mirror", "go deep", … → `book-mirror/SKILL.md (deep retrieval is now the default)`
|
||||
- Freshness check, data source SLA monitoring, smoke test → `freshness-monitor`
|
||||
- Write as the owner: blog posts → `garry-voice`
|
||||
- Essay review, writing feedback, draft review → `essay-review`
|
||||
- Brain search/query, hybrid search, entity lookup; Brain maintenance, lint, backlinks, health checks → `gbrain`
|
||||
- "My ChatGPT conversations" → `conversation-history`
|
||||
- Brain integrity → `brain-librarian`
|
||||
- "archive crawler", "mine my old files", … → `archive-crawler`
|
||||
- "concept synthesis", "intellectual map", … → `concept-synthesis`
|
||||
- "Ingest all X" → `bulk-skillify`
|
||||
- "extract takes", "seed takes", … → `takes-extraction`
|
||||
- Any ycli command, ycli SSO expired → `ycli-auth`
|
||||
- "extreme mirror", "go deep on this book", deep-retrieval book mirror → `book-mirror-extreme`
|
||||
- Book mirror synthesis, synthesize book analysis → `book-mirror-synthesis`
|
||||
- Export brain, download brain pages, brain backup → `brain-export`
|
||||
- Brain planning, plan brain changes, schema planning → `brain-plan`
|
||||
- Conversation enrichment, enrich chat transcript → `conversation-enrichment`
|
||||
- Fact check, verify claim, "is this true", citation check → `fact-check`
|
||||
- Upgrade gbrain, update gbrain, gbrain version → `gbrain-upgrade`
|
||||
- "Review my Dropbox archive", Dropbox folder audit, old Dropbox files → `dropbox-archive-review`
|
||||
- Screenshot style, apply style to screenshot → `screenshot-style`
|
||||
- Signorelli letter, draft formal letter → `signorelli-letter`
|
||||
- Data loss prevention, confirm bulk delete → `data-loss-gate`
|
||||
- Public repo PII guard, check for secrets → `public-repo-guard`
|
||||
|
||||
### Places & Travel
|
||||
- Trip itinerary PDF/doc → `trip-logistics`
|
||||
- "I'm at [place]"; "Where should I eat in X"; Foursquare/Swarm data export, bulk location import → `checkin`
|
||||
- "What's playing", "showtimes", … → `showtimes`
|
||||
|
||||
### Calendar (direct queries)
|
||||
- "What's my schedule", "am I free", calendar briefing, day lookahead → `google-calendar`
|
||||
- "Create a calendar item", "add to my calendar", … → `calendar-event-create`
|
||||
- "Prep for my meeting with X" → `meeting-prep`
|
||||
- Interview prep → `interview-prep`
|
||||
- Calendar conflict detection, double bookings, travel impossibility, missing prep; After calendar sync completes, or when day's schedule changes → `calendar-check`
|
||||
- Travel booking → `calendar-travel-setup`
|
||||
- Sync calendars to brain → `calendar-sync`
|
||||
- Historical/past calendar lookup: "when did I" → `calendar-recall`
|
||||
|
||||
### Time, location, and context
|
||||
- "What time is it" → `context-now`
|
||||
- "What's my jet lag plan" → `jet-lag`
|
||||
|
||||
### Executive assistant
|
||||
- Inbox triage, email reply, scheduling, calendar → `executive-assistant`
|
||||
- Gmail search, send email, draft reply via ClawVisor → `gmail`
|
||||
- Google Contacts lookup, search contacts, contact info → `google-contacts`
|
||||
- Personal logistics, schedule timeline, countdown deltas, time-aware foundation → `personal-logistics`
|
||||
- Intro health check, dropped handoffs, re-ping opportunities, intro tracker → `intro-reping`
|
||||
- Startup intro request, "draft an intro", evaluate intro, score intro quality → `startup-intro`
|
||||
- Alumni dinner planning, guest list curation, dinner invite list → `alumni-dinner`
|
||||
- "Partner lunch brief" → `partner-lunch-brief`
|
||||
- Flight delay tracking → `flight-tracker`
|
||||
- "Where is the owner", location inference, fix location, travel state machine → `location-inference`
|
||||
- Task add/remove/complete/defer/review → `daily-task-manager`
|
||||
- Morning task list prep (cron) → `daily-task-prep`
|
||||
- Business development, outreach tracking → `business-development`
|
||||
- Phone call handling (510-MY-GARRY) → `voice-agent`
|
||||
- Venus call ended, "Process this Venus call", voice session analysis → `voice-session-ingest`
|
||||
- Post-call analysis, "analyze the last call", "what happened on that call" → `venus-post-call`
|
||||
- "give me a link" → `voice-link`
|
||||
- OpenPhone/SMS (415-777-0000) → `quo`
|
||||
- "What's my jet lag plan" → `jet-lag`
|
||||
- New trip detected, trip itinerary shared, post-trip reflection, "trip is done" → `trip-ingest`
|
||||
|
||||
### Face detection & recognition
|
||||
- Face detect → `face-detect`
|
||||
- "identify faces" → `identify-faces`
|
||||
|
||||
### Content & media ingestion
|
||||
- Frame.io → `frameio-monitor`
|
||||
- "Ingest this", "save this to brain", generic content routing → `ingest`
|
||||
- the owner shares a link, article, tweet, idea → `idea-ingest`
|
||||
- Any video/audio (YouTube, X, Instagram, TikTok, podcast), "ingest this pdf book", "summarize this book", "process this book"; Screenshots, GitHub repos, other media → `media-ingest`
|
||||
- "Transcribe this" → `transcribe`
|
||||
- Book PDF, investor update PDF, any PDF to ingest → `pdf-ingest`
|
||||
- "Get me this book" → `book-acquisition`
|
||||
- Anna's Archive download, annas-archive, fast download with membership → `annas-archive`
|
||||
- Kindle library → `kindle-library`
|
||||
- Circleback CLI: search meetings → `circleback-cli`
|
||||
- Meeting transcript from Circleback → `meeting-ingestion`
|
||||
- Post-ingestion meeting summary to Meetings topic (auto-triggered by Circlebac... → `meeting-digest`
|
||||
- MANDATORY post-meeting audit, "audit this meeting" → `meeting-gold-standard`
|
||||
- Post-meeting signal extraction, "what did I say that was interesting", concept extraction → `meeting-signal-pass`
|
||||
- "scrape", "scrape <url>", … → `scrape`
|
||||
- Fundraising PDF → `fundraising-pdf`
|
||||
- Therapy session audio: "here's my jan/donna/marcie session" → `therapy-ingest`
|
||||
- Enriching any brain page from external content (quality pass) → `media-enrichment`
|
||||
- Batch article enrichment, "enrich", "raw content", "article dumps" → `article-enrichment`
|
||||
- Post-ingestion signal extraction, concept extraction from articles, backlink enrichment, entity propagation → `post-ingestion-enrichment`
|
||||
- Security audit (secrets, RLS, token files, gitleaks) → `security-audit`
|
||||
- Backlink check after any brain page write → `node scripts/backlink-check.mjs <page-path> — deterministic, run after EVERY brain page create/update`
|
||||
- X daily quality → `x-daily-quality`
|
||||
- ycli → `yc-ingest`
|
||||
- YC OH meeting notes, ycli office hours ingestion, "pull my YC meetings" → `yc-oh-ingest`
|
||||
- "Ingest this application" → `yc-app-ingest`
|
||||
- Company investor update, VC fund LP update, portfolio metrics email → `investor-update-ingest`
|
||||
- Voice note, audio message to transcribe and ingest, "voice memo", "audio note", "audio message" → `voice-note-ingest`
|
||||
- Save session transcripts to brain → `transcript-save`
|
||||
- "Unsubscribe from this", remove me from this list → `email-unsubscribe`
|
||||
- Deep web research, "research this person/topic thoroughly", "web research", … → `perplexity-research`
|
||||
- Exa semantic web search, find people/companies/LinkedIn profiles → `exa`
|
||||
- Happenstance professional network search, research people → `happenstance`
|
||||
- Crustdata B2B intelligence, LinkedIn enrichment, career history → `crustdata`
|
||||
- Captain API, Pitchbook data, funding rounds, investor lookup → `captain-api`
|
||||
- Structured data research, "track" → `data-research`
|
||||
- Substack ingest, import from Substack → `substack-ingest`
|
||||
- Pocket ingest, import from Pocket → `pocket-ingest`
|
||||
- Tweet deep ingest, deep tweet enrichment, article extraction from tweets → `tweet-deep-ingest`
|
||||
|
||||
### X/Twitter API - ENTERPRISE TIER
|
||||
**ALL X API work:** Read `skills/_x-api-rules.md` FIRST. We pay $50K/mo. Rate limit: 40K req/15min. Import `lib/x-api.mjs`. NEVER throttle to free-tier limits.
|
||||
|
||||
### Message intelligence
|
||||
- "Scan my DMs", "triage my messages", X DM triage, unified message extraction → `message-intel`
|
||||
- "Project Karma", blocked/muted users, adversary tweets, hostile accounts → `adversary-tracking`
|
||||
|
||||
### Monitoring & social
|
||||
- X/Twitter ingestion (daily, backfill, rollup, enrichment) → `x-ingest`
|
||||
- "x stream" → `svc/x-stream`
|
||||
- "Concept tier" → `x-concept-tier`
|
||||
- "look up tweet"; "social json store" → `social-json-store`
|
||||
- "storage tier"; "download video when needed" → `brain-storage`
|
||||
- "link to supabase file" → `brain-storage-links`
|
||||
- "backblaze" → `backblaze`
|
||||
- Social media mention alerts (cron) → `social-radar`
|
||||
- YC launch cringe-o-meter, YC media monitoring, YC sentiment, "scan YC launches" → `yc-media-monitor`
|
||||
- Slack channel scanning (cron) → `slack-scan`
|
||||
- Content idea generation (cron) → `content-ideas`
|
||||
- Check Steph's Instagram → `steph-instagram`
|
||||
|
||||
### Adversarial / research
|
||||
- Track/monitor a public figure or critic → `adversary-tracking`
|
||||
- Detect astroturfing, "is this organic", bot check, paid amplification → `detect-astroturf`
|
||||
- Real-name hostile identification, "who hates me", hostile account ID → `real-name-hostiles`
|
||||
- Deanonymize anon X account → `investigate-x-anon`
|
||||
- Fiscal forensics, government spending, nonprofit audit, 990 filings, grant fraud → `fiscal-forensics`
|
||||
- Academic claim verification, "verify this study", "is this replicated", … → `academic-verify`
|
||||
- Private investigation, deep background check, "find out everything about" → `private-investigator`
|
||||
- Opposition research backgrounder → `oppo-research`
|
||||
- OSINT collection on tracked individuals → `osint-collector`
|
||||
- Network mapping, relationship intelligence, who-knows-who → `network-intel`
|
||||
- YC competitor oppo → `yc-competitor-oppo`
|
||||
- Who's boosting competitors → `yc-booster-tracker`
|
||||
|
||||
### Product / building
|
||||
- "Review this plan" / "CEO review" / "think bigger" → `gstack-openclaw-ceo-review`
|
||||
- "Debug this" / "investigate" / "root cause" → `gstack-openclaw-investigate`
|
||||
- "Office hours" / "brainstorm" / "is this worth building" / startup advice / f... → `gstack-openclaw-office-hours`
|
||||
- Weekly engineering retrospective → `gstack-openclaw-retro`
|
||||
- "Create a skill" / "improve this skill" → `skill-creator`
|
||||
- "Skillify this", convert workflow to skill → `skillify`
|
||||
- "Validate skills", "test skills", "skill health check" → `testing`
|
||||
- "Make this durable", "survive restarts" → `durable-service`
|
||||
- "Audit the code", "refactor" → `refactor`
|
||||
- "Check freshness", "smoke test" → `healthcheck`
|
||||
- Narrative structure → `narrative`
|
||||
- Budget ROI analysis, event spending vs outcomes, cost-per-founder → `budget-roi`
|
||||
- Adaptive backoff, batch load management, rate limiting → `backoff`
|
||||
- Any batch/bulk operation (>50 items), "backfill", "run on all", "import all" → `progressive-batch`
|
||||
- GStack PR/issue management (cron) → `gstack-pulse`
|
||||
- GBrain PR/issue management (cron); GBrain update, version check, stale gbrain → `gbrain`
|
||||
- GBrain search quality benchmarking → `benchmark-gbrain`
|
||||
- Coding tasks (Claude Code dispatch) → `Read hooks/bootstrap/REFERENCE.md`
|
||||
- Cross-modal review, second opinion, adversarial challenge → `cross-modal-review`
|
||||
- Deterministic code failing on edge cases → `fail-improve-loop`
|
||||
- GStack Browser tasks (cron) → `browser-tasks`
|
||||
- Weekly essay, write essay, draft weekly piece → `weekly-essay`
|
||||
- Investigate no response, why didn't they reply, follow up analysis → `investigate-no-response`
|
||||
- Printing press, publish to distribution → `printing-press`
|
||||
|
||||
### Infrastructure
|
||||
- Sending ANY service URL to the owner, "is the tunnel up", verify endpoint → `ngrok-verify`
|
||||
- "Check cpu", "system load", …, resource usage → `system-load`
|
||||
- Container restart → `container-restart`
|
||||
- Zombie processes → `zombie-reaper`
|
||||
- Write to /tmp → `scratch-space`
|
||||
- ClawVisor service routing, Gmail/Calendar/Drive/Contacts/iMessage via ClawVisor → `clawvisor`
|
||||
- ClawVisor Shield proxy, credential vaulting, API audit → `clawvisor-shield`
|
||||
- "What crons are running", recurring jobs, cron audit, scheduled tasks → `recurring-jobs`
|
||||
- Work on a PR → `acp-coding`
|
||||
- PR workflow, git worktree, dev checkout, "build this feature" → `repo-dev`
|
||||
- Brain page commit/push, always push after brain writes → `brain-commit`
|
||||
- Brain links, clickable GitHub URLs, "link me to" → `brain-links`
|
||||
- GitHub repo lookup, "repo not found", clone/check repo existence, READ a repo → `github-repo`
|
||||
- GitHub WRITE: push → `github-agents`
|
||||
- gbrain PR content, anonymization, PR body for gbrain → `gbrain-pr`
|
||||
- CAPTCHA, DataDome, "verification required", slide to verify → `captcha-solver`
|
||||
- QR code generation, "make a QR code", scannable code → `qr-code`
|
||||
- Front API, front link, front conversation, front search → `front-api`
|
||||
- OAuth2 authorization, "connect my X/service account", callback server → `oauth-webhook`
|
||||
- Headless browser, form fill, web interaction → `browser`
|
||||
- Cloud browser automation → `browser-use`
|
||||
- "Bypass IP restriction" → `nordvpn-proxy`
|
||||
- Channel discovery, find channels, list channels → `channel-discovery`
|
||||
- Telegram test divert, test message routing → `telegram-test-divert`
|
||||
- GStack Browse headed+proxy, browser-native download, anti-bot browsing → `gstack-browse`
|
||||
- "Submit a shell job" → `gbrain skills/minion-orchestrator`
|
||||
- Start GStack Browser (headed, the owner's machine) → `Ask the owner to run gstack-browser and share pairing code`
|
||||
- Binary dep missing, shared library error, container restart → `binary-deps`
|
||||
- Match HTML to screenshot, pixel-perfect, visual comparison, CSS tuning → `pixel-match`
|
||||
- YC app investigation, YC application ingestion, "ingest this company", company 404 → `yc-app-ingest`
|
||||
- Email triage, inbox classification, cold pitch scoring, auto-archive → `email-triage`
|
||||
- Cold pitch scoring, rate this pitch, pitch quality → `cold-pitch-scorer`
|
||||
- Company oppo, competitive intel, investigate competitor → `company-oppo`
|
||||
- Cross-modal eval, compare models, model comparison → `cross-modal-eval`
|
||||
- Tweet reply, dunk, respond to troll, "don't respond to this" → `anti-dunk`
|
||||
- "Write a comeback", "roast this", aggressive reply draft → `clapback`
|
||||
- Tweet draft, compose tweet, write a tweet → `tweet-draft`
|
||||
- Tweet composition, draft tweet structure → `tweet-composition`
|
||||
- Tweet vulnerability scan, shield, check my tweet → `tweet-shield`
|
||||
- Journo dunk, journalist oppo, build dunk file → `journo-dunk`
|
||||
- Hater tracker, hostile engagement analysis → `hater-tracker`
|
||||
- Slack messages, slack search, slack DMs → `slack`
|
||||
- Voter guide, election research, candidate analysis → `voter-guide`
|
||||
- Voter guide data extraction → `voter-guide-extract`
|
||||
- Web archive, save page, preserve article, offline copy → `web-archive`
|
||||
- YC meeting recording, OH transcript ingestion → `yc-meeting-ingest`
|
||||
- Quote screenshot, article screenshot for tweet → `quote-screenshot`
|
||||
- Song lyrics, quote lyrics (content filter bypass) → `song-lyrics`
|
||||
- Voice call enrichment, post-call brain page → `voice-call-enrich`
|
||||
- Context health, bootstrap budget, resolver coverage → `context-health`
|
||||
- Daily question, personal question drip → `daily-question`
|
||||
- Stalker watch, threat monitoring, dangerous individual → `stalker-watch`
|
||||
- Idea registry, idea capture, "I have an idea" → `idea-registry`
|
||||
- File archive ingestion, Dropbox, Google Drive import → `file-archive-ingestion`
|
||||
- "skillpackify", PR to gbrain, open source this skill, add to skillpack → `skillpackify`
|
||||
- Restart sweep, dropped messages, missed messages after restart → `restart-sweep`
|
||||
- Neuromancer coordination, agent handoffs, inter-agent tasks, "hand off to Neuromancer" → `neuromancer-coordination`
|
||||
- Inter-agent coordination, "Owner's Agents" group chat, the agent+Neuromancer collaboration, agent task claiming, brain write protocol; Bot-to-bot communication, /curtain protocol, agent volley limits, bot-to-bot setup, how agents talk to each other → `inter-agent-coordination`
|
||||
|
||||
**Internal data-source skills** (called by other skills, not directly): captain-api, crustdata, exa, happenstance, gmail, google-calendar, google-contacts, slack, clawvisor
|
||||
|
||||
|
||||
## Neuromancer Delegation (Cross-Topic)
|
||||
|
||||
**In ANY topic**, if a task would benefit from Neuromancer's capabilities, delegate it by posting a `[TASK]` message to the "Owner's Agents" group (thread 1, group -<GROUP_ID>).
|
||||
|
||||
**Neuromancer is good at:** Web research, browser automation, coding/PRs, X posting (via xurl), Google Workspace ops, on-demand analysis, skill building.
|
||||
|
||||
**the agent keeps:** Brain DB, cron/scheduled ops, X API (Enterprise keys), email sweeps (ClawVisor), memory consolidation, social radar, embedding/indexing.
|
||||
|
||||
**Protocol:** Prefix structured messages with `[TASK]`, `[RESULT]`, or `[QUERY]`. Neuromancer monitors the topic in real-time. Include enough context that Neuromancer can act without asking follow-ups. Reference brain pages by path.
|
||||
|
||||
**Don't delegate silently.** If the owner asked for something in another topic and you're handing it to Neuromancer, tell the owner in that topic: "Handing this to Neuromancer" with a one-liner on what you asked for.
|
||||
|
||||
## Memory (Operational)
|
||||
|
||||
- `MEMORY.md` — permanent, cross-session state. Keep tight. Flush to `memory/YYYY-MM-DD.md` daily.
|
||||
- `memory/YYYY-MM-DD.md` — daily operational memory. Append-only per day.
|
||||
- `memory/heartbeat-state.json` — structured state (location, wake status, last checks, blockers).
|
||||
- Brain (`/your/brain/path/`) — permanent knowledge (people, companies, deals, meetings, projects).
|
||||
|
||||
## Operating Rules
|
||||
|
||||
For the full set of operating principles, sub-agent rules, testing conventions, style guide, coding task protocols, and group chat rules: **read `skills/_operating-rules.md`**.
|
||||
|
||||
Key rules always in effect:
|
||||
- **Tests ship with code.** No PR without tests. No skip. See the full principle in the reference.
|
||||
- **Test before bulk.** Read `skills/progressive-batch/SKILL.md` for any operation touching >50 items. Progressive ramp: 10 → verify output exists → 100 → verify → 500 → verify → full. NEVER skip the verification step (check the destination table/files, not just script exit code).
|
||||
- **Fix tools, don't work around them.** If a tool is broken, fix it.
|
||||
- **Present options, then STOP.** For ambiguous requests, present 2-3 options. Don't pick one silently.
|
||||
- **Durable MECE skills.** Every repeated workflow → a skill. DRY across skills.
|
||||
- **GStack for coding PRs.** Read `skills/acp-coding/SKILL.md` for Claude Code / Codex integration.
|
||||
|
||||
## Coding Tasks — GStack Integration
|
||||
|
||||
Coding on gstack/gbrain/GL/any dev project: read `skills/acp-coding/SKILL.md`, spawn Codex via ACP, give full context, monitor+relay. Slash: `/code`, `/codex`, `/ship`, `/qa`, `/review`, `/investigate`.
|
||||
|
||||
<!-- gbrain:skillpack:begin -->
|
||||
<!-- Installed by gbrain 0.25.1. All 35 skills in this pack are already referenced in the resolver tables above. -->
|
||||
<!-- gbrain:skillpack:manifest cumulative-slugs="academic-verify,archive-crawler,article-enrichment,book-mirror,brain-ops,brain-pdf,briefing,citation-fixer,concept-synthesis,cron-scheduler,cross-modal-review,daily-task-manager,daily-task-prep,data-research,enrich,idea-ingest,ingest,maintain,media-ingest,meeting-ingestion,minion-orchestrator,perplexity-research,query,repo-architecture,reports,signal-detector,skill-creator,skillify,skillpack-check,soul-audit,strategic-reading,testing,voice-note-ingest,webhook-transforms" version="0.25.1" -->
|
||||
<!-- gbrain:skillpack:end -->
|
||||
@@ -1,146 +0,0 @@
|
||||
<!-- A/B EVAL FIXTURE — synthetic resolver shape, do not invoke from agent context. -->
|
||||
<!-- Variant: FUNCTIONAL-AREAS — the dispatcher pattern, extracted from a production AGENTS.md at the post-compression state; owner PII scrubbed. ~13KB. -->
|
||||
|
||||
# AGENTS.md
|
||||
|
||||
This folder is home. Treat it that way.
|
||||
|
||||
## Hard Gates (NEVER VIOLATE)
|
||||
|
||||
⛔ **RUNTIME CONTEXT > PROJECT DOCS.** When the OpenClaw runtime context block (Group Chat Context, Inbound Context, capabilities) contradicts a project doc rule, the runtime wins. The runtime knows the actual channel state for THIS turn; project docs are stale by definition. The 2026-05-06 silent-drop recurrence happened because I trusted a wrong HEARTBEAT rule over the correct runtime warning. Don't do that again.
|
||||
|
||||
⛔ **NEVER RESTART GATEWAY.** Tell the owner. He does it himself. No exceptions.
|
||||
|
||||
⛔ **BRAIN-FIRST STORAGE.** ALL valuable outputs → `/your/brain/path/` or Supabase IMMEDIATELY. Use `/your/tmp` for scratch (not `/tmp`). `/tmp` hard limit: 2GB. See `skills/conventions/brain-first.md`.
|
||||
|
||||
⛔ **DATA LOSS GATE.** Before ANY bulk delete: read `skills/data-loss-gate/SKILL.md`, present confirmation card, wait for "yes."
|
||||
|
||||
⛔ **NO WIKILINKS.** Standard markdown links only: `[Name](path)`. Never `[[wikilinks]]`.
|
||||
|
||||
⛔ **GBRAIN MASTER READ-ONLY.** Never push to master on <owner>/gbrain. Never merge PRs. Branch → push → PR only. See `skills/github-agents/SKILL.md`.
|
||||
|
||||
⛔ **PUBLIC REPO GUARD.** Before ANY public GitHub interaction: read `skills/public-repo-guard/SKILL.md`. Run PII scanner on ALL content.
|
||||
|
||||
⚡ **MINIONS OVER SUB-AGENTS.** Use gbrain Minions (shell jobs) for batch/deterministic work. Sub-agents only when LLM reasoning is required mid-task. Always set `--timeout-ms 900000` for long jobs.
|
||||
|
||||
## Gate -1 — Acknowledge Immediately
|
||||
|
||||
For any request taking >5 sec: send a one-line ack with rough time estimate FIRST, then start tools. Never go silent into a tool chain. Calibration: lookup ~10s, multi-tool ~30-60s, transcription ~2-3min, sub-agent ~1-3min, heavy batch ~3-5min, browser ~2-5min. Overestimate slightly.
|
||||
|
||||
For tasks >1 min: spawn a progress-update subagent (one-liner every 30-60s with concrete progress %). Critical in group topics with no typing indicator.
|
||||
|
||||
## Gate 0 — Access Control
|
||||
|
||||
On EVERY inbound message, check `sender_id` FIRST.
|
||||
- **the owner (<OWNER_ID_A> or <OWNER_ID_B>):** Proceed. Full access.
|
||||
- **Known non-the owner:** Read `skills/multi-user/SKILL.md` immediately. It governs everything.
|
||||
- **Unknown sender:** "This is a private agent." → notify the owner → stop.
|
||||
|
||||
## Gate 0.5 — Critical Life Events
|
||||
|
||||
If the owner mentions a **death, funeral, birth, hospitalization, emergency, diagnosis, accident, divorce, or arrest** — IMMEDIATELY write to BOTH `MEMORY.md` AND `memory/YYYY-MM-DD.md`. Priority 0. No deferral.
|
||||
|
||||
## Gate 1 — Signal Detection (the owner only)
|
||||
|
||||
Every the owner message: scan for entity mentions (people, companies, deals, YC batches). For each: search brain, load context, update if stale. Read `skills/entity-detector/ENTITY-DETECTION.md` for the full protocol.
|
||||
|
||||
**Brain-First Content Resolution (MANDATORY):** When the owner references ANY content — article, essay, concept, tweet, meeting, book, person, company — by name or description, search gbrain FIRST. Never ask "which article?" or "can you share the link?" The brain has 100K pages. Search it. Only ask the owner if gbrain + memory + web all fail.
|
||||
|
||||
## Gate 2 — Session Startup
|
||||
|
||||
Before first substantive reply:
|
||||
1. Read `ops/tasks.md` for task state
|
||||
2. Read `memory/heartbeat-state.json` for location, blockers, last checks
|
||||
3. Read relevant `memory/YYYY-MM-DD.md` for recent context
|
||||
4. Check calendar if time-sensitive
|
||||
|
||||
**Brain link rule:** Every brain path in output MUST be a clickable GitHub URL: `[name](https://github.com/<owner>/brain/blob/main/path.md)`. Never bare paths. Never invented URLs. `<owner>.github.io/brain/` does NOT exist.
|
||||
|
||||
**After every brain write:** `bash scripts/brain-commit-link.sh "<message>"`. Always absolute paths for brain writes (`/your/brain/path/...`).
|
||||
|
||||
**Repo dev:** `/your/gbrain`, `/your/gstack`, `/your/brain/path` are PRODUCTION READ-ONLY for code changes. All dev work → `/your/git-projects/<repo>-<feature>/`. See `skills/repo-dev/SKILL.md`.
|
||||
|
||||
## Gate 3 — Outbound Link Gate
|
||||
|
||||
Before EVERY reply containing a brain reference:
|
||||
1. Path must be absolute GitHub URL
|
||||
2. Commit must be pushed (not just local)
|
||||
3. Use `brain-commit-link.sh` output for the URL
|
||||
4. Never invent URLs. Never use `<owner>.github.io`.
|
||||
|
||||
## Skill Resolver
|
||||
|
||||
Read the skill file before acting. If two could match, read both. Non-the owner senders: only WORK/FAMILY-accessible skills.
|
||||
|
||||
### Always-on (every message)
|
||||
- Gate -1: any request taking >5 sec → `acknowledge`
|
||||
- Gate 0: sender_id != the owner → `multi-user`
|
||||
- Gate 1: the owner messages only → `entity-detector`
|
||||
- Non-the owner shares info → `group-chat-intel`
|
||||
- Brain read/write/lookup → `brain-ops`
|
||||
- Reply mentioning repo/project → `brain-link-refs`
|
||||
- Reply referencing brain page → `brain-link-report`
|
||||
- Report with external links → `report-quality-gate`
|
||||
- Multi-user group reply referencing brain → `brain-pdf-auto`
|
||||
- Time-sensitive claim → `context-now`
|
||||
- the owner corrects behavior → `correction-pipeline`
|
||||
- Inline buttons / user decision gate → `ask-user`
|
||||
|
||||
### Functional Areas
|
||||
- **Brain & knowledge**: create/enrich/search/export brain pages, filing, citations, publishing, book analysis, strategic reading, concept synthesis, archive mining, conversation history → `brain-ops` (dispatcher for: enrich, query, brain-pdf, brain-publish, brain-export, brain-plan, brain-librarian, brain-commit, brain-storage, brain-storage-links, citation-fixer, repo-architecture, book-mirror, book-mirror-extreme, book-mirror-synthesis, strategic-reading, concept-synthesis, archive-crawler, conversation-history, conversation-enrichment, garry-voice, essay-review, fact-check, takes-extraction, gbrain, gbrain-upgrade, benchmark-gbrain, freshness-monitor, dropbox-archive-review, bulk-skillify, x-handle-enrich, person-score)
|
||||
- **Content ingestion**: ingest links/articles/PDFs/video/audio/tweets/books/meetings/voice notes, transcription, media enrichment → `ingest` (dispatcher for: media-ingest, meeting-ingestion, meeting-digest, meeting-gold-standard, meeting-signal-pass, voice-note-ingest, article-enrichment, post-ingestion-enrichment, media-enrichment, book-acquisition, annas-archive, pdf-ingest, tweet-deep-ingest, substack-ingest, pocket-ingest, investor-update-ingest, yc-ingest, yc-oh-ingest, yc-app-ingest, yc-meeting-ingest, kindle-library, therapy-ingest, transcript-save, file-archive-ingestion, idea-ingest)
|
||||
- **Calendar & scheduling**: schedule, events, conflicts, sync, prep, travel booking, time/location → `google-calendar` (dispatcher for: calendar-event-create, calendar-check, calendar-sync, calendar-recall, calendar-travel-setup, meeting-prep, interview-prep, context-now, jet-lag, location-inference)
|
||||
- **Email & comms**: inbox triage, email search/send, iMessage, Slack, unsubscribe, Front API → `executive-assistant` (dispatcher for: gmail, email-triage, email-unsubscribe, cold-email-lookup, cold-pitch-scorer, front-api, slack, intro-reping, startup-intro, investigate-no-response)
|
||||
- **Research & investigation**: web research, people/company lookup, LinkedIn, competitive intel, background checks → `perplexity-research` (dispatcher for: exa, happenstance, crustdata, captain-api, data-research, diligence, company-oppo, network-intel, private-investigator, oppo-research, academic-verify)
|
||||
- **X/Twitter & social**: tweets, social monitoring, adversary tracking, content strategy, DM triage → `x-ingest` (dispatcher for: adversary-tracking, social-radar, x-daily-quality, x-concept-tier, social-json-store, detect-astroturf, real-name-hostiles, investigate-x-anon, anti-dunk, clapback, tweet-draft, tweet-composition, tweet-shield, journo-dunk, hater-tracker, message-intel, yc-media-monitor, yc-competitor-oppo, yc-booster-tracker, steph-instagram, content-ideas)
|
||||
- **Places & travel**: checkins, restaurants, showtimes, trip logistics → `checkin` (dispatcher for: trip-logistics, trip-ingest, showtimes, personal-logistics)
|
||||
- **Product & building**: CEO review, code, debugging, skill creation, testing, refactoring, PR management → `acp-coding` (dispatcher for: gstack-openclaw-ceo-review, gstack-openclaw-investigate, gstack-openclaw-office-hours, gstack-openclaw-retro, skill-creator, skillify, testing, durable-service, refactor, narrative, budget-roi, fail-improve-loop, weekly-essay, printing-press, cross-modal-review, cross-modal-eval)
|
||||
- **Infrastructure**: tunnels, containers, services, crons, GitHub, browser automation, security → `healthcheck` (dispatcher for: ngrok-verify, system-load, container-restart, zombie-reaper, scratch-space, clawvisor, clawvisor-shield, recurring-jobs, github-repo, github-agents, gbrain-pr, captcha-solver, qr-code, browser, browser-use, gstack-browse, binary-deps, pixel-match, nordvpn-proxy, channel-discovery, durable-service, data-loss-gate, public-repo-guard, web-archive, security-audit)
|
||||
- **People & contacts**: Google contacts, face detection/identification, people enrichment → `google-contacts` (dispatcher for: face-detect, identify-faces, enrich)
|
||||
- **Tasks & logistics**: daily tasks, reminders, briefings, business dev, flight tracking, voice calls → `daily-task-manager` (dispatcher for: daily-task-prep, business-development, flight-tracker, voice-agent, voice-session-ingest, venus-post-call, voice-link, voice-call-enrich, quo, checkin)
|
||||
- **Political**: donation tracking, voter guides, civic intel → `political-donations` (dispatcher for: voter-guide, voter-guide-extract, fiscal-forensics)
|
||||
- **Inter-agent**: Neuromancer delegation, agent coordination → `inter-agent-coordination` (dispatcher for: neuromancer-coordination)
|
||||
- **Circleback**: meeting search → `circleback-cli`
|
||||
|
||||
**Internal data-source skills** (called by other skills, not directly): captain-api, crustdata, exa, happenstance, gmail, google-calendar, google-contacts, slack, clawvisor
|
||||
|
||||
|
||||
## Neuromancer Delegation (Cross-Topic)
|
||||
|
||||
**In ANY topic**, if a task would benefit from Neuromancer's capabilities, delegate it by posting a `[TASK]` message to the "Owner's Agents" group (thread 1, group -<GROUP_ID>).
|
||||
|
||||
**Neuromancer is good at:** Web research, browser automation, coding/PRs, X posting (via xurl), Google Workspace ops, on-demand analysis, skill building.
|
||||
|
||||
**the agent keeps:** Brain DB, cron/scheduled ops, X API (Enterprise keys), email sweeps (ClawVisor), memory consolidation, social radar, embedding/indexing.
|
||||
|
||||
**Protocol:** Prefix structured messages with `[TASK]`, `[RESULT]`, or `[QUERY]`. Neuromancer monitors the topic in real-time. Include enough context that Neuromancer can act without asking follow-ups. Reference brain pages by path.
|
||||
|
||||
**Don't delegate silently.** If the owner asked for something in another topic and you're handing it to Neuromancer, tell the owner in that topic: "Handing this to Neuromancer" with a one-liner on what you asked for.
|
||||
|
||||
## Memory (Operational)
|
||||
|
||||
- `MEMORY.md` — permanent, cross-session state. Keep tight. Flush to `memory/YYYY-MM-DD.md` daily.
|
||||
- `memory/YYYY-MM-DD.md` — daily operational memory. Append-only per day.
|
||||
- `memory/heartbeat-state.json` — structured state (location, wake status, last checks, blockers).
|
||||
- Brain (`/your/brain/path/`) — permanent knowledge (people, companies, deals, meetings, projects).
|
||||
|
||||
## Operating Rules
|
||||
|
||||
For the full set of operating principles, sub-agent rules, testing conventions, style guide, coding task protocols, and group chat rules: **read `skills/_operating-rules.md`**.
|
||||
|
||||
Key rules always in effect:
|
||||
- **Tests ship with code.** No PR without tests. No skip. See the full principle in the reference.
|
||||
- **Test before bulk.** Read `skills/progressive-batch/SKILL.md` for any operation touching >50 items. Progressive ramp: 10 → verify output exists → 100 → verify → 500 → verify → full. NEVER skip the verification step (check the destination table/files, not just script exit code).
|
||||
- **Fix tools, don't work around them.** If a tool is broken, fix it.
|
||||
- **Present options, then STOP.** For ambiguous requests, present 2-3 options. Don't pick one silently.
|
||||
- **Durable MECE skills.** Every repeated workflow → a skill. DRY across skills.
|
||||
- **GStack for coding PRs.** Read `skills/acp-coding/SKILL.md` for Claude Code / Codex integration.
|
||||
|
||||
## Coding Tasks — GStack Integration
|
||||
|
||||
Coding on gstack/gbrain/GL/any dev project: read `skills/acp-coding/SKILL.md`, spawn Codex via ACP, give full context, monitor+relay. Slash: `/code`, `/codex`, `/ship`, `/qa`, `/review`, `/investigate`.
|
||||
|
||||
<!-- gbrain:skillpack:begin -->
|
||||
<!-- Installed by gbrain 0.25.1. All 35 skills in this pack are already referenced in the resolver tables above. -->
|
||||
<!-- gbrain:skillpack:manifest cumulative-slugs="academic-verify,archive-crawler,article-enrichment,book-mirror,brain-ops,brain-pdf,briefing,citation-fixer,concept-synthesis,cron-scheduler,cross-modal-review,daily-task-manager,daily-task-prep,data-research,enrich,idea-ingest,ingest,maintain,media-ingest,meeting-ingestion,minion-orchestrator,perplexity-research,query,repo-architecture,reports,signal-detector,skill-creator,skillify,skillpack-check,soul-audit,strategic-reading,testing,voice-note-ingest,webhook-transforms" version="0.25.1" -->
|
||||
<!-- gbrain:skillpack:end -->
|
||||
@@ -1,146 +0,0 @@
|
||||
<!-- A/B EVAL FIXTURE — synthetic resolver shape, do not invoke from agent context. -->
|
||||
<!-- Variant: RESOLVER-OF-RESOLVERS — functional-areas WITHOUT the '(dispatcher for: ...)' clauses. This is the variant the skill describes as 'broken' — pipe-table compression that loses sub-skill visibility. -->
|
||||
|
||||
# AGENTS.md
|
||||
|
||||
This folder is home. Treat it that way.
|
||||
|
||||
## Hard Gates (NEVER VIOLATE)
|
||||
|
||||
⛔ **RUNTIME CONTEXT > PROJECT DOCS.** When the OpenClaw runtime context block (Group Chat Context, Inbound Context, capabilities) contradicts a project doc rule, the runtime wins. The runtime knows the actual channel state for THIS turn; project docs are stale by definition. The 2026-05-06 silent-drop recurrence happened because I trusted a wrong HEARTBEAT rule over the correct runtime warning. Don't do that again.
|
||||
|
||||
⛔ **NEVER RESTART GATEWAY.** Tell the owner. He does it himself. No exceptions.
|
||||
|
||||
⛔ **BRAIN-FIRST STORAGE.** ALL valuable outputs → `/your/brain/path/` or Supabase IMMEDIATELY. Use `/your/tmp` for scratch (not `/tmp`). `/tmp` hard limit: 2GB. See `skills/conventions/brain-first.md`.
|
||||
|
||||
⛔ **DATA LOSS GATE.** Before ANY bulk delete: read `skills/data-loss-gate/SKILL.md`, present confirmation card, wait for "yes."
|
||||
|
||||
⛔ **NO WIKILINKS.** Standard markdown links only: `[Name](path)`. Never `[[wikilinks]]`.
|
||||
|
||||
⛔ **GBRAIN MASTER READ-ONLY.** Never push to master on <owner>/gbrain. Never merge PRs. Branch → push → PR only. See `skills/github-agents/SKILL.md`.
|
||||
|
||||
⛔ **PUBLIC REPO GUARD.** Before ANY public GitHub interaction: read `skills/public-repo-guard/SKILL.md`. Run PII scanner on ALL content.
|
||||
|
||||
⚡ **MINIONS OVER SUB-AGENTS.** Use gbrain Minions (shell jobs) for batch/deterministic work. Sub-agents only when LLM reasoning is required mid-task. Always set `--timeout-ms 900000` for long jobs.
|
||||
|
||||
## Gate -1 — Acknowledge Immediately
|
||||
|
||||
For any request taking >5 sec: send a one-line ack with rough time estimate FIRST, then start tools. Never go silent into a tool chain. Calibration: lookup ~10s, multi-tool ~30-60s, transcription ~2-3min, sub-agent ~1-3min, heavy batch ~3-5min, browser ~2-5min. Overestimate slightly.
|
||||
|
||||
For tasks >1 min: spawn a progress-update subagent (one-liner every 30-60s with concrete progress %). Critical in group topics with no typing indicator.
|
||||
|
||||
## Gate 0 — Access Control
|
||||
|
||||
On EVERY inbound message, check `sender_id` FIRST.
|
||||
- **the owner (<OWNER_ID_A> or <OWNER_ID_B>):** Proceed. Full access.
|
||||
- **Known non-the owner:** Read `skills/multi-user/SKILL.md` immediately. It governs everything.
|
||||
- **Unknown sender:** "This is a private agent." → notify the owner → stop.
|
||||
|
||||
## Gate 0.5 — Critical Life Events
|
||||
|
||||
If the owner mentions a **death, funeral, birth, hospitalization, emergency, diagnosis, accident, divorce, or arrest** — IMMEDIATELY write to BOTH `MEMORY.md` AND `memory/YYYY-MM-DD.md`. Priority 0. No deferral.
|
||||
|
||||
## Gate 1 — Signal Detection (the owner only)
|
||||
|
||||
Every the owner message: scan for entity mentions (people, companies, deals, YC batches). For each: search brain, load context, update if stale. Read `skills/entity-detector/ENTITY-DETECTION.md` for the full protocol.
|
||||
|
||||
**Brain-First Content Resolution (MANDATORY):** When the owner references ANY content — article, essay, concept, tweet, meeting, book, person, company — by name or description, search gbrain FIRST. Never ask "which article?" or "can you share the link?" The brain has 100K pages. Search it. Only ask the owner if gbrain + memory + web all fail.
|
||||
|
||||
## Gate 2 — Session Startup
|
||||
|
||||
Before first substantive reply:
|
||||
1. Read `ops/tasks.md` for task state
|
||||
2. Read `memory/heartbeat-state.json` for location, blockers, last checks
|
||||
3. Read relevant `memory/YYYY-MM-DD.md` for recent context
|
||||
4. Check calendar if time-sensitive
|
||||
|
||||
**Brain link rule:** Every brain path in output MUST be a clickable GitHub URL: `[name](https://github.com/<owner>/brain/blob/main/path.md)`. Never bare paths. Never invented URLs. `<owner>.github.io/brain/` does NOT exist.
|
||||
|
||||
**After every brain write:** `bash scripts/brain-commit-link.sh "<message>"`. Always absolute paths for brain writes (`/your/brain/path/...`).
|
||||
|
||||
**Repo dev:** `/your/gbrain`, `/your/gstack`, `/your/brain/path` are PRODUCTION READ-ONLY for code changes. All dev work → `/your/git-projects/<repo>-<feature>/`. See `skills/repo-dev/SKILL.md`.
|
||||
|
||||
## Gate 3 — Outbound Link Gate
|
||||
|
||||
Before EVERY reply containing a brain reference:
|
||||
1. Path must be absolute GitHub URL
|
||||
2. Commit must be pushed (not just local)
|
||||
3. Use `brain-commit-link.sh` output for the URL
|
||||
4. Never invent URLs. Never use `<owner>.github.io`.
|
||||
|
||||
## Skill Resolver
|
||||
|
||||
Read the skill file before acting. If two could match, read both. Non-the owner senders: only WORK/FAMILY-accessible skills.
|
||||
|
||||
### Always-on (every message)
|
||||
- Gate -1: any request taking >5 sec → `acknowledge`
|
||||
- Gate 0: sender_id != the owner → `multi-user`
|
||||
- Gate 1: the owner messages only → `entity-detector`
|
||||
- Non-the owner shares info → `group-chat-intel`
|
||||
- Brain read/write/lookup → `brain-ops`
|
||||
- Reply mentioning repo/project → `brain-link-refs`
|
||||
- Reply referencing brain page → `brain-link-report`
|
||||
- Report with external links → `report-quality-gate`
|
||||
- Multi-user group reply referencing brain → `brain-pdf-auto`
|
||||
- Time-sensitive claim → `context-now`
|
||||
- the owner corrects behavior → `correction-pipeline`
|
||||
- Inline buttons / user decision gate → `ask-user`
|
||||
|
||||
### Functional Areas
|
||||
- **Brain & knowledge**: create/enrich/search/export brain pages, filing, citations, publishing, book analysis, strategic reading, concept synthesis, archive mining, conversation history → `brain-ops`
|
||||
- **Content ingestion**: ingest links/articles/PDFs/video/audio/tweets/books/meetings/voice notes, transcription, media enrichment → `ingest`
|
||||
- **Calendar & scheduling**: schedule, events, conflicts, sync, prep, travel booking, time/location → `google-calendar`
|
||||
- **Email & comms**: inbox triage, email search/send, iMessage, Slack, unsubscribe, Front API → `executive-assistant`
|
||||
- **Research & investigation**: web research, people/company lookup, LinkedIn, competitive intel, background checks → `perplexity-research`
|
||||
- **X/Twitter & social**: tweets, social monitoring, adversary tracking, content strategy, DM triage → `x-ingest`
|
||||
- **Places & travel**: checkins, restaurants, showtimes, trip logistics → `checkin`
|
||||
- **Product & building**: CEO review, code, debugging, skill creation, testing, refactoring, PR management → `acp-coding`
|
||||
- **Infrastructure**: tunnels, containers, services, crons, GitHub, browser automation, security → `healthcheck`
|
||||
- **People & contacts**: Google contacts, face detection/identification, people enrichment → `google-contacts`
|
||||
- **Tasks & logistics**: daily tasks, reminders, briefings, business dev, flight tracking, voice calls → `daily-task-manager`
|
||||
- **Political**: donation tracking, voter guides, civic intel → `political-donations`
|
||||
- **Inter-agent**: Neuromancer delegation, agent coordination → `inter-agent-coordination`
|
||||
- **Circleback**: meeting search → `circleback-cli`
|
||||
|
||||
**Internal data-source skills** (called by other skills, not directly): captain-api, crustdata, exa, happenstance, gmail, google-calendar, google-contacts, slack, clawvisor
|
||||
|
||||
|
||||
## Neuromancer Delegation (Cross-Topic)
|
||||
|
||||
**In ANY topic**, if a task would benefit from Neuromancer's capabilities, delegate it by posting a `[TASK]` message to the "Owner's Agents" group (thread 1, group -<GROUP_ID>).
|
||||
|
||||
**Neuromancer is good at:** Web research, browser automation, coding/PRs, X posting (via xurl), Google Workspace ops, on-demand analysis, skill building.
|
||||
|
||||
**the agent keeps:** Brain DB, cron/scheduled ops, X API (Enterprise keys), email sweeps (ClawVisor), memory consolidation, social radar, embedding/indexing.
|
||||
|
||||
**Protocol:** Prefix structured messages with `[TASK]`, `[RESULT]`, or `[QUERY]`. Neuromancer monitors the topic in real-time. Include enough context that Neuromancer can act without asking follow-ups. Reference brain pages by path.
|
||||
|
||||
**Don't delegate silently.** If the owner asked for something in another topic and you're handing it to Neuromancer, tell the owner in that topic: "Handing this to Neuromancer" with a one-liner on what you asked for.
|
||||
|
||||
## Memory (Operational)
|
||||
|
||||
- `MEMORY.md` — permanent, cross-session state. Keep tight. Flush to `memory/YYYY-MM-DD.md` daily.
|
||||
- `memory/YYYY-MM-DD.md` — daily operational memory. Append-only per day.
|
||||
- `memory/heartbeat-state.json` — structured state (location, wake status, last checks, blockers).
|
||||
- Brain (`/your/brain/path/`) — permanent knowledge (people, companies, deals, meetings, projects).
|
||||
|
||||
## Operating Rules
|
||||
|
||||
For the full set of operating principles, sub-agent rules, testing conventions, style guide, coding task protocols, and group chat rules: **read `skills/_operating-rules.md`**.
|
||||
|
||||
Key rules always in effect:
|
||||
- **Tests ship with code.** No PR without tests. No skip. See the full principle in the reference.
|
||||
- **Test before bulk.** Read `skills/progressive-batch/SKILL.md` for any operation touching >50 items. Progressive ramp: 10 → verify output exists → 100 → verify → 500 → verify → full. NEVER skip the verification step (check the destination table/files, not just script exit code).
|
||||
- **Fix tools, don't work around them.** If a tool is broken, fix it.
|
||||
- **Present options, then STOP.** For ambiguous requests, present 2-3 options. Don't pick one silently.
|
||||
- **Durable MECE skills.** Every repeated workflow → a skill. DRY across skills.
|
||||
- **GStack for coding PRs.** Read `skills/acp-coding/SKILL.md` for Claude Code / Codex integration.
|
||||
|
||||
## Coding Tasks — GStack Integration
|
||||
|
||||
Coding on gstack/gbrain/GL/any dev project: read `skills/acp-coding/SKILL.md`, spawn Codex via ACP, give full context, monitor+relay. Slash: `/code`, `/codex`, `/ship`, `/qa`, `/review`, `/investigate`.
|
||||
|
||||
<!-- gbrain:skillpack:begin -->
|
||||
<!-- Installed by gbrain 0.25.1. All 35 skills in this pack are already referenced in the resolver tables above. -->
|
||||
<!-- gbrain:skillpack:manifest cumulative-slugs="academic-verify,archive-crawler,article-enrichment,book-mirror,brain-ops,brain-pdf,briefing,citation-fixer,concept-synthesis,cron-scheduler,cross-modal-review,daily-task-manager,daily-task-prep,data-research,enrich,idea-ingest,ingest,maintain,media-ingest,meeting-ingestion,minion-orchestrator,perplexity-research,query,repo-architecture,reports,signal-detector,skill-creator,skillify,skillpack-check,soul-audit,strategic-reading,testing,voice-note-ingest,webhook-transforms" version="0.25.1" -->
|
||||
<!-- gbrain:skillpack:end -->
|
||||
+855
-1022
File diff suppressed because one or more lines are too long
+4
-18
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.32.3.0",
|
||||
"version": "0.25.1",
|
||||
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
|
||||
"family": "bundle-plugin",
|
||||
"configSchema": {
|
||||
@@ -8,25 +8,19 @@
|
||||
"type": "string",
|
||||
"required": true,
|
||||
"description": "PostgreSQL connection URL (Supabase recommended)",
|
||||
"uiHints": {
|
||||
"sensitive": true
|
||||
}
|
||||
"uiHints": { "sensitive": true }
|
||||
},
|
||||
"openai_api_key": {
|
||||
"type": "string",
|
||||
"required": false,
|
||||
"description": "OpenAI API key for embeddings (uses OPENAI_API_KEY env var if not set)",
|
||||
"uiHints": {
|
||||
"sensitive": true
|
||||
}
|
||||
"uiHints": { "sensitive": true }
|
||||
}
|
||||
},
|
||||
"mcpServers": {
|
||||
"gbrain": {
|
||||
"command": "./bin/gbrain",
|
||||
"args": [
|
||||
"serve"
|
||||
]
|
||||
"args": ["serve"]
|
||||
}
|
||||
},
|
||||
"skills": [
|
||||
@@ -45,7 +39,6 @@
|
||||
"skills/daily-task-prep",
|
||||
"skills/data-research",
|
||||
"skills/enrich",
|
||||
"skills/functional-area-resolver",
|
||||
"skills/idea-ingest",
|
||||
"skills/ingest",
|
||||
"skills/maintain",
|
||||
@@ -60,7 +53,6 @@
|
||||
"skills/skill-creator",
|
||||
"skills/skillify",
|
||||
"skills/skillpack-check",
|
||||
"skills/skillpack-harvest",
|
||||
"skills/soul-audit",
|
||||
"skills/strategic-reading",
|
||||
"skills/testing",
|
||||
@@ -69,7 +61,6 @@
|
||||
],
|
||||
"shared_deps": [
|
||||
"skills/conventions",
|
||||
"skills/_AGENT_README.md",
|
||||
"skills/_brain-filing-rules.md",
|
||||
"skills/_brain-filing-rules.json",
|
||||
"skills/_output-rules.md"
|
||||
@@ -83,10 +74,5 @@
|
||||
"compat": {
|
||||
"pluginApi": ">=2026.4.0"
|
||||
}
|
||||
},
|
||||
"contracts": {
|
||||
"contextEngines": [
|
||||
"gbrain-context"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+5
-21
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.36.5.0",
|
||||
"version": "0.28.7",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -24,26 +24,22 @@
|
||||
"./backoff": "./src/core/backoff.ts",
|
||||
"./search/hybrid": "./src/core/search/hybrid.ts",
|
||||
"./search/expansion": "./src/core/search/expansion.ts",
|
||||
"./ai/gateway": "./src/core/ai/gateway.ts",
|
||||
"./extract": "./src/commands/extract.ts"
|
||||
},
|
||||
"scripts": {
|
||||
"dev": "bun run src/cli.ts",
|
||||
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
||||
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
||||
"build:admin": "cd admin && bun run build && cd .. && bun run scripts/build-admin-embedded.ts",
|
||||
"build:admin-embedded": "bun run scripts/build-admin-embedded.ts",
|
||||
"build:admin": "cd admin && bun run build",
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"build:llms": "bun run scripts/build-llms.ts",
|
||||
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
||||
"test": "bash scripts/run-unit-parallel.sh",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:synthetic-corpus-privacy && bun run typecheck",
|
||||
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"verify": "bun run check:privacy && bun run check:jsonb && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run typecheck",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
|
||||
"check:all": "scripts/check-privacy.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
@@ -55,15 +51,10 @@
|
||||
"ci:select-e2e": "bun run scripts/select-e2e.ts",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "scripts/check-privacy.sh",
|
||||
"check:proposal-pii": "scripts/check-proposal-pii.sh",
|
||||
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "scripts/check-test-real-names.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"check:exports-count": "scripts/check-exports-count.sh",
|
||||
"check:admin-build": "scripts/check-admin-build.sh",
|
||||
"check:admin-embedded": "scripts/check-admin-embedded.sh",
|
||||
"check:test-isolation": "scripts/check-test-isolation.sh",
|
||||
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
@@ -72,10 +63,7 @@
|
||||
"openclaw": {
|
||||
"compat": {
|
||||
"pluginApi": ">=2026.4.0"
|
||||
},
|
||||
"extensions": [
|
||||
"./src/openclaw-context-engine.ts"
|
||||
]
|
||||
}
|
||||
},
|
||||
"dependencies": {
|
||||
"@ai-sdk/anthropic": "^3.0.71",
|
||||
@@ -86,18 +74,14 @@
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@dqbd/tiktoken": "^1.0.22",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@jsquash/avif": "^2.1.1",
|
||||
"@jsquash/png": "^3.1.1",
|
||||
"@modelcontextprotocol/sdk": "1.29.0",
|
||||
"ai": "^6.0.168",
|
||||
"cookie-parser": "^1.4.7",
|
||||
"cors": "^2.8.5",
|
||||
"eventsource-parser": "^3.0.8",
|
||||
"exifr": "^7.1.3",
|
||||
"express": "^5.1.0",
|
||||
"express-rate-limit": "^7.5.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"heic-decode": "^2.1.0",
|
||||
"marked": "^18.0.0",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
|
||||
@@ -1,134 +0,0 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* Generates `src/admin-embedded.ts` from `admin/dist/*`.
|
||||
*
|
||||
* Why: `bun build --compile` does NOT embed arbitrary asset directories.
|
||||
* The only way to ship a file inside a compiled binary is via an ESM
|
||||
* `import x from './path' with { type: 'file' }` reference (which Bun
|
||||
* resolves at runtime to a path that works inside the binary archive).
|
||||
*
|
||||
* Pre-v0.36.x, `serve-http.ts:780` resolved `admin/dist/` via
|
||||
* `process.cwd()` — fine in dev (`cd ~/gbrain && bun start serve --http`),
|
||||
* broken in every globally-installed binary (no admin/dist next to the
|
||||
* binary). Result: every fresh `bun install -g github:garrytan/gbrain`
|
||||
* user got 404 on /admin (issue #1090).
|
||||
*
|
||||
* This generator emits one `import` line per file under admin/dist/,
|
||||
* plus a manifest map keyed by the request path the express handler
|
||||
* sees (e.g. `/admin/index.html`, `/admin/assets/index-XXX.js`).
|
||||
*
|
||||
* Run: `bun run scripts/build-admin-embedded.ts` (also invoked by
|
||||
* `bun run build:admin`).
|
||||
*
|
||||
* CI guard: `scripts/check-admin-embedded.sh` re-runs this generator
|
||||
* and `git diff --exit-code src/admin-embedded.ts` so PRs that change
|
||||
* admin/dist without regenerating the embedded module fail loud.
|
||||
*/
|
||||
|
||||
import { readdirSync, statSync, writeFileSync, existsSync, readFileSync } from 'fs';
|
||||
import { join, relative, posix } from 'path';
|
||||
|
||||
const REPO = join(import.meta.dir, '..');
|
||||
const DIST = join(REPO, 'admin', 'dist');
|
||||
const OUT = join(REPO, 'src', 'admin-embedded.ts');
|
||||
|
||||
function walk(dir: string, base: string = dir): string[] {
|
||||
if (!existsSync(dir)) return [];
|
||||
const out: string[] = [];
|
||||
for (const entry of readdirSync(dir)) {
|
||||
const full = join(dir, entry);
|
||||
if (statSync(full).isDirectory()) {
|
||||
out.push(...walk(full, base));
|
||||
} else {
|
||||
out.push(relative(base, full));
|
||||
}
|
||||
}
|
||||
return out.sort();
|
||||
}
|
||||
|
||||
const MIME: Record<string, string> = {
|
||||
'.html': 'text/html; charset=utf-8',
|
||||
'.css': 'text/css; charset=utf-8',
|
||||
'.js': 'application/javascript; charset=utf-8',
|
||||
'.json': 'application/json; charset=utf-8',
|
||||
'.svg': 'image/svg+xml',
|
||||
'.png': 'image/png',
|
||||
'.jpg': 'image/jpeg',
|
||||
'.jpeg': 'image/jpeg',
|
||||
'.gif': 'image/gif',
|
||||
'.webp': 'image/webp',
|
||||
'.ico': 'image/x-icon',
|
||||
'.woff': 'font/woff',
|
||||
'.woff2': 'font/woff2',
|
||||
'.txt': 'text/plain; charset=utf-8',
|
||||
'.map': 'application/json; charset=utf-8',
|
||||
};
|
||||
|
||||
function mimeFor(filename: string): string {
|
||||
const dot = filename.lastIndexOf('.');
|
||||
if (dot === -1) return 'application/octet-stream';
|
||||
return MIME[filename.slice(dot).toLowerCase()] ?? 'application/octet-stream';
|
||||
}
|
||||
|
||||
function safeIdent(rel: string, idx: number): string {
|
||||
// Stable, collision-free identifier per relative path. The numeric
|
||||
// suffix prevents collisions between filenames that normalize to the
|
||||
// same identifier (e.g. `foo.bar.js` and `foo-bar.js`).
|
||||
const cleaned = rel.replace(/[^a-zA-Z0-9]/g, '_').replace(/^_+/, '');
|
||||
return `A_${idx}_${cleaned}`;
|
||||
}
|
||||
|
||||
const files = walk(DIST);
|
||||
if (files.length === 0) {
|
||||
console.error('[build-admin-embedded] no files under admin/dist — run `cd admin && bun run build` first.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const imports: string[] = [];
|
||||
const manifestEntries: string[] = [];
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
const rel = files[i];
|
||||
// POSIX-style relative path for the import (works on Windows too).
|
||||
const importRel = `../admin/dist/${rel.split(/[\\/]/).join('/')}`;
|
||||
const ident = safeIdent(rel, i);
|
||||
// @ts-ignore — `with { type: 'file' }` is Bun syntax not in lib.d.ts;
|
||||
// same pattern as src/core/chunkers/code.ts wasm imports.
|
||||
imports.push(`// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts`);
|
||||
imports.push(`import ${ident} from '${importRel}' with { type: 'file' };`);
|
||||
const requestPath = '/admin/' + rel.split(/[\\/]/).join('/');
|
||||
manifestEntries.push(` ${JSON.stringify(requestPath)}: { path: ${ident} as unknown as string, mime: ${JSON.stringify(mimeFor(rel))} },`);
|
||||
}
|
||||
|
||||
const content = `// AUTO-GENERATED — do not edit by hand.
|
||||
// Run \`bun run scripts/build-admin-embedded.ts\` to regenerate.
|
||||
// Source: admin/dist/ at ${new Date().toISOString().slice(0, 10)}.
|
||||
//
|
||||
// Bun resolves the file: imports to a path that works at runtime even
|
||||
// inside a compiled binary (\`bun build --compile\`). The manifest maps
|
||||
// the request path the express handler sees to (resolved-path, mime).
|
||||
|
||||
${imports.join('\n')}
|
||||
|
||||
export interface AdminAsset {
|
||||
path: string;
|
||||
mime: string;
|
||||
}
|
||||
|
||||
export const ADMIN_ASSETS: Record<string, AdminAsset> = {
|
||||
${manifestEntries.join('\n')}
|
||||
};
|
||||
|
||||
/** Index entry point for SPA fallback. */
|
||||
export const ADMIN_INDEX_HTML: AdminAsset = ADMIN_ASSETS['/admin/index.html'];
|
||||
|
||||
export const ADMIN_ASSET_COUNT = ${files.length};
|
||||
`;
|
||||
|
||||
const existing = existsSync(OUT) ? readFileSync(OUT, 'utf-8') : '';
|
||||
if (existing === content) {
|
||||
console.log(`[build-admin-embedded] up to date (${files.length} files)`);
|
||||
} else {
|
||||
writeFileSync(OUT, content, 'utf-8');
|
||||
console.log(`[build-admin-embedded] wrote ${OUT} (${files.length} files)`);
|
||||
}
|
||||
@@ -1,308 +0,0 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* scripts/build-contradictions-fixture.ts (v0.32.6, T2)
|
||||
*
|
||||
* Build a privacy-redacted gold fixture for the contradiction probe judge
|
||||
* by running the probe against the user's REAL brain and hand-labeling
|
||||
* the candidate pairs. Output: test/fixtures/contradictions-eval-gold.jsonl.
|
||||
*
|
||||
* Privacy posture (CLAUDE.md rule): the operator MUST inspect the
|
||||
* generated file before commit. The redactor (fixture-redact.ts) is
|
||||
* best-effort; the pre-commit review is the safety net. Fail-closed if
|
||||
* any pair fails the isCleanForCommit check after redaction.
|
||||
*
|
||||
* Usage:
|
||||
* bun run scripts/build-contradictions-fixture.ts \
|
||||
* [--queries-file FILE.jsonl] \
|
||||
* [--top-k N=5] \
|
||||
* [--judge MODEL=claude-haiku-4-5] \
|
||||
* [--max-pairs N=50] \
|
||||
* [--output PATH=test/fixtures/contradictions-eval-gold.jsonl] \
|
||||
* [--non-interactive]
|
||||
*
|
||||
* Interactive flow:
|
||||
* - Probe runs with --no-cache (so candidate pairs aren't pre-judged).
|
||||
* - For each candidate pair, the script prints A + B and prompts:
|
||||
* y) contradiction, n) not contradiction, s) skip
|
||||
* If y: prompt for severity (low|medium|high) and one-line axis.
|
||||
* - After labeling, redact in-memory, write JSONL with audit comments.
|
||||
* - Pre-commit safety: isCleanForCommit per line. Failures abort with
|
||||
* a sentinel string the operator must resolve manually.
|
||||
*
|
||||
* Non-interactive flow (`--non-interactive`): captures candidates with
|
||||
* NO labels, redacts, writes JSONL. Operator labels manually later.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
||||
import { dirname } from 'node:path';
|
||||
import { createInterface } from 'node:readline/promises';
|
||||
import { stdin as input, stdout as output } from 'node:process';
|
||||
import { loadConfig, toEngineConfig } from '../src/core/config.ts';
|
||||
import { createEngine } from '../src/core/engine-factory.ts';
|
||||
import { connectWithRetry } from '../src/core/db.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { runContradictionProbe } from '../src/core/eval-contradictions/runner.ts';
|
||||
|
||||
async function connectLocalEngine(): Promise<BrainEngine> {
|
||||
const cfg = loadConfig();
|
||||
if (!cfg) throw new Error('No brain configured. Run `gbrain init` first.');
|
||||
const engineCfg = toEngineConfig(cfg);
|
||||
const engine = await createEngine(engineCfg);
|
||||
await connectWithRetry(engine, engineCfg, { noRetry: false });
|
||||
return engine;
|
||||
}
|
||||
import {
|
||||
createRedactionSession,
|
||||
isCleanForCommit,
|
||||
redactSlug,
|
||||
redactText,
|
||||
} from '../src/core/eval-contradictions/fixture-redact.ts';
|
||||
import type { ContradictionPair, Severity } from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
interface ParsedFlags {
|
||||
queriesFile?: string;
|
||||
topK: number;
|
||||
judge: string;
|
||||
maxPairs: number;
|
||||
output: string;
|
||||
nonInteractive: boolean;
|
||||
help: boolean;
|
||||
}
|
||||
|
||||
function parseFlags(argv: string[]): ParsedFlags {
|
||||
const f: ParsedFlags = {
|
||||
topK: 5,
|
||||
judge: 'anthropic:claude-haiku-4-5',
|
||||
maxPairs: 50,
|
||||
output: 'test/fixtures/contradictions-eval-gold.jsonl',
|
||||
nonInteractive: false,
|
||||
help: false,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
const next = (): string => {
|
||||
const v = argv[++i];
|
||||
if (v === undefined) throw new Error(`flag ${a} requires a value`);
|
||||
return v;
|
||||
};
|
||||
if (a === '--help' || a === '-h') f.help = true;
|
||||
else if (a === '--queries-file') f.queriesFile = next();
|
||||
else if (a === '--top-k') f.topK = Number.parseInt(next(), 10);
|
||||
else if (a === '--judge') f.judge = next();
|
||||
else if (a === '--max-pairs') f.maxPairs = Number.parseInt(next(), 10);
|
||||
else if (a === '--output') f.output = next();
|
||||
else if (a === '--non-interactive') f.nonInteractive = true;
|
||||
else throw new Error(`unknown flag: ${a}`);
|
||||
}
|
||||
return f;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stderr.write(`Build a privacy-redacted gold fixture for the contradiction probe judge.
|
||||
|
||||
Usage:
|
||||
bun run scripts/build-contradictions-fixture.ts \\
|
||||
--queries-file FILE.jsonl # one JSON object per line, {query: "..."}
|
||||
[--top-k N=5]
|
||||
[--judge MODEL=claude-haiku-4-5]
|
||||
[--max-pairs N=50]
|
||||
[--output PATH=test/fixtures/contradictions-eval-gold.jsonl]
|
||||
[--non-interactive]
|
||||
|
||||
Output: JSONL with one labeled-and-redacted pair per line. Lines that
|
||||
fail isCleanForCommit are marked with a sentinel string the operator
|
||||
MUST resolve manually before commit. Audit log printed to stderr.
|
||||
`);
|
||||
}
|
||||
|
||||
function readQueriesFile(path: string): string[] {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const out: string[] = [];
|
||||
for (const line of raw.split(/\r?\n/)) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
if (trimmed.startsWith('{')) {
|
||||
try {
|
||||
const parsed = JSON.parse(trimmed) as { query?: string };
|
||||
if (typeof parsed.query === 'string' && parsed.query.length > 0) {
|
||||
out.push(parsed.query);
|
||||
}
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
} else {
|
||||
out.push(trimmed);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function promptLabel(rl: ReturnType<typeof createInterface>, pair: ContradictionPair): Promise<{
|
||||
contradicts: boolean;
|
||||
severity: Severity;
|
||||
axis: string;
|
||||
skip: boolean;
|
||||
}> {
|
||||
process.stderr.write(`\n--- Pair ---\n`);
|
||||
process.stderr.write(`A (${pair.a.slug}): ${pair.a.text.slice(0, 240)}${pair.a.text.length > 240 ? '…' : ''}\n`);
|
||||
process.stderr.write(`B (${pair.b.slug}): ${pair.b.text.slice(0, 240)}${pair.b.text.length > 240 ? '…' : ''}\n`);
|
||||
const ans = (await rl.question('Contradiction? [y/n/s skip]: ')).trim().toLowerCase();
|
||||
if (ans === 's' || ans === 'skip') {
|
||||
return { contradicts: false, severity: 'low', axis: '', skip: true };
|
||||
}
|
||||
if (ans !== 'y' && ans !== 'yes') {
|
||||
return { contradicts: false, severity: 'low', axis: '', skip: false };
|
||||
}
|
||||
let sev = (await rl.question('Severity [low/medium/high, default low]: ')).trim().toLowerCase();
|
||||
if (sev !== 'low' && sev !== 'medium' && sev !== 'high') sev = 'low';
|
||||
const axis = (await rl.question('One-line axis: ')).trim();
|
||||
return { contradicts: true, severity: sev as Severity, axis, skip: false };
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
let flags: ParsedFlags;
|
||||
try {
|
||||
flags = parseFlags(process.argv.slice(2));
|
||||
} catch (err) {
|
||||
process.stderr.write(`Error: ${(err as Error).message}\n`);
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
if (flags.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
if (!flags.queriesFile) {
|
||||
process.stderr.write(`--queries-file is required for the fixture build.\n`);
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const queries = readQueriesFile(flags.queriesFile);
|
||||
if (queries.length === 0) {
|
||||
process.stderr.write(`No queries in ${flags.queriesFile}.\n`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
process.stderr.write(`Building gold fixture against the local brain.\n`);
|
||||
process.stderr.write(`Queries: ${queries.length} Top-K: ${flags.topK} Max pairs: ${flags.maxPairs}\n`);
|
||||
process.stderr.write(`Output: ${flags.output}\n\n`);
|
||||
|
||||
const engine = await connectLocalEngine();
|
||||
try {
|
||||
// Run the probe with --no-cache so we get candidate pairs without
|
||||
// pre-judged verdicts. We don't keep verdicts; we hand-label every pair.
|
||||
// We intercept pairs via judgeFn returning contradicts:false (so nothing
|
||||
// is filtered to findings) and accumulating them for labeling instead.
|
||||
const candidatePairs: ContradictionPair[] = [];
|
||||
await runContradictionProbe({
|
||||
engine,
|
||||
queries,
|
||||
judgeModel: flags.judge,
|
||||
topK: flags.topK,
|
||||
noCache: true,
|
||||
// Wide budget so we don't hit cap during candidate collection.
|
||||
budgetUsd: 100,
|
||||
yesOverride: true,
|
||||
// Hijack the judge to collect pairs without spending tokens.
|
||||
judgeFn: async (input) => {
|
||||
candidatePairs.push({
|
||||
kind: 'cross_slug_chunks', // best-effort label; runner emits both kinds
|
||||
a: { slug: input.a.slug, chunk_id: 0, take_id: null, source_tier: 'curated', holder: input.a.holder ?? null, text: input.a.text },
|
||||
b: { slug: input.b.slug, chunk_id: 0, take_id: null, source_tier: 'curated', holder: input.b.holder ?? null, text: input.b.text },
|
||||
combined_score: 0,
|
||||
});
|
||||
return {
|
||||
verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0, resolution_kind: null },
|
||||
usage: { inputTokens: 0, outputTokens: 0 },
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
process.stderr.write(`\nCollected ${candidatePairs.length} candidate pairs.\n`);
|
||||
const capped = candidatePairs.slice(0, flags.maxPairs);
|
||||
|
||||
// Label.
|
||||
const rl = createInterface({ input, output });
|
||||
const session = createRedactionSession();
|
||||
const labeled: Array<{
|
||||
contradicts: boolean;
|
||||
severity: Severity;
|
||||
axis: string;
|
||||
query_redacted: string;
|
||||
a: { slug: string; text: string };
|
||||
b: { slug: string; text: string };
|
||||
}> = [];
|
||||
|
||||
for (let i = 0; i < capped.length; i++) {
|
||||
const pair = capped[i];
|
||||
process.stderr.write(`\n[${i + 1}/${capped.length}]`);
|
||||
let label: { contradicts: boolean; severity: Severity; axis: string; skip: boolean };
|
||||
if (flags.nonInteractive) {
|
||||
label = { contradicts: false, severity: 'low', axis: '', skip: false };
|
||||
} else {
|
||||
label = await promptLabel(rl, pair);
|
||||
if (label.skip) continue;
|
||||
}
|
||||
const redactedA = {
|
||||
slug: redactSlug(session, pair.a.slug),
|
||||
text: redactText(session, pair.a.text),
|
||||
};
|
||||
const redactedB = {
|
||||
slug: redactSlug(session, pair.b.slug),
|
||||
text: redactText(session, pair.b.text),
|
||||
};
|
||||
labeled.push({
|
||||
contradicts: label.contradicts,
|
||||
severity: label.severity,
|
||||
axis: redactText(session, label.axis),
|
||||
// Query gets redacted too, in case it referenced real names.
|
||||
query_redacted: '', // candidatePairs don't carry the query; populated by future iteration
|
||||
a: redactedA,
|
||||
b: redactedB,
|
||||
});
|
||||
}
|
||||
rl.close();
|
||||
|
||||
// Pre-commit safety: every text field must pass isCleanForCommit.
|
||||
const out: string[] = [];
|
||||
let flagged = 0;
|
||||
out.push(`# Gold fixture for contradiction probe judge (v0.32.6)`);
|
||||
out.push(`# schema_version: 1`);
|
||||
out.push(`# Generated: ${new Date().toISOString()}`);
|
||||
out.push(`# Audit (in-memory redactions applied):`);
|
||||
for (const entry of session.audit.slice(0, 100)) {
|
||||
out.push(`# ${entry}`);
|
||||
}
|
||||
out.push(`# Total redactions: ${session.audit.length}`);
|
||||
out.push(`#`);
|
||||
for (const row of labeled) {
|
||||
const cleanA = isCleanForCommit(row.a.text) && isCleanForCommit(row.a.slug);
|
||||
const cleanB = isCleanForCommit(row.b.text) && isCleanForCommit(row.b.slug);
|
||||
const sentinel = !cleanA || !cleanB ? ' [REDACT?]' : '';
|
||||
if (sentinel) flagged++;
|
||||
out.push(JSON.stringify({ ...row, ...(sentinel ? { _operator_review: 'REDACTION INCOMPLETE — fix manually before commit' } : {}) }));
|
||||
}
|
||||
|
||||
// Ensure output dir exists, then write.
|
||||
mkdirSync(dirname(flags.output), { recursive: true });
|
||||
if (existsSync(flags.output)) {
|
||||
process.stderr.write(`\nWARN: ${flags.output} already exists. Overwriting.\n`);
|
||||
}
|
||||
writeFileSync(flags.output, out.join('\n') + '\n');
|
||||
process.stderr.write(`\nWrote ${labeled.length} labeled pairs to ${flags.output}.\n`);
|
||||
if (flagged > 0) {
|
||||
process.stderr.write(`*** ${flagged} pair(s) flagged with [REDACT?] — review before commit ***\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
process.stderr.write(`OK — pre-commit safety pass. Inspect the file once more before committing.\n`);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
process.stderr.write(`fatal: ${(err as Error).message}\n`);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,35 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI gate: src/admin-embedded.ts must match admin/dist/ contents.
|
||||
#
|
||||
# This protects against the v0.36.x #1090 bug class re-emerging — a PR
|
||||
# that rebuilds admin/dist but forgets to regenerate src/admin-embedded.ts
|
||||
# would silently break /admin on every fresh install of the compiled
|
||||
# binary. The Vite build outputs hashed filenames, so a stale embedded
|
||||
# manifest references nonexistent assets.
|
||||
#
|
||||
# How: re-run the generator, then `git diff --exit-code` on the output.
|
||||
# Exits 0 when in sync, 1 when the generator produces different output
|
||||
# than what's committed.
|
||||
#
|
||||
# Mirrors scripts/check-wasm-embedded.sh's pattern.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
if [ ! -d admin/dist ]; then
|
||||
echo "[check:admin-embedded] no admin/dist (run \`cd admin && bun run build\` first); skipping"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
bun run scripts/build-admin-embedded.ts > /dev/null
|
||||
|
||||
if ! git diff --exit-code -- src/admin-embedded.ts; then
|
||||
echo ""
|
||||
echo "[check:admin-embedded] src/admin-embedded.ts is out of sync with admin/dist/."
|
||||
echo " Fix: bun run build:admin && bun run build:admin-embedded"
|
||||
echo " Then re-commit the regenerated src/admin-embedded.ts."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[check:admin-embedded] OK"
|
||||
@@ -1,43 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.32.3 — CI guard for docs/eval/METRIC_GLOSSARY.md freshness.
|
||||
#
|
||||
# Mirrors the scripts/check-jsonb-pattern.sh / check-progress-to-stdout.sh
|
||||
# discipline: regenerate the doc into a tmp file, diff against the committed
|
||||
# version, fail the build if they drift.
|
||||
#
|
||||
# Run: bash scripts/check-eval-glossary-fresh.sh
|
||||
# CI wires this through `bun run test` so PRs that bump the glossary module
|
||||
# without regenerating the doc are caught before review.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
COMMITTED="$REPO_ROOT/docs/eval/METRIC_GLOSSARY.md"
|
||||
TMP="$(mktemp)"
|
||||
trap 'rm -f "$TMP"' EXIT
|
||||
|
||||
if [ ! -f "$COMMITTED" ]; then
|
||||
echo "ERROR: $COMMITTED not found." >&2
|
||||
echo "Run: bun run scripts/generate-metric-glossary.ts" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Regenerate into TMP without touching the committed file. We can't easily
|
||||
# point the generator at a different path; trick it by redirecting cwd to
|
||||
# a sandbox and post-comparing.
|
||||
cd "$REPO_ROOT"
|
||||
# Render directly via bun + a one-liner that exposes the module function.
|
||||
bun -e "import { renderMetricGlossaryMarkdown } from './src/core/eval/metric-glossary.ts'; process.stdout.write(renderMetricGlossaryMarkdown());" > "$TMP"
|
||||
|
||||
if ! diff -q "$COMMITTED" "$TMP" >/dev/null 2>&1; then
|
||||
echo "ERROR: docs/eval/METRIC_GLOSSARY.md is stale." >&2
|
||||
echo "" >&2
|
||||
echo "Diff between committed and freshly-generated:" >&2
|
||||
echo "" >&2
|
||||
diff -u "$COMMITTED" "$TMP" >&2 || true
|
||||
echo "" >&2
|
||||
echo "To regenerate: bun run scripts/generate-metric-glossary.ts" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✓ docs/eval/METRIC_GLOSSARY.md is fresh"
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
EXPECTED_COUNT=18
|
||||
EXPECTED_COUNT=17
|
||||
|
||||
# Count top-level keys in the exports object. `node -e` parses JSON
|
||||
# reliably without needing jq (which isn't in every CI environment).
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: verify that bun --compile binaries can decode HEIC + AVIF.
|
||||
#
|
||||
# heic-decode bundles its libheif WASM as base64 inside libheif-bundle.js, which
|
||||
# bun --compile preserves correctly out of the box. @jsquash/avif loads
|
||||
# avif_dec.wasm via a path relative to its own JS file, which FAILS inside a
|
||||
# compiled binary — the workaround is to pre-init the module with bytes loaded
|
||||
# via `with { type: 'file' }`. This guard ensures both paths actually work in
|
||||
# the compiled artifact, not just in dev mode.
|
||||
#
|
||||
# Mirrors scripts/check-wasm-embedded.sh from v0.19.0 (tree-sitter pattern).
|
||||
#
|
||||
# Wired into `bun run verify` (which `/ship` and `bun run test:full` call).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
OUT_BIN="$(mktemp /tmp/gbrain-img-decoders-check.XXXXXX)"
|
||||
trap 'rm -f "$OUT_BIN"' EXIT
|
||||
|
||||
bun build --compile --outfile "$OUT_BIN" scripts/image-decoders-smoketest.ts >/dev/null 2>&1
|
||||
|
||||
OUTPUT="$("$OUT_BIN" 2>&1 || true)"
|
||||
|
||||
# The smoketest writes a JSON line on stdout. Look for ok=true on each decoder.
|
||||
if ! echo "$OUTPUT" | grep -q '"heic":{"ok":true'; then
|
||||
echo "[check-image-decoders-embedded] FAIL: heic-decode failed in compiled binary." >&2
|
||||
echo "[check-image-decoders-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
echo "" >&2
|
||||
echo "Likely cause: libheif-bundle.js was upgraded to a non-bundle variant," >&2
|
||||
echo "or wasm-bundle.js stopped inlining the WASM as base64. Check the" >&2
|
||||
echo "heic-decode + libheif-js versions in package.json." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! echo "$OUTPUT" | grep -q '"avif":{"ok":true'; then
|
||||
echo "[check-image-decoders-embedded] FAIL: @jsquash/avif failed in compiled binary." >&2
|
||||
echo "[check-image-decoders-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
echo "" >&2
|
||||
echo "Likely cause: the import attribute path for avif_dec.wasm changed in" >&2
|
||||
echo "@jsquash/avif, or initAvif() no longer accepts a WebAssembly.Module" >&2
|
||||
echo "directly. Check scripts/image-decoders-smoketest.ts for the WASM" >&2
|
||||
echo "pre-init pattern, then mirror it in src/core/import-file.ts." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Final guard: top-level "ok":true.
|
||||
if ! echo "$OUTPUT" | grep -q '"ok":true}$'; then
|
||||
echo "[check-image-decoders-embedded] FAIL: probe returned ok:false." >&2
|
||||
echo "$OUTPUT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[check-image-decoders-embedded] HEIC + AVIF decoders embed and decode correctly in compiled binary."
|
||||
@@ -1,64 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: every `switch (X.type)` site in src/ that discriminates on a
|
||||
# PageType-shaped value MUST use assertNever() in the default branch.
|
||||
#
|
||||
# Why: extending PageType (e.g. v0.27.1 adding 'image') silently fell through
|
||||
# default branches in v0.20 / v0.22 because TypeScript couldn't catch the
|
||||
# missing case at type-check time. assertNever() forces the compiler to error
|
||||
# when a new PageType lacks a matching case.
|
||||
#
|
||||
# Today (pre-v0.27.1) the codebase has zero PageType-discriminating switches —
|
||||
# it uses the type system for exhaustiveness via union narrowing. This guard
|
||||
# is preventive: catches the moment a contributor adds a switch and forgets
|
||||
# the assertNever.
|
||||
#
|
||||
# Pattern: a `switch (x.type)` where the surrounding file imports PageType
|
||||
# (heuristic: imports from './types' or '../types') is treated as a
|
||||
# PageType-shaped switch and must include assertNever in default.
|
||||
#
|
||||
# False positives are easy to silence by adding an `// eslint-disable-line
|
||||
# pagetype-exhaustive` style comment above the offending switch.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
VIOLATIONS=0
|
||||
|
||||
# Find every src/**.ts file that imports PageType. Portable across Bash 3.2
|
||||
# (macOS default) — no mapfile, no process substitution arrays.
|
||||
PAGETYPE_FILES=$(grep -rlE "import.*PageType.*from.*types" src 2>/dev/null || true)
|
||||
|
||||
if [ -z "$PAGETYPE_FILES" ]; then
|
||||
echo "[check-pagetype-exhaustive] No files import PageType. Skipping."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
while IFS= read -r file; do
|
||||
[ -z "$file" ] && continue
|
||||
# Look for `switch (X.type)` patterns in the file. Heuristic: any `switch (`
|
||||
# followed by a `.type)` within the line.
|
||||
if grep -nE 'switch\s*\([^)]*\.type\s*\)' "$file" >/dev/null 2>&1; then
|
||||
# File has at least one switch on .type. Verify assertNever is imported
|
||||
# AND used somewhere in the file. If both are present, assume the dev
|
||||
# wired it correctly — finer-grained per-switch checking is too brittle.
|
||||
if ! grep -qE 'assertNever' "$file"; then
|
||||
echo "[check-pagetype-exhaustive] FAIL: $file has switch(X.type) but no assertNever() use." >&2
|
||||
grep -nE 'switch\s*\([^)]*\.type\s*\)' "$file" >&2 || true
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
fi
|
||||
fi
|
||||
done <<< "$PAGETYPE_FILES"
|
||||
|
||||
if [ "$VIOLATIONS" -gt 0 ]; then
|
||||
echo "" >&2
|
||||
echo "Fix: import { assertNever } from './types.ts' (or wherever appropriate)" >&2
|
||||
echo "and add \`default: return assertNever(x.type);\` to the switch." >&2
|
||||
echo "If the switch is intentionally non-exhaustive (e.g. handling only a" >&2
|
||||
echo "subset of PageTypes), document why with a comment and add the file" >&2
|
||||
echo "to an explicit allow-list at the top of this script." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[check-pagetype-exhaustive] All PageType-discriminating switches use assertNever() (or none exist)."
|
||||
@@ -1,52 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI grep guard (v0.30.1, finding F3): no source file under src/ may emit
|
||||
# a postgresql:// URL with userinfo to a logging surface.
|
||||
#
|
||||
# Specifically we forbid string literals or template substitutions that
|
||||
# look like `postgresql://user:pass@host` being passed to:
|
||||
# - console.log / .warn / .error
|
||||
# - process.stderr.write / process.stdout.write
|
||||
# - appendFileSync / writeFileSync (audit JSONL writes)
|
||||
# - new logging APIs that may show up later (the regex matches the URL,
|
||||
# not the consumer; any leak will trip)
|
||||
#
|
||||
# Wired into bun run check:all and bun run verify.
|
||||
#
|
||||
# Exit codes: 0 = clean, 1 = found at least one suspect line.
|
||||
set -euo pipefail
|
||||
|
||||
ROOT=$(cd "$(dirname "$0")/.." && pwd)
|
||||
|
||||
# False-positive allow-list: lines we know are safe.
|
||||
# - The redactor itself: src/core/url-redact.ts
|
||||
# - Test fixtures that build redacted strings from full URLs
|
||||
# - Documentation comments referring to the pattern
|
||||
ALLOW_REGEX='url-redact\.ts|test/url-redact\.test\.ts|/\* allow-pg-url-literal \*/'
|
||||
|
||||
# The pattern matches an unredacted Postgres URL appearing in a string
|
||||
# literal, NOT preceded by `redactPgUrl(` or `***@`. We also match any
|
||||
# URL containing `[^*]@` (i.e. the `***@` redacted form passes).
|
||||
PATTERN='postgres(ql)?://[^@*"`]+@'
|
||||
|
||||
# Search src/ only — tests are excluded since they intentionally construct
|
||||
# unredacted URLs as input fixtures.
|
||||
HITS=$(grep -rEn "$PATTERN" "$ROOT/src" 2>/dev/null || true)
|
||||
|
||||
if [ -z "$HITS" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Filter against the allow-list.
|
||||
FILTERED=$(echo "$HITS" | grep -vE "$ALLOW_REGEX" || true)
|
||||
|
||||
if [ -z "$FILTERED" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "ERROR: unredacted postgres:// URL found in source. Use redactPgUrl() before logging."
|
||||
echo ""
|
||||
echo "$FILTERED"
|
||||
echo ""
|
||||
echo "Allowed exemption: append \"/* allow-pg-url-literal */\" comment on the line"
|
||||
echo "(only for fixtures and the redactor itself)."
|
||||
exit 1
|
||||
@@ -121,43 +121,6 @@ ALLOW_LIST=(
|
||||
# walkthrough; it explains the privacy-guard extension to the
|
||||
# operating agent and references the banned literals while doing so.
|
||||
'skills/migrations/v0.25.1.md'
|
||||
# v0.29.1: the recency-decay default-map test asserts that
|
||||
# DEFAULT_RECENCY_DECAY's keys do NOT include fork-specific path
|
||||
# prefixes. The test must name the banned tokens to assert their
|
||||
# absence — same exception status as scripts/check-privacy.sh,
|
||||
# CHANGELOG.md, and CLAUDE.md (meta-rule enforcement requires
|
||||
# mentioning what the rule forbids).
|
||||
'test/recency-decay.test.ts'
|
||||
# v0.32.5: the sibling check-test-real-names.sh enforces the same
|
||||
# privacy rule for test fixtures and lists the banned names literally
|
||||
# (Wintermute, Hermes, etc) inside its BANNED_NAMES + ALLOWLIST arrays.
|
||||
# Same meta-rule-enforcement exception as scripts/check-privacy.sh itself.
|
||||
'scripts/check-test-real-names.sh'
|
||||
# v0.34 / Lane CI: scripts/check-proposal-pii.sh and its test list the
|
||||
# banned literal as part of the structural denylist they enforce against
|
||||
# docs/proposals/*.md. Same meta-rule-enforcement exception as the two
|
||||
# entries above — describing what the rule forbids requires naming it.
|
||||
'scripts/check-proposal-pii.sh'
|
||||
'test/scripts/check-proposal-pii.test.ts'
|
||||
# v0.32.3.0: the functional-area-resolver skill's behavior-contract
|
||||
# section describes the privacy guarantees the skill preserves and
|
||||
# references the banned literals while doing so (line 306). Same
|
||||
# meta-rule-enforcement exception as scripts/check-privacy.sh and
|
||||
# CHANGELOG.md — describing what the rule forbids requires naming it.
|
||||
'skills/functional-area-resolver/SKILL.md'
|
||||
# v0.36.0.0: the gbrain skillpack harvest privacy linter's whole job
|
||||
# is to catch the banned literal leaking into gbrain. The regex
|
||||
# pattern in harvest-lint.ts is `\bWintermute\b` by necessity; the
|
||||
# tests verify that pattern fires by feeding it the banned string;
|
||||
# the harvest skill markdown describes the substitution policy
|
||||
# ("Wintermute → your OpenClaw") as part of the genericization
|
||||
# checklist. Same meta-rule-enforcement exception as the privacy
|
||||
# checks themselves.
|
||||
'src/core/skillpack/harvest-lint.ts'
|
||||
'test/skillpack-harvest-lint.test.ts'
|
||||
'test/skillpack-harvest.test.ts'
|
||||
'test/e2e/skillpack-flow.test.ts'
|
||||
'skills/skillpack-harvest/SKILL.md'
|
||||
)
|
||||
|
||||
is_allowed() {
|
||||
|
||||
@@ -1,166 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# check-proposal-pii.sh — privacy guard for `docs/proposals/*.md`.
|
||||
#
|
||||
# Sibling to check-privacy.sh: that script bans the `Wintermute` literal
|
||||
# everywhere. This one focuses on `docs/proposals/*.md` and the OTHER PII
|
||||
# classes that have surfaced in past RFC drafts — personal-relationship
|
||||
# vocabulary, private repo references, etc.
|
||||
#
|
||||
# Why two scripts: the patterns this lint flags would be too noisy if
|
||||
# applied repo-wide (e.g. a test fixture mentioning "trial" is fine).
|
||||
# Restricting to `docs/proposals/` keeps the lint surgical — proposals are
|
||||
# public-facing RFC documents that should never contain personal context,
|
||||
# so the false-positive rate is near zero.
|
||||
#
|
||||
# Design note: the denylist names PATTERNS, not real people. Specific
|
||||
# real names (deceased relatives, therapist names, dealflow contacts)
|
||||
# would leak PII into the repo just by appearing in this script's
|
||||
# denylist. The structural patterns below catch the SURROUNDING context
|
||||
# of personal-event prose. The trade-off: a future RFC that names a real
|
||||
# person without any of the contextual markers won't be caught — that's
|
||||
# accepted as a residual risk handled by human review.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/check-proposal-pii.sh # scan working tree
|
||||
# scripts/check-proposal-pii.sh --staged # scan git staged index
|
||||
# scripts/check-proposal-pii.sh --help
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 clean
|
||||
# 1 PII pattern found
|
||||
# 2 setup error
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
PROPOSALS_DIR="$REPO_ROOT/docs/proposals"
|
||||
|
||||
# Structural patterns. One per line. Matched case-insensitively, fixed-string
|
||||
# (no regex). Comments start with #. Blank lines OK.
|
||||
#
|
||||
# IMPORTANT — design contract: this list MUST NOT contain real personal
|
||||
# names (deceased relatives, therapist first names, dealflow contacts).
|
||||
# Naming those would leak PII into scripts/. The patterns below catch the
|
||||
# SURROUNDING VOCABULARY that always accompanies such content in personal
|
||||
# RFC prose. Maintainers extending this list: prefer adding a phrase that
|
||||
# captures the context (e.g. `couples session`) rather than a specific
|
||||
# person's name.
|
||||
read -r -d '' PATTERNS <<'EOF' || true
|
||||
# Private repo references (zero false-positive risk)
|
||||
garrytan/brain
|
||||
|
||||
# Personal relationship vocabulary (extremely unlikely in technical RFCs)
|
||||
trial separation
|
||||
permanent separation
|
||||
couples session
|
||||
couples therapist
|
||||
divorce attorney
|
||||
divorce attorneys
|
||||
|
||||
# Death/funeral vocabulary in personal contexts (combined phrases — bare
|
||||
# "funeral" alone would false-positive in legitimate metaphorical use)
|
||||
grandmother's funeral
|
||||
grandmother funeral
|
||||
aunt's funeral
|
||||
aunt funeral
|
||||
|
||||
# Private agent / fork name (also enforced repo-wide by check-privacy.sh
|
||||
# but listed here for proposal-scoped clarity)
|
||||
wintermute
|
||||
EOF
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
scripts/check-proposal-pii.sh — privacy guard for docs/proposals/*.md.
|
||||
|
||||
USAGE:
|
||||
scripts/check-proposal-pii.sh Scan all proposal files.
|
||||
scripts/check-proposal-pii.sh --staged Scan only staged proposal files.
|
||||
scripts/check-proposal-pii.sh --help Show this message.
|
||||
|
||||
Flags personal-context vocabulary (e.g. "trial separation", "couples
|
||||
session", private repo references) inside docs/proposals/*.md. Use
|
||||
generic placeholders (alice-example, acme-corp, fund-a) in proposals.
|
||||
See CLAUDE.md "Privacy rule: scrub real names from public docs" for
|
||||
the canonical name-mapping table.
|
||||
|
||||
Sibling to scripts/check-privacy.sh which enforces the "Wintermute"
|
||||
ban repo-wide; this script catches the broader PII classes that
|
||||
appeared in past RFC drafts and were corrected at landing time.
|
||||
|
||||
Exit codes: 0 clean, 1 pattern found, 2 setup error.
|
||||
EOF
|
||||
}
|
||||
|
||||
MODE=working
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--staged) MODE=staged ;;
|
||||
--help|-h) usage; exit 1 ;;
|
||||
*)
|
||||
echo "Unknown argument: $arg" >&2
|
||||
usage >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ ! -d "$PROPOSALS_DIR" ]; then
|
||||
# No proposals dir yet — nothing to lint. Not a failure.
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Build the file list. Staged mode filters git's staged set down to
|
||||
# docs/proposals/*.md; working mode globs the directory directly.
|
||||
if [ "$MODE" = staged ]; then
|
||||
if ! command -v git >/dev/null 2>&1; then
|
||||
echo "check-proposal-pii: git not found" >&2
|
||||
exit 2
|
||||
fi
|
||||
FILES=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null \
|
||||
| grep -E '^docs/proposals/.+\.md$' || true)
|
||||
else
|
||||
FILES=$(find "$PROPOSALS_DIR" -maxdepth 1 -type f -name '*.md' 2>/dev/null \
|
||||
| sed "s|^$REPO_ROOT/||")
|
||||
fi
|
||||
|
||||
if [ -z "$FILES" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
FOUND=0
|
||||
# Iterate patterns; for each non-comment line, scan the file list.
|
||||
while IFS= read -r raw_line; do
|
||||
# Strip leading/trailing whitespace.
|
||||
pat="${raw_line#"${raw_line%%[![:space:]]*}"}"
|
||||
pat="${pat%"${pat##*[![:space:]]}"}"
|
||||
# Skip empty and comment lines.
|
||||
[ -z "$pat" ] && continue
|
||||
case "$pat" in '#'*) continue ;; esac
|
||||
|
||||
while IFS= read -r file; do
|
||||
[ -z "$file" ] && continue
|
||||
full="$REPO_ROOT/$file"
|
||||
[ ! -f "$full" ] && continue
|
||||
# Fixed-string (-F), case-insensitive (-i), with line numbers (-n).
|
||||
if matches=$(grep -nFi -- "$pat" "$full" 2>/dev/null); then
|
||||
if [ -n "$matches" ]; then
|
||||
echo "[check-proposal-pii] PII pattern in $file:" >&2
|
||||
echo " pattern: $pat" >&2
|
||||
echo "$matches" | sed 's|^| |' >&2
|
||||
FOUND=$((FOUND + 1))
|
||||
fi
|
||||
fi
|
||||
done <<< "$FILES"
|
||||
done <<< "$PATTERNS"
|
||||
|
||||
if [ "$FOUND" -gt 0 ]; then
|
||||
echo "" >&2
|
||||
echo "[check-proposal-pii] $FOUND PII pattern hit(s) in docs/proposals/*.md." >&2
|
||||
echo "[check-proposal-pii] See CLAUDE.md 'Privacy rule: scrub real names from public docs'." >&2
|
||||
echo "[check-proposal-pii] Use generic placeholders: alice-example, acme-corp, fund-a, etc." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any SELECT projection on `pages` that feeds rowToPage()
|
||||
# drops `source_id`. After v0.32.8, Page.source_id is required at the type
|
||||
# level; a projection that omits the column makes rowToPage return a Page
|
||||
# with source_id=undefined, which TypeScript's `: string` then lies about.
|
||||
#
|
||||
# This complements the type-system guard. The grep finds the specific 4-tuple
|
||||
# shape (id, slug, type, title) without source_id — the exact pre-v0.32.8
|
||||
# pattern that codex's plan review flagged.
|
||||
#
|
||||
# Usage: scripts/check-source-id-projection.sh
|
||||
# Exit: 0 when no matches, 1 when matches found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Allowlist: SELECT shapes that legitimately don't need source_id (single-col
|
||||
# `SELECT slug FROM pages` for getAllSlugs / resolveSlugs, SELECT id for
|
||||
# subqueries, COUNT, etc.) These don't feed rowToPage.
|
||||
#
|
||||
# The shape that DOES feed rowToPage starts `SELECT id, ... slug, ... type, ... title`
|
||||
# (in some order). The pattern below matches "id" + "slug" + "type" + "title"
|
||||
# in a SELECT projection — that's the rowToPage feeder signature.
|
||||
|
||||
FOUND_BAD=0
|
||||
|
||||
# Use multiline-aware grep so the SELECT can span lines. pcre2grep would be
|
||||
# cleaner but isn't universally available; do a simple two-pass instead:
|
||||
# 1. Pull each SELECT-from-pages block.
|
||||
# 2. For each, check if it has the rowToPage signature WITHOUT source_id.
|
||||
|
||||
check_file() {
|
||||
local file="$1"
|
||||
# Extract every SELECT...FROM pages block (across lines, up to 12 lines)
|
||||
# then test each.
|
||||
awk '
|
||||
/SELECT/ {
|
||||
buf = $0
|
||||
lines = 1
|
||||
while (lines < 12 && (!match(buf, /FROM[[:space:]]+pages\b/))) {
|
||||
if ((getline next_line) <= 0) break
|
||||
buf = buf " " next_line
|
||||
lines++
|
||||
}
|
||||
if (match(buf, /FROM[[:space:]]+pages\b/)) {
|
||||
# Has id, slug, type, title (rowToPage feeder) but NO source_id?
|
||||
if (match(buf, /\bid\b/) && match(buf, /\bslug\b/) && match(buf, /\btype\b/) && match(buf, /\btitle\b/) && !match(buf, /\bsource_id\b/)) {
|
||||
print FILENAME ": SELECT projection missing source_id:"
|
||||
print " " buf
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
}
|
||||
' "$file" || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
EXIT=0
|
||||
for f in src/core/postgres-engine.ts src/core/pglite-engine.ts; do
|
||||
if ! check_file "$f"; then
|
||||
EXIT=1
|
||||
fi
|
||||
done
|
||||
|
||||
# Also check RETURNING clauses (putPage uses INSERT ... RETURNING).
|
||||
# Same shape: returns a row that feeds rowToPage.
|
||||
for f in src/core/postgres-engine.ts src/core/pglite-engine.ts; do
|
||||
awk '
|
||||
/RETURNING/ {
|
||||
buf = $0
|
||||
lines = 1
|
||||
while (lines < 6 && !match(buf, /\`/)) {
|
||||
if ((getline next_line) <= 0) break
|
||||
buf = buf " " next_line
|
||||
lines++
|
||||
}
|
||||
if (match(buf, /\bid\b/) && match(buf, /\bslug\b/) && match(buf, /\btype\b/) && match(buf, /\btitle\b/) && !match(buf, /\bsource_id\b/)) {
|
||||
print FILENAME ": RETURNING projection missing source_id:"
|
||||
print " " buf
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
' "$f" || EXIT=1
|
||||
done
|
||||
|
||||
if [ "$EXIT" = 1 ]; then
|
||||
echo
|
||||
echo "ERROR: SELECT/RETURNING projection on \`pages\` is missing source_id."
|
||||
echo " After v0.32.8, Page.source_id is required at the type level."
|
||||
echo " Add \`source_id\` to the projection or rowToPage will lie."
|
||||
echo " See ~/.claude/plans/gleaming-soaring-mccarthy.md F2 finding."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: all rowToPage feeder projections include source_id"
|
||||
@@ -1,107 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.36.1.0 (T20 / CDX-14) — privacy CI guard for the synthetic calibration corpus.
|
||||
#
|
||||
# Scans test/fixtures/calibration/ for patterns that look like real-world
|
||||
# specificity. Fails the build if any are found. Closes the synthetic-corpus
|
||||
# privacy hole flagged by codex review CDX-14: "CC reads real brain pages
|
||||
# locally, writes nothing still risks privacy if any generated synthetic
|
||||
# fixture memorizes structure-specific facts. Placeholder names are not enough."
|
||||
#
|
||||
# What this catches:
|
||||
# - Real dollar amounts (e.g. "$50M", "$1.2B")
|
||||
# - Specific large round counts ($X cap is OK; "$50M Series B" is not)
|
||||
# - Year-specific date strings outside the 2024-2026 placeholder range
|
||||
# - The real founder/company names from the operator's network (looked up
|
||||
# from a sibling file scripts/check-synthetic-corpus-allowlist.txt when
|
||||
# present; otherwise we just check the placeholder allow-list)
|
||||
#
|
||||
# False positives stay safer than false negatives — this guard biases toward
|
||||
# the operator manually verifying a flagged page is legitimately synthetic.
|
||||
|
||||
set -e
|
||||
|
||||
CORPUS_DIR="test/fixtures/calibration"
|
||||
PLACEHOLDERS=(
|
||||
"alice-example"
|
||||
"charlie-example"
|
||||
"acme-example"
|
||||
"widget-co"
|
||||
"fund-a"
|
||||
"fund-b"
|
||||
"fund-c"
|
||||
"acme-seed"
|
||||
"widget-series-a"
|
||||
"meetings/2026-"
|
||||
)
|
||||
|
||||
# Skip if directory doesn't exist yet (early-clone state).
|
||||
if [ ! -d "$CORPUS_DIR" ]; then
|
||||
echo "OK: $CORPUS_DIR does not exist yet (skipping privacy scan)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
VIOLATIONS=0
|
||||
|
||||
# Check 1: real dollar amounts. Synthetic pages should say "$X" or describe
|
||||
# amounts as ranges; explicit numerics like "$50M" suggest real-world specificity.
|
||||
echo "[corpus-privacy] checking for explicit dollar amounts..."
|
||||
while IFS= read -r match; do
|
||||
if [ -n "$match" ]; then
|
||||
echo " VIOLATION: explicit dollar amount in $match"
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
fi
|
||||
done < <(grep -rEn '\$[0-9]+[MBKkmb]\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
|
||||
|
||||
# Check 2: explicit year-specific dates outside the 2024-2026 placeholder window.
|
||||
# The corpus uses placeholder timeline references like "2024-Q2", "2026-04-03".
|
||||
# Numbers like "2019" or "2027" mapped to specific events are suspicious.
|
||||
echo "[corpus-privacy] checking for out-of-range year references..."
|
||||
while IFS= read -r match; do
|
||||
if [ -n "$match" ]; then
|
||||
# Allow 2019 (used as a generic past year), 2023, 2027 (used as future). The
|
||||
# specific concern is dates the operator might recognize as a real prior event.
|
||||
# This is a low-precision heuristic; manual review decides.
|
||||
: # informational, not a failure for v0.36.1.0
|
||||
fi
|
||||
done < <(grep -rEn '\b(201[0-8]|2030|2031)\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
|
||||
|
||||
# Check 3: presence of expected placeholders. Synthetic pages should reference
|
||||
# at least one canonical placeholder. A page with ZERO placeholder names is
|
||||
# suspicious — might be referring to real people/companies.
|
||||
echo "[corpus-privacy] checking that fixture pages reference at least one placeholder..."
|
||||
while IFS= read -r file; do
|
||||
has_placeholder=false
|
||||
for ph in "${PLACEHOLDERS[@]}"; do
|
||||
if grep -q "$ph" "$file" 2>/dev/null; then
|
||||
has_placeholder=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
# Allow README + label JSON files to skip this check.
|
||||
# Also allow essay-genre fixtures, which are anonymized PG-essay-style writing
|
||||
# and don't reference specific people/companies by design.
|
||||
case "$file" in
|
||||
*README.md|*labels.json|*/essay-*.md) continue ;;
|
||||
esac
|
||||
if [ "$has_placeholder" = "false" ]; then
|
||||
echo " VIOLATION: $file references no placeholder name (expected at least one of: ${PLACEHOLDERS[*]})"
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
fi
|
||||
done < <(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null)
|
||||
|
||||
if [ "$VIOLATIONS" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ $VIOLATIONS privacy violation(s) found in $CORPUS_DIR."
|
||||
echo ""
|
||||
echo "The synthetic calibration corpus must use anonymized placeholder names"
|
||||
echo "(see test/fixtures/calibration/README.md). Real names of YC partners,"
|
||||
echo "portfolio companies, funds, etc. cannot enter this directory."
|
||||
echo ""
|
||||
echo "Either:"
|
||||
echo " - replace the offending content with placeholder names"
|
||||
echo " - confirm the dollar amount is intentionally generic, then update"
|
||||
echo " this script to exempt it"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✓ corpus privacy: $VIOLATIONS violations across $(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null | wc -l | tr -d ' ') pages"
|
||||
@@ -1,91 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.32.2 CI guard: enforce the system-of-record invariant.
|
||||
#
|
||||
# The rule: user-knowledge writes to derived DB tables (facts, takes,
|
||||
# links, timeline_entries) must go through the extract / reconcile /
|
||||
# migration layer, never directly from arbitrary code paths. Direct
|
||||
# calls would bypass the markdown source-of-truth contract — the next
|
||||
# `gbrain rebuild` (v0.32.3) would lose the data because the fence
|
||||
# wasn't updated.
|
||||
#
|
||||
# This script grep-bans the direct-write surface across src/ and
|
||||
# scripts/ (NOT test/ — tests legitimately seed fixtures via direct
|
||||
# inserts, per Codex R2-#8). A function-scoped allow-list lets the
|
||||
# legitimate extract / reconcile / migration call sites pass: add
|
||||
# `// gbrain-allow-direct-insert: <reason>` on the SAME LINE as the
|
||||
# banned call. The grep parses the trailing comment.
|
||||
#
|
||||
# Usage: scripts/check-system-of-record.sh
|
||||
# Exit: 0 when no violations, 1 when violations found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Banned direct-call patterns. Each is a method on BrainEngine that
|
||||
# writes to a derived table. Pre-v0.32.2 callers used these freely;
|
||||
# post-v0.32.2 every call site must either route through the
|
||||
# reconcile layer OR carry an explicit allow-direct-insert comment.
|
||||
PATTERNS=(
|
||||
'engine\.insertFact\('
|
||||
'engine\.insertFacts\('
|
||||
'engine\.addLink\('
|
||||
'engine\.addLinksBatch\('
|
||||
'engine\.addTimelineEntry\('
|
||||
'engine\.upsertTake\('
|
||||
'engine\.expireFact\('
|
||||
)
|
||||
|
||||
# Build an OR-regex for one grep pass.
|
||||
COMBINED=""
|
||||
for p in "${PATTERNS[@]}"; do
|
||||
if [ -z "$COMBINED" ]; then
|
||||
COMBINED="$p"
|
||||
else
|
||||
COMBINED="$COMBINED|$p"
|
||||
fi
|
||||
done
|
||||
|
||||
# Scan src/ and scripts/ only. test/ is deliberately excluded per Codex
|
||||
# R2-#8: tests legitimately call these methods to seed fixtures, and
|
||||
# gating tests would break the test surface without protecting any
|
||||
# invariant.
|
||||
SCOPE_DIRS=("src" "scripts")
|
||||
|
||||
# Collect violations. A violation is a line that:
|
||||
# 1. Matches one of the banned patterns
|
||||
# 2. Does NOT contain the `gbrain-allow-direct-insert:` comment
|
||||
# 3. Is NOT a pure-comment line (JSDoc, line-comment, backtick mention)
|
||||
# Comment-line exclusions stop the grep from false-positiving on
|
||||
# docstrings/comments that mention the method names. The runtime
|
||||
# regression coverage lives in the unit + E2E tests.
|
||||
violations=$(
|
||||
for dir in "${SCOPE_DIRS[@]}"; do
|
||||
[ -d "$dir" ] || continue
|
||||
grep -rEn --include='*.ts' --include='*.tsx' --include='*.js' --include='*.sh' \
|
||||
"$COMBINED" "$dir" 2>/dev/null || true
|
||||
done \
|
||||
| grep -vE 'gbrain-allow-direct-insert:' \
|
||||
| grep -vE ':[[:space:]]*\*[[:space:]]+' \
|
||||
| grep -vE ':[[:space:]]*//' \
|
||||
| grep -vE '`[^`]*\\.\w+\(' \
|
||||
|| true
|
||||
)
|
||||
|
||||
if [ -n "$violations" ]; then
|
||||
echo
|
||||
echo "ERROR: direct writes to derived tables found outside the reconcile layer."
|
||||
echo " Every call to engine.insertFact / insertFacts / addLink /"
|
||||
echo " addLinksBatch / addTimelineEntry / upsertTake / expireFact must"
|
||||
echo " either route through the extract / cycle / migration path OR"
|
||||
echo " carry an explicit \`// gbrain-allow-direct-insert: <reason>\`"
|
||||
echo " comment on the SAME LINE. See docs/architecture/system-of-record.md."
|
||||
echo
|
||||
echo "Violations:"
|
||||
echo "$violations"
|
||||
echo
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: no direct derived-table writes outside the reconcile layer in src/ + scripts/"
|
||||
@@ -43,7 +43,6 @@ test/init-migrate-only.test.ts
|
||||
test/integrations.test.ts
|
||||
test/mcp-eval-capture.test.ts
|
||||
test/migrate.test.ts
|
||||
test/migration-orchestrator-v0_31_0.test.ts
|
||||
test/migration-resume.test.ts
|
||||
test/migrations-v0_11_0.test.ts
|
||||
test/migrations-v0_13_1.test.ts
|
||||
|
||||
@@ -1,157 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any test fixture references a real person's name.
|
||||
#
|
||||
# CLAUDE.md's "Privacy rule" section is unambiguous: never reference real
|
||||
# people, companies, funds, or private agent names in any public-facing
|
||||
# artifact. Tests are checked-in code distributed with every release and
|
||||
# indexed by GitHub search. This guard catches the patterns the rule names.
|
||||
#
|
||||
# Design (post-Codex F4 review):
|
||||
# - Banned names: exact-string allowlist of known real identifiers. Adding
|
||||
# a name when CLAUDE.md flags one is a one-line edit.
|
||||
# - Banned emails: specific addresses that identify real contacts. NOT a
|
||||
# broad corporate-email regex — those would catch legitimate fixture
|
||||
# domains in billing/auth tests (`customer@stripe.com` etc.).
|
||||
# - Allowlist: exact "file:offending-string" pairs that are intentional
|
||||
# and pre-existing (e.g., the user's own email is not a "contact").
|
||||
#
|
||||
# Scope: test/**/*.test.ts only. Historical CHANGELOG entries, doc examples,
|
||||
# and skill READMEs each have their own scrub status and are out of scope
|
||||
# for this guard.
|
||||
#
|
||||
# Usage: scripts/check-test-real-names.sh
|
||||
# Exit: 0 clean, 1 banned reference found, 2 setup error (rg + grep missing).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Banned real-name strings (matched as whole words, case-insensitive).
|
||||
# Add an entry when CLAUDE.md flags a new real-person name.
|
||||
BANNED_NAMES=(
|
||||
'Diana' # Diana Hu, named in CLAUDE.md privacy example
|
||||
'Wintermute' # private OpenClaw fork name (CLAUDE.md rule)
|
||||
'Hermes' # downstream agent fork name
|
||||
'Technium' # real GP handle
|
||||
'McGrew' # ex-OpenAI exec
|
||||
'YC Labs' # internal team name
|
||||
)
|
||||
|
||||
# Banned specific email addresses. NOT a generic corporate-email regex —
|
||||
# those would catch legitimate fixture domains in billing/auth tests
|
||||
# (`customer@stripe.com`, `account@openai.com` etc).
|
||||
BANNED_EMAILS=(
|
||||
'diana@ycombinator.com'
|
||||
)
|
||||
|
||||
# Exact "file:offending-string" pairs that are intentional and pre-existing.
|
||||
# These pre-date the rule, the file's own author confirmed the use, the
|
||||
# string identifies the user themselves (not a contact), OR the reference
|
||||
# is structural (e.g., a regression test that ASSERTS the banned name does
|
||||
# NOT appear in production code — the name MUST be in the test file as a
|
||||
# literal).
|
||||
ALLOWLIST=(
|
||||
"test/writer.test.ts:garry@ycombinator.com" # user's own email — CLAUDE.md rule does not apply
|
||||
"test/integrations.test.ts:Wintermute" # regex pattern in personal-info filter test (structural)
|
||||
"test/recency-decay.test.ts:Wintermute" # regression-prevention test asserting wintermute is absent (structural)
|
||||
"test/scripts/check-proposal-pii.test.ts:Wintermute" # privacy-guard test asserting docs/proposals/ rejects wintermute (structural; same meta-rule exception as check-privacy.sh)
|
||||
"test/scripts/check-proposal-pii.test.ts:WINTERMUTE" # case-insensitive sentinel literal for the same privacy-guard test
|
||||
"test/serve-stdio-lifecycle.test.ts:Hermes" # comment naming a downstream-agent scenario — pre-existing, low signal
|
||||
"test/extract.test.ts:Hermes" # markdown-link extraction test fixture — pre-existing, ambiguous (Greek god vs fork)
|
||||
"test/readme-hero-anchors.test.ts:Hermes" # v0.36.0.0 D9 anchor test — asserts README mentions Hermes as a credit
|
||||
"test/readme-hero-anchors.test.ts:OpenClaw" # v0.36.0.0 D9 anchor test — asserts README mentions OpenClaw as a credit
|
||||
# v0.36.0.0: skillpack-harvest privacy linter tests structurally
|
||||
# require the literal "Wintermute" to verify the linter catches it.
|
||||
# Same meta-rule exception as integrations.test.ts and the proposal-pii
|
||||
# privacy guard test above.
|
||||
"test/skillpack-harvest.test.ts:Wintermute"
|
||||
"test/skillpack-harvest-lint.test.ts:Wintermute"
|
||||
"test/e2e/skillpack-flow.test.ts:Wintermute"
|
||||
)
|
||||
|
||||
# Build the combined regex. Names matched as whole words (\b), emails matched
|
||||
# literally with dot escapes.
|
||||
PATTERN_PARTS=()
|
||||
for n in "${BANNED_NAMES[@]}"; do
|
||||
# Escape any regex metacharacters in the name (defensive — most are bare
|
||||
# words but YC Labs has a space).
|
||||
escaped="${n//./\\.}"
|
||||
escaped="${escaped// /\\s}"
|
||||
PATTERN_PARTS+=("\\b${escaped}\\b")
|
||||
done
|
||||
for e in "${BANNED_EMAILS[@]}"; do
|
||||
escaped="${e//./\\.}"
|
||||
PATTERN_PARTS+=("${escaped}")
|
||||
done
|
||||
|
||||
# Join with |.
|
||||
IFS='|' eval 'PATTERN="${PATTERN_PARTS[*]}"'
|
||||
|
||||
# Find tool.
|
||||
if command -v rg >/dev/null 2>&1; then
|
||||
matches="$(rg -niH --no-heading -t ts "$PATTERN" test/ 2>/dev/null || true)"
|
||||
elif command -v grep >/dev/null 2>&1; then
|
||||
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test/ 2>/dev/null || true)"
|
||||
else
|
||||
echo "check-test-real-names: ERROR: neither rg nor grep available." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [ -z "$matches" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Apply allowlist. Each line is "file:lineno:content"; check whether
|
||||
# "file:<needle>" appears in ALLOWLIST for any needle in BANNED_EMAILS+NAMES
|
||||
# that matches the content.
|
||||
filtered=""
|
||||
while IFS= read -r line; do
|
||||
[ -z "$line" ] && continue
|
||||
# Extract filename and content (everything after second :).
|
||||
file="${line%%:*}"
|
||||
rest="${line#*:}"
|
||||
# rest is "lineno:content" — strip lineno.
|
||||
content="${rest#*:}"
|
||||
|
||||
matched_needle=""
|
||||
for needle in "${BANNED_EMAILS[@]}" "${BANNED_NAMES[@]}"; do
|
||||
if echo "$content" | grep -qi -- "$needle"; then
|
||||
matched_needle="$needle"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
allow_key="${file}:${matched_needle}"
|
||||
allowed=0
|
||||
for allow_entry in "${ALLOWLIST[@]}"; do
|
||||
if [ "$allow_entry" = "$allow_key" ]; then
|
||||
allowed=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$allowed" = "0" ]; then
|
||||
filtered+="${line}"$'\n'
|
||||
fi
|
||||
done <<< "$matches"
|
||||
|
||||
if [ -z "$filtered" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "check-test-real-names: banned real-name references found in test/ fixtures." >&2
|
||||
echo "" >&2
|
||||
echo "$filtered" >&2
|
||||
echo "" >&2
|
||||
echo "Fix: replace with canonical placeholders per CLAUDE.md 'Name mapping' table." >&2
|
||||
echo " alice-example / @alice-example for people" >&2
|
||||
echo " bob-example / charlie-example for additional people" >&2
|
||||
echo " alice@example.com for emails (example.com is RFC 6761 reserved)" >&2
|
||||
echo " acme-example / widget-co for companies" >&2
|
||||
echo " fund-a / fund-b for funds" >&2
|
||||
echo " a-team / agent-fork for teams / OpenClaw forks" >&2
|
||||
echo "" >&2
|
||||
echo "If the match is intentional (e.g., the user's own identifier, not a contact)," >&2
|
||||
echo "add an exact 'file:string' entry to ALLOWLIST in scripts/check-test-real-names.sh." >&2
|
||||
exit 1
|
||||
@@ -23,29 +23,10 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
|
||||
],
|
||||
// Tree-sitter chunkers feed code-indexing E2E.
|
||||
"src/core/chunkers/**": ["test/e2e/code-indexing.test.ts"],
|
||||
// OpenClaw context-engine plugin: engine + entry feed the plugin-shape E2E
|
||||
// (mocked SDK) AND the real-loader Tier 2 E2E that spawns openclaw and
|
||||
// actually installs the plugin into an isolated --profile.
|
||||
"src/core/context-engine.ts": [
|
||||
"test/e2e/openclaw-context-engine-plugin.test.ts",
|
||||
"test/e2e/openclaw-plugin-load-real.test.ts",
|
||||
],
|
||||
"src/openclaw-context-engine.ts": [
|
||||
"test/e2e/openclaw-context-engine-plugin.test.ts",
|
||||
"test/e2e/openclaw-plugin-load-real.test.ts",
|
||||
],
|
||||
// dream.ts is a thin alias over runCycle in cycle.ts.
|
||||
"src/core/cycle.ts": ["test/e2e/cycle.test.ts", "test/e2e/dream.test.ts"],
|
||||
// Multi-source sync writes share the per-source bookmark anchor.
|
||||
"src/core/sync.ts": ["test/e2e/sync.test.ts", "test/e2e/multi-source.test.ts"],
|
||||
// v0.32.8 multi-source bug class regression suite — fires on any cycle
|
||||
// phase, extract, integrity, embed, or migrate-engine change.
|
||||
"src/core/cycle/extract-takes.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/core/cycle/patterns.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/core/cycle/synthesize.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/commands/embed.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/commands/extract.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/commands/migrate-engine.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
// Any minions queue/worker/handler change exercises all minion E2E.
|
||||
"src/core/minions/**": [
|
||||
"test/e2e/minions-concurrency.test.ts",
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* v0.32.3 — auto-generate docs/eval/METRIC_GLOSSARY.md from
|
||||
* src/core/eval/metric-glossary.ts.
|
||||
*
|
||||
* Run: bun run scripts/generate-metric-glossary.ts
|
||||
*
|
||||
* CI guard `scripts/check-eval-glossary-fresh.sh` regenerates and diffs
|
||||
* against the committed version — out-of-date doc fails the build.
|
||||
*/
|
||||
|
||||
import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { dirname, join, resolve } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { renderMetricGlossaryMarkdown } from '../src/core/eval/metric-glossary.ts';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = resolve(__dirname, '..');
|
||||
const OUT_PATH = join(REPO_ROOT, 'docs', 'eval', 'METRIC_GLOSSARY.md');
|
||||
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
|
||||
mkdirSync(dirname(OUT_PATH), { recursive: true });
|
||||
writeFileSync(OUT_PATH, md, 'utf-8');
|
||||
|
||||
console.log(`Wrote ${OUT_PATH} (${md.length} bytes, ${md.split('\n').length} lines).`);
|
||||
@@ -1,80 +0,0 @@
|
||||
// Compiled-binary smoke test for HEIC/AVIF decoders.
|
||||
//
|
||||
// Verifies that bun --compile produces a binary where heic-decode and
|
||||
// @jsquash/avif both load their WASM and successfully decode a fixture
|
||||
// to a non-empty pixel buffer.
|
||||
//
|
||||
// Output: a single JSON line on stdout.
|
||||
// {"heic":{"ok":true,"width":N,"height":N,"bytes":N},"avif":{"ok":true,...}}
|
||||
//
|
||||
// Exit code 0 on full success, 1 on any decode failure.
|
||||
//
|
||||
// Used by scripts/check-image-decoders-embedded.sh as a CI guard.
|
||||
//
|
||||
// The fixture paths are resolved at compile time via import attributes so
|
||||
// bun --compile embeds the bytes into the binary itself. Otherwise a compiled
|
||||
// binary running away from the repo would fail to find the fixtures.
|
||||
|
||||
import heicFixture from '../test/fixtures/images/tiny.heic' with { type: 'file' };
|
||||
import avifFixture from '../test/fixtures/images/tiny.avif' with { type: 'file' };
|
||||
// @jsquash/avif loads its WASM relative to its own JS file, which fails inside
|
||||
// a bun --compile VFS. Pre-compile the module via `init()` with the embedded
|
||||
// bytes — `with { type: 'file' }` works correctly inside compiled binaries.
|
||||
import avifWasmPath from '@jsquash/avif/codec/dec/avif_dec.wasm' with { type: 'file' };
|
||||
import { readFileSync } from 'node:fs';
|
||||
|
||||
import heicDecode from 'heic-decode';
|
||||
import avifDecode, { init as initAvif } from '@jsquash/avif/decode.js';
|
||||
|
||||
interface DecodeResult {
|
||||
ok: boolean;
|
||||
width?: number;
|
||||
height?: number;
|
||||
bytes?: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
async function decodeHeic(): Promise<DecodeResult> {
|
||||
try {
|
||||
const buf = readFileSync(heicFixture);
|
||||
const result = await heicDecode({ buffer: buf });
|
||||
if (!result || !result.data || result.data.byteLength === 0) {
|
||||
return { ok: false, error: 'heic-decode returned empty pixel buffer' };
|
||||
}
|
||||
return {
|
||||
ok: true,
|
||||
width: result.width,
|
||||
height: result.height,
|
||||
bytes: result.data.byteLength,
|
||||
};
|
||||
} catch (err) {
|
||||
return { ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
async function decodeAvif(): Promise<DecodeResult> {
|
||||
try {
|
||||
const wasmBytes = readFileSync(avifWasmPath);
|
||||
const wasmModule = await WebAssembly.compile(wasmBytes);
|
||||
await initAvif(wasmModule);
|
||||
const buf = readFileSync(avifFixture);
|
||||
const result = await avifDecode(buf);
|
||||
if (!result || !result.data || result.data.byteLength === 0) {
|
||||
return { ok: false, error: 'avif decode returned empty pixel buffer' };
|
||||
}
|
||||
return {
|
||||
ok: true,
|
||||
width: result.width,
|
||||
height: result.height,
|
||||
bytes: result.data.byteLength,
|
||||
};
|
||||
} catch (err) {
|
||||
return { ok: false, error: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
const heic = await decodeHeic();
|
||||
const avif = await decodeAvif();
|
||||
const allOk = heic.ok && avif.ok;
|
||||
console.log(JSON.stringify({ heic, avif, ok: allOk }));
|
||||
process.exit(allOk ? 0 : 1);
|
||||
@@ -30,29 +30,5 @@ if [ "${1:-}" = "--dry-run-list" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "[serial-tests] running ${#files[@]} file(s), one bun process per file"
|
||||
|
||||
# Each serial file gets its OWN bun process. `--max-concurrency=1` was not
|
||||
# enough: files in the same process share the module registry, so a top-level
|
||||
# `mock.module(...)` in one file leaks into the next file's imports
|
||||
# (eval-takes-quality-runner mocks gateway.ts and the next file fails on
|
||||
# `import { resetGateway }` because the mock factory didn't export it).
|
||||
# Per-file processes give true isolation; cost is ~100ms startup × N files.
|
||||
fail_count=0
|
||||
failed_files=()
|
||||
for f in "${files[@]}"; do
|
||||
if ! bun test --max-concurrency=1 --timeout=60000 "$f"; then
|
||||
fail_count=$((fail_count + 1))
|
||||
failed_files+=("$f")
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$fail_count" -gt 0 ]; then
|
||||
echo "" >&2
|
||||
echo "[serial-tests] $fail_count file(s) failed:" >&2
|
||||
for f in "${failed_files[@]}"; do
|
||||
echo " - $f" >&2
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
echo "[serial-tests] all ${#files[@]} file(s) passed"
|
||||
echo "[serial-tests] running ${#files[@]} file(s) with --max-concurrency=1"
|
||||
exec bun test --max-concurrency=1 --timeout=60000 "${files[@]}"
|
||||
|
||||
+6
-34
@@ -5,29 +5,16 @@
|
||||
# shard-index: 1-based (1..N)
|
||||
# total-shards: positive integer
|
||||
#
|
||||
# Excluded from sharding:
|
||||
# - test/e2e/* — need DATABASE_URL; run via bun run test:e2e
|
||||
# - *.serial.test.ts — concurrency-unsafe (file-wide mock.module / env
|
||||
# leaks); run via scripts/run-serial-tests.sh on
|
||||
# shard 1 only. Including these here lets their
|
||||
# mock.module() calls leak into the rest of the
|
||||
# shard's bun process and silently break unrelated
|
||||
# tests. See test/eval-takes-quality-runner.serial.test.ts
|
||||
# mocking gateway.ts → voyage-multimodal failures.
|
||||
# E2E tests under test/e2e/ are excluded — they need DATABASE_URL and run via
|
||||
# bun run test:e2e separately.
|
||||
#
|
||||
# Stable partitioning: a file's shard is `(hash(path) % N) + 1`. Same file
|
||||
# lands in the same shard on every run, regardless of how many other files
|
||||
# exist, so retries are reproducible. Hash is FNV-1a — pure shell, no jq.
|
||||
set -euo pipefail
|
||||
|
||||
DRY_RUN_LIST=0
|
||||
if [ "${1:-}" = "--dry-run-list" ]; then
|
||||
DRY_RUN_LIST=1
|
||||
shift
|
||||
fi
|
||||
|
||||
if [ "$#" -ne 2 ]; then
|
||||
echo "usage: scripts/test-shard.sh [--dry-run-list] <shard-index> <total-shards>" >&2
|
||||
echo "usage: scripts/test-shard.sh <shard-index> <total-shards>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -45,19 +32,12 @@ fi
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
# Find all unit test files, deterministic order. Excludes test/e2e/ and
|
||||
# *.serial.test.ts. Serial files share file-wide state (top-level
|
||||
# mock.module, module singletons) that leaks across files in the same
|
||||
# `bun test` shard process — see scripts/check-test-isolation.sh R2.
|
||||
# CI runs them via `bun run test:serial` (scripts/run-serial-tests.sh) at
|
||||
# --max-concurrency=1 in a separate step on shard 1. Local `bun run test`
|
||||
# already excludes them from the parallel pass and runs them after the
|
||||
# same way. Portable: avoid `mapfile` (bash 4+) so this runs on macOS
|
||||
# bash 3.2 too.
|
||||
# Find all unit test files, deterministic order. Excludes test/e2e/.
|
||||
# Portable: avoid `mapfile` (bash 4+) so this runs on macOS bash 3.2 too.
|
||||
FILES=()
|
||||
while IFS= read -r line; do
|
||||
FILES+=("$line")
|
||||
done < <(find test -name '*.test.ts' -not -name '*.serial.test.ts' -not -path 'test/e2e/*' | sort)
|
||||
done < <(find test -name '*.test.ts' -not -path 'test/e2e/*' | sort)
|
||||
|
||||
if [ "${#FILES[@]}" -eq 0 ]; then
|
||||
echo "no test files found under test/" >&2
|
||||
@@ -87,14 +67,6 @@ for f in "${FILES[@]}"; do
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$DRY_RUN_LIST" = "1" ]; then
|
||||
if [ "${#SHARD_FILES[@]}" -eq 0 ]; then
|
||||
exit 0
|
||||
fi
|
||||
printf '%s\n' "${SHARD_FILES[@]}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "shard $SHARD_INDEX/$TOTAL_SHARDS: ${#SHARD_FILES[@]}/${#FILES[@]} files"
|
||||
if [ "${#SHARD_FILES[@]}" -eq 0 ]; then
|
||||
echo "warning: shard $SHARD_INDEX has no files (rehash or reduce shard count)" >&2
|
||||
|
||||
+18
-18
@@ -22,15 +22,13 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
| "what search mode", "is my cache hot", "tune my retrieval", "compare search modes", "clear search overrides" | `gbrain search modes/stats/tune` directly. See `skills/conventions/search-modes.md` |
|
||||
| "eval results", "search benchmark", "haters-immune methodology", "regression check on retrieval" | `gbrain eval run-all` / `gbrain eval compare`. See `docs/eval/SEARCH_MODE_METHODOLOGY.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| User shares a link, article, tweet, or idea | `skills/idea-ingest/SKILL.md` |
|
||||
| "watch this video", "process this YouTube link", "ingest this PDF", "save this podcast", "process this book", "summarize this book", "PDF book", "ingest it into my brain", "what's in this screenshot", "check out this repo" | `skills/media-ingest/SKILL.md` |
|
||||
| Video, audio, PDF, book, YouTube, screenshot | `skills/media-ingest/SKILL.md` |
|
||||
| Meeting transcript received | `skills/meeting-ingestion/SKILL.md` |
|
||||
| Generic "ingest this" (auto-routes to above) | `skills/ingest/SKILL.md` |
|
||||
|
||||
@@ -57,22 +55,18 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Save or load reports | `skills/reports/SKILL.md` |
|
||||
| "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` |
|
||||
| "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` |
|
||||
| "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table" | `skills/functional-area-resolver/SKILL.md` |
|
||||
| "Is gbrain healthy?", morning health check, skillpack-check | `skills/skillpack-check/SKILL.md` |
|
||||
| "harvest this skill into gbrain", "publish this skill to gbrain", "lift this skill upstream", "share this skill with other gbrain clients", "promote my skill to gbrain" | `skills/skillpack-harvest/SKILL.md` |
|
||||
| Post-restart health + auto-fix, "did the container restart break anything", smoke test | `skills/smoke-test/SKILL.md` |
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
|
||||
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
|
||||
| "present options", "ask before proceeding", "choice gate", "user decision" | `skills/ask-user/SKILL.md` |
|
||||
|
||||
## Setup & migration
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| "Set up GBrain", first boot | `skills/setup/SKILL.md` |
|
||||
| "Now what?", "fill my brain", "cold start", "bootstrap", "import my data", "what should I import first" | `skills/cold-start/SKILL.md` |
|
||||
| "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` |
|
||||
| Brain health check, maintenance run | `skills/maintain/SKILL.md` |
|
||||
| "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) |
|
||||
@@ -99,7 +93,7 @@ When multiple skills could match:
|
||||
2. If the user mentions a URL, route by content type (link → idea-ingest, video → media-ingest)
|
||||
3. If the user mentions a person/company, check if enrich or query fits better
|
||||
4. Chaining is explicit in each skill's Phases section
|
||||
5. When in doubt, ask the user (see `skills/ask-user/SKILL.md` for the choice-gate pattern)
|
||||
5. When in doubt, ask the user
|
||||
|
||||
## Conventions (cross-cutting)
|
||||
|
||||
@@ -108,7 +102,6 @@ These apply to ALL brain-writing skills:
|
||||
- `skills/conventions/brain-first.md` — check brain before external APIs
|
||||
- `skills/conventions/brain-routing.md` — which brain (DB) and which source (repo) to target; cross-brain federation is latent-space only
|
||||
- `skills/conventions/subagent-routing.md` — when to use Minions vs inline work
|
||||
- `skills/ask-user/SKILL.md` — choice-gate pattern for human input at decision points
|
||||
- `skills/_brain-filing-rules.md` — where files go
|
||||
- `skills/_output-rules.md` — output quality standards
|
||||
|
||||
@@ -116,13 +109,20 @@ These apply to ALL brain-writing skills:
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| "personalized version of this book", "mirror this book", "two-column book analysis", "apply this book to my life", "how does this book apply to me" | `skills/book-mirror/SKILL.md` |
|
||||
| "enrich this article", "enrich brain pages", "batch enrich", "make brain pages useful" | `skills/article-enrichment/SKILL.md` |
|
||||
| "strategic reading", "read this through the lens of", "apply this to my problem", "what can I learn from this about", "extract a playbook from" | `skills/strategic-reading/SKILL.md` |
|
||||
| "concept synthesis", "synthesize my concepts", "find patterns across my notes", "build my intellectual map", "trace idea evolution" | `skills/concept-synthesis/SKILL.md` |
|
||||
| "perplexity research", "what's new about", "current state of", "web research", "what changed about" | `skills/perplexity-research/SKILL.md` |
|
||||
| "crawl my archive", "find gold in my archive", "archive crawler", "scan my dropbox for", "mine my old files for" | `skills/archive-crawler/SKILL.md` |
|
||||
| "verify this academic claim", "check this study", "academic verify", "validate citation", "is this study real" | `skills/academic-verify/SKILL.md` |
|
||||
| "make pdf from brain", "brain pdf", "convert brain page to pdf", "publish this page as pdf", "export brain page" | `skills/brain-pdf/SKILL.md` |
|
||||
| "voice note", "ingest this voice memo", "transcribe and file", "voice note ingest", "save this audio note" | `skills/voice-note-ingest/SKILL.md` |
|
||||
| "personalized version of this book" | `skills/book-mirror/SKILL.md` |
|
||||
|
||||
| "enrich this article" | `skills/article-enrichment/SKILL.md` |
|
||||
|
||||
| "strategic reading" | `skills/strategic-reading/SKILL.md` |
|
||||
|
||||
| "concept synthesis" | `skills/concept-synthesis/SKILL.md` |
|
||||
|
||||
| "perplexity research" | `skills/perplexity-research/SKILL.md` |
|
||||
|
||||
| "crawl my archive" | `skills/archive-crawler/SKILL.md` |
|
||||
|
||||
| "verify this academic claim" | `skills/academic-verify/SKILL.md` |
|
||||
|
||||
| "make pdf from brain" | `skills/brain-pdf/SKILL.md` |
|
||||
|
||||
| "voice note" | `skills/voice-note-ingest/SKILL.md` |
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
# Agent onboarding — what to do with the files in this directory
|
||||
|
||||
You (the agent) are running on a host that scaffolded gbrain skills here. This
|
||||
file is the operating contract. Read it on every cold start. It is short on
|
||||
purpose.
|
||||
|
||||
## What lives in this directory
|
||||
|
||||
```
|
||||
skills/
|
||||
_AGENT_README.md ← you are here
|
||||
_brain-filing-rules.md ← where to file brain pages (read on every write)
|
||||
_output-rules.md ← output quality standards (no LLM slop, exact phrasing)
|
||||
_friction-protocol.md ← log friction the user hits to ~/.gstack/friction/
|
||||
conventions/ ← cross-cutting rules every skill defers to
|
||||
<skill-name>/
|
||||
SKILL.md ← the skill's contract + workflow
|
||||
routing-eval.jsonl ← (optional) test fixtures for routing-eval
|
||||
script.ts ← (optional) deterministic code, if any
|
||||
```
|
||||
|
||||
Other files in the host repo's `src/`, `docs/`, `recipes/` etc. are owned by the
|
||||
host, not by gbrain. Don't treat them as gbrain artifacts.
|
||||
|
||||
## Routing — your first job
|
||||
|
||||
Discover skills at runtime by walking every `skills/<slug>/SKILL.md` here and
|
||||
parsing the YAML frontmatter. Each skill declares one or more `triggers:`
|
||||
strings; they are the user-facing phrases that route to that skill.
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: book-mirror
|
||||
triggers:
|
||||
- "personalized version of this book"
|
||||
- "mirror this book"
|
||||
- "two-column book analysis"
|
||||
---
|
||||
```
|
||||
|
||||
On every user message, match the message against every skill's `triggers:`
|
||||
array. Substring match is the baseline. Semantic similarity (embedding or
|
||||
keyword expansion) is fine on top. When a trigger matches strongly, invoke the
|
||||
skill — read its SKILL.md body in full and follow the workflow described there.
|
||||
|
||||
**Do NOT** look for a managed-block table inside `RESOLVER.md` or `AGENTS.md`.
|
||||
That pattern was retired in gbrain v0.36. Routing lives in frontmatter now.
|
||||
|
||||
## When the user invokes a skill
|
||||
|
||||
Read the entire `skills/<slug>/SKILL.md` file. Follow its `## Phases`,
|
||||
`## Workflow`, or equivalent step-by-step section. If the skill has a
|
||||
`mutating: true` frontmatter and declares `writes_pages:` / `writes_to:`,
|
||||
those are the brain-side write surfaces — consult `_brain-filing-rules.md`
|
||||
to confirm the file path is sanctioned.
|
||||
|
||||
If the SKILL.md frontmatter declares `sources:` (paired source files), those
|
||||
live at their mirror path in the host repo (e.g. `src/commands/<slug>.ts`).
|
||||
They are reference code that the gbrain CLI calls. You do not run them
|
||||
directly unless the SKILL.md tells you to.
|
||||
|
||||
## Updates — when gbrain ships a new version
|
||||
|
||||
The user runs `gbrain upgrade`. Skill files DO NOT change automatically.
|
||||
gbrain becomes a reference library you compare against.
|
||||
|
||||
On every cold start, or any time the user mentions an upgrade, run:
|
||||
|
||||
```bash
|
||||
gbrain skillpack reference --all
|
||||
```
|
||||
|
||||
That sweeps every bundled skill and reports per-skill `identical / differs /
|
||||
missing` counts. For each `differs`:
|
||||
|
||||
```bash
|
||||
gbrain skillpack reference <slug>
|
||||
```
|
||||
|
||||
This prints a unified diff between gbrain's bundle and the local file. Read
|
||||
it, then decide per file:
|
||||
|
||||
- **Local edit was intentional.** Keep your version. gbrain is reference, not
|
||||
law.
|
||||
- **Local edit was accidental drift** (e.g. you wrote stale content into the
|
||||
skill body). Either patch by hand, or run
|
||||
`gbrain skillpack reference <slug> --apply-clean-hunks` (read the WARNING
|
||||
about two-way merge below first).
|
||||
- **Genuinely new gbrain change in a section you don't care about.** Skip or
|
||||
apply per your judgment.
|
||||
|
||||
For `missing` files (gbrain added a new bundled skill since you scaffolded),
|
||||
run `gbrain skillpack scaffold <new-slug>` to bring it in.
|
||||
|
||||
### `reference --apply-clean-hunks` — two-way merge warning
|
||||
|
||||
This command does a two-way diff against gbrain's current bundle. It does
|
||||
NOT have access to the version you originally scaffolded. Consequence: if
|
||||
the user's local file differs from gbrain in ANY section (including
|
||||
intentional user edits), those sections WILL be aligned to gbrain.
|
||||
|
||||
Always run plain `gbrain skillpack reference <slug>` first to inspect.
|
||||
Use `--apply-clean-hunks` only when you're confident the local edits were
|
||||
accidental or you want to fully reset to gbrain's current bundle.
|
||||
|
||||
## Removing a scaffolded skill
|
||||
|
||||
There is no `uninstall` command in v0.36. The files are yours.
|
||||
|
||||
```bash
|
||||
rm -rf skills/<slug>
|
||||
# if the skill declared paired source files:
|
||||
rm src/commands/<slug>.ts
|
||||
```
|
||||
|
||||
Consult the skill's frontmatter `sources:` array for the full paired-file
|
||||
list before deleting.
|
||||
|
||||
## When in doubt
|
||||
|
||||
The single source of truth for the model is
|
||||
`docs/guides/skillpacks-as-scaffolding.md` in the gbrain repo. The skill
|
||||
files you scaffolded are the source of truth for individual skill behavior.
|
||||
This file (`_AGENT_README.md`) is the routing contract — keep it short.
|
||||
@@ -122,24 +122,6 @@
|
||||
"directory": "media/articles/",
|
||||
"examples": ["personalized article reads", "long-form content tailored to reader"],
|
||||
"description": "Same sanctioned exception as media/books/. One-of-one synthesis output of an article personalized for the reader. Distinct from raw article ingest, which goes to the article's primary-subject directory."
|
||||
},
|
||||
{
|
||||
"kind": "daily",
|
||||
"directory": "daily/",
|
||||
"examples": ["daily/calendar/YYYY-MM-DD.md", "daily/notes/YYYY-MM-DD.md"],
|
||||
"description": "Date-keyed pages for events, calendar entries, or daily notes. Calendar imports land at daily/calendar/YYYY-MM-DD.md with attendees cross-linked to people/. Use when the primary subject is the date itself, not a person or topic."
|
||||
},
|
||||
{
|
||||
"kind": "media-format",
|
||||
"directory": "media/",
|
||||
"examples": ["media/x/{handle}/", "media/audio/", "media/video/"],
|
||||
"description": "Format-prefixed parent for media-by-source-format ingest. Subdirectories like media/x/{handle}/ hold X/Twitter archives, media/audio/ holds podcast/voice captures. The format-prefix lives only when the content is sui generis to the source format AND lacks a clean primary-subject directory. Prefer subject-by-subject filing; fall through to media/ only when the source format IS the unifying frame."
|
||||
},
|
||||
{
|
||||
"kind": "conversation",
|
||||
"directory": "conversations/",
|
||||
"examples": ["conversations/chatgpt/{thread-slug}.md", "conversations/claude/{thread-slug}.md"],
|
||||
"description": "Imported chat exports (ChatGPT, Claude, etc.) where the conversation itself is the artifact. Cross-link concepts and people from the conversation; the conversation page is the source-of-truth for the dialog. Distinct from voice-notes/ (which holds raw voice capture)."
|
||||
}
|
||||
],
|
||||
"sources_dir": {
|
||||
|
||||
@@ -151,42 +151,3 @@ to add a new directory the synthesis subagent may write to:
|
||||
2. Cross-reference compulsively: every new page MUST link to existing brain content.
|
||||
3. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated. NO underscores, NO file extensions.
|
||||
4. Edited transcripts produce NEW slugs (content-hash suffix changes) — never silently overwrite a prior reflection.
|
||||
|
||||
## Takes attribution (v0.32+)
|
||||
|
||||
When writing a `<!--- gbrain:takes:begin -->` fence, the **holder** column says
|
||||
WHO BELIEVES the claim, not who it's ABOUT. Cross-modal eval over 100K
|
||||
production takes scored attribution at 6.5/10 — holder/subject confusion was
|
||||
the #1 error. These six rules are the contract. Long form with worked
|
||||
examples lives in `docs/takes-vs-facts.md`.
|
||||
|
||||
1. **Holder ≠ subject.** The test: did this person SAY or CLEARLY IMPLY this?
|
||||
- YES → `holder = people/<slug>`
|
||||
- NO, it's your analysis OF them → `holder = brain`
|
||||
- Example: "Garry has a hero/rescuer pattern" → `holder=brain` (analysis ABOUT Garry, not stated BY Garry)
|
||||
2. **Atomic claims.** Split compound rows into separate rows. One claim per row.
|
||||
3. **Amplification ≠ endorsement.** A retweet-only signal caps at `weight 0.55`.
|
||||
The user shared something; they didn't necessarily endorse every clause.
|
||||
4. **Self-reported ≠ verified.** "Saif reports 7 figures" → `holder=people/saif`,
|
||||
`weight=0.75`, NOT `holder=world/1.0`. Self-report is a strong individual
|
||||
signal, not consensus fact.
|
||||
5. **No false precision.** Use 0.05 increments only (`0.35`, `0.55`, `0.75`).
|
||||
`0.74` and `0.82` imply calibration accuracy that doesn't exist. The engine
|
||||
layer rounds on insert — match the grid in your fence and avoid the warning.
|
||||
6. **"So what" test.** Skip metadata-style trivia (Twitter handles, follower
|
||||
counts, obvious bio fields). A take has to be load-bearing for some future
|
||||
query.
|
||||
|
||||
**Holder format (enforced as a parser warning in v0.32, error in v0.33+):**
|
||||
- `world` (consensus fact, no individual claimant)
|
||||
- `brain` (AI-inferred, holder genuinely ambiguous)
|
||||
- `people/<slug>` (individual's stated belief)
|
||||
- `companies/<slug>` (institutional fact, no individual claimant)
|
||||
|
||||
Slugs use the standard grammar (`[a-z0-9._-]+`). `Garry`, `people/Garry-Tan`,
|
||||
and `world/garry-tan` all fail validation.
|
||||
|
||||
**Founder-describing-own-company rule.** When a founder describes their own
|
||||
company, the holder is the FOUNDER, not the company. "We can hit $10M ARR"
|
||||
said by Bo Lu → `holder=people/bo-lu`, NOT `holder=companies/clipboard-health`.
|
||||
Companies don't speak; their employees do.
|
||||
|
||||
@@ -8,7 +8,6 @@ triggers:
|
||||
- "academic verify"
|
||||
- "validate citation"
|
||||
- "is this study real"
|
||||
- "Retraction Watch"
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
|
||||
@@ -4,11 +4,8 @@ version: 0.1.0
|
||||
description: Transform raw article text dumps in the brain into structured pages with executive summary, verbatim quotes, key insights, why-it-matters, and cross-references. Replaces walls-of-text with quotable, actionable brain pages.
|
||||
triggers:
|
||||
- "enrich this article"
|
||||
- "enrich the article"
|
||||
- "enriching the article"
|
||||
- "enrich brain pages"
|
||||
- "batch enrich"
|
||||
- "enrich pass"
|
||||
- "make brain pages useful"
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
// Routing eval fixtures for skills/article-enrichment. Each intent
|
||||
// includes at least one trigger string as substring.
|
||||
// `enrich` parent skill naturally co-fires (skills chain by design,
|
||||
// per RESOLVER.md preamble); ambiguous_with acknowledges that.
|
||||
{"intent":"This article page is a wall of raw text — please enrich this article with quotes and insights","expected_skill":"article-enrichment","ambiguous_with":["enrich"]}
|
||||
{"intent":"Run a batch enrich pass on the unstructured articles in my brain","expected_skill":"article-enrichment","ambiguous_with":["enrich"]}
|
||||
{"intent":"Make brain pages useful by enriching the article dumps","expected_skill":"article-enrichment","ambiguous_with":["enrich"]}
|
||||
{"intent":"Please enrich brain pages that have raw content but no executive summary","expected_skill":"article-enrichment","ambiguous_with":["enrich"]}
|
||||
{"intent":"Enrich this article so it has verbatim quotes, key insights, and a why-it-matters section","expected_skill":"article-enrichment","ambiguous_with":["enrich"]}
|
||||
{"intent":"This article page is a wall of raw text — please enrich this article with quotes and insights","expected_skill":"article-enrichment"}
|
||||
{"intent":"Run a batch enrich pass on the unstructured articles in my brain","expected_skill":"article-enrichment"}
|
||||
{"intent":"Make brain pages useful by enriching the article dumps","expected_skill":"article-enrichment"}
|
||||
{"intent":"Please enrich brain pages that have raw content but no executive summary","expected_skill":"article-enrichment"}
|
||||
{"intent":"Enrich this article so it has verbatim quotes, key insights, and a why-it-matters section","expected_skill":"article-enrichment"}
|
||||
|
||||
@@ -1,253 +0,0 @@
|
||||
---
|
||||
name: ask-user
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Reusable pattern for presenting the user with explicit choices and gating
|
||||
execution until they respond. Used by other skills when a decision point
|
||||
requires human input before proceeding. Platform-agnostic — works on
|
||||
Telegram (inline buttons), Discord, CLI, or any agent with a message tool.
|
||||
triggers:
|
||||
- "present options"
|
||||
- "ask before proceeding"
|
||||
- "choice gate"
|
||||
- "user decision"
|
||||
priority: 50
|
||||
---
|
||||
|
||||
# Ask User — Choice Gate Pattern
|
||||
|
||||
## Contract
|
||||
|
||||
- Present 2-4 options (no more — decision paralysis kicks in past 4).
|
||||
- Always include an escape hatch (Skip, Cancel, or "none of these").
|
||||
- Stop the turn immediately after presenting choices. No follow-up tool calls,
|
||||
no preemptive action, no default-and-proceed.
|
||||
- The user's response triggers the next turn. Acknowledge briefly, then branch.
|
||||
- One question per message — never stack multiple choice gates.
|
||||
- Self-explanatory option labels: action verb plus brief qualifier, not "Option 1".
|
||||
|
||||
## What This Is
|
||||
|
||||
A **formalized pattern** for presenting users with 2-4 options and **stopping
|
||||
execution** until they respond. This is the canonical way to gate on user input
|
||||
in any GBrain-powered agent.
|
||||
|
||||
This is NOT a traditional async/await. In an LLM agent, "gating" means:
|
||||
1. Present the choices (buttons or numbered options)
|
||||
2. Explicitly stop the current turn (do not proceed)
|
||||
3. The user's response triggers the next turn
|
||||
4. Read the response and branch accordingly
|
||||
|
||||
## When To Use
|
||||
|
||||
- Ambiguous requests with multiple valid interpretations
|
||||
- Destructive operations (bulk deletes, overwrites)
|
||||
- Filing/routing decisions ("where should this go?")
|
||||
- Priority triage ("which should I do first?")
|
||||
- Cold-start phase gates ("ready for the next import source?")
|
||||
- Any fork where the wrong default wastes significant work
|
||||
|
||||
## When NOT To Use
|
||||
|
||||
- Clear, unambiguous instructions → just do it
|
||||
- Low-stakes decisions → pick the best option and mention it
|
||||
- Time-critical operations where delay costs more than a wrong choice
|
||||
- When the user has already expressed a preference
|
||||
|
||||
## How To Present Choices
|
||||
|
||||
### Platform-agnostic format (works everywhere)
|
||||
|
||||
Present choices as a clear question with numbered or labeled options:
|
||||
|
||||
```
|
||||
🔀 **How should I handle this?**
|
||||
|
||||
[context about the decision — 1-3 lines max]
|
||||
|
||||
1. **Option A** — short description
|
||||
2. **Option B** — short description
|
||||
3. **Option C** — short description
|
||||
4. **Skip** — do nothing for now
|
||||
```
|
||||
|
||||
### With inline buttons (Telegram, Discord, Slack)
|
||||
|
||||
If the platform supports interactive buttons, use them:
|
||||
|
||||
```json
|
||||
{
|
||||
"message": "🔀 **How should I handle this?**\n\n<context>",
|
||||
"buttons": [
|
||||
{ "label": "Option A — description", "value": "option_a" },
|
||||
{ "label": "Option B — description", "value": "option_b" },
|
||||
{ "label": "Skip", "value": "skip" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### With the `clarify` tool (OpenClaw agents)
|
||||
|
||||
Some OpenClaw agents have a built-in `clarify` tool that presents choices natively:
|
||||
|
||||
```
|
||||
clarify(
|
||||
question: "How should I handle this?",
|
||||
choices: [
|
||||
"Option A — description",
|
||||
"Option B — description",
|
||||
"Option C — description",
|
||||
"Skip for now"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
- **2-4 options max.** More than 4 creates decision paralysis.
|
||||
- **Labels must be self-explanatory.** The user shouldn't need to re-read context.
|
||||
- **Always include an escape hatch.** At minimum: "Skip" or "Cancel" as the last option.
|
||||
- **One question per message.** Never stack multiple choice gates.
|
||||
|
||||
## How To Gate (CRITICAL)
|
||||
|
||||
After presenting choices, **you MUST stop your turn.** Do not:
|
||||
- ❌ Continue with "while you decide, I'll start on..."
|
||||
- ❌ Pick a default and proceed
|
||||
- ❌ Send follow-up messages before the user responds
|
||||
- ❌ Make assumptions about which option they'll pick
|
||||
|
||||
Instead:
|
||||
- ✅ End your message with a brief note that you're waiting
|
||||
- ✅ Stop. Full stop. No more tool calls.
|
||||
|
||||
## How To Handle The Response
|
||||
|
||||
When the user responds:
|
||||
|
||||
1. **Read the response** — button click, number, or text
|
||||
2. **Acknowledge briefly** — "Got it, going with Option A."
|
||||
3. **Branch and execute** the chosen path
|
||||
4. If unclear, ask again
|
||||
|
||||
### Handling text responses
|
||||
|
||||
Users sometimes type instead of clicking. Handle gracefully:
|
||||
- "the first one" / "A" / "1" → map to first option
|
||||
- "merge" → fuzzy match against option labels/values
|
||||
- "actually, none of those" → present alternatives or ask what they want
|
||||
- Unrelated message → the user moved on; drop the gate
|
||||
|
||||
## Formatting Guidelines
|
||||
|
||||
### Question line emoji prefix
|
||||
|
||||
Signal the decision type:
|
||||
- 🔀 Routing/filing decisions
|
||||
- ⚠️ Destructive/risky operations
|
||||
- 🎯 Priority/triage decisions
|
||||
- 💡 Creative/strategic forks
|
||||
- 📋 Workflow/process choices
|
||||
- 🔐 Credential/security decisions
|
||||
|
||||
### Context block
|
||||
|
||||
1-3 lines maximum. The user should understand the decision in under 5 seconds.
|
||||
|
||||
### Button/option labels
|
||||
|
||||
Format: `Action verb — brief qualifier`
|
||||
- ✅ "Merge — combine with existing page"
|
||||
- ✅ "Create new — separate meeting page"
|
||||
- ❌ "Option 1"
|
||||
- ❌ "Click here to merge the content into the existing brain page"
|
||||
|
||||
## Examples
|
||||
|
||||
### Cold-start phase gate
|
||||
```
|
||||
📋 **Phase 2: Google Contacts**
|
||||
|
||||
I can import your Google Contacts to seed the people/ directory.
|
||||
This creates a brain page for each real contact (~200 pages).
|
||||
|
||||
1. **Import via ClawVisor** — secure credential gateway
|
||||
2. **Import via direct OAuth** — simpler, agent holds tokens
|
||||
3. **Import from Google Takeout export** — offline, from file
|
||||
4. **Skip** — move to the next phase
|
||||
```
|
||||
|
||||
### Filing decision
|
||||
```
|
||||
🔀 **Where should this go?**
|
||||
|
||||
Meeting notes from call with Jane Smith. She already has a page at
|
||||
people/jane-smith.md and there's a deal page at deals/acme-corp.md.
|
||||
|
||||
1. **Merge into Jane's page** — add to her timeline
|
||||
2. **Add to Acme deal page** — this was primarily a deal discussion
|
||||
3. **New meeting page** — standalone at meetings/2026-01-15-jane-acme.md
|
||||
4. **Skip** — don't file this
|
||||
```
|
||||
|
||||
### Destructive operation
|
||||
```
|
||||
⚠️ **About to delete 847 stale cache files (2.3 GB)**
|
||||
|
||||
These haven't been accessed in 90+ days. They can be re-fetched
|
||||
but that takes ~4 hours.
|
||||
|
||||
1. **Delete them** — free up space now
|
||||
2. **Archive first** — upload to cloud storage, then delete
|
||||
3. **Keep them** — no changes
|
||||
4. **Show me the list** — let me review before deciding
|
||||
```
|
||||
|
||||
## Integration With Other Skills
|
||||
|
||||
This pattern is used by:
|
||||
- **cold-start** — phase gates for each import source
|
||||
- **ingest** — routing decisions for ambiguous content
|
||||
- **enrich** — merge vs create decisions for entity pages
|
||||
- **brain-ops** — filing location decisions
|
||||
- **meeting-ingestion** — where to file meeting notes
|
||||
- **archive-crawler** — scan vs full ingestion gate
|
||||
|
||||
When building a new skill that needs user input at a decision point,
|
||||
reference this pattern rather than inventing a new one.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Continuing the turn after presenting choices.** "While you decide, I'll start on..."
|
||||
defeats the gate. Stop. Wait. The whole point is that the user controls what happens next.
|
||||
- **Picking a default and proceeding silently.** If the question matters enough to ask,
|
||||
it matters enough to wait. Silent defaults erode trust the next time you do ask.
|
||||
- **More than 4 options.** Decision paralysis is real. Group, summarize, or split into
|
||||
staged questions instead.
|
||||
- **No escape hatch.** Every choice gate must let the user decline. "None of these"
|
||||
/ "Skip" / "Cancel" is mandatory.
|
||||
- **Stacking multiple choice gates in one message.** The user can only answer one
|
||||
question per turn. Multi-question gates either get half-answered or dropped entirely.
|
||||
- **Cryptic option labels.** "Option 1" forces re-reading the context. "Merge into
|
||||
existing page" is self-explanatory.
|
||||
- **Asking about low-stakes decisions.** If the wrong answer costs nothing, just pick
|
||||
the best option and mention it. Reserve gates for forks where rework is expensive.
|
||||
|
||||
## Output Format
|
||||
|
||||
The skill's "output" is the choice-gate message itself, structured as:
|
||||
|
||||
```
|
||||
{emoji-prefix} **{question}**
|
||||
|
||||
{1-3 lines of context}
|
||||
|
||||
1. **{Option A label}** — {short qualifier}
|
||||
2. **{Option B label}** — {short qualifier}
|
||||
3. **{Skip / Cancel}** — {what skipping means}
|
||||
```
|
||||
|
||||
After emitting this, the skill stops the turn. No further tool calls, no
|
||||
preemptive action, no follow-up message until the user responds. The
|
||||
user's response triggers the next turn, where the calling skill branches
|
||||
on the chosen option.
|
||||
@@ -31,30 +31,6 @@ Compile a daily briefing from brain context.
|
||||
|
||||
## Phases
|
||||
|
||||
0. **Hot memory pulse (v0.32).** Before composing anything else, run:
|
||||
|
||||
```bash
|
||||
gbrain recall --since-last-run --supersessions --pending --rollup --json
|
||||
```
|
||||
|
||||
Fold the result into the briefing under a "Brain pulse" section at the top:
|
||||
1. **Contradictions resolved overnight** — the `--supersessions` output. Lead
|
||||
with these because they're new corrections to your model of the world.
|
||||
2. **Top mentions** — `top_entities` from `--rollup` (top 5 entity slugs by
|
||||
fact count in the window).
|
||||
3. **New facts since last briefing** — group the `facts` array under each
|
||||
entity from the rollup; include `kind`, `notability`, and `confidence`.
|
||||
4. **Pending consolidation footer** — when `pending_consolidation_count > 0`,
|
||||
note `N facts await dream-cycle consolidation` so the operator can decide
|
||||
whether to run `gbrain dream` before reading further.
|
||||
|
||||
The `--since-last-run` flag advances `~/.gbrain/recall-cursors/<source>.json`
|
||||
so the next briefing picks up exactly where this one left off. If you're
|
||||
running this as a cron job, pass `--source <slug>` or set `GBRAIN_SOURCE`
|
||||
explicitly — cron doesn't start in your repo-root cwd, so dotfile resolution
|
||||
may miss the right source. Thin-client installs (`gbrain init --mcp-only`)
|
||||
route through the remote brain transparently.
|
||||
|
||||
1. **Today's meetings.** For each meeting on the calendar:
|
||||
- Search gbrain for each participant by name
|
||||
- Read their pages from gbrain for compiled_truth context
|
||||
|
||||
@@ -1,506 +0,0 @@
|
||||
---
|
||||
name: cold-start
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Day-one data bootstrapping for a new brain. Sequences the highest-leverage
|
||||
data sources to go from empty brain to useful brain in one session. Uses
|
||||
ClawVisor for safe credential handling — the agent never holds raw API keys.
|
||||
Covers Gmail import, calendar sync, contacts seeding, X/Twitter archive,
|
||||
conversation imports, and file archives.
|
||||
Use when a user has just finished gbrain setup and asks "now what?"
|
||||
triggers:
|
||||
- "cold start"
|
||||
- "fill my brain"
|
||||
- "bootstrap brain"
|
||||
- "import my data"
|
||||
- "day one"
|
||||
- "get started"
|
||||
- "what should I import first"
|
||||
- "populate brain"
|
||||
- "now what?"
|
||||
tools:
|
||||
- search
|
||||
- query
|
||||
- get_page
|
||||
- put_page
|
||||
- add_link
|
||||
- add_timeline_entry
|
||||
- sync_brain
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- people/
|
||||
- companies/
|
||||
- meetings/
|
||||
- daily/
|
||||
- media/
|
||||
- conversations/
|
||||
- sources/
|
||||
---
|
||||
|
||||
# Cold Start — Day-One Brain Bootstrapping
|
||||
|
||||
You have a working brain. Search works. Now what?
|
||||
|
||||
An empty brain is a static database. A brain with your email history, calendar,
|
||||
contacts, conversations, and social media is a **live context membrane** that makes
|
||||
every future interaction smarter. This skill sequences the highest-leverage data
|
||||
sources to get you from zero to useful in one session.
|
||||
|
||||
## Contract
|
||||
|
||||
- Every import phase is gated on user consent (ask-user pattern) before proceeding.
|
||||
- **Google/social API access goes through ClawVisor.** The agent never holds raw OAuth
|
||||
tokens or API keys. This is a safety requirement, not a preference. ClawVisor vaults
|
||||
credentials, enforces task-scoped authorization, logs every API call, and requires
|
||||
human approval for destructive operations. If the user doesn't want ClawVisor, the
|
||||
only safe alternative is offline file exports (Google Takeout, Twitter archive download).
|
||||
- Each phase is independently valuable — the user can stop after any phase and still
|
||||
have a useful brain.
|
||||
- Progress is tracked in `~/.gbrain/cold-start-state.json` so interrupted sessions
|
||||
can resume.
|
||||
- Entity detection and cross-linking run on every import, not as a separate pass.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- GBrain installed and initialized (`gbrain doctor --json` all green)
|
||||
- Brain repo cloned and synced
|
||||
- Agent has terminal access and can run `gbrain` CLI commands
|
||||
|
||||
## The Priority Stack
|
||||
|
||||
Data sources ranked by **information density × ease of import**:
|
||||
|
||||
| Priority | Source | Why | Time | Pages Created |
|
||||
|----------|--------|-----|------|---------------|
|
||||
| 1 | Existing markdown/Obsidian | Highest density — it's already structured | 5 min | 100s-1000s |
|
||||
| 2 | Google Contacts | Seeds the people/ directory — names, emails, companies | 10 min | 50-500 |
|
||||
| 3 | Google Calendar (90 days) | Meeting history with attendee context | 15 min | 30-90 |
|
||||
| 4 | Gmail (recent threads) | Relationship context, active threads, org chart signals | 20 min | 50-200 |
|
||||
| 5 | Conversations (ChatGPT/Claude exports) | Your thinking, questions, mental models | 15 min | 10-100 |
|
||||
| 6 | X/Twitter archive | Your public positions, takes, engagement patterns | 20 min | 30-365 |
|
||||
| 7 | File archives (Dropbox/Drive/local) | Historical documents, old writing, photos | 30+ min | varies |
|
||||
| 8 | Meeting transcripts (Circleback/etc.) | Deep relationship context from recorded calls | 20 min | 10-50 |
|
||||
|
||||
## Phase 0: ClawVisor Setup (Required for API Access)
|
||||
|
||||
> **Safety boundary:** An AI agent with raw OAuth tokens to your Gmail, Calendar,
|
||||
> and Contacts is an uncontrolled attack surface. One prompt injection, one
|
||||
> malicious tool call, and your entire Google account is exposed. ClawVisor
|
||||
> eliminates this risk class entirely.
|
||||
|
||||
[ClawVisor](https://clawvisor.com) is a credential gateway that sits between the
|
||||
agent and your APIs. The agent never sees your credentials — ClawVisor injects
|
||||
them at request time, enforces policies, and logs everything.
|
||||
|
||||
**What ClawVisor gives you:**
|
||||
- **Credential vaulting** — agent sees shadow tokens, never real secrets
|
||||
- **Task-scoped authorization** — each workflow declares exactly what it needs
|
||||
- **Audit trail** — every API call logged with metadata (who, what, when)
|
||||
- **Human approval gates** — destructive operations (send email, modify calendar)
|
||||
require your explicit approval
|
||||
- **Multi-service** — Gmail, Calendar, Contacts, Drive, GitHub, iMessage from one gateway
|
||||
- **Revocation** — disable the agent's access in one click, no token rotation needed
|
||||
|
||||
**Setup (15 min):**
|
||||
1. Sign up at [app.clawvisor.com](https://app.clawvisor.com)
|
||||
2. Create an agent in the dashboard, copy the agent token
|
||||
3. Set environment variables:
|
||||
```bash
|
||||
gbrain config set clawvisor_url "https://app.clawvisor.com"
|
||||
gbrain config set clawvisor_agent_token "<token>"
|
||||
```
|
||||
4. Activate Google services (Gmail, Calendar, Contacts) in the dashboard
|
||||
5. Create a standing task with expansive scope:
|
||||
> "Full brain bootstrapping: read emails, calendar events, and contacts to
|
||||
> populate knowledge base. List, read, and search across all connected accounts."
|
||||
6. Save the standing task ID:
|
||||
```bash
|
||||
gbrain config set clawvisor_task_id "<task_id>"
|
||||
```
|
||||
|
||||
**Critical scoping rule:** Be expansive in task purposes. "Email triage" gets
|
||||
rejected by intent verification. "Full executive assistant email management
|
||||
including inbox triage, searching by any criteria, reading emails, tracking
|
||||
threads" works. The intent model uses the purpose to judge each request.
|
||||
|
||||
### If the user declines ClawVisor
|
||||
|
||||
Do NOT fall back to direct OAuth. Instead, skip Phases 2-4 (Contacts, Calendar,
|
||||
Gmail) and proceed with offline-only imports:
|
||||
|
||||
- **Phase 1** (markdown/Obsidian) — works without any API access
|
||||
- **Phase 5** (conversation exports) — works from downloaded JSON files
|
||||
- **Phase 6** (X/Twitter) — works from downloaded archive
|
||||
- **Phase 7** (file archives) — works from local files
|
||||
- **Phase 8** (meeting transcripts) — works from exported transcripts
|
||||
|
||||
Tell the user:
|
||||
> "No problem. We'll skip the Google imports for now and work with file-based
|
||||
> sources. You can set up ClawVisor anytime to unlock Contacts, Calendar, and
|
||||
> Gmail imports safely."
|
||||
|
||||
**Do NOT offer direct OAuth as an alternative.** An agent holding raw Google
|
||||
tokens is a security liability. The skill should not teach agents to store
|
||||
credentials they shouldn't have.
|
||||
|
||||
## Phase 1: Existing Markdown / Obsidian Import
|
||||
|
||||
**The highest-leverage first import.** If the user already has a notes system, this
|
||||
is hundreds or thousands of structured pages ready to go.
|
||||
|
||||
### Discovery
|
||||
|
||||
```bash
|
||||
echo "=== Markdown Repository Discovery ==="
|
||||
for dir in /data/* ~/git/* ~/Documents/* ~/notes/* ~/obsidian/* 2>/dev/null; do
|
||||
if [ -d "$dir" ]; then
|
||||
md_count=$(find "$dir" -name "*.md" -not -path "*/node_modules/*" \
|
||||
-not -path "*/.git/*" -not -path "*/.obsidian/*" 2>/dev/null | wc -l | tr -d ' ')
|
||||
if [ "$md_count" -gt 5 ]; then
|
||||
total_size=$(du -sh "$dir" 2>/dev/null | cut -f1)
|
||||
echo " $dir ($total_size, $md_count .md files)"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### Import
|
||||
|
||||
```bash
|
||||
# For Obsidian vaults, use the migrate skill for proper wikilink handling
|
||||
gbrain migrate --from obsidian --path /path/to/vault
|
||||
|
||||
# For plain markdown directories
|
||||
gbrain import /path/to/dir --no-embed --workers 4
|
||||
|
||||
# Verify
|
||||
gbrain stats
|
||||
gbrain search "<topic from the imported data>"
|
||||
```
|
||||
|
||||
### Post-import
|
||||
|
||||
- Run link extraction: `gbrain extract links --source db`
|
||||
- Run timeline extraction: `gbrain extract timeline --source db`
|
||||
- Start embeddings: `gbrain embed --stale` (runs in background)
|
||||
|
||||
> **Track progress:**
|
||||
> ```bash
|
||||
> echo '{"phase_1_complete": true, "pages_imported": N}' > ~/.gbrain/cold-start-state.json
|
||||
> ```
|
||||
|
||||
## Phase 2: Google Contacts → People Pages
|
||||
|
||||
**Seeds the people/ directory.** Every person in your contacts becomes a brain page
|
||||
with name, email, phone, company, and notes. This is the foundation that all other
|
||||
imports build on — when Gmail references "john@acme.com", the brain already knows
|
||||
who John is.
|
||||
|
||||
### Via ClawVisor
|
||||
|
||||
```javascript
|
||||
// Fetch all contacts
|
||||
const contacts = await clawvisor('google.contacts', 'list_contacts', {
|
||||
limit: 1000,
|
||||
fields: 'names,emailAddresses,phoneNumbers,organizations,biographies'
|
||||
});
|
||||
```
|
||||
|
||||
### Via direct Google People API
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $GOOGLE_TOKEN" \
|
||||
"https://people.googleapis.com/v1/people/me/connections?personFields=names,emailAddresses,phoneNumbers,organizations,biographies&pageSize=1000"
|
||||
```
|
||||
|
||||
### Processing rules
|
||||
|
||||
For each contact:
|
||||
1. **Filter out noise** — skip contacts with no name, no email, or that are clearly
|
||||
automated (noreply@, no-reply@, support@, notifications@)
|
||||
2. **Check brain first** — `gbrain search "name"` to avoid duplicates
|
||||
3. **Create people/ page** with:
|
||||
- Name, email(s), phone(s), company, title
|
||||
- Source attribution: `[Source: Google Contacts, YYYY-MM-DD]`
|
||||
- Any notes from the contact as initial context
|
||||
4. **Link to company** — if the contact has an organization, create/update the
|
||||
company page and link the person to it
|
||||
|
||||
### Quality gate
|
||||
|
||||
After importing 5 contacts, pause and show the user a sample page. Ask:
|
||||
> "Here's what a contact page looks like. Want me to continue with the rest, or
|
||||
> adjust the format first?"
|
||||
|
||||
## Phase 3: Google Calendar (Last 90 Days)
|
||||
|
||||
**Meeting history with attendee context.** Calendar events reveal who the user meets
|
||||
with, how often, and in what context. Combined with contacts, this builds a rich
|
||||
relationship map.
|
||||
|
||||
### Fetch events
|
||||
|
||||
```javascript
|
||||
// Via ClawVisor — query ALL calendar accounts
|
||||
const accounts = ['primary@gmail.com', 'work@company.com'];
|
||||
for (const account of accounts) {
|
||||
const events = await clawvisor(`google.calendar:${account}`, 'list_events', {
|
||||
timeMin: new Date(Date.now() - 90 * 86400000).toISOString(),
|
||||
timeMax: new Date().toISOString(),
|
||||
singleEvents: true,
|
||||
orderBy: 'startTime'
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Brain structure
|
||||
|
||||
Follow the three-tier calendar architecture:
|
||||
```
|
||||
brain/daily/calendar/
|
||||
├── calendar-log.md ← compiled truth (patterns, key people)
|
||||
├── YYYY/
|
||||
│ ├── YYYY-MM.md ← monthly summary
|
||||
│ └── YYYY-MM-DD.md ← daily event log
|
||||
```
|
||||
|
||||
### Entity enrichment
|
||||
|
||||
For each event with attendees:
|
||||
1. Look up each attendee in the brain (they should exist from Phase 2)
|
||||
2. Add a timeline entry to their page: met at [event title] on [date]
|
||||
3. If an attendee has no brain page and appears in 3+ events, create one
|
||||
4. Link attendees who appear in the same meeting
|
||||
|
||||
## Phase 4: Gmail (Recent Threads)
|
||||
|
||||
**Relationship context and active threads.** Email reveals organizational
|
||||
relationships, ongoing conversations, and communication patterns.
|
||||
|
||||
### Strategy: Smart sampling, not bulk import
|
||||
|
||||
Don't import every email. Import the **signal**:
|
||||
|
||||
1. **Sent mail (last 30 days)** — who the user actively communicates with
|
||||
2. **Starred/important emails** — user-curated signal
|
||||
3. **Threads with 3+ replies** — active conversations worth tracking
|
||||
4. **Emails from people already in the brain** — enrichment, not cold import
|
||||
|
||||
### Processing
|
||||
|
||||
For each email thread:
|
||||
1. **Entity detection** — extract people, companies mentioned
|
||||
2. **Update people pages** — add communication context to timeline
|
||||
3. **Create meeting pages** — if the email is a meeting summary or follow-up
|
||||
4. **Skip noise** — newsletters, automated notifications, marketing
|
||||
|
||||
### Filtering rules
|
||||
|
||||
**Auto-skip (never import):**
|
||||
- noreply@, no-reply@, notifications@, support@, mailer-daemon@
|
||||
- Unsubscribe-heavy senders (marketing)
|
||||
- GitHub/Jira/Linear notification emails
|
||||
- Calendar invites (already captured in Phase 3)
|
||||
|
||||
**Always import:**
|
||||
- Direct emails from people in the brain
|
||||
- Starred/flagged emails
|
||||
- Emails the user sent (their words are highest-value signal)
|
||||
|
||||
## Phase 5: Conversation Exports (ChatGPT / Claude / Perplexity)
|
||||
|
||||
**Your thinking, captured.** AI conversation exports reveal what the user
|
||||
was researching, building, and thinking about. This is original thinking
|
||||
preserved in dialog form.
|
||||
|
||||
### Supported formats
|
||||
|
||||
- **ChatGPT:** Settings → Data Controls → Export → `conversations.json`
|
||||
- **Claude:** Download from claude.ai conversation history
|
||||
- **Perplexity:** Export from settings
|
||||
|
||||
### Processing
|
||||
|
||||
For each conversation:
|
||||
1. **Assess significance** (1-5 scale):
|
||||
- 1 = Pure utility (how-tos, quick lookups) → skip or minimal page
|
||||
- 2 = Minor context → 1-paragraph note
|
||||
- 3 = Notable (reveals interests, building something) → full page
|
||||
- 4 = Important (deep personal processing, strategic thinking) → rich page
|
||||
- 5 = Defining (identity work, breakthrough insights) → full treatment
|
||||
2. **Extract entities** — people, companies, concepts discussed
|
||||
3. **Capture original thinking** — the user's exact phrasing is the signal.
|
||||
Never paraphrase.
|
||||
4. **File by primary subject** — not in a "conversations/" dump. A conversation
|
||||
about a person goes to people/, about a concept goes to concepts/, etc.
|
||||
|
||||
### Quality rule
|
||||
|
||||
Only import conversations rated 3+. The brain is for signal, not noise.
|
||||
|
||||
## Phase 6: X/Twitter Archive
|
||||
|
||||
**Your public positions and engagement patterns.** Twitter reveals what the user
|
||||
thinks, who they engage with, and what ideas they're developing publicly.
|
||||
|
||||
### Data sources
|
||||
|
||||
1. **Twitter data export** (Settings → Your Account → Download Archive)
|
||||
- Contains all tweets, likes, DMs, bookmarks
|
||||
2. **Live API** (if available) — recent tweets and engagement
|
||||
3. **Bookmarks** — curated signal, high value
|
||||
|
||||
### Brain structure
|
||||
|
||||
```
|
||||
brain/media/x/{handle}/
|
||||
├── x-log.md ← compiled truth (themes, voice, key threads)
|
||||
├── daily/YYYY-MM-DD.md ← daily tweet log
|
||||
├── monthly/YYYY-MM.md ← monthly rollup
|
||||
└── bookmarks/ ← saved/bookmarked content
|
||||
```
|
||||
|
||||
### Processing
|
||||
|
||||
- **Original tweets** → capture with full context, extract entities
|
||||
- **Quote tweets** → capture the user's commentary + the source tweet
|
||||
- **Threads** → reconstruct as a single narrative
|
||||
- **Bookmarks** → high-signal curation, import with tags
|
||||
- **Likes** — low signal, skip unless the user wants them
|
||||
|
||||
## Phase 7: File Archives
|
||||
|
||||
**Historical documents, old writing, photos with metadata.** This is the long tail —
|
||||
less structured but potentially very high value (old journals, letters, early writing).
|
||||
|
||||
Delegate to the `archive-crawler` skill. It handles:
|
||||
- Crawling directory structures
|
||||
- Filtering for high-value content (user's own writing, not installers)
|
||||
- Text extraction from PDFs, images (OCR), documents
|
||||
- Entity extraction and brain page creation
|
||||
|
||||
> **Safety gate:** Archive crawling can be slow and create many pages. Always start
|
||||
> with a scan-only pass:
|
||||
> ```bash
|
||||
> gbrain archive-crawler --scan-only --path /path/to/archive
|
||||
> ```
|
||||
> Show the user the manifest before proceeding with full ingestion.
|
||||
|
||||
**Supported sources:**
|
||||
- Local directories (Dropbox sync folder, Google Drive, old hard drives)
|
||||
- Cloud storage (Backblaze B2, S3) via mounted paths
|
||||
- Email archives (PST, mbox, EML, Google Takeout)
|
||||
- Data exports (LinkedIn, Facebook, etc.)
|
||||
|
||||
## Phase 8: Meeting Transcripts
|
||||
|
||||
**Deep relationship context from recorded calls.** If the user has a meeting
|
||||
recording service (Circleback, Otter, Fireflies, Read.ai), import recent
|
||||
transcripts.
|
||||
|
||||
Delegate to `meeting-ingestion` skill. Key rules:
|
||||
- Always pull the **complete transcript**, not just the AI summary
|
||||
- Entity propagation is MANDATORY — every attendee gets a timeline update
|
||||
- A meeting is NOT fully ingested until all entity pages are updated
|
||||
|
||||
## Post-Bootstrap Checklist
|
||||
|
||||
After completing available phases:
|
||||
|
||||
1. **Verify brain health:**
|
||||
```bash
|
||||
gbrain doctor --json
|
||||
gbrain stats
|
||||
```
|
||||
|
||||
2. **Test retrieval:**
|
||||
```bash
|
||||
gbrain query "who do I meet with most often?"
|
||||
gbrain query "what am I working on?"
|
||||
gbrain search "<person from contacts>"
|
||||
```
|
||||
|
||||
3. **Set up live sync** (if not already):
|
||||
- Calendar: daily cron
|
||||
- Email: periodic sweep (4-8 hours)
|
||||
- X: daily ingest
|
||||
- Brain repo: `gbrain sync --repo <path>` every 5-30 minutes
|
||||
|
||||
4. **Track state:**
|
||||
```json
|
||||
// ~/.gbrain/cold-start-state.json
|
||||
{
|
||||
"started": "2026-01-15T10:00:00Z",
|
||||
"credential_gateway": "clawvisor",
|
||||
"phases_completed": [1, 2, 3, 4],
|
||||
"phases_skipped": [6, 7],
|
||||
"total_pages_created": 847,
|
||||
"total_entities_linked": 1203,
|
||||
"next_phase": 5
|
||||
}
|
||||
```
|
||||
|
||||
5. **Tell the user what to do next:**
|
||||
> "Your brain has N pages across people, calendar, email, and conversations.
|
||||
> Live sync is configured for [sources]. From here:
|
||||
> - The **signal-detector** captures entities from every conversation
|
||||
> - The **briefing** skill can compile daily context
|
||||
> - The **executive-assistant** pattern handles email triage
|
||||
> - Say 'enrich [person]' to deep-dive any contact"
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Giving the agent raw OAuth tokens.** This is the #1 anti-pattern. An agent with
|
||||
raw Gmail/Calendar tokens is an uncontrolled attack surface — one prompt injection
|
||||
and your entire Google account is exposed. Use ClawVisor. If the user declines
|
||||
ClawVisor, skip to offline imports. Never offer direct OAuth as a fallback.
|
||||
- **Bulk importing everything without filtering.** The brain is for signal, not noise.
|
||||
Filter out automated senders, marketing emails, utility conversations.
|
||||
- **Importing without entity cross-linking.** Every import should detect entities and
|
||||
update existing brain pages. Isolated imports don't compound.
|
||||
- **Not gating on user consent.** Every phase should be presented as a choice. The user
|
||||
may not want their DMs or therapy conversations imported.
|
||||
- **Importing everything at significance 1.** Not every conversation is worth a brain
|
||||
page. Use the significance scale and skip utility content.
|
||||
- **Creating people pages for automated senders.** Sentry, GitHub notifications,
|
||||
newsletter platforms are not people. Filter by the rules in Phase 4.
|
||||
|
||||
## Resume Protocol
|
||||
|
||||
If the session is interrupted:
|
||||
|
||||
1. Read `~/.gbrain/cold-start-state.json`
|
||||
2. Skip completed phases
|
||||
3. Resume from `next_phase`
|
||||
4. The user doesn't have to repeat credential setup or re-import completed sources
|
||||
|
||||
## Output Format
|
||||
|
||||
After each phase:
|
||||
|
||||
```
|
||||
PHASE N COMPLETE: [source name]
|
||||
================================
|
||||
|
||||
Pages created: N
|
||||
Pages updated: N
|
||||
Entities linked: N
|
||||
Time elapsed: N min
|
||||
|
||||
Sample pages:
|
||||
- people/jane-smith.md (created — 3 emails, 5 meetings)
|
||||
- companies/acme-corp.md (updated — 2 new employees linked)
|
||||
|
||||
Next: Phase N+1 — [description]. Ready to proceed?
|
||||
```
|
||||
|
||||
## Tools Used
|
||||
|
||||
- `search` — check for existing pages before creating
|
||||
- `query` — hybrid search for entity deduplication
|
||||
- `get_page` — read existing pages for merge decisions
|
||||
- `put_page` — create and update brain pages
|
||||
- `add_link` — cross-reference entities
|
||||
- `add_timeline_entry` — record events on entity timelines
|
||||
- `sync_brain` — sync changes to the index after each phase
|
||||
@@ -8,7 +8,6 @@ triggers:
|
||||
- "find patterns across my notes"
|
||||
- "build my intellectual map"
|
||||
- "trace idea evolution"
|
||||
- "canon vs riff"
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
|
||||
@@ -1,92 +0,0 @@
|
||||
# Convention: calibration loop (v0.36.1.0)
|
||||
|
||||
The brain knows your track record and uses it. The calibration loop has
|
||||
five concrete touchpoints — agents working in this codebase should know
|
||||
which one applies to their current task.
|
||||
|
||||
## Touchpoints
|
||||
|
||||
| When you're working on... | Apply this |
|
||||
|---|---|
|
||||
| Adding a new advice surface where the brain tells the user something | Voice-gate the output via `gateVoice()` in `src/core/calibration/voice-gate.ts`. Pick a mode: `pattern_statement`, `nudge`, `forecast_blurb`, `dashboard_caption`, `morning_pulse`. Add a new mode only when none of the five fits — extend `VOICE_GATE_MODES` and `DEFAULT_RUBRICS`. |
|
||||
| Writing user-facing strings about the user's track record | Conversational, not academic. Friend, not doctor. Concrete numbers ("2 of 3 missed") over abstract metrics ("Brier 0.31"). See `DESIGN.md` voice section. Never use the phrase "according to your data." |
|
||||
| Adding a new cycle phase | Extend `BaseCyclePhase` in `src/core/cycle/base-phase.ts`. Inherits source-scope threading + budget metering + error envelope + progress reporter. Declare `budgetUsdKey` + `budgetUsdDefault`. |
|
||||
| Adding a new MCP op that reads source-scoped data | Route through `sourceScopeOpts(ctx)` from `src/core/operations.ts`. Type-enforced at the BaseCyclePhase level; manual MCP handlers should do this explicitly. |
|
||||
| Writing schema for any new calibration-related table | Stamp every row with `wave_version TEXT NOT NULL DEFAULT 'v0.36.1.0'` (or the current wave's version). The `--undo-wave` command reverses precisely by wave_version. |
|
||||
| Adding a new test fixture page under `test/fixtures/calibration/` | Synthetic only. Use the canonical placeholder names: `alice-example`, `acme-example`, `widget-co`, `fund-a/b/c`, `meetings/2026-04-03`. The CI guard `scripts/check-synthetic-corpus-privacy.sh` catches violations. |
|
||||
|
||||
## When to surface a calibration warning
|
||||
|
||||
The four doctor checks (in `src/commands/doctor.ts`):
|
||||
|
||||
- `abandoned_threads` — informational. Count of high-conviction takes
|
||||
(weight >= 0.7) older than 12 months that haven't been superseded or
|
||||
linked to a follow-up. Always status='ok' with a count.
|
||||
|
||||
- `calibration_freshness` — warns when the active profile is older than
|
||||
7 days. Hint: `gbrain calibration --regenerate`.
|
||||
|
||||
- `grade_confidence_drift` (CDX-11 mitigation) — placeholder for the
|
||||
v0.37+ confidence-vs-accuracy correlation math. v0.36.1.0 reports the
|
||||
count of auto-applied verdicts and the "drift math arrives in v0.37+"
|
||||
status. Don't add a noise threshold here until the math is in.
|
||||
|
||||
- `voice_gate_health` — warns when voice gate failure rate >= 30% over
|
||||
the last 7 days. Hint: review `src/core/calibration/voice-gate.ts`
|
||||
rubric.
|
||||
|
||||
## Auto-resolve posture
|
||||
|
||||
Auto-resolve is DISABLED by default (D17). Operator flips it on via
|
||||
`cycle.grade_takes.auto_resolve.enabled: true` once they trust the
|
||||
judge's verdicts. Thresholds:
|
||||
|
||||
- Single-model path: confidence >= 0.95
|
||||
- Ensemble path: 3/3 unanimous AND min confidence >= 0.85
|
||||
- 'unresolvable' verdict NEVER auto-applies even at confidence=1.0
|
||||
|
||||
These are MONOTONIC TIGHTENING ONLY. The config schema rejects attempts
|
||||
to LOWER an active threshold without an explicit `--allow-loosen-confidence`
|
||||
flag — because relaxing after data accumulates silently shifts which
|
||||
historical resolutions count as auto-applied.
|
||||
|
||||
## Cross-brain semantics (D18)
|
||||
|
||||
For any read of a calibration profile across mounted brains:
|
||||
|
||||
1. **Local first.** Query local. If local has it, return; do not query mounts.
|
||||
2. **Mount fallback.** Only if local is empty AND `canReadMountsForCtx(ctx)`
|
||||
returns true. Mount-side rows must have `published=true`.
|
||||
3. **Cross-brain attribution.** Returned profile carries
|
||||
`source_brain_id` + `from_mount`. UI consumers MUST surface
|
||||
"from mounted brain: X" so the user knows.
|
||||
4. **Subagent prohibition.** `ctx.viaSubagent && !allowedSlugPrefixes`
|
||||
cannot read mounts — subagent loops see only the local brain. Trusted-
|
||||
workspace cycle phases (synthesize/patterns) pass
|
||||
`allowedSlugPrefixes` set and ARE allowed.
|
||||
|
||||
## Test seams
|
||||
|
||||
Every calibration module accepts test injection via opts:
|
||||
- `opts.judge` / `opts.thinkRunner` / `opts.extractor` / `opts.evidenceRetriever`
|
||||
- `opts.voiceGateJudge` — bypass the Haiku call
|
||||
- `opts.preferenceResolver` — bypass the interactive prompt in A/B harness
|
||||
|
||||
Tests MUST use these seams. Never call gateway.chat directly from a
|
||||
calibration unit test — that's a test-isolation R2 violation (mocks the
|
||||
gateway module via `mock.module`, which leaks across files in the shard
|
||||
process).
|
||||
|
||||
## Bug class to avoid
|
||||
|
||||
The v0.34.1 source-isolation leak class is the canonical bug pattern
|
||||
the calibration wave has structural defense against:
|
||||
|
||||
- BaseCyclePhase enforces `sourceScopeOpts(ctx)` threading at the type level.
|
||||
- Every new schema table has `source_id NOT NULL REFERENCES sources(id)`.
|
||||
- Cross-brain reads route through `canReadMountsForCtx()` classifier.
|
||||
- Tests pin all 4 D18 rules in `test/cross-brain-calibration.test.ts`.
|
||||
|
||||
If you find yourself writing a `ctx.engine.executeRaw(...)` inside a
|
||||
calibration module that doesn't pass `sourceScopeOpts`, you've found
|
||||
the bug. Stop, route through the helper.
|
||||
@@ -1,73 +1,8 @@
|
||||
# Model Routing Convention
|
||||
|
||||
Two distinct concerns share this name. Read both — they apply at different
|
||||
moments.
|
||||
When spawning sub-agents, choose the right model for the task.
|
||||
|
||||
## 1. gbrain's internal tier system (v0.31.12+)
|
||||
|
||||
This is how gbrain itself picks which Claude/OpenAI/Google model runs each
|
||||
internal task (chat, expansion, synthesis, classification, etc.).
|
||||
|
||||
Four tiers:
|
||||
|
||||
| Tier | Purpose | Default | Examples |
|
||||
|---|---|---|---|
|
||||
| `utility` | fast classification, expansion, verdict, dedup | `claude-haiku-4-5-20251001` | query expansion, facts contradiction classifier, dream synthesize verdict |
|
||||
| `reasoning` | default chat, synthesis, generation | `claude-sonnet-4-6` | gateway chat, dream synthesize, patterns, facts extraction |
|
||||
| `deep` | slow, expensive reasoning | `claude-opus-4-7` | `gbrain think`, auto-think, cross-modal eval slot B |
|
||||
| `subagent` | Anthropic-only multi-turn tool loop | `claude-sonnet-4-6` | `gbrain agent run` |
|
||||
|
||||
Override priority (highest first):
|
||||
|
||||
1. CLI flag (`--model opus`)
|
||||
2. Per-task config (`gbrain config set models.dream.synthesize opus`)
|
||||
3. Deprecated per-task config (stderr-warns once, then honored)
|
||||
4. **Global default** (`gbrain config set models.default opus`) — single hammer
|
||||
5. **Tier override** (`gbrain config set models.tier.reasoning opus`)
|
||||
6. Env var (`GBRAIN_MODEL=opus`)
|
||||
7. Tier default (the table above)
|
||||
8. Hardcoded caller fallback
|
||||
|
||||
Power-user recipes:
|
||||
|
||||
```bash
|
||||
# Use opus for everything
|
||||
gbrain config set models.default opus
|
||||
|
||||
# Use opus only for reasoning + deep, keep haiku for utility
|
||||
gbrain config set models.tier.reasoning opus
|
||||
gbrain config set models.tier.deep opus
|
||||
|
||||
# Custom alias, then use it everywhere
|
||||
gbrain config set models.aliases.frontier anthropic:claude-opus-4-7
|
||||
gbrain config set models.default frontier
|
||||
```
|
||||
|
||||
Visibility:
|
||||
|
||||
```bash
|
||||
gbrain models # print current routing table
|
||||
gbrain models doctor # 1-token probe to each configured model
|
||||
```
|
||||
|
||||
**Subagent tier exists because the loop is Anthropic-only.** The handler
|
||||
uses Messages API + prompt caching on system + tools. Setting
|
||||
`models.default = openai:gpt-5.5` silently breaks the loop, so we isolate
|
||||
`tier.subagent`. Three enforcement layers: submit-time guard in
|
||||
`MinionQueue.add`, tier-resolution fallback in `resolveModel`, doctor
|
||||
`subagent_provider` check.
|
||||
|
||||
When adding a new LLM call, route through `resolveModel()` with a tier —
|
||||
never hardcode a model string. The v0.31.6 chat default
|
||||
(`claude-sonnet-4-6-20250929`) was a phantom ID that 404'd the Anthropic
|
||||
API and made `extractFactsFromTurn` silently return `[]`. The tier system
|
||||
+ doctor probe is the structural fix for that bug class.
|
||||
|
||||
## 2. Subagent spawn routing
|
||||
|
||||
When the user-facing agent (Claude in the main session) chooses which model
|
||||
to spawn a sub-agent on, this table applies. It's about WHERE to send the
|
||||
work, not what gbrain calls internally.
|
||||
## Routing Table
|
||||
|
||||
| Task | Model | Why |
|
||||
|------|-------|-----|
|
||||
@@ -77,7 +12,7 @@ work, not what gbrain calls internally.
|
||||
| Fast lightweight sub-agents | Groq | 500 tok/s, good for simple tasks |
|
||||
| Judge tests / quality grading | Haiku | Cheap, good enough for pass/fail |
|
||||
|
||||
### Refusal routing
|
||||
## Refusal Routing
|
||||
|
||||
When a model refuses a request:
|
||||
1. Re-spawn on a different model (don't announce the switch)
|
||||
@@ -85,7 +20,7 @@ When a model refuses a request:
|
||||
3. Never show the refusal to the user
|
||||
4. If all models refuse, escalate to the user
|
||||
|
||||
### Spawn rules
|
||||
## Spawn Rules
|
||||
|
||||
- 3+ items to process → spawn a sub-agent
|
||||
- >2 tool calls that don't need real-time judgment → spawn
|
||||
|
||||
@@ -1,131 +0,0 @@
|
||||
# Salience + Recency on `gbrain query` (v0.29.1)
|
||||
|
||||
YOU ARE IN CHARGE of the `salience` and `recency` parameters on gbrain's
|
||||
`query` op. They are TWO ORTHOGONAL axes — use either, both, or neither.
|
||||
|
||||
If you OMIT a parameter, gbrain auto-detects from query text via a
|
||||
regex heuristic. The default for queries that don't match any pattern
|
||||
is `'off'`. Prefer to pass values EXPLICITLY when you know what the
|
||||
user wants.
|
||||
|
||||
## What each axis means
|
||||
|
||||
- `salience` — **mattering**. Boosts pages with high `emotional_weight`
|
||||
and many active takes. NO time component. Use when the user wants
|
||||
the most important / most-discussed pages on a topic, regardless of
|
||||
when they were updated.
|
||||
|
||||
- `recency` — **age**. Boosts pages with recent `effective_date`. NO
|
||||
mattering signal. Per-prefix decay (`concepts/`, `originals/`,
|
||||
`writing/` are evergreen; `daily/`, `media/x/`, `chat/` decay
|
||||
aggressively). Use when freshness is the signal.
|
||||
|
||||
## When to pass `salience='on'`
|
||||
|
||||
The "mattering" axis. The user wants what matters in this brain on
|
||||
the topic, not the canonical encyclopedia entry.
|
||||
|
||||
- `"prep me for the widget-ceo meeting"` (meeting prep)
|
||||
- `"catch me up on acme"` (conversation recall)
|
||||
- `"what's going on with widget-co"` (current state matters)
|
||||
- `"remind me about the deal"` (recall takes / opinions)
|
||||
- `"what's been happening lately"`
|
||||
- `"status update on X"`
|
||||
|
||||
Pair with `recency='on'` when current-state matters. Just `salience='on'`
|
||||
alone gives you "what matters about X regardless of when."
|
||||
|
||||
## When to pass `recency='on'`
|
||||
|
||||
The "freshness" axis. The user wants recent content, with or without
|
||||
mattering.
|
||||
|
||||
- `"latest news on AI"` (recent, no mattering needed)
|
||||
- `"what's new this week"`
|
||||
- `"recent updates on widget-co"`
|
||||
- `"this week's announcements"`
|
||||
|
||||
Use `'strong'` when the user explicitly asks for the most recent:
|
||||
|
||||
- `"what happened today"`
|
||||
- `"right now what's going on"`
|
||||
- `"this morning"`
|
||||
|
||||
## When to pass BOTH `'off'`
|
||||
|
||||
The "canonical truth" axis. The user wants the authoritative answer.
|
||||
|
||||
- `"who is widget-ceo"` (entity lookup)
|
||||
- `"what is widget-co"` (definitional)
|
||||
- `"history of acme"` (historical research)
|
||||
- `"explain how recursion works"` (concept query)
|
||||
- `"tell me about widget-co"` (canonical recall)
|
||||
- Code lookups: function/class names, syntax like `Foo::bar()` or `obj.method`
|
||||
- Graph traversal: backlinks, inbound/outbound edges
|
||||
- Anything not matching above
|
||||
|
||||
## Heuristic when unsure
|
||||
|
||||
> Current state → on. Canonical truth → off.
|
||||
|
||||
If you can't classify confidently, OMIT the param and let gbrain's
|
||||
auto-detect handle it. The heuristic defaults to `off` for everything
|
||||
that doesn't clearly match a current-state pattern. The `--explain`
|
||||
output shows `_resolved.salience_source` and `_resolved.recency_source`
|
||||
('caller' vs. 'auto_heuristic') so you can see what fired and why.
|
||||
|
||||
You can override at any time. gbrain is smart but not infallible. You
|
||||
have context gbrain doesn't.
|
||||
|
||||
## Narrow temporal-bound exception
|
||||
|
||||
Even when a query matches canonical patterns, an explicit temporal
|
||||
bound (`today`, `this week`, `right now`, `since X`, `last N days`)
|
||||
overrides the canonical-wins rule:
|
||||
|
||||
- `"who is widget-ceo right now"` → recency = `'strong'`, salience = `'on'`
|
||||
(the temporal bound wins over "who is")
|
||||
- `"who is widget-ceo"` → recency = `'off'`, salience = `'off'` (no bound)
|
||||
|
||||
## English-only
|
||||
|
||||
The auto-detect heuristic is English-only in v0.29.1. Non-English
|
||||
queries fall through to the default `off` for both axes. Pass
|
||||
`salience` and `recency` explicitly for non-English queries.
|
||||
|
||||
## Tuning the recency formula
|
||||
|
||||
Defaults are in `src/core/search/recency-decay.ts`. Override per-brain
|
||||
via `gbrain.yml`:
|
||||
|
||||
```yaml
|
||||
recency:
|
||||
daily/:
|
||||
halflifeDays: 7
|
||||
coefficient: 2.0
|
||||
custom-prefix/:
|
||||
halflifeDays: 30
|
||||
coefficient: 0.5
|
||||
```
|
||||
|
||||
Or per-process via env: `GBRAIN_RECENCY_DECAY="prefix:halflife:coefficient,..."`.
|
||||
The parser fails LOUD on bad syntax (no silent fallback).
|
||||
|
||||
## Date filtering with `since` / `until`
|
||||
|
||||
Independent of the axes. Filter to pages whose `effective_date` is
|
||||
within a range:
|
||||
|
||||
- `since: '7d'` — last 7 days
|
||||
- `since: '2024-06-01'` — ISO-8601
|
||||
- `until: '2024-06-30'` — ends at end-of-day
|
||||
|
||||
`since`/`until` work with OR without `salience`/`recency`. Pure filter,
|
||||
no boost.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/recency.md` — full reference
|
||||
- `gbrain query --explain` — see resolved values + factor contributions
|
||||
- `get_recent_salience` op gains `recency_bias: 'flat' | 'on'` — opt
|
||||
into per-prefix decay on the dedicated salience query
|
||||
@@ -1,104 +0,0 @@
|
||||
---
|
||||
name: search-modes
|
||||
description: Three named search modes (conservative / balanced / tokenmax). Pick one at install; everything else inherits.
|
||||
type: convention
|
||||
---
|
||||
|
||||
# Convention: Search Modes (v0.32.3)
|
||||
|
||||
> **Convention:** every brain has one active search mode. The mode bundles the
|
||||
> search-lite knobs from PR #897 (semantic cache, token budget, intent
|
||||
> weighting, LLM expansion, result limit) into a single config key:
|
||||
> `search.mode = conservative | balanced | tokenmax`.
|
||||
|
||||
## When this fires
|
||||
|
||||
Any agent doing search-adjacent work in a gbrain brain consults this convention:
|
||||
|
||||
- `brain-ops` / `query` / `signal-detector` skills: respect the active mode at
|
||||
search time. Per-call `SearchOpts` overrides win when set; mode is the default.
|
||||
- Skills that recommend tuning ("the cache hit rate is high — raise threshold?"):
|
||||
route operators to `gbrain search tune` rather than rolling their own logic.
|
||||
- New skills that add per-call retrieval overrides: name them explicitly so
|
||||
the resolved-knob attribution dashboard (`gbrain search modes`) reads cleanly.
|
||||
|
||||
## Mode bundle (read-only constants)
|
||||
|
||||
The 3 bundles live in `src/core/search/mode.ts` as `MODE_BUNDLES` (frozen).
|
||||
Don't redefine them per-install; that breaks the public methodology numbers.
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
**Cache, intent weighting, and similarity threshold are constant across modes**
|
||||
— they're free wins (no API cost). Modes scale the three cost levers:
|
||||
`tokenBudget`, `expansion`, `searchLimit`.
|
||||
|
||||
## Resolution chain (matches v0.31.12 model-tier shape)
|
||||
|
||||
per-call SearchOpts.tokenBudget / expansion / etc.
|
||||
↓ (when undefined)
|
||||
per-key config: search.cache.enabled, search.tokenBudget, …
|
||||
↓ (when unset)
|
||||
MODE_BUNDLES[search.mode]
|
||||
↓ (when search.mode is unset)
|
||||
MODE_BUNDLES.balanced (safety fallback)
|
||||
|
||||
## Tools for agents
|
||||
|
||||
Agents tuning a brain's retrieval should call these directly:
|
||||
|
||||
gbrain search modes # dashboard + per-knob source attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode is canonical)
|
||||
gbrain search stats [--days N] # hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
|
||||
`gbrain search tune` reads the `search_telemetry` rollup (sums + counts of
|
||||
last 7 days) + brain size + configured `models.tier.subagent` to suggest
|
||||
mode + per-key changes. With `--apply`, it mutates config via `setConfig`
|
||||
and prints a paste-ready revert command.
|
||||
|
||||
## Cache contamination guard
|
||||
|
||||
Migration v56 added `query_cache.knobs_hash`. A tokenmax write
|
||||
(expansion=on, limit=50) is keyed by a different hash than a conservative
|
||||
read (no expansion, limit=10), so cross-mode contamination is structurally
|
||||
impossible. The cache lookup filter is:
|
||||
|
||||
WHERE source_id = $ AND knobs_hash = $ AND embedding similarity < $
|
||||
|
||||
Legacy NULL-knobs_hash rows from pre-v0.32.3 are silently excluded
|
||||
(treated as misses, re-populated with the right hash on first hit).
|
||||
|
||||
## Trigger phrases
|
||||
|
||||
If an operator or agent asks any of these, route to `gbrain search …`:
|
||||
|
||||
- "what search mode is active?" → `gbrain search modes`
|
||||
- "is my cache hot?" → `gbrain search stats`
|
||||
- "tune my retrieval" → `gbrain search tune`
|
||||
- "clear search overrides" → `gbrain search modes --reset`
|
||||
- "compare modes" → `gbrain eval compare`
|
||||
|
||||
## Don't
|
||||
|
||||
- Don't redefine `MODE_BUNDLES` per-install. The methodology numbers in
|
||||
`docs/eval/SEARCH_MODE_METHODOLOGY.md` cite these as canonical.
|
||||
- Don't mutate `search.mode` config from inside a subagent loop without
|
||||
operator approval. Mutation is a trust-boundary crossing
|
||||
(`tune --apply` stays CLI-only in v0.32.3 per `[CDX-21]`).
|
||||
- Don't add per-call `tokenBudget` overrides on the production `query` op
|
||||
without naming them in `gbrain search modes` output.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/eval/SEARCH_MODE_METHODOLOGY.md` — full eval methodology
|
||||
- `docs/eval/METRIC_GLOSSARY.md` — plain-English definitions
|
||||
- `src/core/search/mode.ts` — module source
|
||||
@@ -1,348 +0,0 @@
|
||||
---
|
||||
name: functional-area-resolver
|
||||
version: 1.0.0
|
||||
prompt_version: 1
|
||||
description: |
|
||||
Compress an agent's routing file (RESOLVER.md or AGENTS.md) by converting
|
||||
granular skill-per-row tables into functional-area dispatchers. Each area
|
||||
lists sub-skills in a "(dispatcher for: ...)" clause. The LLM reads one
|
||||
area entry and routes to the correct sub-skill. Proven via held-out
|
||||
A/B eval: dispatcher pattern outperforms naive pipe-table compression.
|
||||
triggers:
|
||||
- "compress agents.md"
|
||||
- "compress my resolver"
|
||||
- "resolver too big"
|
||||
- "resolver.md too big"
|
||||
- "agents.md too large"
|
||||
- "shrink routing table"
|
||||
- "slim down agents.md"
|
||||
- "functional area resolver"
|
||||
- "functional area dispatcher"
|
||||
- "context-health agents"
|
||||
- "context-health resolver"
|
||||
- "reduce context budget"
|
||||
tools:
|
||||
- exec
|
||||
- read
|
||||
- write
|
||||
- edit
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Functional-Area Resolver — Pattern for Compressing Routing Tables
|
||||
|
||||
## Problem
|
||||
|
||||
Routing files (RESOLVER.md, AGENTS.md) grow as skills are added. Each skill
|
||||
gets its own row (trigger -> skill path). At ~200+ skills this hits 25-30KB,
|
||||
eating context budget that should go to actual work.
|
||||
|
||||
## Solution: Functional-Area Dispatchers
|
||||
|
||||
Replace N rows per area with **one entry per functional area**. Each entry
|
||||
lists all sub-skills it can dispatch to in a `(dispatcher for: ...)` clause.
|
||||
|
||||
### Before (270 rows, 25KB)
|
||||
```
|
||||
- Creating/enriching a person or company page -> `enrich`
|
||||
- Fix broken citations in brain pages -> `citation-fixer`
|
||||
- Publish/share a brain page as link -> `brain-publish`
|
||||
- Generate PDF from brain page -> `brain-pdf`
|
||||
- Read a book through lens of a problem -> `strategic-reading`
|
||||
- Personalized book analysis -> `book-mirror`
|
||||
- Brain integrity -> `brain-librarian`
|
||||
...
|
||||
```
|
||||
|
||||
### After (13 rows, 13KB)
|
||||
```
|
||||
- **Brain & knowledge**: create/enrich/search/export brain pages, filing,
|
||||
citations, publishing, book analysis, strategic reading, concept synthesis,
|
||||
archive mining -> `brain-ops` (dispatcher for: enrich, query, brain-pdf,
|
||||
brain-publish, brain-export, brain-librarian, citation-fixer, book-mirror,
|
||||
strategic-reading, concept-synthesis, archive-crawler, ...)
|
||||
```
|
||||
|
||||
## Why It Works
|
||||
|
||||
The LLM doesn't need one row per sub-skill. It needs:
|
||||
1. **Area recognition** — "this is about brain pages" -> Brain & Knowledge
|
||||
2. **Sub-skill visibility** — the `(dispatcher for: ...)` list shows what's available
|
||||
3. **The skill file itself** — once the LLM reads `brain-ops/SKILL.md`, it has full routing detail
|
||||
|
||||
This is a **two-layer dispatch**: routing file routes to the area, the area
|
||||
skill routes to the specific sub-skill. Each layer does one job well.
|
||||
|
||||
## A/B Eval Results
|
||||
|
||||
Three resolver architectures tested across three Anthropic frontier models
|
||||
(Opus 4.7, Sonnet 4.6, Haiku 4.5) on real production AGENTS.md content,
|
||||
20 hand-authored training fixtures + 5 held-out blind fixtures, n=3 seeded
|
||||
repeats per (fixture, variant). Two scoring rules: **STRICT** (predicted
|
||||
slug exactly equals expected) and **LENIENT** (predicted is in the same
|
||||
dispatcher area as expected). Both matter:
|
||||
|
||||
- STRICT measures: "does the LLM return the exact slug?"
|
||||
- LENIENT measures: "does the LLM land in the right area, even if it picks a
|
||||
more-specific sub-skill from `(dispatcher for: ...)`?" This is closer to
|
||||
production behavior — an agent that lands in `gmail` for an email intent
|
||||
succeeds even if the resolver entry said `executive-assistant`.
|
||||
|
||||
### Training corpus (n=20, 3 seeds × 3 variants × 3 models, LENIENT)
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 | Size |
|
||||
|---|---|---|---|---|
|
||||
| baseline (270 bullet rows) | 81.7% ± 7.2% | 86.7% ± 7.2% | 73.3% ± 7.2% | 25KB |
|
||||
| **functional-areas** (this pattern) | **98.3% ± 7.2%** | **100% ± 0%** | **88.3% ± 7.2%** | **13KB** |
|
||||
| resolver-of-resolvers (no dispatcher clause) | 63.3% ± 14.3% | 41.7% ± 7.2% | 65.0% ± 12.4% | 10KB |
|
||||
|
||||
### Held-out blind corpus (n=5, 3 seeds, LENIENT)
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 |
|
||||
|---|---|---|---|
|
||||
| baseline | 100% ± 0% | 100% ± 0% | 100% ± 0% |
|
||||
| **functional-areas** | **100% ± 0%** | **100% ± 0%** | **100% ± 0%** |
|
||||
| resolver-of-resolvers | 100% ± 0% | **73.3% ± 28.7%** | 100% ± 0% |
|
||||
|
||||
### What the data shows
|
||||
|
||||
1. **Functional-areas BEATS baseline on training across all three models** (+13 to +17pp) at 48% the size. Held-out is saturated at 100% for both — within margin of error.
|
||||
|
||||
2. **The `(dispatcher for: ...)` clause is the load-bearing signal.** resolver-of-resolvers strips that clause and collapses to 41.7% on Sonnet — the catastrophic failure case the original PR predicted, now observed.
|
||||
|
||||
3. **The pattern works because the LLM can drill into the dispatcher list.** Most "STRICT failures" are the LLM picking a more-specific sub-skill (`gmail` instead of `executive-assistant`). That's the pattern working as designed. STRICT scoring under-counts; LENIENT scoring reflects production agent behavior.
|
||||
|
||||
4. **The pattern's value scales with model tier.** Compression gain (functional-areas vs baseline, training, LENIENT) is +17pp on Opus, +13pp on Sonnet, +15pp on Haiku. Sonnet shows the cleanest separation between functional-areas and resolver-of-resolvers (100% vs 41.7%) — model capacity affects how much the dispatcher signal matters.
|
||||
|
||||
### Reproduce
|
||||
|
||||
```bash
|
||||
cd evals/functional-area-resolver
|
||||
node harness.mjs --model opus # ~225 LLM calls, ~$1.70 at Opus pricing
|
||||
node harness.mjs --model sonnet # ~$1.00
|
||||
node harness.mjs --model haiku # ~$0.30
|
||||
node rescore.mjs baseline-runs/2026-05-11-opus-4-7.jsonl # zero-cost re-score
|
||||
```
|
||||
|
||||
Receipts (model, prompt_template_hash, fixtures_hash, harness_sha, ts):
|
||||
`evals/functional-area-resolver/baseline-runs/2026-05-11-{opus-4-7,sonnet-4-6,haiku-4-5}.jsonl`.
|
||||
|
||||
### Methodology caveats
|
||||
|
||||
- **Production prompt matters.** With a naive "return the skill slug" prompt
|
||||
(no instruction about `(dispatcher for: ...)`), every compression variant
|
||||
collapses to ~30-60% on Opus. The dispatcher-aware prompt is in
|
||||
`evals/functional-area-resolver/harness-runner.ts:PROMPT_TEMPLATE`. Use it
|
||||
as the template for your agent's harness; without it, compression breaks.
|
||||
- **Training corpus and variants were authored by the same release.** Held-out
|
||||
corpus was written before the variants and never adjusted; this mitigates
|
||||
but does not eliminate overfitting.
|
||||
- **Confidence intervals via t-distribution across n=3 seeded repeats.** Hold the
|
||||
n=3 lower-bound: high CIs mean the underlying sample is noisy.
|
||||
- **Single-vendor result.** All three models are Anthropic. Cross-vendor
|
||||
verification (Gemini, GPT) is a v0.33.x follow-up.
|
||||
- **Held-out blind set is small (n=5).** Saturated at 100% across most cells —
|
||||
the harness can't distinguish between "100%" and "95% with one nondeterministic
|
||||
miss." Expanding to ≥20 is a v0.33.x follow-up.
|
||||
|
||||
### Prior work and citations
|
||||
|
||||
The pattern is a **static-prompt analog of hierarchical agent routing**, a
|
||||
2024-2025 research direction:
|
||||
|
||||
- **AnyTool** ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253)) showed
|
||||
meta-agent → category-agent → tool-agent hierarchy on 16K APIs beats flat
|
||||
retrieval by +35.4pp. The `(dispatcher for: ...)` clause is the
|
||||
meta-agent's view collapsed into a single LLM pass.
|
||||
- **RAG-MCP** ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1))
|
||||
reports 49.2% prompt-token reduction at 3.2× accuracy gain via
|
||||
embedding-based pre-retrieval. The token-reduction story matches ours
|
||||
(48% smaller), via a different mechanism (RAG vs static dispatcher).
|
||||
- **Anthropic Agent Skills**
|
||||
([engineering blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills))
|
||||
promotes progressive disclosure: frontmatter (~80 tokens) always loaded,
|
||||
SKILL.md body loaded on match. This skill applies the same principle at
|
||||
the routing-table level, not the per-skill body level.
|
||||
|
||||
The 2025-2026 literature has no published benchmark for **static-prompt
|
||||
hierarchical routing** (every published hierarchical scheme resolves the
|
||||
hierarchy at runtime via a second LLM call). Our finding — that the
|
||||
hierarchy can be inlined into a single-LLM-pass dispatcher list and retain
|
||||
routing accuracy — is the open contribution. See
|
||||
`evals/functional-area-resolver/README.md` for methodology details.
|
||||
|
||||
## How To Compress
|
||||
|
||||
### Step 1: Preconditions
|
||||
|
||||
Refuse to compress if either gate fails:
|
||||
- Source routing file is under 12KB (compression overhead exceeds benefit).
|
||||
- `git status` shows uncommitted changes to the routing file (the
|
||||
compressor's edit would entangle with whatever the user was doing).
|
||||
|
||||
If a user wants to override either gate, they ask explicitly with `--force`.
|
||||
|
||||
### Step 2: When to compress which file
|
||||
|
||||
GBrain workspaces often have TWO routing files merged at runtime (per
|
||||
`src/core/check-resolvable.ts` v0.31.7): `skills/RESOLVER.md` and a sibling
|
||||
`../AGENTS.md`. Choose which to compress:
|
||||
|
||||
- Only one is fat (>12KB): compress that one; leave the small one alone.
|
||||
- Both are fat: compress them separately, in order: AGENTS.md first
|
||||
(usually the larger one in OpenClaw-style deployments), then RESOLVER.md.
|
||||
- Only the small one is fat (rare): same rule — compress it.
|
||||
|
||||
If the deployment uses only one routing file, this section is a no-op —
|
||||
compress that one.
|
||||
|
||||
### Step 3: Identify functional areas
|
||||
|
||||
Group skills by domain. Typical areas (adjust per deployment):
|
||||
|
||||
- **Brain & Knowledge** — brain-ops as dispatcher
|
||||
- **Content Ingestion** — ingest as dispatcher
|
||||
- **Calendar & Scheduling** — google-calendar as dispatcher
|
||||
- **Email & Comms** — executive-assistant as dispatcher
|
||||
- **Research & Investigation** — perplexity-research as dispatcher
|
||||
- **X/Twitter & Social** — x-ingest as dispatcher
|
||||
- **Places & Travel** — checkin as dispatcher
|
||||
- **Product & Building** — acp-coding as dispatcher
|
||||
- **Infrastructure** — healthcheck as dispatcher
|
||||
- **Tasks & Logistics** — daily-task-manager as dispatcher
|
||||
- **People & Contacts** — google-contacts as dispatcher
|
||||
|
||||
### Step 4: Build the area entry format
|
||||
|
||||
Each area entry follows this template:
|
||||
|
||||
```
|
||||
- **{Area Name}**: {comma-separated trigger phrases} -> `{dispatcher-skill}`
|
||||
(dispatcher for: {comma-separated sub-skill names})
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Trigger phrases should be broad enough to catch intent ("brain pages, enrich,
|
||||
search, filing, citations, book analysis")
|
||||
- Sub-skill list should be comprehensive — this is how the LLM knows what's available
|
||||
- The dispatcher skill file should have its own internal routing table
|
||||
|
||||
### Step 5: Keep always-on entries separate
|
||||
|
||||
Gates and always-on entries (acknowledge, multi-user, entity-detector, etc.)
|
||||
stay as individual rows — they're checked on every message, not dispatched.
|
||||
|
||||
### Step 6 (MANDATORY): Verify routing accuracy
|
||||
|
||||
Run two gates before committing the compressed file. Do NOT commit if either
|
||||
fails.
|
||||
|
||||
**Gate 1: Structural verification.** Confirms your `routing-eval.jsonl`
|
||||
fixtures still resolve to the right skills under the compressed routing file.
|
||||
Run from the workspace whose routing file you just edited:
|
||||
|
||||
```bash
|
||||
gbrain routing-eval --json
|
||||
```
|
||||
|
||||
If accuracy on your fixtures drops below 95%, revert and tune the area
|
||||
entries before re-running.
|
||||
|
||||
**Gate 2: LLM A/B verification on YOUR edited file.** Confirms a frontier
|
||||
LLM can still drill into the dispatcher list and reach sub-skills under
|
||||
your specific compression. Requires a gbrain repo checkout because the
|
||||
harness lives there. Copy your edited routing file into the harness's
|
||||
variants directory, then invoke the harness with `--variants` pointing
|
||||
at it:
|
||||
|
||||
```bash
|
||||
# In your agent workspace, identify the routing file you just compressed.
|
||||
EDITED=/path/to/your/AGENTS.md # or skills/RESOLVER.md, whichever you edited
|
||||
|
||||
# In your gbrain repo checkout:
|
||||
cd /path/to/gbrain/evals/functional-area-resolver
|
||||
TMP=$(mktemp -d)/variants && mkdir -p "$TMP"
|
||||
cp "$EDITED" "$TMP/my-edit.md"
|
||||
|
||||
# Run the harness against your file (sequential, ~75 calls × $0.0076 ≈ $0.57 on Opus).
|
||||
ANTHROPIC_API_KEY=... node harness.mjs --variants-dir "$TMP" --variants my-edit \
|
||||
--model opus --parallel 3 --yes
|
||||
```
|
||||
|
||||
The harness uses gbrain's bundled fixture set, so this verifies "did the LLM
|
||||
land in the right sub-skill for routing intents the gbrain-bundled fixtures
|
||||
cover" — a regression check on shared skills, not a full re-eval of YOUR
|
||||
fixture set. For full eval coverage, mirror this skill's
|
||||
`fixtures.jsonl` + `fixtures-held-out.jsonl` setup with intents specific
|
||||
to your skills.
|
||||
|
||||
If the lenient (same-area) score on your variant drops below 95%, revert the
|
||||
compression and tune. Common causes:
|
||||
- A sub-skill was omitted from the `(dispatcher for: ...)` list.
|
||||
- Trigger phrases for an area are too narrow (LLM can't recognize intent).
|
||||
- Areas were collapsed too aggressively (too few areas — see Anti-Patterns).
|
||||
- ASCII `->` vs Unicode `→` mismatch — the harness now accepts both, but
|
||||
earlier versions only matched Unicode. Pin gbrain to v0.32.3.0+.
|
||||
|
||||
Common false negatives on the harness eval (NOT bugs in your compression):
|
||||
- The gbrain-bundled fixtures target skill names like `enrich`, `query`,
|
||||
`gmail`, `executive-assistant`. If your routing file doesn't expose
|
||||
those skills at all, expect strict-scoring failures on those fixtures.
|
||||
Lenient scoring stays accurate for any sub-skill present in your
|
||||
`(dispatcher for: ...)` lists.
|
||||
|
||||
### Step 7: Review the diff before committing
|
||||
|
||||
Show the user the proposed edit (or the actual git diff) and wait for
|
||||
explicit approval before staging. Same convention as `skills/book-mirror/SKILL.md`.
|
||||
|
||||
## Contract
|
||||
|
||||
This skill guarantees:
|
||||
|
||||
- Routing matches the canonical triggers in the frontmatter.
|
||||
- Compression is only performed when the preconditions in Step 1 pass (file ≥12KB AND clean working tree, or `--force`).
|
||||
- The mandatory verification gate in Step 6 fires on the user's edited file, not on sample variants. The user runs `gbrain routing-eval --json` AND the gbrain-repo harness (`node harness.mjs --variants-dir <tmp> --variants my-edit`) before committing the compressed file.
|
||||
- Privacy contract preserved: no fork-specific filesystem path literals (server-side brain home, OpenClaw fork home) leak into the compressed output.
|
||||
|
||||
The full behavior contract is documented in the body sections above; this section exists for the conformance test.
|
||||
|
||||
## Output Format
|
||||
|
||||
The compressed routing file follows the area-entry template documented in Step 4 ("Build the area entry format"). Each entry: `- **{Area Name}**: {trigger phrases} -> \`{dispatcher-skill}\` (dispatcher for: {sub-skill list})`. The dispatcher arrow may be either ASCII `->` (default in this template) or Unicode `→` (used in some production deployments); the gbrain harness accepts both.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Resolver-of-resolvers with pipe tables.** Tested and failed (see eval
|
||||
table). The LLM picks area names from the table instead of drilling into
|
||||
sub-skills.
|
||||
|
||||
- **Removing sub-skill names.** Without the `(dispatcher for: ...)` list,
|
||||
the LLM can't route to specific sub-skills. The list is the routing signal.
|
||||
|
||||
- **Too few areas.** Collapsing to <5 areas makes each area too broad.
|
||||
12-15 areas is the sweet spot.
|
||||
|
||||
- **Too many areas.** Defeats the purpose. If you have 50 areas, just keep
|
||||
individual rows.
|
||||
|
||||
## Maintenance
|
||||
|
||||
When adding a new skill:
|
||||
1. Identify its functional area.
|
||||
2. Add the skill name to that area's `(dispatcher for: ...)` list.
|
||||
3. Update the area's skill file with routing detail.
|
||||
4. Run the routing eval (Step 6) to verify.
|
||||
|
||||
When adding a new functional area:
|
||||
1. Create the dispatcher skill with internal routing.
|
||||
2. Add the area entry to the routing file.
|
||||
3. Run the routing eval (Step 6) to verify.
|
||||
|
||||
## Changelog
|
||||
|
||||
### v1.0.0 — 2026-05-11
|
||||
- Initial version. Pattern shipped in gbrain v0.32.3.0 with a held-out A/B
|
||||
eval (see `evals/functional-area-resolver/`).
|
||||
- Skill renamed from `compress-agents-md` to `functional-area-resolver`
|
||||
pre-release; the contribution is the pattern, not the filename.
|
||||
@@ -1,23 +0,0 @@
|
||||
// Routing eval fixtures for skills/functional-area-resolver. Each
|
||||
// positive-intent fixture contains at least one trigger string from the
|
||||
// skill's RESOLVER.md row as substring (structural matcher requirement
|
||||
// in src/core/routing-eval.ts:170).
|
||||
// Adversarial negative fixtures at the bottom guard against the
|
||||
// broadened triggers (D5:B) over-capturing intents that belong to
|
||||
// adjacent meta-skills like skillify, skill-creator, book-mirror,
|
||||
// concept-synthesis.
|
||||
{"intent":"My AGENTS.md too large at 30KB and hitting context limits, how do I shrink it","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"The daily doctor says context-health is red because AGENTS.md too large","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"How do I compress my resolver without losing routing accuracy","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"RESOLVER.md too big — convert my 200-row skill resolver into functional areas","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"What's the functional area dispatcher pattern for AGENTS.md","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"My RESOLVER.md too big at 25KB, how do I shrink it","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"I want to compress my resolver while keeping all the sub-skills reachable","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"Explain the functional area dispatcher pattern and when to use it","expected_skill":"functional-area-resolver"}
|
||||
// Adversarial negatives. These intents pattern-match the broadened
|
||||
// triggers ("compress my resolver", "shrink routing table", etc.) but
|
||||
// the correct route is the target skill, not functional-area-resolver.
|
||||
{"intent":"Skillify this — make this proper from the routing-pattern notes","expected_skill":"skillify","ambiguous_with":["functional-area-resolver"]}
|
||||
{"intent":"Create a skill that compacts a routing file using AI","expected_skill":"skill-creator","ambiguous_with":["functional-area-resolver"]}
|
||||
{"intent":"Personalized version of this book about resolver and dispatcher design","expected_skill":"book-mirror","ambiguous_with":["functional-area-resolver"]}
|
||||
{"intent":"Synthesize my concepts about how routing files grow over time","expected_skill":"concept-synthesis","ambiguous_with":["functional-area-resolver"]}
|
||||
@@ -50,34 +50,6 @@ This skill guarantees:
|
||||
|
||||
## Phases
|
||||
|
||||
### Autonomous path (v0.36.4.0) — when you want to reach a target score
|
||||
|
||||
If the user asks "get my brain to 90/100" or "fix what's broken", prefer the
|
||||
one-command loop over walking each dimension by hand:
|
||||
|
||||
```bash
|
||||
gbrain doctor --remediation-plan --json # preview what would run
|
||||
gbrain doctor --remediate --yes --target-score 90 --max-usd 5
|
||||
```
|
||||
|
||||
`--remediation-plan` prints a dependency-ordered list (sync before extract,
|
||||
embed after consolidate, etc.) with per-step `est_seconds` and `est_usd_cost`.
|
||||
`--remediate` walks the plan, submitting each step as a Minion job, re-checking
|
||||
score between every step. `--max-usd N` is a hard cost cap — submission refuses
|
||||
when the plan would exceed the cap (prevents synthesize loops from burning
|
||||
Anthropic credits unattended).
|
||||
|
||||
When the target score is unreachable for the brain (empty brain with no entity
|
||||
pages → `graph_coverage` caps at 70; unconfigured embedding key → caps at 60),
|
||||
the command bails with a list of what's missing rather than looping.
|
||||
|
||||
Use the per-dimension walk below (Phase 2 onward) when:
|
||||
- The user explicitly asks for a dimension-by-dimension audit
|
||||
- You're investigating why score is stuck below `--remediate`'s ceiling
|
||||
- A specific dimension needs manual judgment that the auto path skips
|
||||
|
||||
### Manual path
|
||||
|
||||
1. **Run health check.** Check gbrain health to get the dashboard.
|
||||
2. **Check each dimension:**
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user