mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
v0.37.3.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out (supersedes #1206) (#1215)
* v0.37.1.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out Cathedral wave superseding PR #1206. Doctor now scans every SKILL.md for external-lookup tools (web_search / web_fetch / exa / perplexity / happenstance / crustdata / captain_api / firecrawl) and warns when the skill has no brain- first compliance signal. gbrain doctor --fix auto-inserts the canonical > **Convention:** see [conventions/brain-first.md](...) callout via the dry-fix.ts MISSING_RULE_PATTERNS extension (sharing safety gates with the existing REPLACE patterns). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches authorship; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. Key design decisions locked via /plan-eng-review + codex outside-voice review: - A1: frontmatter ships only brain_first: exempt (no required/n/a enum) - A2: snapshot+diff audit at ~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl with transition-only writes (stable brains = 0 lines/run) - A3: scaffold template pre-inserts callout; skillify check fails (exit 1) on external + no callout + no exempt - A4: position-relative gate is BODY-ONLY (frontmatter tools: [web_search] declaration doesn't false-flag the skill) - Q1: single pure analyzeSkillBrainFirst() helper consumed by 3 surfaces - CMT1: no upgrade migration — doctor surfaces hint, --fix applies via dry-fix safety gates (user stays in loop) - CMT2: dropped tools+writes_pages auto-exemption (was hiding mixed-class skills like idea-ingest/meeting-ingestion/data-research) Trio: VERSION + package.json + CHANGELOG aligned at 0.37.1.0. 56 unit cases + 12 E2E cases pass. 170 related existing tests pass unchanged. Self-dogfood: gbrain doctor against this repo's skills/ reports skill_brain_first: ok across 43 skills (compliant or exempt). functional-area-resolver and strategic-reading skills gained brain_first: exempt to validate the declarative opt-out in production code (both name perplexity in dispatcher prose without calling it). Co-Authored-By: garrytan-agents <noreply@github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md for v0.37.1.0 skill_brain_first wave Added Key Files entries for the four new modules: - src/core/skill-frontmatter.ts (shared parser) - src/core/skill-brain-first.ts (analyzer + FORMERLY_HARDCODED_EXEMPT) - src/core/skill-fix-gates.ts (extracted safety primitives) - src/core/audit-skill-brain-first.ts (snapshot+diff JSONL) Extended existing entries: - src/core/filing-audit.ts: rewired to shared parser - src/core/dry-fix.ts: MISSING_RULE_PATTERNS INSERT pattern type - src/commands/doctor.ts: skill_brain_first check + tweet-shield framing - src/commands/skillify-check.ts: required item 12 + scaffold pre-insert Added test inventory entries: - test/skill-brain-first.test.ts (56 unit cases) - test/e2e/skill-brain-first.test.ts (12 E2E cases) Regenerated llms-full.txt via bun run build:llms. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): skill_brain_first guard uses doctor --fast to skip engine connect CI run #76881161092 failed because scripts/check-skill-brain-first.sh invoked plain `gbrain doctor --json`, which routes through connectEngine(). With no ~/.gbrain/config.json present (CI's case — runner is bun-only, no brain init), connectEngine() exits 1 with "No brain configured." and emits zero stdout. The python parser sees an empty file and returns parse_error, failing the verify gate. Fix: pass --fast to doctor. --fast routes through runDoctor(null, ...) which runs the filesystem-only check set (resolver_health, skill_conformance, skill_brain_first) and emits the standard single-line JSON envelope the parser expects. skill_brain_first is filesystem-only by design (scans SKILL.md, no DB touch), so --fast is the correct knob, not a workaround. Verified by reproducing the CI failure mode locally with GBRAIN_HOME=/tmp/empty-... — gate now passes both with and without a configured brain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.37.1.0 → v0.37.3.0 (queue collision with #1214) PR #1214 (brainstorm + lsd) claimed v0.37.1.0 concurrently with #1215. Skipping 0.37.2.0 leaves a buffer for #1214's adjacent slot. Trio (VERSION + package.json + CHANGELOG header + inline "To take advantage of v0.37.3.0" block) aligned at 0.37.3.0. No behavior changes — version metadata only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <noreply@github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
garrytan-agents
Claude Opus 4.7
garrytan-agents
parent
9a4ae0962e
commit
772253ef44
+101
@@ -2,6 +2,107 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.37.3.0] - 2026-05-19
|
||||
|
||||
**Your agent now catches skills that would call the web before checking the brain. The same class of miss that flagged Garry's own Palantir tweet as a risk because none of the three eval models knew he built it.**
|
||||
|
||||
Real story: on 2026-05-19, the cross-modal eval looked at a tweet about Palantir's Finance UI and said "this is risky, none of us recognize this work." But Garry's brain already had pages explaining he designed that Finance UI and shipped 150+ PSDs in 2006. The brain knew. The eval skill went to the web first and never asked the brain. A static SKILL.md check catches the authorship side of that failure mode: any skill that calls `web_search`, Perplexity, Exa, Crustdata, Happenstance, or Captain API now has to declare either how it consults the brain first, or that it intentionally doesn't need to.
|
||||
|
||||
How to turn it on (it's already on after upgrade):
|
||||
|
||||
```
|
||||
gbrain doctor # warns on skills that need brain-first declaration
|
||||
gbrain doctor --fix # auto-adds the canonical Convention callout to each
|
||||
gbrain doctor --fix --dry-run # preview without writing
|
||||
```
|
||||
|
||||
What a flagged skill looks like in your `~/.openclaw/workspace/skills/`:
|
||||
|
||||
| Skill | What the doctor sees | Easy fix |
|
||||
|---|---|---|
|
||||
| Author called `perplexity` for research but never `gbrain search` | warn — `missing_brain_first` | `gbrain doctor --fix` adds `> **Convention:** see [conventions/brain-first.md](...)` near the top |
|
||||
| Pure infra (cron schedulers, container managers, ask-user prompters) | warn — same | Add `brain_first: exempt` to frontmatter; that's it |
|
||||
| Author typed `brain-first: exempt` (kebab-case typo) | warn + paste-ready hint | Switch to snake_case `brain_first: exempt` |
|
||||
| Already carries `> **Convention:** see conventions/brain-first.md` | ok — `compliant_callout` | no action |
|
||||
| First `gbrain search` reference comes before first `web_search` in body | ok — `compliant_position` | no action |
|
||||
|
||||
The cathedral piece that makes this stick: `gbrain doctor --fix` runs through the same git-safety gates that `dry-fix.ts` already shipped (refuses to write to a dirty working tree, refuses on non-git files, refuses against the install-tree fallback). So you can run `--fix` on your OpenClaw workspace without losing the audit trail.
|
||||
|
||||
Things to watch:
|
||||
- Audit log lives at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` (ISO-week rotated). Stable brains write zero lines per doctor run; only state transitions (new violation, resolved violation, applied fix) get recorded. So `tail -20` actually shows you signal, not noise.
|
||||
- The snapshot at `~/.gbrain/audit/skill-brain-first-snapshot.json` is last-writer-wins under concurrent doctor runs. If two doctor processes race, one snapshot wins; the next run reconciles.
|
||||
- New skills scaffolded via `gbrain skillify scaffold <name>` get the canonical Convention callout pre-inserted. `gbrain skillify check <skill>` fails (exit 1) when external-lookup-without-callout-or-exempt is detected, so non-compliant skills can't sneak through scaffold-then-merge.
|
||||
- This catches the AUTHORSHIP miss class. The RUNTIME analog (intercepting MCP tool dispatch when a subagent calls `web_search` without an earlier `gbrain search` in the same turn) is filed as a v0.37+ TODO. Both layers eventually.
|
||||
|
||||
What we caught and fixed before merging:
|
||||
- Codex's outside-voice review (gpt-5.5) found 18 issues against an earlier draft. Three of them changed the design materially:
|
||||
1. The original plan shipped a one-shot upgrade migration that would have silently edited host SKILL.md files outside the `--fix` git-safety contract. Codex flagged that as smuggling the PR allowlist back in as data. Dropped the migration; doctor surfaces the hint and `--fix` applies via the existing safety gates.
|
||||
2. The original plan auto-exempted skills with `tools: [search, query, put_page]` + `writes_pages: true`. Codex pointed out this covers up the riskiest class (ingest skills that write pages AND call Perplexity). Dropped the rule entirely; brain-tool ownership doesn't prove brain-first compliance.
|
||||
3. The first-pass detection regex scanned the whole file. Codex pointed out `tools: [web_search]` in YAML frontmatter would be the "first external reference" and false-flag the skill. Detection now strips frontmatter before any position-relative scan.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
|
||||
- **`brain_first: exempt` frontmatter field** — declarative opt-out for skills that don't need brain-first. The single canonical form (`brain_first: exempt`, snake_case, lowercase, unquoted). Near-misses (`brain-first`, `BrainFirst`, quoted values, unknown values) trigger a paste-ready doctor hint instead of silent failure.
|
||||
- **`skill_brain_first` doctor check** — scans every SKILL.md under the configured skills dir, detects external-lookup patterns (web_search / web_fetch / exa / perplexity / happenstance / crustdata / captain_api / firecrawl), confirms compliance via canonical Convention callout / explicit Phase 1 brain heading / position-relative brain reference / `brain_first: exempt` opt-out / absence of external pattern. Surfaces structured `Check.issues[]` for JSON tooling; emits formerly-EXEMPT_SKILLS hints for PR #1206's historical 40-name allowlist.
|
||||
- **`gbrain doctor --fix` MISSING_RULE_PATTERNS** — `dry-fix.ts` gains an INSERT pattern type alongside the existing REPLACE patterns. Auto-inserts `> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) for the lookup chain (search → query → get_page → external).` at the `after-h1-paragraph` site of any flagged skill. Idempotent (re-runs detect the existing callout and skip).
|
||||
- **`gbrain skillify scaffold` pre-insert** — `src/core/skillify/templates.ts` now writes the canonical Convention callout into new SKILL.md scaffolds by default. Zero-friction compliance for new skills.
|
||||
- **`gbrain skillify check` required item 12** — brain-first compliance is now a real gate, not informational. Exit 1 when an external-lookup skill lacks both the callout and the `brain_first: exempt` declaration.
|
||||
- **Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`** — transition-only writes. Stable brains produce 0 audit lines per doctor run. `readRecentBrainFirstEvents(7)` exposes the audit for future trend tooling.
|
||||
- **Live OpenClaw dev script** at `scripts/live-brain-first-check.ts` — opt-in (`$OPENCLAW_WORKSPACE`-gated) shape report against the live deployment. Not in CI; run manually during dev / QA / post-`--fix` validation.
|
||||
- **CI guard `scripts/check-skill-brain-first.sh`** — JSON-parses `gbrain doctor --json` to gate `bun run verify` on `warn` (doctor's exit code only flags `fail`).
|
||||
|
||||
#### Changed
|
||||
|
||||
- **`src/core/skill-frontmatter.ts`** — NEW shared content-based frontmatter parser. Replaces `filing-audit.ts`'s private path-based `parseFrontmatter`. Adds `tools?`, `triggers?`, `brain_first?: 'exempt'`, and a typed `brain_first_typo` field that surfaces near-miss declarations.
|
||||
- **`src/core/skill-fix-gates.ts`** — NEW shared safety primitives module (working-tree check, code-fence guard, etc.). Both REPLACE and INSERT auto-fix patterns consume from here; `dry-fix.ts` re-exports for back-compat.
|
||||
- **`skills/conventions/brain-first.md`** — extended with declarative opt-out documentation. New "Declarative opt-out (v0.36.x)" section covers the strict canonical form, typo behavior, and when the opt-out is unnecessary.
|
||||
- **PR #1206 hardcoded `EXEMPT_SKILLS` allowlist** — replaced with structural-signal inference + explicit `brain_first: exempt` opt-out. The 40-name list is preserved as `FORMERLY_HARDCODED_EXEMPT` in `src/core/skill-brain-first.ts` purely for doctor-hint flow (CMT1).
|
||||
- **`functional-area-resolver` and `strategic-reading` skills in this repo** — gained `brain_first: exempt` frontmatter. Both name `perplexity` in dispatcher prose (sub-skill cross-references) without actually calling external APIs; the regex tripped on word boundary. Declarative opt-out is the canonical fix for this false-positive class.
|
||||
|
||||
#### Tests
|
||||
|
||||
- New: `test/skill-brain-first.test.ts` (56 cases — frontmatter parser, analyzer ladder across 9 fixtures, offset helpers, regex shape, audit snapshot+diff, PR #1206 regression absorption).
|
||||
- New: `test/fixtures/brain-first-skills/*/SKILL.md` (9 fixtures driving the unit + E2E suites).
|
||||
- New: `test/e2e/skill-brain-first.test.ts` (12 cases — shape assertions, `--fix` dry-run/apply cycle, idempotency, audit transition signal).
|
||||
- Existing: 170 cases in `test/filing-audit.test.ts`, `test/dry-fix.test.ts`, `test/doctor*.test.ts` pass unchanged (regression-preservation).
|
||||
|
||||
#### Removed
|
||||
|
||||
- Nothing user-facing. The `parseFrontmatter` function inside `filing-audit.ts` is now a thin wrapper over the new shared parser — internal refactor only.
|
||||
|
||||
#### For contributors
|
||||
|
||||
- The brain-first regex is intentionally permissive (word-boundary `\bperplexity\b` etc.). False-positives on name mentions in dispatcher prose are expected and answered by the declarative opt-out. Tightening to require API-call shape is a v0.36.x+ TODO.
|
||||
- The runtime MCP-dispatch brain-first gate is the bigger follow-up wave. Static-check covers authorship; runtime covers compliance. Filed as v0.37+ TODO.
|
||||
- Co-Authored-By: garrytan-agents (PR #1206 contributor) — the EXEMPT_SKILLS list shape, regex set, and tweet-shield incident framing carry forward verbatim.
|
||||
|
||||
## To take advantage of v0.37.3.0
|
||||
|
||||
`gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`.
|
||||
This release has no schema migrations — every change is filesystem-only — so the
|
||||
upgrade is purely the binary swap.
|
||||
|
||||
1. **Verify the new check landed:**
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "skill_brain_first")'
|
||||
```
|
||||
2. **If your skills dir flags any violators**, the easiest fix is the auto-fix:
|
||||
```bash
|
||||
gbrain doctor --fix --dry-run # preview
|
||||
gbrain doctor --fix # apply (writes the canonical callout)
|
||||
```
|
||||
Or for genuine infra skills, add `brain_first: exempt` to the frontmatter manually.
|
||||
3. **Your agent reads `skills/conventions/brain-first.md` the next time you interact with it.** The new "Declarative opt-out" section documents the contract; no manual agent prompt update needed.
|
||||
4. **If `gbrain doctor` reports unexpected violations or any step fails,** file an issue: https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor --json | jq '.checks[] | select(.name == "skill_brain_first")'`
|
||||
- the SKILL.md of the surprising flag
|
||||
- whether `--fix` cleaned it up
|
||||
|
||||
The brain-first detection is regex-based and will hit false-positives on
|
||||
skills that NAME but don't CALL the external tools. Declarative opt-out
|
||||
(`brain_first: exempt`) is the canonical answer for that class.
|
||||
## [0.37.2.0] - 2026-05-19
|
||||
|
||||
**Your grading script writes "unresolvable" verdicts now. Before this fix, every single one was rejected at the database layer — 0 of 34 writes landed in a recent production run.**
|
||||
|
||||
@@ -1,6 +1,56 @@
|
||||
# TODOS
|
||||
|
||||
|
||||
## skill_brain_first wave follow-ups (v0.36.4+)
|
||||
|
||||
- [ ] **v0.37+: Runtime brain-first gate at MCP dispatch.** The v0.36.x
|
||||
`skill_brain_first` doctor check is purely static — it scans SKILL.md
|
||||
authorship for canonical Convention callouts, `brain_first: exempt`
|
||||
frontmatter, or position-relative brain references. The motivating
|
||||
incident (2026-05-19 tweet-shield) was a RUNTIME failure: an agent
|
||||
called Perplexity / cross-modal eval to assess Garry's Palantir tweet
|
||||
without ever checking the brain, which already had "designed the
|
||||
entire Finance product UI" and "150+ PSDs from April-December 2006."
|
||||
A runtime gate would hook MCP tool dispatch: when a subagent invokes
|
||||
`web_search` / `perplexity` / `exa` / etc., require that a `search`,
|
||||
`query`, or `get_page` call landed earlier in the same agent turn.
|
||||
Subagent-isolation aware (the gate scope is per-turn, per-agent).
|
||||
Touches: `src/mcp/dispatch.ts` (tool-call entry seam, would gate before
|
||||
routing to external-tool handlers), `src/core/minions/handlers/subagent.ts`
|
||||
(per-turn tracking), `src/core/operations.ts` (cross-reference the
|
||||
brain-tool ops). Full wave on its own (~3-5 days human / ~1-2h CC).
|
||||
Out of scope for the static-check wave because the surface area is
|
||||
fundamentally different. Closes the tweet-shield root cause at the
|
||||
enforcement layer instead of just the authorship layer.
|
||||
|
||||
- [ ] **v0.36.x: Audit trend doctor check `skill_brain_first_trend`.** The
|
||||
v0.36.x snapshot+diff audit JSONL at
|
||||
`~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected /
|
||||
resolved / fixed events as transitions. The data is reachable via
|
||||
`readRecentBrainFirstEvents(7)` in `src/core/audit-skill-brain-first.ts`
|
||||
but no doctor surface consumes it yet. Add a `skill_brain_first_trend`
|
||||
check (~30 LOC) that reads recent events, aggregates added vs resolved
|
||||
counts per week, warns when violations are rising (e.g. >3 added, 0
|
||||
resolved over 4 weeks). Cheap to land once audit logs accumulate
|
||||
multiple weeks of data (no point shipping it with zero baseline data).
|
||||
Mirrors the doctor check pattern in `src/commands/doctor.ts`. Filed
|
||||
during /plan-eng-review as TODO-2.
|
||||
|
||||
- [ ] **v0.36.x: Tighten the external-lookup regex to reduce false-positive
|
||||
rate from name mentions.** v0.36.x ships with word-boundary regex on
|
||||
`perplexity`, `exa`, `web_search`, etc. This matches "perplexity"
|
||||
inside `perplexity-research` (a sub-skill name in dispatcher prose, not
|
||||
an API call). Two skills in this repo's own `skills/` (functional-area-
|
||||
resolver, strategic-reading) hit this false-positive and ship with
|
||||
`brain_first: exempt`. Possible mitigation: tighten the pattern to
|
||||
require an API-call shape like `perplexity\.|perplexity[\s._-]?(?:api|search|query)`.
|
||||
Whack-a-mole risk — the negation-prose false-positive class can't be
|
||||
reliably caught with regex either. Tracking as a follow-up; the
|
||||
declarative `brain_first: exempt` opt-out is the canonical answer for
|
||||
the false-positive cases. Decide based on real-world hit rate after
|
||||
the v0.36.x wave is in production for a few weeks.
|
||||
|
||||
|
||||
## v0.35.6.0 floor-ratio gate follow-ups (v0.36.x+)
|
||||
|
||||
- [ ] **v0.36.x: Run gbrain-side floor-ratio ablation before flipping any mode-bundle default.** v0.35.6.0 ships the gate default-off (`MODE_BUNDLES[*].floor_ratio = undefined`) because the SkyTwin labeled-retrieval ablation that surfaced the regression isn't reproducible on gbrain's own eval surfaces from outside. Before any mode-bundle default flip, run the gate at `floor_ratio: undefined`, 0.85, 0.90, 0.95 across `gbrain eval longmemeval`, `gbrain eval whoknows`, `gbrain eval suspected-contradictions`, and the BrainBench-Real replay (sibling gbrain-evals repo). Quantify per-mode P@k / R@k / nDCG@k / top-1 stability deltas. Look for: regression on queries that genuinely need the long-tail boost (specific entity lookups, low-frequency topics) vs improvement on queries where weak-overlap pages were leapfrogging. The corpus-level finding determines whether tokenmax (most exposure to the failure mode) should flip first, or whether the gate stays a per-call opt-in indefinitely. Filed during v0.35.6.0 codex outside-voice review.
|
||||
|
||||
+10
-4
File diff suppressed because one or more lines are too long
+4
-3
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.37.2.0",
|
||||
"version": "0.37.3.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -38,12 +38,13 @@
|
||||
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
||||
"test": "bash scripts/run-unit-parallel.sh",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:synthetic-corpus-privacy && bun run typecheck",
|
||||
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run typecheck",
|
||||
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
|
||||
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh",
|
||||
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
|
||||
Executable
+100
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard for the v0.36.x skill_brain_first doctor check.
|
||||
#
|
||||
# Runs `gbrain doctor --json` against this repo's own skills/ and parses
|
||||
# the JSON to assert `checks[name=skill_brain_first].status !== "warn"`.
|
||||
# Doctor's exit code only flags `fail`, not `warn`, so explicit JSON-
|
||||
# parsing is required to gate `bun run verify` on this warning-class check
|
||||
# (F15 from /plan-eng-review).
|
||||
#
|
||||
# When this fires, the brain-first compliance check found new offenders
|
||||
# in this repo's skills. Either:
|
||||
# - add `brain_first: exempt` to the flagged skill's frontmatter (if it
|
||||
# legitimately doesn't need brain-first), or
|
||||
# - add a canonical `> **Convention:** see [conventions/brain-first.md]`
|
||||
# callout near the top of the skill body.
|
||||
#
|
||||
# Usage: scripts/check-skill-brain-first.sh
|
||||
# Exit: 0 on ok; 1 on warn or unexpected.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Run doctor with this repo's own skills as the explicit target.
|
||||
#
|
||||
# --fast is REQUIRED here. Without it, doctor calls connectEngine() which
|
||||
# exits 1 when no ~/.gbrain/config.json exists (the CI runner's case — no
|
||||
# brain init), producing zero stdout and tripping the parser's
|
||||
# `parse_error` fallback. --fast routes through runDoctor(null, ...) which
|
||||
# runs filesystem-only checks (resolver_health, skill_conformance,
|
||||
# skill_brain_first) and emits the standard JSON envelope. The
|
||||
# skill_brain_first check is filesystem-only by design, so --fast is the
|
||||
# correct knob, not a workaround.
|
||||
#
|
||||
# Capturing JSON output; redirect stderr to keep progress noise out of the
|
||||
# parse.
|
||||
TMPOUT="$(mktemp -t gbrain-doctor-XXXXXXXX)"
|
||||
# shellcheck disable=SC2064
|
||||
trap "rm -f \"$TMPOUT\"" EXIT
|
||||
|
||||
GBRAIN_SKILLS_DIR="$ROOT/skills" bun run src/cli.ts doctor --fast --json >"$TMPOUT" 2>/dev/null || true
|
||||
|
||||
# Extract the skill_brain_first check status. Use python3 (already a
|
||||
# repo-wide dependency via image-decoders + admin tooling) so we don't
|
||||
# add jq to the verify chain.
|
||||
STATUS=$(python3 -c "
|
||||
import json, sys
|
||||
with open('$TMPOUT') as fp:
|
||||
for line in fp:
|
||||
line = line.strip()
|
||||
if not (line.startswith('{') and line.endswith('}')):
|
||||
continue
|
||||
try:
|
||||
report = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
for c in report.get('checks', []):
|
||||
if c.get('name') == 'skill_brain_first':
|
||||
print(c.get('status', 'missing'))
|
||||
sys.exit(0)
|
||||
print('missing')
|
||||
sys.exit(0)
|
||||
print('parse_error')
|
||||
" 2>/dev/null || echo "parse_error")
|
||||
|
||||
case "$STATUS" in
|
||||
ok)
|
||||
echo "OK: skill_brain_first check passes against this repo's skills/"
|
||||
exit 0
|
||||
;;
|
||||
warn)
|
||||
echo
|
||||
echo "ERROR: skill_brain_first check found violations in this repo's skills/."
|
||||
echo
|
||||
echo "Re-run for details:"
|
||||
echo " GBRAIN_SKILLS_DIR=\"\$(pwd)/skills\" bun run src/cli.ts doctor"
|
||||
echo
|
||||
echo "Fix options per skill:"
|
||||
echo " 1. Add 'brain_first: exempt' to frontmatter (declarative opt-out)"
|
||||
echo " 2. Add a > **Convention:** see [conventions/brain-first.md] callout"
|
||||
echo " 3. Run 'gbrain doctor --fix' to auto-add the canonical callout"
|
||||
exit 1
|
||||
;;
|
||||
fail)
|
||||
echo "ERROR: skill_brain_first check returned status=fail (unexpected)."
|
||||
exit 1
|
||||
;;
|
||||
missing)
|
||||
echo "ERROR: skill_brain_first check not present in doctor output."
|
||||
echo " This guard expected the check to run. Investigate doctor.ts wiring."
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: skill_brain_first guard could not parse doctor --json output."
|
||||
echo " Status: $STATUS"
|
||||
cat "$TMPOUT" | head -20
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,175 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* scripts/live-brain-first-check.ts — opt-in dev script for the v0.36.x
|
||||
* skill_brain_first wave (T10 from /plan-eng-review).
|
||||
*
|
||||
* Runs the doctor brain-first check against the LIVE OpenClaw deployment
|
||||
* (or whatever `$OPENCLAW_WORKSPACE` points at) and produces a human-
|
||||
* readable + machine-readable report. NOT part of `bun run verify` — the
|
||||
* deployment isn't this repo's content, so coupling CI to it would
|
||||
* produce drift the moment OpenClaw evolves independently.
|
||||
*
|
||||
* Use this manually during dev / QA / after `gbrain doctor --fix` runs
|
||||
* to validate the wave against the real deployment.
|
||||
*
|
||||
* Usage:
|
||||
* $OPENCLAW_WORKSPACE=~/.openclaw/workspace bun run scripts/live-brain-first-check.ts
|
||||
* $OPENCLAW_WORKSPACE=~/.openclaw/workspace bun run scripts/live-brain-first-check.ts --json
|
||||
* $OPENCLAW_WORKSPACE=~/.openclaw/workspace bun run scripts/live-brain-first-check.ts --fix-preview
|
||||
*
|
||||
* Exit codes:
|
||||
* 0 — workspace clean (no violators) OR workspace not configured
|
||||
* 1 — workspace configured + has violators (informational; no CI gate)
|
||||
* 2 — usage error (couldn't resolve workspace, doctor crash, etc.)
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
import { autoDetectSkillsDirReadOnly } from '../src/core/repo-root.ts';
|
||||
import { skillBrainFirstCheck } from '../src/commands/doctor.ts';
|
||||
import { autoFixDryViolations } from '../src/core/dry-fix.ts';
|
||||
import { parseSkillFrontmatter } from '../src/core/skill-frontmatter.ts';
|
||||
import {
|
||||
analyzeSkillBrainFirst,
|
||||
buildBrainFirstSummaryLine,
|
||||
FORMERLY_HARDCODED_EXEMPT,
|
||||
} from '../src/core/skill-brain-first.ts';
|
||||
import { loadOrDeriveManifest } from '../src/core/skill-manifest.ts';
|
||||
|
||||
function main(): number {
|
||||
const args = process.argv.slice(2);
|
||||
const jsonMode = args.includes('--json');
|
||||
const fixPreview = args.includes('--fix-preview');
|
||||
|
||||
if (!process.env.OPENCLAW_WORKSPACE && !process.env.GBRAIN_SKILLS_DIR) {
|
||||
process.stderr.write(
|
||||
'[live-brain-first] No skills dir source set. Set OPENCLAW_WORKSPACE or GBRAIN_SKILLS_DIR.\n' +
|
||||
' $OPENCLAW_WORKSPACE=~/.openclaw/workspace bun run scripts/live-brain-first-check.ts\n',
|
||||
);
|
||||
return 0; // not configured = not a failure
|
||||
}
|
||||
|
||||
const detected = autoDetectSkillsDirReadOnly();
|
||||
if (!detected || !detected.dir || !existsSync(detected.dir)) {
|
||||
process.stderr.write(
|
||||
`[live-brain-first] Could not resolve skills dir from OPENCLAW_WORKSPACE / GBRAIN_SKILLS_DIR / cwd walk-up.\n`,
|
||||
);
|
||||
return 2;
|
||||
}
|
||||
|
||||
process.stderr.write(`[live-brain-first] Scanning ${detected.dir} (source: ${detected.source})\n\n`);
|
||||
|
||||
const check = skillBrainFirstCheck(detected.dir);
|
||||
const violators = check.issues ?? [];
|
||||
|
||||
// Shape assertions for human eyeballs (per T1 from /plan-eng-review):
|
||||
// structurally-compliant skills MUST NOT appear in the violator list.
|
||||
// We surface these as informational warnings if any shape invariant
|
||||
// breaks; the script never fails on shape breach (that's the unit-test
|
||||
// suite's job).
|
||||
const shapeWarnings: string[] = [];
|
||||
|
||||
// Walk manifest once to collect classification info for the report.
|
||||
const manifest = loadOrDeriveManifest(detected.dir);
|
||||
const compliantViaCallout: string[] = [];
|
||||
const exemptByFrontmatter: string[] = [];
|
||||
const exemptByNoExternal: string[] = [];
|
||||
const flaggedFormerly: string[] = [];
|
||||
const flaggedNew: string[] = [];
|
||||
|
||||
for (const entry of manifest.skills) {
|
||||
const skillPath = join(detected.dir, entry.path);
|
||||
if (!existsSync(skillPath)) continue;
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(skillPath, 'utf-8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const a = analyzeSkillBrainFirst(content, entry.name, fm);
|
||||
if (a.status === 'ok') {
|
||||
if (a.reason === 'compliant_callout' || a.reason === 'compliant_phase' || a.reason === 'compliant_position') {
|
||||
compliantViaCallout.push(a.skill);
|
||||
} else if (a.reason === 'exempt_explicit') {
|
||||
exemptByFrontmatter.push(a.skill);
|
||||
} else if (a.reason === 'exempt_no_external') {
|
||||
exemptByNoExternal.push(a.skill);
|
||||
}
|
||||
} else {
|
||||
if (a.formerly_hardcoded_exempt) flaggedFormerly.push(a.skill);
|
||||
else flaggedNew.push(a.skill);
|
||||
}
|
||||
}
|
||||
|
||||
if (jsonMode) {
|
||||
const report = {
|
||||
schema_version: 1,
|
||||
workspace: detected.dir,
|
||||
source: detected.source,
|
||||
status: check.status,
|
||||
total_skills: manifest.skills.length,
|
||||
compliant_via_callout: compliantViaCallout.sort(),
|
||||
exempt_by_frontmatter: exemptByFrontmatter.sort(),
|
||||
exempt_by_no_external: exemptByNoExternal.sort(),
|
||||
flagged_formerly_exempt: flaggedFormerly.sort(),
|
||||
flagged_new: flaggedNew.sort(),
|
||||
total_violators: violators.length,
|
||||
};
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
} else {
|
||||
console.log(`Workspace: ${detected.dir}`);
|
||||
console.log(`Source: ${detected.source}`);
|
||||
console.log(`Total skills scanned: ${manifest.skills.length}`);
|
||||
console.log('');
|
||||
console.log(`Compliant via callout/phase/position: ${compliantViaCallout.length}`);
|
||||
console.log(`Exempt by frontmatter (brain_first: exempt): ${exemptByFrontmatter.length}`);
|
||||
console.log(`Exempt by no-external-pattern: ${exemptByNoExternal.length}`);
|
||||
console.log('');
|
||||
console.log(`Flagged (formerly hardcoded-exempt in PR #1206): ${flaggedFormerly.length}`);
|
||||
if (flaggedFormerly.length > 0) {
|
||||
for (const s of flaggedFormerly.sort()) console.log(` - ${s}`);
|
||||
}
|
||||
console.log('');
|
||||
console.log(`Flagged (genuinely new violators): ${flaggedNew.length}`);
|
||||
if (flaggedNew.length > 0) {
|
||||
for (const s of flaggedNew.sort()) console.log(` - ${s}`);
|
||||
}
|
||||
console.log('');
|
||||
|
||||
if (violators.length === 0) {
|
||||
console.log('STATUS: ok — no brain-first violators in the live deployment');
|
||||
} else {
|
||||
console.log('STATUS: warn — fix with:');
|
||||
console.log(' gbrain doctor --fix # auto-add canonical Convention callout (writes files)');
|
||||
console.log(' gbrain doctor --fix --dry-run # preview without writing');
|
||||
console.log(' or add `brain_first: exempt` to each flagged skill\'s frontmatter');
|
||||
}
|
||||
}
|
||||
|
||||
// --fix-preview: also run autoFixDryViolations against the live workspace
|
||||
// in dry-run mode to show what callouts the auto-fix would insert.
|
||||
if (fixPreview) {
|
||||
process.stderr.write('\n[live-brain-first] Running auto-fix dry-run preview...\n');
|
||||
const report = autoFixDryViolations(detected.dir, { dryRun: true });
|
||||
const brainFirstProposed = report.fixed.filter(
|
||||
f => f.status === 'proposed' && f.patternLabel === 'brain-first compliance',
|
||||
);
|
||||
if (brainFirstProposed.length === 0) {
|
||||
console.log('No brain-first auto-fix proposals.');
|
||||
} else {
|
||||
console.log(`\nWould insert canonical Convention callout into ${brainFirstProposed.length} skill(s):`);
|
||||
for (const p of brainFirstProposed.slice(0, 10)) {
|
||||
console.log(` - ${p.skill}`);
|
||||
}
|
||||
if (brainFirstProposed.length > 10) {
|
||||
console.log(` ... and ${brainFirstProposed.length - 10} more`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return violators.length > 0 ? 1 : 0;
|
||||
}
|
||||
|
||||
process.exit(main());
|
||||
@@ -73,3 +73,46 @@ If you spawn your own sub-agents, include this line in their task prompt:
|
||||
> Read `skills/conventions/brain-first.md` before starting work.
|
||||
|
||||
This ensures the convention propagates through any depth of sub-agent chain.
|
||||
|
||||
## Declarative opt-out (v0.36.x)
|
||||
|
||||
A skill can declare it does not need brain-first by adding this line to its
|
||||
frontmatter:
|
||||
|
||||
brain_first: exempt
|
||||
|
||||
Use this for pure-infra skills (cron schedulers, container managers,
|
||||
ask-user prompters, browser drivers) whose entire job is to operate without
|
||||
consulting the brain. The doctor `skill_brain_first` check honors this opt-
|
||||
out; the `gbrain doctor --fix` auto-add of the canonical Convention callout
|
||||
skips opted-out skills.
|
||||
|
||||
**Strict canonical form (the parser is loud about typos):**
|
||||
|
||||
| Form | Result |
|
||||
|---|---|
|
||||
| `brain_first: exempt` | ✅ matches |
|
||||
| `brain-first: exempt` | ⚠ doctor hint — snake_case required |
|
||||
| `BrainFirst: exempt` | ⚠ doctor hint — snake_case required |
|
||||
| `brain_first: "exempt"` | ⚠ doctor hint — drop the quotes |
|
||||
| `brain_first: Exempt` | ⚠ doctor hint — value must be lowercase |
|
||||
| `brain_first: required` | ⚠ doctor hint — only `exempt` is supported in v0.36 |
|
||||
|
||||
A near-miss prints a paste-ready fix line and the skill stays flagged
|
||||
until the canonical form lands. Silent typos would be the worst outcome
|
||||
("I declared exempt and it still flags!"), so the parser refuses to guess.
|
||||
|
||||
**You do NOT need to declare `brain_first: exempt` when:**
|
||||
|
||||
- The skill ALREADY includes the canonical Convention callout above
|
||||
(this file's path). The compliance check matches `> **Convention:**`
|
||||
blockquotes referencing `brain-first.md` and short-circuits to OK.
|
||||
`brain-ops`, `signal-detector`, `idea-ingest`, `enrich`,
|
||||
`perplexity-research`, and `academic-verify` all pass via this path.
|
||||
- The skill has no external-lookup references at all (`web_search`,
|
||||
`exa`, `perplexity`, `happenstance`, `crustdata`, `captain-api`,
|
||||
`firecrawl`). Trivially exempt.
|
||||
|
||||
When in doubt: declare `brain_first: exempt` explicitly OR add the
|
||||
canonical Convention callout near the top of the skill body. Both are
|
||||
zero-friction one-line operations.
|
||||
|
||||
@@ -27,6 +27,11 @@ tools:
|
||||
- write
|
||||
- edit
|
||||
mutating: true
|
||||
# This skill names other skills (perplexity-research, brain-publish,
|
||||
# etc.) in its dispatcher prose; the v0.36.x brain-first regex matches
|
||||
# the word `perplexity` but the skill never actually calls external
|
||||
# APIs. It rewrites local routing tables. Declarative opt-out.
|
||||
brain_first: exempt
|
||||
---
|
||||
|
||||
# Functional-Area Resolver — Pattern for Compressing Routing Tables
|
||||
|
||||
@@ -13,6 +13,12 @@ writes_pages: true
|
||||
writes_to:
|
||||
- concepts/
|
||||
- projects/
|
||||
# Brain-first analyzer matches `web_fetch` in a diagram on line 106
|
||||
# (illustrating fetch shape, not an API call) and `perplexity` in a
|
||||
# cross-reference to perplexity-research. The skill itself works on
|
||||
# uploaded source text + writes brain pages; it doesn't call external
|
||||
# APIs. Declarative opt-out.
|
||||
brain_first: exempt
|
||||
---
|
||||
|
||||
# strategic-reading — Applied Analysis from Source Texts
|
||||
|
||||
@@ -4,6 +4,19 @@ import { LATEST_VERSION, getIdleBlockers } from '../core/migrate.ts';
|
||||
import { checkResolvable } from '../core/check-resolvable.ts';
|
||||
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
|
||||
import { autoDetectSkillsDirReadOnly } from '../core/repo-root.ts';
|
||||
import { loadOrDeriveManifest } from '../core/skill-manifest.ts';
|
||||
import { parseSkillFrontmatter } from '../core/skill-frontmatter.ts';
|
||||
import {
|
||||
analyzeSkillBrainFirst,
|
||||
buildBrainFirstSummaryLine,
|
||||
type BrainFirstAnalysis,
|
||||
} from '../core/skill-brain-first.ts';
|
||||
import {
|
||||
loadSnapshot,
|
||||
writeSnapshotAtomically,
|
||||
diffAgainstSnapshot,
|
||||
appendAuditEventsForTransitions,
|
||||
} from '../core/audit-skill-brain-first.ts';
|
||||
import { loadCompletedMigrations } from '../core/preferences.ts';
|
||||
import { compareVersions } from './migrations/index.ts';
|
||||
import { createProgress, startHeartbeat, type ProgressReporter } from '../core/progress.ts';
|
||||
@@ -1466,6 +1479,21 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
checks.push(conformanceResult);
|
||||
}
|
||||
|
||||
// 2b. Skill brain-first compliance (v0.36.x, supersedes PR #1206).
|
||||
// Scans every SKILL.md for external-lookup tools (web_search, exa,
|
||||
// perplexity, etc.) and warns when the skill doesn't declare
|
||||
// `brain_first: exempt` AND doesn't carry a canonical Convention
|
||||
// callout / Phase 1 brain heading / position-relative brain-first
|
||||
// reference. Motivated by the 2026-05-19 tweet-shield incident.
|
||||
//
|
||||
// Audit trail: snapshot+diff at ~/.gbrain/audit/skill-brain-first-
|
||||
// snapshot.json. Writes one detected/resolved JSONL line per state
|
||||
// transition + one fixed line per applied --fix. Stable brain → zero
|
||||
// audit writes per doctor run.
|
||||
if (skillsDir) {
|
||||
checks.push(skillBrainFirstCheck(skillsDir));
|
||||
}
|
||||
|
||||
// 3. Half-migrated Minions detection (filesystem-only).
|
||||
// If completed.jsonl has any status:"partial" entry with no later
|
||||
// status:"complete" for the same version, the install is mid-migration.
|
||||
@@ -3606,6 +3634,149 @@ function checkSkillConformance(skillsDir: string): Check {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36.x skill_brain_first doctor check (supersedes PR #1206).
|
||||
*
|
||||
* Walks the skills manifest, runs the pure `analyzeSkillBrainFirst()`
|
||||
* helper on each, surfaces violators with structured issues[]. Snapshot-
|
||||
* diff against the previous run drives audit JSONL writes (transition-
|
||||
* only) — stable brains produce zero audit churn per doctor invocation.
|
||||
*
|
||||
* Exit shape:
|
||||
* - 0 violators → status: 'ok', message: '<n> skills compliant or exempt'
|
||||
* - any violator → status: 'warn', message + per-skill summary lines +
|
||||
* formerly-EXEMPT_SKILLS hint when applicable (CMT1 replaces the
|
||||
* dropped upgrade migration with a guided opt-in)
|
||||
*
|
||||
* Test seam: pure function, no `process.exit`. Direct call from tests
|
||||
* with a synthetic skills dir under tempdir.
|
||||
*/
|
||||
export function skillBrainFirstCheck(skillsDir: string): Check {
|
||||
let manifest: ReturnType<typeof loadOrDeriveManifest>;
|
||||
try {
|
||||
manifest = loadOrDeriveManifest(skillsDir);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
return {
|
||||
name: 'skill_brain_first',
|
||||
status: 'warn',
|
||||
message: `Could not load skills manifest from ${skillsDir} (${msg})`,
|
||||
};
|
||||
}
|
||||
if (manifest.skills.length === 0) {
|
||||
return {
|
||||
name: 'skill_brain_first',
|
||||
status: 'ok',
|
||||
message: 'No skills found — skill_brain_first not applicable',
|
||||
};
|
||||
}
|
||||
|
||||
const violators: BrainFirstAnalysis[] = [];
|
||||
const typoSkills: BrainFirstAnalysis[] = [];
|
||||
|
||||
for (const entry of manifest.skills) {
|
||||
const skillPath = join(skillsDir, entry.path);
|
||||
if (!existsSync(skillPath)) continue; // resolver_health already reports
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(skillPath, 'utf-8');
|
||||
} catch {
|
||||
continue; // best-effort; permissions etc.
|
||||
}
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, entry.name, fm);
|
||||
if (result.typo_hint) typoSkills.push(result);
|
||||
if (result.status === 'warn') violators.push(result);
|
||||
}
|
||||
|
||||
// --- Snapshot + diff audit (A2 contract) ---------------------------------
|
||||
// Best-effort: snapshot/audit failures don't poison the check result.
|
||||
const violatorSlugs = new Set(violators.map(v => v.skill));
|
||||
const patternsBySlug = new Map<string, string[]>();
|
||||
for (const v of violators) {
|
||||
patternsBySlug.set(v.skill, v.external_patterns_matched);
|
||||
}
|
||||
let priorSnapshotPresent = true;
|
||||
try {
|
||||
const snapshot = loadSnapshot();
|
||||
priorSnapshotPresent = snapshot.present;
|
||||
const diff = diffAgainstSnapshot(violatorSlugs, snapshot.violators);
|
||||
const doctorRunId = `${process.pid}-${Date.now()}`;
|
||||
if (snapshot.present) {
|
||||
// Steady-state path: write events only for transitions.
|
||||
appendAuditEventsForTransitions(diff, patternsBySlug, doctorRunId);
|
||||
} else {
|
||||
// First run / corrupt snapshot: bootstrap by writing one
|
||||
// `detected` line per current violator. This is the only path
|
||||
// that writes more than `diff.added.length` lines in a single
|
||||
// doctor invocation.
|
||||
const bootstrapDiff = { added: Array.from(violatorSlugs).sort(), removed: [], unchanged: [] };
|
||||
appendAuditEventsForTransitions(bootstrapDiff, patternsBySlug, doctorRunId);
|
||||
}
|
||||
writeSnapshotAtomically(violatorSlugs);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[gbrain] skill_brain_first audit step failed (${msg}); check continues\n`);
|
||||
}
|
||||
|
||||
// --- Build the check result ---------------------------------------------
|
||||
if (violators.length === 0) {
|
||||
const typoNote = typoSkills.length > 0
|
||||
? ` (note: ${typoSkills.length} skill(s) have brain_first typo hints: ${typoSkills.map(t => t.skill).join(', ')})`
|
||||
: '';
|
||||
return {
|
||||
name: 'skill_brain_first',
|
||||
status: 'ok',
|
||||
message: `${manifest.skills.length} skill(s) compliant or exempt${typoNote}`,
|
||||
};
|
||||
}
|
||||
|
||||
// Sort for deterministic message + issues order.
|
||||
violators.sort((a, b) => a.skill.localeCompare(b.skill));
|
||||
|
||||
const formerlyExempt = violators.filter(v => v.formerly_hardcoded_exempt);
|
||||
const summary: string[] = [];
|
||||
summary.push(
|
||||
`${violators.length} skill(s) do external lookups without a brain-first compliance signal. ` +
|
||||
`Fix via 'gbrain doctor --fix' (adds canonical Convention callout) ` +
|
||||
`or set 'brain_first: exempt' in skill frontmatter for genuine infra skills.`,
|
||||
);
|
||||
if (formerlyExempt.length > 0) {
|
||||
summary.push(
|
||||
`Of these, ${formerlyExempt.length} were hardcoded-exempt in PR #1206 (${formerlyExempt.map(v => v.skill).slice(0, 6).join(', ')}${formerlyExempt.length > 6 ? ', ...' : ''}). ` +
|
||||
`These need explicit opt-out now: run 'gbrain doctor --fix' to add the canonical callout, ` +
|
||||
`or add 'brain_first: exempt' to frontmatter for skills that genuinely shouldn't consult the brain.`,
|
||||
);
|
||||
}
|
||||
if (typoSkills.length > 0) {
|
||||
summary.push(
|
||||
`${typoSkills.length} skill(s) have brain_first typo hints: ` +
|
||||
typoSkills.slice(0, 6).map(t => `${t.skill} — ${t.typo_hint}`).join('; ') +
|
||||
(typoSkills.length > 6 ? '; ...' : ''),
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
name: 'skill_brain_first',
|
||||
status: 'warn',
|
||||
message: summary.join(' '),
|
||||
issues: violators.map(v => ({
|
||||
type: 'skill_missing_brain_first',
|
||||
skill: v.skill,
|
||||
action: v.formerly_hardcoded_exempt
|
||||
? `Add canonical Convention callout OR set 'brain_first: exempt' (was hardcoded-exempt in PR #1206)`
|
||||
: `Add canonical Convention callout OR set 'brain_first: exempt'`,
|
||||
fix: {
|
||||
kind: 'add-convention-callout',
|
||||
external_patterns: v.external_patterns_matched,
|
||||
typo_hint: v.typo_hint,
|
||||
formerly_hardcoded_exempt: v.formerly_hardcoded_exempt,
|
||||
summary_line: buildBrainFirstSummaryLine(v),
|
||||
},
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
function outputResults(checks: Check[], json: boolean): boolean {
|
||||
const hasFail = checks.some(c => c.status === 'fail');
|
||||
const hasWarn = checks.some(c => c.status === 'warn');
|
||||
|
||||
@@ -33,6 +33,11 @@ import {
|
||||
findReceiptForSkill,
|
||||
type ReceiptStatus,
|
||||
} from '../core/cross-modal-eval/receipt-name.ts';
|
||||
import { parseSkillFrontmatter } from '../core/skill-frontmatter.ts';
|
||||
import {
|
||||
analyzeSkillBrainFirst,
|
||||
buildBrainFirstSummaryLine,
|
||||
} from '../core/skill-brain-first.ts';
|
||||
|
||||
interface CheckItem {
|
||||
name: string;
|
||||
@@ -283,6 +288,17 @@ function runSkillifyCheckTarget(target: string, root: string): CheckResult {
|
||||
),
|
||||
);
|
||||
|
||||
// Item 12: brain-first compliance (v0.36.x, REQUIRED — A3 + F9 from
|
||||
// /plan-eng-review). Skills that reference external lookup tools
|
||||
// (web_search, exa, perplexity, etc.) must declare brain-first stance
|
||||
// explicitly: either via the canonical Convention callout, or via
|
||||
// 'brain_first: exempt' in frontmatter, or by absence of external
|
||||
// refs entirely. This is a REQUIRED gate — non-compliant skills fail
|
||||
// `gbrain skillify check` with exit 1 so new skills can't be born
|
||||
// non-compliant.
|
||||
const brainFirst = checkBrainFirstCompliance(skillMd, skillName);
|
||||
items.push(check('Brain-first compliance', brainFirst.passed, brainFirst.detail));
|
||||
|
||||
const passed = items.filter(i => i.passed).length;
|
||||
const total = items.length;
|
||||
const missing = items.filter(i => !i.passed && i.required).map(i => i.name);
|
||||
@@ -339,6 +355,50 @@ function lookupCrossModalReceipt(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Item 12 helper: brain-first compliance gate.
|
||||
*
|
||||
* Reads the skill's SKILL.md, parses frontmatter, runs the pure
|
||||
* `analyzeSkillBrainFirst()` analyzer. A `warn` status fails this
|
||||
* required item; an `ok` status (any exemption or compliance reason)
|
||||
* passes.
|
||||
*
|
||||
* When SKILL.md doesn't exist, the check passes — item 1 already
|
||||
* reported the missing file; we don't pile-on with a second failure.
|
||||
*
|
||||
* Detail string follows the same shape as the doctor check message
|
||||
* (via `buildBrainFirstSummaryLine`) so the two surfaces stay
|
||||
* consistent for skill authors learning the contract.
|
||||
*/
|
||||
function checkBrainFirstCompliance(
|
||||
skillMdPath: string,
|
||||
skillName: string,
|
||||
): { passed: boolean; detail: string } {
|
||||
if (!existsSync(skillMdPath)) {
|
||||
return { passed: true, detail: 'no SKILL.md — covered by item 1' };
|
||||
}
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(skillMdPath, 'utf-8');
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
// Read failure is best-effort — don't double-fail; item 1 / 2 would
|
||||
// already report the problem. Pass with a note.
|
||||
return { passed: true, detail: `could not read SKILL.md: ${msg}` };
|
||||
}
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const analysis = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
if (analysis.status === 'ok') {
|
||||
return { passed: true, detail: `${analysis.reason} (${skillName})` };
|
||||
}
|
||||
return {
|
||||
passed: false,
|
||||
detail:
|
||||
buildBrainFirstSummaryLine(analysis) +
|
||||
` — Fix: add canonical Convention callout (see conventions/brain-first.md), or set 'brain_first: exempt' in frontmatter.`,
|
||||
};
|
||||
}
|
||||
|
||||
function recentlyModified(root: string, days: number = 7): string[] {
|
||||
const candidates: string[] = [];
|
||||
const cutoff = Date.now() - days * 24 * 60 * 60 * 1000;
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
/**
|
||||
* audit-skill-brain-first.ts — Snapshot+diff audit trail for the v0.36.x
|
||||
* `skill_brain_first` doctor check (A2 + F11 + F12 from /plan-eng-review).
|
||||
*
|
||||
* **Why snapshot+diff, not append-every-run:** doctor runs ~20-50x/day on
|
||||
* a working brain (autopilot cycle, dev flow, CI). Writing every detected
|
||||
* violation per run produces ~2K lines/day of churn on a 42-violator
|
||||
* deployment — pure noise, no trend signal. Instead:
|
||||
*
|
||||
* - Load the last-known snapshot of violator slugs.
|
||||
* - Diff against the current detection.
|
||||
* - Write JSONL audit lines ONLY for transitions (added/removed slugs).
|
||||
* - On `--fix` apply, write a `fixed` event per applied fix.
|
||||
* - Write the new snapshot atomically.
|
||||
*
|
||||
* Result: stable brain produces 0 audit writes per doctor run. The audit
|
||||
* log becomes signal, not noise. `tail -20 audit-YYYY-Www.jsonl` shows
|
||||
* real events.
|
||||
*
|
||||
* **Race handling (F12):** concurrent doctor runs (autopilot + dev + CI)
|
||||
* are real. `writeSnapshotAtomically` uses an mkstemp-style unique tmpfile
|
||||
* + `rename()`. Last-writer-wins is the explicit semantic — concurrent
|
||||
* snapshot writes do NOT collude. Acceptable for read-mostly audit data:
|
||||
* the worst case is one doctor run misses a transition because another
|
||||
* run rewrote the snapshot first; the next run reconciles.
|
||||
*
|
||||
* Mirrors the ISO-week filename pattern from `audit-slug-fallback.ts` and
|
||||
* `minions/handlers/supervisor-audit.ts` so all gbrain audit channels share
|
||||
* one rotation discipline.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { resolveAuditDir } from './minions/handlers/shell-audit.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type BrainFirstAuditEventKind = 'detected' | 'resolved' | 'fixed';
|
||||
|
||||
export interface BrainFirstAuditEvent {
|
||||
ts: string;
|
||||
event: BrainFirstAuditEventKind;
|
||||
skill: string;
|
||||
/** External-lookup patterns matched at the moment of the event. */
|
||||
external_patterns?: string[];
|
||||
/** Optional run correlation id (e.g. `${pid}-${startEpochMs}`). */
|
||||
doctor_run_id?: string;
|
||||
/** Stable code consumed by future doctor `skill_brain_first_trend` check. */
|
||||
code: 'SKILL_BRAIN_FIRST';
|
||||
severity: 'info';
|
||||
}
|
||||
|
||||
export interface SnapshotDiff {
|
||||
added: string[]; // slugs newly in violation
|
||||
removed: string[]; // slugs no longer in violation (resolved / removed)
|
||||
unchanged: string[]; // slugs in both (no audit write)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// ISO-week filename helpers (parity with audit-slug-fallback.ts)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* `skill-brain-first-YYYY-Www.jsonl` — ISO-8601 week math, identical to
|
||||
* `computeSlugFallbackAuditFilename()` so file rotation is consistent
|
||||
* across all audit channels.
|
||||
*/
|
||||
export function computeBrainFirstAuditFilename(now: Date = new Date()): string {
|
||||
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
|
||||
const dayNum = (d.getUTCDay() + 6) % 7;
|
||||
d.setUTCDate(d.getUTCDate() - dayNum + 3);
|
||||
const isoYear = d.getUTCFullYear();
|
||||
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
|
||||
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
|
||||
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
|
||||
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
|
||||
const ww = String(weekNum).padStart(2, '0');
|
||||
return `skill-brain-first-${isoYear}-W${ww}.jsonl`;
|
||||
}
|
||||
|
||||
const SNAPSHOT_FILENAME = 'skill-brain-first-snapshot.json';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Snapshot I/O
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface SnapshotFile {
|
||||
/** Schema version — bump when shape changes. */
|
||||
schema_version: 1;
|
||||
/** ISO-8601 timestamp of when the snapshot was last written. */
|
||||
written_at: string;
|
||||
/** Sorted array of violator slugs (canonical for diff stability). */
|
||||
violators: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Load the snapshot file. Returns an empty Set + `present: false` flag
|
||||
* when the file is missing OR corrupt JSON. Callers (doctor) use the
|
||||
* `present` flag to decide whether to bootstrap-write all current
|
||||
* violators as `detected` events on first run.
|
||||
*/
|
||||
export function loadSnapshot(): { violators: Set<string>; present: boolean } {
|
||||
const file = path.join(resolveAuditDir(), SNAPSHOT_FILENAME);
|
||||
let content: string;
|
||||
try {
|
||||
content = fs.readFileSync(file, 'utf8');
|
||||
} catch {
|
||||
return { violators: new Set(), present: false };
|
||||
}
|
||||
try {
|
||||
const parsed = JSON.parse(content) as SnapshotFile;
|
||||
if (
|
||||
typeof parsed !== 'object' || parsed === null ||
|
||||
!Array.isArray(parsed.violators)
|
||||
) {
|
||||
// Corrupt shape — once-per-process warn, treat as missing.
|
||||
warnOnce(`[gbrain] snapshot corrupt: ${file} (unexpected shape); treating as missing`);
|
||||
return { violators: new Set(), present: false };
|
||||
}
|
||||
const slugs = new Set<string>();
|
||||
for (const v of parsed.violators) {
|
||||
if (typeof v === 'string' && v.length > 0) slugs.add(v);
|
||||
}
|
||||
return { violators: slugs, present: true };
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
warnOnce(`[gbrain] snapshot corrupt: ${file} (${msg}); treating as missing`);
|
||||
return { violators: new Set(), present: false };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the added/removed/unchanged diff against the previous snapshot.
|
||||
* Pure function (no I/O) so callers can compose it with caching layers.
|
||||
*/
|
||||
export function diffAgainstSnapshot(
|
||||
current: Set<string>,
|
||||
previous: Set<string>,
|
||||
): SnapshotDiff {
|
||||
const added: string[] = [];
|
||||
const removed: string[] = [];
|
||||
const unchanged: string[] = [];
|
||||
for (const slug of current) {
|
||||
if (previous.has(slug)) unchanged.push(slug);
|
||||
else added.push(slug);
|
||||
}
|
||||
for (const slug of previous) {
|
||||
if (!current.has(slug)) removed.push(slug);
|
||||
}
|
||||
added.sort();
|
||||
removed.sort();
|
||||
unchanged.sort();
|
||||
return { added, removed, unchanged };
|
||||
}
|
||||
|
||||
/**
|
||||
* Write the snapshot atomically. Uses a process-unique tmpfile suffix
|
||||
* (`<pid>-<epoch_ms>-<random>`) followed by `rename()`, the standard
|
||||
* POSIX atomic-replace idiom.
|
||||
*
|
||||
* **Last-writer-wins (F12):** two concurrent doctor runs may both write
|
||||
* snapshots in arbitrary order; whichever lands last sticks. The
|
||||
* intervening run's transitions are correctly captured in its own
|
||||
* append-only JSONL — only the snapshot state is "lost," and the next
|
||||
* run reconciles by diffing against the now-current snapshot. This is
|
||||
* documented behavior, not a bug; the alternative (cross-process file
|
||||
* locking) is overkill for read-mostly audit data.
|
||||
*
|
||||
* Write failures log to stderr but never throw — audit is best-effort.
|
||||
*/
|
||||
export function writeSnapshotAtomically(violators: Set<string>, now: Date = new Date()): void {
|
||||
const dir = resolveAuditDir();
|
||||
const finalPath = path.join(dir, SNAPSHOT_FILENAME);
|
||||
const tmpSuffix = `${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
|
||||
const tmpPath = path.join(dir, `${SNAPSHOT_FILENAME}.tmp.${tmpSuffix}`);
|
||||
const sorted = Array.from(violators).sort();
|
||||
const payload: SnapshotFile = {
|
||||
schema_version: 1,
|
||||
written_at: now.toISOString(),
|
||||
violators: sorted,
|
||||
};
|
||||
try {
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(tmpPath, JSON.stringify(payload, null, 2) + '\n', { encoding: 'utf8' });
|
||||
fs.renameSync(tmpPath, finalPath);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[gbrain] snapshot write failed (${msg}); doctor continues\n`);
|
||||
// Best-effort cleanup of the tmpfile if rename failed mid-flight.
|
||||
try { fs.unlinkSync(tmpPath); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Audit JSONL writer
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Append a brain-first audit event to the current week's JSONL.
|
||||
*
|
||||
* Best-effort — write failures log to stderr but never throw. Doctor
|
||||
* continues even if the audit dir is read-only or the disk is full.
|
||||
*/
|
||||
export function logBrainFirstEvent(
|
||||
partial: Omit<BrainFirstAuditEvent, 'ts' | 'severity' | 'code'>,
|
||||
now: Date = new Date(),
|
||||
): void {
|
||||
const event: BrainFirstAuditEvent = {
|
||||
ts: now.toISOString(),
|
||||
severity: 'info',
|
||||
code: 'SKILL_BRAIN_FIRST',
|
||||
...partial,
|
||||
};
|
||||
const dir = resolveAuditDir();
|
||||
const file = path.join(dir, computeBrainFirstAuditFilename(now));
|
||||
try {
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.appendFileSync(file, JSON.stringify(event) + '\n', { encoding: 'utf8' });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[gbrain] brain-first audit write failed (${msg}); doctor continues\n`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Append `detected`/`resolved` events for an entire diff in one call.
|
||||
* Bounded: writes at most `diff.added.length + diff.removed.length` lines.
|
||||
* A no-transition diff writes nothing (the A2 contract).
|
||||
*/
|
||||
export function appendAuditEventsForTransitions(
|
||||
diff: SnapshotDiff,
|
||||
patternsBySlug: Map<string, string[]>,
|
||||
doctor_run_id?: string,
|
||||
now: Date = new Date(),
|
||||
): void {
|
||||
for (const slug of diff.added) {
|
||||
logBrainFirstEvent({
|
||||
event: 'detected',
|
||||
skill: slug,
|
||||
external_patterns: patternsBySlug.get(slug),
|
||||
doctor_run_id,
|
||||
}, now);
|
||||
}
|
||||
for (const slug of diff.removed) {
|
||||
logBrainFirstEvent({
|
||||
event: 'resolved',
|
||||
skill: slug,
|
||||
doctor_run_id,
|
||||
}, now);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Reader (consumed by future skill_brain_first_trend doctor check, TODO-2)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Read recent (`days` window, default 7) brain-first audit events from the
|
||||
* current + previous ISO-week JSONLs. Missing files / corrupt rows are
|
||||
* skipped silently — audit reads are forensic, not blocking.
|
||||
*/
|
||||
export function readRecentBrainFirstEvents(
|
||||
days = 7,
|
||||
now: Date = new Date(),
|
||||
): BrainFirstAuditEvent[] {
|
||||
const dir = resolveAuditDir();
|
||||
const cutoff = now.getTime() - days * 86400000;
|
||||
const out: BrainFirstAuditEvent[] = [];
|
||||
const filenames = [
|
||||
computeBrainFirstAuditFilename(now),
|
||||
computeBrainFirstAuditFilename(new Date(now.getTime() - 7 * 86400000)),
|
||||
];
|
||||
for (const filename of filenames) {
|
||||
const file = path.join(dir, filename);
|
||||
let content: string;
|
||||
try {
|
||||
content = fs.readFileSync(file, 'utf8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const line of content.split('\n')) {
|
||||
if (line.length === 0) continue;
|
||||
try {
|
||||
const ev = JSON.parse(line) as BrainFirstAuditEvent;
|
||||
const ts = Date.parse(ev.ts);
|
||||
if (Number.isFinite(ts) && ts >= cutoff) out.push(ev);
|
||||
} catch {
|
||||
// Corrupt row — skip.
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Once-per-process warn (corrupt snapshot bootstrap path)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const _warnedSet = new Set<string>();
|
||||
|
||||
function warnOnce(message: string): void {
|
||||
if (_warnedSet.has(message)) return;
|
||||
_warnedSet.add(message);
|
||||
process.stderr.write(`${message}\n`);
|
||||
}
|
||||
|
||||
/** Test-only: reset the once-per-process warning gate. */
|
||||
export function _resetWarnedSetForTests(): void {
|
||||
_warnedSet.clear();
|
||||
}
|
||||
+246
-40
@@ -14,8 +14,7 @@
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join } from 'path';
|
||||
import {
|
||||
CROSS_CUTTING_PATTERNS,
|
||||
DRY_PROXIMITY_LINES,
|
||||
@@ -23,6 +22,17 @@ import {
|
||||
type CrossCuttingPattern,
|
||||
} from './check-resolvable.ts';
|
||||
import { loadOrDeriveManifest } from './skill-manifest.ts';
|
||||
import {
|
||||
getWorkingTreeStatus as _getWorkingTreeStatus,
|
||||
isInsideCodeFence as _isInsideCodeFence,
|
||||
isWorkingTreeDirty as _isWorkingTreeDirty,
|
||||
type WorkingTreeStatus as _WorkingTreeStatus,
|
||||
} from './skill-fix-gates.ts';
|
||||
import { parseSkillFrontmatter } from './skill-frontmatter.ts';
|
||||
import {
|
||||
analyzeSkillBrainFirst,
|
||||
CONVENTION_CALLOUT_RE,
|
||||
} from './skill-brain-first.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
@@ -60,6 +70,60 @@ export interface AutoFixReport {
|
||||
skipped: FixOutcome[]; // skips and errors
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// MISSING_RULE pattern type — v0.36.x INSERT-new-callout flow (T2 + T4)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* INSERT-missing-rule pattern type. Sibling of `CrossCuttingPattern` but
|
||||
* with INSERT semantics instead of REPLACE:
|
||||
* - `detect` decides whether THIS skill is missing the rule.
|
||||
* - `callout` is the literal Convention-callout line to insert.
|
||||
* - `idempotentCheck` decides whether the rule is ALREADY present
|
||||
* (so we don't double-insert on re-runs).
|
||||
*
|
||||
* Insertion site: `findInsertionLine(content)` (after frontmatter close
|
||||
* `---`, after first H1 paragraph if present, before first H2). The
|
||||
* shared safety gates (working-tree, code-fence, install-path) apply
|
||||
* exactly the same way as for REPLACE patterns.
|
||||
*/
|
||||
export interface MissingRulePattern {
|
||||
/** Stable label for reporting (e.g. 'brain-first compliance'). */
|
||||
label: string;
|
||||
/** Return true when the rule is MISSING for this skill (needs insert). */
|
||||
detect: (content: string, skillName: string) => boolean;
|
||||
/** Return true when this skill already declares the rule (skip insert). */
|
||||
idempotentCheck: (content: string) => boolean;
|
||||
/** The literal callout line to insert. */
|
||||
callout: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36.x missing-rule patterns. Currently one entry — the brain-first
|
||||
* Convention callout, motivated by the 2026-05-19 tweet-shield incident
|
||||
* (no model knew Garry built Palantir's Finance UI; brain did).
|
||||
*
|
||||
* The detector calls `analyzeSkillBrainFirst()` (the pure helper) so the
|
||||
* detector here, the doctor check, and the skillify-check gate all share
|
||||
* the same compliance ladder. One source of truth.
|
||||
*
|
||||
* The callout shape matches the existing compliant skills (brain-ops,
|
||||
* perplexity-research, academic-verify) — `> **Convention:** see
|
||||
* conventions/brain-first.md ...` with a brief explanation of the lookup
|
||||
* chain so a reader landing on it knows what to do next.
|
||||
*/
|
||||
export const MISSING_RULE_PATTERNS: MissingRulePattern[] = [
|
||||
{
|
||||
label: 'brain-first compliance',
|
||||
detect: (content, skillName) => {
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
return analyzeSkillBrainFirst(content, skillName, fm).status === 'warn';
|
||||
},
|
||||
idempotentCheck: (content) => CONVENTION_CALLOUT_RE.test(content),
|
||||
callout: '> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) for the lookup chain (search → query → get_page → external).',
|
||||
},
|
||||
];
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Block-expansion strategy map
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -146,46 +210,20 @@ export const expanders: Record<BlockShape, (lines: string[], lineIdx: number) =>
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Guards
|
||||
// Guards (re-exported from src/core/skill-fix-gates.ts for back-compat)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** True when the match offset sits inside a fenced code block (``` ... ```).
|
||||
* Counts triple-backtick fences at line starts. Odd count = inside. */
|
||||
export function isInsideCodeFence(content: string, offset: number): boolean {
|
||||
const before = content.slice(0, offset);
|
||||
const fenceRe = /^```/gm;
|
||||
const fenceCount = (before.match(fenceRe) || []).length;
|
||||
return fenceCount % 2 === 1;
|
||||
}
|
||||
|
||||
export type WorkingTreeStatus = 'clean' | 'dirty' | 'not_a_repo';
|
||||
|
||||
/** Check the git state of a skill file. Three distinct outcomes — callers
|
||||
* must NOT conflate "not a repo" with "clean", because the auto-fix
|
||||
* contract is "git is the backup" and writing to a file outside any repo
|
||||
* destroys user data with no recovery path.
|
||||
*
|
||||
* `execFileSync` with array args bypasses the shell entirely, so paths
|
||||
* with odd characters from a manifest can't inject commands. */
|
||||
export function getWorkingTreeStatus(skillPath: string): WorkingTreeStatus {
|
||||
try {
|
||||
const out = execFileSync('git', ['status', '--porcelain', '--', skillPath], {
|
||||
encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
cwd: dirname(skillPath),
|
||||
});
|
||||
return out.trim().length > 0 ? 'dirty' : 'clean';
|
||||
} catch {
|
||||
// git exits 128 when not inside a repo; treat any non-zero the same.
|
||||
return 'not_a_repo';
|
||||
}
|
||||
}
|
||||
|
||||
/** Legacy wrapper. Callers that need to distinguish not_a_repo from clean
|
||||
* should use getWorkingTreeStatus() directly. */
|
||||
export function isWorkingTreeDirty(skillPath: string): boolean {
|
||||
return getWorkingTreeStatus(skillPath) === 'dirty';
|
||||
}
|
||||
/**
|
||||
* v0.36.x extracted the working-tree + code-fence safety primitives to
|
||||
* `src/core/skill-fix-gates.ts` so both REPLACE (CROSS_CUTTING_PATTERNS)
|
||||
* and INSERT (MISSING_RULE_PATTERNS) auto-fix flows can share them. The
|
||||
* re-exports below preserve the public symbol names for tests + external
|
||||
* callers that imported these from `dry-fix.ts` directly.
|
||||
*/
|
||||
export const isInsideCodeFence = _isInsideCodeFence;
|
||||
export const getWorkingTreeStatus = _getWorkingTreeStatus;
|
||||
export const isWorkingTreeDirty = _isWorkingTreeDirty;
|
||||
export type WorkingTreeStatus = _WorkingTreeStatus;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Manifest loading delegated to src/core/skill-manifest.ts. Using the
|
||||
@@ -256,11 +294,179 @@ export function autoFixDryViolations(
|
||||
skipped.push(outcome);
|
||||
}
|
||||
}
|
||||
|
||||
// v0.36.x INSERT-missing-rule patterns. Run AFTER REPLACE so a
|
||||
// freshly-inserted Convention callout from REPLACE doesn't get a
|
||||
// second INSERT layered on top by the brain-first detector.
|
||||
// (Belt + suspenders: brain-first's detect() reads the current file
|
||||
// content and calls analyzeSkillBrainFirst, which already short-
|
||||
// circuits on CONVENTION_CALLOUT_RE match.)
|
||||
for (const mrp of MISSING_RULE_PATTERNS) {
|
||||
const outcome = attemptInsertFix(skill.name, skillPath, content, mrp, opts);
|
||||
if (!outcome) continue;
|
||||
if (outcome.status === 'applied' || outcome.status === 'proposed') {
|
||||
fixed.push(outcome);
|
||||
if (outcome.status === 'applied') {
|
||||
try {
|
||||
content = readFileSync(skillPath, 'utf-8');
|
||||
} catch {
|
||||
break;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
skipped.push(outcome);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { fixed, skipped };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// INSERT expander — v0.36.x missing-rule auto-fix
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Find the line index at which to insert a new Convention callout.
|
||||
*
|
||||
* Insertion strategy `after-h1-paragraph`:
|
||||
* 1. After frontmatter closing `---`
|
||||
* 2. After the first `# Title` H1 if present
|
||||
* 3. After the leading paragraph following the H1 if present
|
||||
* 4. Before the first `## H2` heading
|
||||
* 5. Fallback: append at body end if no H2 exists
|
||||
*
|
||||
* Returns a 0-indexed line number where the new callout should be
|
||||
* inserted. Callers splice `[callout, ''] + ` at this position.
|
||||
*
|
||||
* Exported for unit tests.
|
||||
*/
|
||||
export function findInsertionLine(content: string): number {
|
||||
const lines = content.split('\n');
|
||||
let cursor = 0;
|
||||
|
||||
// Step 1: Skip leading frontmatter fence if present.
|
||||
if (lines[0] === '---') {
|
||||
for (let i = 1; i < lines.length; i++) {
|
||||
if (lines[i] === '---') {
|
||||
cursor = i + 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Step 2: Skip blank lines after frontmatter.
|
||||
while (cursor < lines.length && lines[cursor].trim() === '') cursor++;
|
||||
|
||||
// Step 3: If there's a leading H1, advance past it.
|
||||
if (cursor < lines.length && /^#\s+/.test(lines[cursor])) {
|
||||
cursor++;
|
||||
// Step 4: Skip blank lines + the leading paragraph following the H1.
|
||||
while (cursor < lines.length && lines[cursor].trim() === '') cursor++;
|
||||
// Paragraph: contiguous non-blank, non-heading, non-fence lines.
|
||||
while (
|
||||
cursor < lines.length &&
|
||||
lines[cursor].trim() !== '' &&
|
||||
!/^##+\s+/.test(lines[cursor]) &&
|
||||
!/^---\s*$/.test(lines[cursor])
|
||||
) {
|
||||
cursor++;
|
||||
}
|
||||
// Skip the trailing blank lines after the leading paragraph.
|
||||
while (cursor < lines.length && lines[cursor].trim() === '') cursor++;
|
||||
}
|
||||
|
||||
// Step 5: Cursor is now at first H2 OR end of file. Either way, insert here.
|
||||
return cursor;
|
||||
}
|
||||
|
||||
/**
|
||||
* Attempt an INSERT-missing-rule fix for one skill+pattern.
|
||||
*
|
||||
* Mirrors `attemptFix()` (the REPLACE-in-place path) for safety gates but
|
||||
* applies INSERT semantics: refuses when the rule is already present,
|
||||
* inserts at `findInsertionLine(content)` otherwise.
|
||||
*
|
||||
* Returns:
|
||||
* - null when the detector decides this skill doesn't need
|
||||
* the rule (skip silently — not every skill needs every
|
||||
* missing-rule pattern).
|
||||
* - 'skipped' outcome with reason when a safety gate blocks the write.
|
||||
* - 'proposed' outcome (dryRun) with before/after preview.
|
||||
* - 'applied' outcome on successful write.
|
||||
* - 'error' outcome on write failure.
|
||||
*/
|
||||
function attemptInsertFix(
|
||||
skillName: string,
|
||||
skillPath: string,
|
||||
content: string,
|
||||
mrp: MissingRulePattern,
|
||||
opts: AutoFixOptions
|
||||
): FixOutcome | null {
|
||||
const base = {
|
||||
skill: skillName,
|
||||
skillPath,
|
||||
patternLabel: mrp.label,
|
||||
};
|
||||
|
||||
// Detector gate: does this skill NEED the rule inserted?
|
||||
if (!mrp.detect(content, skillName)) return null;
|
||||
|
||||
// Idempotency: is the rule already declared somehow? Belt+suspenders;
|
||||
// detect() should already short-circuit, but double-check at the
|
||||
// insertion gate so a future detector that misses callout cases doesn't
|
||||
// produce double-inserts.
|
||||
if (mrp.idempotentCheck(content)) {
|
||||
return { ...base, status: 'skipped', reason: 'already_delegated' };
|
||||
}
|
||||
|
||||
// Safety gates (shared with REPLACE path).
|
||||
const treeStatus = getWorkingTreeStatus(skillPath);
|
||||
if (treeStatus === 'dirty') {
|
||||
return { ...base, status: 'skipped', reason: 'working_tree_dirty' };
|
||||
}
|
||||
if (treeStatus === 'not_a_repo') {
|
||||
return { ...base, status: 'skipped', reason: 'no_git_backup' };
|
||||
}
|
||||
|
||||
// Compute insertion site.
|
||||
const insertAt = findInsertionLine(content);
|
||||
const lines = content.split('\n');
|
||||
|
||||
// Build the new file content: splice [callout, ''] at insertAt.
|
||||
// The blank line after the callout keeps the surrounding block
|
||||
// structure readable.
|
||||
const before = lines.slice(0, insertAt);
|
||||
const after = lines.slice(insertAt);
|
||||
const inserted = [...before, mrp.callout, '', ...after];
|
||||
let next = inserted.join('\n');
|
||||
if (content.endsWith('\n') && !next.endsWith('\n')) {
|
||||
next += '\n';
|
||||
}
|
||||
|
||||
if (opts.dryRun) {
|
||||
return {
|
||||
...base,
|
||||
status: 'proposed',
|
||||
before: '(no prior block — inserting new callout)',
|
||||
after: mrp.callout,
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
writeFileSync(skillPath, next, 'utf-8');
|
||||
} catch {
|
||||
return { ...base, status: 'error', reason: 'write_error' };
|
||||
}
|
||||
|
||||
return {
|
||||
...base,
|
||||
status: 'applied',
|
||||
before: '(no prior block — inserted new callout)',
|
||||
after: mrp.callout,
|
||||
};
|
||||
}
|
||||
|
||||
function attemptFix(
|
||||
skillName: string,
|
||||
skillPath: string,
|
||||
|
||||
+26
-35
@@ -24,6 +24,7 @@
|
||||
|
||||
import { existsSync, readFileSync, readdirSync, statSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { parseSkillFrontmatter } from './skill-frontmatter.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
@@ -112,6 +113,19 @@ function normalizeDir(dir: string): string {
|
||||
// Skill frontmatter parsing (minimal, tolerant)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Public surface preserved for back-compat: SkillFrontmatter remains a
|
||||
* narrow alias here, but the underlying parser now lives in
|
||||
* `skill-frontmatter.ts` (`parseSkillFrontmatter`). The wider
|
||||
* `ParsedFrontmatter` type from that module is structurally compatible
|
||||
* with this narrower one — every field on SkillFrontmatter is optional
|
||||
* and present on ParsedFrontmatter.
|
||||
*
|
||||
* If you're writing new code, import `parseSkillFrontmatter` and
|
||||
* `ParsedFrontmatter` from `./skill-frontmatter.ts` directly. This
|
||||
* thin wrapper exists so existing filing-audit callers don't need to
|
||||
* be touched.
|
||||
*/
|
||||
export interface SkillFrontmatter {
|
||||
name?: string;
|
||||
writes_pages?: boolean;
|
||||
@@ -127,41 +141,18 @@ function parseFrontmatter(skillMdPath: string): SkillFrontmatter | null {
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const fmMatch = content.match(/^---\n([\s\S]*?)\n---/);
|
||||
if (!fmMatch) return null;
|
||||
const raw = fmMatch[1];
|
||||
const out: SkillFrontmatter = { raw };
|
||||
|
||||
const nameMatch = raw.match(/^name:\s*["']?([^"'\n]+?)["']?\s*$/m);
|
||||
if (nameMatch) out.name = nameMatch[1].trim();
|
||||
|
||||
const wpMatch = raw.match(/^writes_pages:\s*(true|false)\s*$/m);
|
||||
if (wpMatch) out.writes_pages = wpMatch[1] === 'true';
|
||||
|
||||
const mutMatch = raw.match(/^mutating:\s*(true|false)\s*$/m);
|
||||
if (mutMatch) out.mutating = mutMatch[1] === 'true';
|
||||
|
||||
// writes_to: supports inline `[a, b, c]` OR multi-line block list
|
||||
// writes_to:
|
||||
// - people/
|
||||
// - companies/
|
||||
// AND inline `writes_to: [people/, companies/]`
|
||||
const inlineWtMatch = raw.match(/^writes_to:\s*\[([^\]]*)\]\s*$/m);
|
||||
if (inlineWtMatch) {
|
||||
out.writes_to = inlineWtMatch[1]
|
||||
.split(',')
|
||||
.map(s => s.trim().replace(/^["']|["']$/g, ''))
|
||||
.filter(Boolean);
|
||||
} else {
|
||||
const blockMatch = raw.match(/^writes_to:\s*\n((?:\s+-\s+[^\n]+\n?)+)/m);
|
||||
if (blockMatch) {
|
||||
out.writes_to = blockMatch[1]
|
||||
.split('\n')
|
||||
.map(l => l.replace(/^\s+-\s+/, '').replace(/^["']|["']$/g, '').trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
const parsed = parseSkillFrontmatter(content);
|
||||
if (!parsed) return null;
|
||||
// Project the wider ParsedFrontmatter onto the narrower SkillFrontmatter
|
||||
// shape filing-audit callers expect. Field order matches the original
|
||||
// shape so tests that compare object keys via JSON.stringify stay stable.
|
||||
return {
|
||||
raw: parsed.raw,
|
||||
name: parsed.name,
|
||||
writes_pages: parsed.writes_pages,
|
||||
writes_to: parsed.writes_to,
|
||||
mutating: parsed.mutating,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,399 @@
|
||||
/**
|
||||
* skill-brain-first.ts — Pure analyzer for the v0.36.x `skill_brain_first`
|
||||
* doctor check + skillify-check gate + dry-fix `--fix` MISSING_RULE pattern.
|
||||
*
|
||||
* Motivation (tweet-shield incident, 2026-05-19): cross-modal eval flagged
|
||||
* Garry's Palantir tweet as risky because no model knew he built it. The
|
||||
* brain already had "designed the entire Finance product UI" and "150+
|
||||
* PSDs from April-December 2006." Brain-first compliance — search the brain
|
||||
* before any external API — would have caught it. This module catches
|
||||
* authors who didn't declare brain-first or didn't opt out explicitly.
|
||||
*
|
||||
* One pure function, three consumers (Q1 from /plan-eng-review):
|
||||
* - `runDoctor()` maps it across the manifest for `skill_brain_first`
|
||||
* - `skillify-check` calls it on a single SKILL.md as required item 12
|
||||
* - `dry-fix.ts` MISSING_RULE_PATTERNS calls it as the "should I insert
|
||||
* a Convention callout here?" gate
|
||||
*
|
||||
* Exemption order (top wins):
|
||||
* 1. Frontmatter `brain_first: exempt` → exempt_explicit
|
||||
* 2. No external-lookup pattern in body → exempt_no_external
|
||||
* 3. Otherwise apply compliance detection (3-tier ladder)
|
||||
*
|
||||
* Compliance ladder (any one passes):
|
||||
* a. Canonical `> **Convention:** ... brain-first ...` callout → compliant_callout
|
||||
* b. Explicit `## Phase 1 [brain]` or `## Step 0 [brain]` heading → compliant_phase
|
||||
* c. First brain reference offset < first external reference offset → compliant_position
|
||||
*
|
||||
* **NOTE (CMT2 from /plan-eng-review):** there is NO `tools + writes_pages`
|
||||
* structural exemption. Skills like `idea-ingest`, `meeting-ingestion`,
|
||||
* and `data-research` write pages AND call external APIs — the exact mixed
|
||||
* class brain-first targets. They get flagged so authors declare stance,
|
||||
* not hidden behind a structural rule. Skills that genuinely ARE the brain
|
||||
* (`brain-ops`, `signal-detector`) exempt via compliance detection because
|
||||
* they already carry the canonical callout — no structural rule needed.
|
||||
*
|
||||
* **NOTE (F6 from /plan-eng-review):** all position-relative scanning is
|
||||
* BODY-ONLY. YAML frontmatter is excluded from offset comparison so a
|
||||
* `tools: [web_search]` declaration doesn't false-flag the skill (the
|
||||
* declaration is metadata, not execution). The body extractor returns the
|
||||
* content starting AFTER the closing `---` of the frontmatter fence.
|
||||
*
|
||||
* **NOTE (F7 from /plan-eng-review):** the canonical-callout regex anchors
|
||||
* on the literal `> **Convention:**` + `brain-first` substring, agnostic to
|
||||
* path syntax (backtick / markdown-link / plain text). The three existing
|
||||
* compliant skills (brain-ops, perplexity-research, academic-verify) use
|
||||
* plain-text paths, so this is the load-bearing detection shape, not
|
||||
* `extractDelegationTargets` which only matches backtick paths.
|
||||
*/
|
||||
|
||||
import type { ParsedFrontmatter } from './skill-frontmatter.ts';
|
||||
import { formatBrainFirstTypoHint } from './skill-frontmatter.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type BrainFirstReason =
|
||||
| 'exempt_explicit' // brain_first: exempt in frontmatter
|
||||
| 'exempt_no_external' // no external-lookup pattern present in body
|
||||
| 'compliant_callout' // canonical > **Convention:** ... brain-first ... callout
|
||||
| 'compliant_phase' // explicit ## Phase 1 / Step 0 brain heading
|
||||
| 'compliant_position' // first brain ref appears before first external ref (body)
|
||||
| 'missing_brain_first'; // external pattern + no compliance signal
|
||||
|
||||
export interface BrainFirstAnalysis {
|
||||
/** Stable identifier — the manifest entry name OR the dir name. */
|
||||
skill: string;
|
||||
/** OK if any exemption or compliance path matched; warn otherwise. */
|
||||
status: 'ok' | 'warn';
|
||||
/** Why the analyzer landed where it did. Drives the doctor message. */
|
||||
reason: BrainFirstReason;
|
||||
/**
|
||||
* The literal pattern names that matched external-lookup detection.
|
||||
* Empty when `exempt_no_external` (no external pattern present).
|
||||
* Populated even on compliance paths so callers can surface what the
|
||||
* skill is calling (e.g. "perplexity + exa, compliant via callout").
|
||||
*/
|
||||
external_patterns_matched: string[];
|
||||
/**
|
||||
* Paste-ready typo hint when the frontmatter has a near-miss
|
||||
* declaration (`brain-first: exempt`, `BrainFirst: Exempt`, quoted
|
||||
* values, etc.). Surfaced in the doctor message and skillify-check
|
||||
* error output.
|
||||
*/
|
||||
typo_hint?: string;
|
||||
/**
|
||||
* True when this skill was in the v0.36.x-and-earlier PR #1206
|
||||
* hardcoded EXEMPT_SKILLS allowlist. The doctor message appends a
|
||||
* dedicated hint for these on first detection (CMT1 from plan review:
|
||||
* replaces the dropped upgrade migration with a guided opt-in via
|
||||
* `gbrain doctor --fix`).
|
||||
*/
|
||||
formerly_hardcoded_exempt: boolean;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pattern constants (exported for shared use in tests + dry-fix.ts detect)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* External-lookup tools that trigger the brain-first compliance gate.
|
||||
* Each entry pairs a stable identifier (for telemetry) with a regex.
|
||||
*
|
||||
* Word-boundary anchored so a comment about `web_search_history` won't
|
||||
* match `web_search`. Case-insensitive because skill authors capitalize
|
||||
* inconsistently (`Perplexity`, `PERPLEXITY`, `perplexity`).
|
||||
*
|
||||
* Captain API specifically allows `captain api`, `captain_api`,
|
||||
* `captain-api`, and `captainapi` because the actual product is
|
||||
* referenced with all four shapes across skills.
|
||||
*/
|
||||
export const EXTERNAL_LOOKUP_PATTERNS: ReadonlyArray<{ name: string; re: RegExp }> = [
|
||||
{ name: 'web_search', re: /\bweb_search\b/i },
|
||||
{ name: 'web_fetch', re: /\bweb_fetch\b/i },
|
||||
{ name: 'exa', re: /\bexa[\s._-]/i },
|
||||
{ name: 'perplexity', re: /\bperplexity\b/i },
|
||||
{ name: 'happenstance', re: /\bhappenstance\b/i },
|
||||
{ name: 'crustdata', re: /\bcrustdata\b/i },
|
||||
{ name: 'captain_api', re: /\bcaptain[\s._-]?api\b/i },
|
||||
{ name: 'firecrawl', re: /\bfirecrawl\b/i },
|
||||
];
|
||||
|
||||
/**
|
||||
* Brain-reference patterns that signal compliance. These match the
|
||||
* canonical brain-tool invocations developers actually write.
|
||||
*/
|
||||
export const BRAIN_REFERENCE_PATTERNS: ReadonlyArray<RegExp> = [
|
||||
/\bgbrain[\s_]+search\b/i,
|
||||
/\bgbrain[\s_]+query\b/i,
|
||||
/\bgbrain[\s_]+get[_-]?page\b/i,
|
||||
/\bgbrain[\s_]+find[_-]?experts\b/i,
|
||||
/\bgbrain[\s_]+get[_-]?backlinks\b/i,
|
||||
/\bgbrain[\s_]+get[_-]?timeline\b/i,
|
||||
/\bgbrain[\s_]+traverse[_-]?graph\b/i,
|
||||
// Bare tool names (subagent context, OpenClaw plugin form):
|
||||
/\bsearch\s+the\s+brain\b/i,
|
||||
/\bquery\s+the\s+brain\b/i,
|
||||
/\bcheck\s+the\s+brain\b/i,
|
||||
];
|
||||
|
||||
/**
|
||||
* Canonical Convention callout regex. Anchored at start-of-line on a
|
||||
* blockquote that contains BOTH the literal `**Convention:**` marker
|
||||
* AND the `brain-first` substring (with optional `.md` extension).
|
||||
*
|
||||
* Path syntax is intentionally ignored — the three existing compliant
|
||||
* skills use varied forms (`> **Convention:** see skills/conventions/
|
||||
* brain-first.md ...`, `> **Convention:** see conventions/brain-first.md
|
||||
* ...`, `> **Convention:** see [conventions/brain-first.md](...) ...`).
|
||||
* The literal `**Convention:**` + `brain-first` is the load-bearing
|
||||
* signal, not the path shape.
|
||||
*/
|
||||
export const CONVENTION_CALLOUT_RE = /^>\s*\*\*Convention:\*\*[^\n]*brain-first/im;
|
||||
|
||||
/**
|
||||
* Explicit phase-heading regex. Matches `## Phase 1: Brain-First Lookup`,
|
||||
* `### Step 0: Brain Context`, etc. — any H2+ heading that names a brain
|
||||
* phase as step 0 or phase 1.
|
||||
*/
|
||||
export const PHASE_HEADING_RE = /^##+\s*(?:Phase\s*1|Step\s*0)\b[^\n]*brain/im;
|
||||
|
||||
/**
|
||||
* Frontmatter fence regex used by body extraction. Conservative match:
|
||||
* leading `---\n` through the next `\n---` (greedy stop). Matches the
|
||||
* shape `parseSkillFrontmatter` already accepts.
|
||||
*/
|
||||
const FRONTMATTER_RE = /^---\n[\s\S]*?\n---\n?/;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Hardcoded EXEMPT_SKILLS (CMT1 — replaces the dropped upgrade migration)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Skills that were in PR #1206's hardcoded EXEMPT_SKILLS allowlist
|
||||
* (committed to v0.36.x-and-earlier `doctor.ts`). The list is
|
||||
* preserved here ONLY for the doctor hint flow: when a skill in this
|
||||
* set newly flags after v0.36.x ships the structural-signal exemption,
|
||||
* the doctor message guides the user to either:
|
||||
* 1. `gbrain doctor --fix` to auto-add the canonical callout, OR
|
||||
* 2. add `brain_first: exempt` to frontmatter (if genuinely infra)
|
||||
*
|
||||
* The list is informational — removing it later doesn't break
|
||||
* compliance detection. It's a guided opt-in surface, NOT an
|
||||
* exemption rule.
|
||||
*
|
||||
* Per CMT1 from /plan-eng-review (codex direction): doctor surfaces +
|
||||
* `--fix` applies via dry-fix safety gates, replacing the originally-
|
||||
* planned silent migration. User stays in the loop.
|
||||
*
|
||||
* Source: PR #1206 `feature/doctor-brain-first-check`,
|
||||
* `src/commands/doctor.ts` (pre-supersede).
|
||||
*/
|
||||
export const FORMERLY_HARDCODED_EXEMPT: ReadonlySet<string> = new Set([
|
||||
// Brain-internal skills (PR rationale: "ARE the brain")
|
||||
'brain-ops', 'brain-commit', 'brain-enrichment-pipeline', 'brain-export',
|
||||
'brain-ingest-gate', 'brain-librarian', 'brain-link-refs', 'brain-link-report',
|
||||
'brain-pdf', 'brain-pdf-auto', 'brain-plan', 'brain-publish', 'brain-storage',
|
||||
'brain-storage-links', 'brain-taxonomist',
|
||||
'gbrain', 'gbrain-pr', 'gbrain-upgrade', 'benchmark-gbrain',
|
||||
// External-tool wrappers (their entire job IS external lookup)
|
||||
'exa', 'happenstance', 'crustdata', 'captain-api',
|
||||
// Pure-infra skills (system, not knowledge)
|
||||
'healthcheck', 'backblaze', 'browser', 'browser-use', 'binary-deps',
|
||||
'captcha-solver', 'container-restart', 'durable-service', 'data-loss-gate',
|
||||
'channel-discovery', 'clawvisor', 'clawvisor-shield',
|
||||
'cron-scheduler', 'cronify', 'correction-pipeline',
|
||||
'acknowledge', 'ask-user', 'backoff',
|
||||
'acp-coding', 'code-pr', 'skill-creator', 'ingest', 'freshness-monitor',
|
||||
]);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pure analyzer
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Analyze a single SKILL.md for brain-first compliance. Pure function —
|
||||
* no I/O, no side effects, no random clocks. Drives the doctor check,
|
||||
* the skillify-check gate, and the dry-fix MISSING_RULE detector.
|
||||
*
|
||||
* @param content — raw SKILL.md content (incl. frontmatter)
|
||||
* @param skillName — stable identifier (manifest entry name or dir name)
|
||||
* @param frontmatter — pre-parsed frontmatter from `parseSkillFrontmatter()`,
|
||||
* or null if the skill has no YAML fence (treated as
|
||||
* empty frontmatter for analysis purposes)
|
||||
*/
|
||||
export function analyzeSkillBrainFirst(
|
||||
content: string,
|
||||
skillName: string,
|
||||
frontmatter: ParsedFrontmatter | null,
|
||||
): BrainFirstAnalysis {
|
||||
const formerly = FORMERLY_HARDCODED_EXEMPT.has(skillName);
|
||||
const typo_hint = frontmatter?.brain_first_typo
|
||||
? formatBrainFirstTypoHint(frontmatter.brain_first_typo) ?? undefined
|
||||
: undefined;
|
||||
|
||||
// Exemption 1: explicit declarative opt-out
|
||||
if (frontmatter?.brain_first === 'exempt') {
|
||||
return {
|
||||
skill: skillName,
|
||||
status: 'ok',
|
||||
reason: 'exempt_explicit',
|
||||
external_patterns_matched: [],
|
||||
typo_hint,
|
||||
formerly_hardcoded_exempt: formerly,
|
||||
};
|
||||
}
|
||||
|
||||
// Body extraction — strip frontmatter so a `tools: [web_search]`
|
||||
// declaration in YAML doesn't false-flag the skill (F6 from review).
|
||||
const body = stripFrontmatter(content);
|
||||
|
||||
// Scan body for external-lookup patterns.
|
||||
const external_patterns_matched = EXTERNAL_LOOKUP_PATTERNS
|
||||
.filter(p => p.re.test(body))
|
||||
.map(p => p.name);
|
||||
|
||||
// Exemption 2: no external pattern present anywhere in body. Trivially
|
||||
// doesn't need brain-first — the skill never reaches for external data.
|
||||
if (external_patterns_matched.length === 0) {
|
||||
return {
|
||||
skill: skillName,
|
||||
status: 'ok',
|
||||
reason: 'exempt_no_external',
|
||||
external_patterns_matched: [],
|
||||
typo_hint,
|
||||
formerly_hardcoded_exempt: formerly,
|
||||
};
|
||||
}
|
||||
|
||||
// External pattern present → apply compliance ladder.
|
||||
|
||||
// Compliance a: canonical Convention callout referencing brain-first.
|
||||
if (CONVENTION_CALLOUT_RE.test(body)) {
|
||||
return {
|
||||
skill: skillName,
|
||||
status: 'ok',
|
||||
reason: 'compliant_callout',
|
||||
external_patterns_matched,
|
||||
typo_hint,
|
||||
formerly_hardcoded_exempt: formerly,
|
||||
};
|
||||
}
|
||||
|
||||
// Compliance b: explicit Phase 1 / Step 0 brain heading.
|
||||
if (PHASE_HEADING_RE.test(body)) {
|
||||
return {
|
||||
skill: skillName,
|
||||
status: 'ok',
|
||||
reason: 'compliant_phase',
|
||||
external_patterns_matched,
|
||||
typo_hint,
|
||||
formerly_hardcoded_exempt: formerly,
|
||||
};
|
||||
}
|
||||
|
||||
// Compliance c: first brain reference appears BEFORE first external
|
||||
// reference in body. Position-relative ordering — body-only by
|
||||
// construction (we're scanning `body`, not `content`).
|
||||
const firstBrainOffset = findFirstBrainRefOffset(body);
|
||||
const firstExternalOffset = findFirstExternalRefOffset(body);
|
||||
if (
|
||||
firstBrainOffset !== -1 &&
|
||||
firstExternalOffset !== -1 &&
|
||||
firstBrainOffset < firstExternalOffset
|
||||
) {
|
||||
return {
|
||||
skill: skillName,
|
||||
status: 'ok',
|
||||
reason: 'compliant_position',
|
||||
external_patterns_matched,
|
||||
typo_hint,
|
||||
formerly_hardcoded_exempt: formerly,
|
||||
};
|
||||
}
|
||||
|
||||
// Otherwise: external pattern present, no compliance signal. Warn.
|
||||
return {
|
||||
skill: skillName,
|
||||
status: 'warn',
|
||||
reason: 'missing_brain_first',
|
||||
external_patterns_matched,
|
||||
typo_hint,
|
||||
formerly_hardcoded_exempt: formerly,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Internal helpers (exported for unit tests)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Return the SKILL.md content with the leading YAML frontmatter fence
|
||||
* stripped. If no fence is present, returns the input unchanged.
|
||||
*
|
||||
* Critical for position-relative scanning (F6): `tools: [web_search]`
|
||||
* in frontmatter must not count as the "first external reference."
|
||||
*/
|
||||
export function stripFrontmatter(content: string): string {
|
||||
return content.replace(FRONTMATTER_RE, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* Offset (0-indexed) of the first brain-reference match in body, or -1.
|
||||
* Scans through BRAIN_REFERENCE_PATTERNS in declared order; returns the
|
||||
* MINIMUM matching offset across all patterns (so multi-pattern hits
|
||||
* still surface the earliest).
|
||||
*/
|
||||
export function findFirstBrainRefOffset(body: string): number {
|
||||
let min = -1;
|
||||
for (const re of BRAIN_REFERENCE_PATTERNS) {
|
||||
const m = body.match(re);
|
||||
if (m && m.index !== undefined) {
|
||||
if (min === -1 || m.index < min) min = m.index;
|
||||
}
|
||||
}
|
||||
return min;
|
||||
}
|
||||
|
||||
/**
|
||||
* Offset (0-indexed) of the first external-reference match in body, or -1.
|
||||
* Sibling of findFirstBrainRefOffset; same minimum-across-patterns semantics.
|
||||
*/
|
||||
export function findFirstExternalRefOffset(body: string): number {
|
||||
let min = -1;
|
||||
for (const { re } of EXTERNAL_LOOKUP_PATTERNS) {
|
||||
const m = body.match(re);
|
||||
if (m && m.index !== undefined) {
|
||||
if (min === -1 || m.index < min) min = m.index;
|
||||
}
|
||||
}
|
||||
return min;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Message builder — used by doctor + skillify-check + dry-fix
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Build a human-readable per-skill summary line. Used by the doctor
|
||||
* message + skillify-check error output. Includes the typo hint and
|
||||
* the formerly-hardcoded-exempt note when applicable.
|
||||
*/
|
||||
export function buildBrainFirstSummaryLine(a: BrainFirstAnalysis): string {
|
||||
if (a.status === 'ok') {
|
||||
return `${a.skill}: ok (${a.reason})`;
|
||||
}
|
||||
const parts: string[] = [
|
||||
`${a.skill}: external lookup (${a.external_patterns_matched.join(', ')}) without brain-first compliance`,
|
||||
];
|
||||
if (a.formerly_hardcoded_exempt) {
|
||||
parts.push(
|
||||
`(was hardcoded-exempt in PR #1206 — opt out explicitly via 'brain_first: exempt' or run 'gbrain doctor --fix' to add the canonical callout)`,
|
||||
);
|
||||
}
|
||||
if (a.typo_hint) {
|
||||
parts.push(`(typo: ${a.typo_hint})`);
|
||||
}
|
||||
return parts.join(' ');
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* skill-fix-gates.ts — Shared safety primitives consumed by every auto-fix
|
||||
* pattern in `dry-fix.ts` (F8 from /plan-eng-review).
|
||||
*
|
||||
* Pre-v0.36.x the dry-fix module had REPLACE-in-place semantics only
|
||||
* (`CROSS_CUTTING_PATTERNS`). v0.36.x adds INSERT-missing-rule semantics
|
||||
* for the brain-first auto-add (`MISSING_RULE_PATTERNS`). Both share the
|
||||
* exact same safety gates:
|
||||
*
|
||||
* - **working-tree check** — refuse writes when the file has uncommitted
|
||||
* changes (the git-is-backup contract).
|
||||
* - **not-a-repo check** — refuse writes when the file isn't under git
|
||||
* (writing would destroy the only copy with no rollback).
|
||||
* - **inside-code-fence check** — don't mangle example prose inside
|
||||
* ``` fences.
|
||||
*
|
||||
* Extracting these as a sibling module is cleaner than duplicating across
|
||||
* REPLACE and INSERT pattern handlers in `dry-fix.ts`. Tests import from
|
||||
* here directly; `dry-fix.ts` re-exports for back-compat with callers that
|
||||
* imported these functions from there pre-v0.36.x.
|
||||
*
|
||||
* The D6 install-path safety gate (refuse `--fix` when skills dir came
|
||||
* from the install-path fallback) lives in `doctor.ts` because it's
|
||||
* specific to the doctor `--fix` flow and consults the
|
||||
* `autoDetectSkillsDirReadOnly` `detected.source` field that doesn't
|
||||
* appear in dry-fix's own argument surface.
|
||||
*/
|
||||
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { dirname } from 'node:path';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Code-fence guard
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* True when the byte offset sits inside a fenced code block (``` ... ```).
|
||||
* Counts triple-backtick fences at line starts before `offset`. Odd count
|
||||
* = inside a fence; even count = outside.
|
||||
*
|
||||
* Used to skip pattern matches that fall inside example prose so the auto-
|
||||
* fix doesn't mangle code samples in skill documentation.
|
||||
*/
|
||||
export function isInsideCodeFence(content: string, offset: number): boolean {
|
||||
const before = content.slice(0, offset);
|
||||
const fenceRe = /^```/gm;
|
||||
const fenceCount = (before.match(fenceRe) || []).length;
|
||||
return fenceCount % 2 === 1;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Working-tree status (git check)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type WorkingTreeStatus = 'clean' | 'dirty' | 'not_a_repo';
|
||||
|
||||
/**
|
||||
* Check the git state of a skill file.
|
||||
*
|
||||
* Three outcomes are deliberately distinct:
|
||||
* - `clean` — file is tracked and has no uncommitted changes.
|
||||
* Safe to write; `git diff` after the write surfaces the change.
|
||||
* - `dirty` — file has uncommitted changes. Refuse to write — auto-fix
|
||||
* would mix its changes with the user's mid-edit work, and
|
||||
* `git stash` / `git checkout -- <file>` wouldn't cleanly
|
||||
* separate them.
|
||||
* - `not_a_repo` — file isn't under git at all. Refuse to write — there
|
||||
* is NO rollback path. The auto-fix contract is "git is
|
||||
* the backup," and writing here breaks the contract.
|
||||
*
|
||||
* `execFileSync` with array args bypasses the shell entirely so paths
|
||||
* with odd characters from a manifest can't inject commands. We change
|
||||
* to the file's parent dir so `git status --porcelain -- <path>`
|
||||
* resolves correctly even when the calling process's cwd is elsewhere.
|
||||
*/
|
||||
export function getWorkingTreeStatus(skillPath: string): WorkingTreeStatus {
|
||||
try {
|
||||
const out = execFileSync('git', ['status', '--porcelain', '--', skillPath], {
|
||||
encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
cwd: dirname(skillPath),
|
||||
});
|
||||
return out.trim().length > 0 ? 'dirty' : 'clean';
|
||||
} catch {
|
||||
// git exits 128 when not inside a repo; treat any non-zero the same.
|
||||
return 'not_a_repo';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Legacy wrapper. Callers that need to distinguish `not_a_repo` from
|
||||
* `clean` should use `getWorkingTreeStatus()` directly. The two-state
|
||||
* boolean here is preserved for the existing dry-fix call site that
|
||||
* was previously content with a coarse "dirty vs everything else"
|
||||
* check.
|
||||
*/
|
||||
export function isWorkingTreeDirty(skillPath: string): boolean {
|
||||
return getWorkingTreeStatus(skillPath) === 'dirty';
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
/**
|
||||
* skill-frontmatter.ts — Single content-based parser for SKILL.md YAML
|
||||
* frontmatter, shared by `filing-audit.ts` (writes_pages / writes_to audit)
|
||||
* and `skill-brain-first.ts` (brain_first compliance check).
|
||||
*
|
||||
* The pre-existing `parseFrontmatter` in `filing-audit.ts` was private +
|
||||
* path-based (took an absolute path, returned `SkillFrontmatter` or null).
|
||||
* Two problems for v0.36.x brain-first work:
|
||||
* 1. Path-based callers can't unit-test against in-memory fixtures
|
||||
* without temp directories. Content-based parsing keeps tests pure.
|
||||
* 2. The two consumers grew different field needs (`tools:` for brain-
|
||||
* first, `writes_pages`/`writes_to` for filing). One parser is the
|
||||
* right abstraction; duplicating it across modules guarantees drift.
|
||||
*
|
||||
* Behavior:
|
||||
* - Tolerant on unknown keys (returns what it parses; ignores the rest).
|
||||
* - STRICT on the v0.36.x `brain_first:` field: only the canonical
|
||||
* `brain_first: exempt` (lowercase snake_case key, lowercase unquoted
|
||||
* value) sets `brain_first`. Near-misses (`brain-first`, `BrainFirst`,
|
||||
* `brain_first: 'exempt'`) populate `brain_first_typo` for doctor to
|
||||
* surface as a paste-ready fix hint.
|
||||
* - Returns null when no frontmatter fence is present (no `---`).
|
||||
*
|
||||
* Strictness rationale (Q3 + F3 from /plan-eng-review 2026-05-19): silent
|
||||
* typos are the worst kind. A developer who writes `brain-first: exempt`
|
||||
* thinking it works deserves a loud hint pointing at the canonical form,
|
||||
* not a silent failure to exempt.
|
||||
*/
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface ParsedFrontmatter {
|
||||
/** Raw YAML between the `---` fences. Empty string when no fence found. */
|
||||
raw: string;
|
||||
/** Skill name; the `name:` field. */
|
||||
name?: string;
|
||||
/** Does this skill write brain pages? */
|
||||
writes_pages?: boolean;
|
||||
/** Allowed brain-page filing directories. */
|
||||
writes_to?: string[];
|
||||
/** Does this skill have side effects? Distinct from writes_pages. */
|
||||
mutating?: boolean;
|
||||
/** Skill tool inventory (e.g. ['search', 'query', 'put_page']). */
|
||||
tools?: string[];
|
||||
/** Routing triggers list. */
|
||||
triggers?: string[];
|
||||
/**
|
||||
* v0.36.x brain-first declarative opt-out. Only the literal canonical
|
||||
* value `'exempt'` (no quotes, lowercase, snake_case key) populates this.
|
||||
* Anything else is a typo and goes into `brain_first_typo`.
|
||||
*/
|
||||
brain_first?: 'exempt';
|
||||
/**
|
||||
* Surfaces near-miss declarations for the doctor typo hint. Examples
|
||||
* the typo detector catches:
|
||||
* - `brain-first: exempt` (kebab-case key)
|
||||
* - `BrainFirst: Exempt` (camelCase key, capitalized value)
|
||||
* - `brain_first: "exempt"` (quoted value)
|
||||
* - `brain_first: required` (unknown value — flagged so we can
|
||||
* communicate "only 'exempt' is supported")
|
||||
*/
|
||||
brain_first_typo?: {
|
||||
/** Original key as written by the author. */
|
||||
key: string;
|
||||
/** Original value as written by the author. */
|
||||
value: string;
|
||||
/** Why this isn't the canonical form. */
|
||||
reason: 'noncanonical_key' | 'quoted_value' | 'unknown_value' | 'capitalized_value';
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Public API
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Parse SKILL.md content. Returns null when no YAML frontmatter is found.
|
||||
*
|
||||
* Content-based (no I/O) so callers control how they load files. Pair with
|
||||
* `readFileSync(path, 'utf-8')` at the boundary.
|
||||
*/
|
||||
export function parseSkillFrontmatter(content: string): ParsedFrontmatter | null {
|
||||
const fmMatch = content.match(/^---\n([\s\S]*?)\n---/);
|
||||
if (!fmMatch) return null;
|
||||
const raw = fmMatch[1];
|
||||
const out: ParsedFrontmatter = { raw };
|
||||
|
||||
// --- name ---
|
||||
const nameMatch = raw.match(/^name:\s*["']?([^"'\n]+?)["']?\s*$/m);
|
||||
if (nameMatch) out.name = nameMatch[1].trim();
|
||||
|
||||
// --- writes_pages / mutating (booleans) ---
|
||||
const wpMatch = raw.match(/^writes_pages:\s*(true|false)\s*$/m);
|
||||
if (wpMatch) out.writes_pages = wpMatch[1] === 'true';
|
||||
|
||||
const mutMatch = raw.match(/^mutating:\s*(true|false)\s*$/m);
|
||||
if (mutMatch) out.mutating = mutMatch[1] === 'true';
|
||||
|
||||
// --- writes_to / tools / triggers (arrays, inline or block) ---
|
||||
out.writes_to = parseArrayField(raw, 'writes_to');
|
||||
out.tools = parseArrayField(raw, 'tools');
|
||||
out.triggers = parseArrayField(raw, 'triggers');
|
||||
|
||||
// --- brain_first (strict canonical) ---
|
||||
parseBrainFirst(raw, out);
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Parse an array-shaped YAML field that may appear inline (`field: [a, b]`)
|
||||
* or as a block list:
|
||||
* field:
|
||||
* - a
|
||||
* - b
|
||||
*
|
||||
* Returns undefined if the field is absent. Returns [] for an explicitly-
|
||||
* empty list (`field: []`).
|
||||
*
|
||||
* The block matcher uses `^\s+- ` so it won't capture comments or other
|
||||
* top-level fields below it. Stops at the first non-indented line.
|
||||
*/
|
||||
function parseArrayField(raw: string, field: string): string[] | undefined {
|
||||
// Inline form: `field: [a, b, c]` or `field: []`
|
||||
const inlineRe = new RegExp(`^${field}:\\s*\\[([^\\]]*)\\]\\s*$`, 'm');
|
||||
const inlineMatch = raw.match(inlineRe);
|
||||
if (inlineMatch) {
|
||||
const inner = inlineMatch[1].trim();
|
||||
if (inner.length === 0) return [];
|
||||
return inner
|
||||
.split(',')
|
||||
.map(s => s.trim().replace(/^["']|["']$/g, ''))
|
||||
.filter(Boolean);
|
||||
}
|
||||
// Block form: `field:` + indented `- value` lines on subsequent lines.
|
||||
const blockRe = new RegExp(`^${field}:\\s*\\n((?:[ \\t]+-[ \\t]+[^\\n]+\\n?)+)`, 'm');
|
||||
const blockMatch = raw.match(blockRe);
|
||||
if (blockMatch) {
|
||||
return blockMatch[1]
|
||||
.split('\n')
|
||||
.map(l => l.replace(/^[ \t]+-[ \t]+/, '').replace(/^["']|["']$/g, '').trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Strict canonical match for `brain_first:` field. Populates either
|
||||
* `out.brain_first = 'exempt'` (canonical hit) or `out.brain_first_typo`
|
||||
* (near-miss the developer probably meant to declare).
|
||||
*
|
||||
* The typo scan is intentionally permissive on the KEY so we catch
|
||||
* `brain-first`, `BrainFirst`, etc. — that's where the surprise lives.
|
||||
* But the canonical match is strict: only `brain_first: exempt` (case-
|
||||
* sensitive key, lowercase unquoted value) sets the typed field.
|
||||
*/
|
||||
function parseBrainFirst(raw: string, out: ParsedFrontmatter): void {
|
||||
// Scan every line that looks like a brain_first declaration (case-
|
||||
// insensitive key, any value). The first match wins; subsequent
|
||||
// duplicates are ignored (YAML loaders also take the last; we
|
||||
// pick the first for determinism and to make the typo hint specific
|
||||
// to what the developer wrote first).
|
||||
const typoRe = /^(brain[-_]?first)\s*:\s*(.+?)\s*$/im;
|
||||
const typoMatch = raw.match(typoRe);
|
||||
if (!typoMatch) return;
|
||||
|
||||
const key = typoMatch[1];
|
||||
const valueRaw = typoMatch[2];
|
||||
|
||||
// Canonical key + canonical value?
|
||||
if (key === 'brain_first' && valueRaw === 'exempt') {
|
||||
out.brain_first = 'exempt';
|
||||
return;
|
||||
}
|
||||
|
||||
// Key is wrong (case or separator).
|
||||
if (key !== 'brain_first') {
|
||||
out.brain_first_typo = {
|
||||
key,
|
||||
value: valueRaw,
|
||||
reason: 'noncanonical_key',
|
||||
};
|
||||
return;
|
||||
}
|
||||
|
||||
// Key is canonical; value is wrong. Classify the value.
|
||||
// Strip outer quotes for comparison so we can detect quoted variants.
|
||||
const unquoted = valueRaw.replace(/^["']|["']$/g, '');
|
||||
const wasQuoted = unquoted !== valueRaw;
|
||||
|
||||
if (wasQuoted && unquoted === 'exempt') {
|
||||
out.brain_first_typo = {
|
||||
key,
|
||||
value: valueRaw,
|
||||
reason: 'quoted_value',
|
||||
};
|
||||
return;
|
||||
}
|
||||
|
||||
if (unquoted.toLowerCase() === 'exempt') {
|
||||
// Capitalized: `Exempt` or `EXEMPT`
|
||||
out.brain_first_typo = {
|
||||
key,
|
||||
value: valueRaw,
|
||||
reason: 'capitalized_value',
|
||||
};
|
||||
return;
|
||||
}
|
||||
|
||||
// Some other value (e.g. `required`, `n/a`, `true`). v0.36 ships only
|
||||
// 'exempt'; flag everything else as an unknown-value typo so the doctor
|
||||
// hint can explain the supported value set.
|
||||
out.brain_first_typo = {
|
||||
key,
|
||||
value: valueRaw,
|
||||
reason: 'unknown_value',
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a paste-ready fix hint string from a `brain_first_typo` field.
|
||||
* Returns null when typo is undefined. The doctor message and the
|
||||
* stderr typo warning both consume this so phrasing stays consistent.
|
||||
*/
|
||||
export function formatBrainFirstTypoHint(typo: ParsedFrontmatter['brain_first_typo']): string | null {
|
||||
if (!typo) return null;
|
||||
switch (typo.reason) {
|
||||
case 'noncanonical_key':
|
||||
return `Found '${typo.key}: ${typo.value}' — did you mean 'brain_first: exempt'? (snake_case key required)`;
|
||||
case 'quoted_value':
|
||||
return `Found 'brain_first: ${typo.value}' — drop the quotes: 'brain_first: exempt'`;
|
||||
case 'capitalized_value':
|
||||
return `Found 'brain_first: ${typo.value}' — value must be lowercase: 'brain_first: exempt'`;
|
||||
case 'unknown_value':
|
||||
return `Found 'brain_first: ${typo.value}' — v0.36 ships only 'brain_first: exempt' (declarative opt-out)`;
|
||||
}
|
||||
}
|
||||
@@ -58,6 +58,18 @@ export function skillMdTemplate(v: ScaffoldVars): string {
|
||||
lines.push('');
|
||||
lines.push(`${v.description}`);
|
||||
lines.push('');
|
||||
// v0.36.x scaffold pre-insert (A3 + F10 from /plan-eng-review). New
|
||||
// skills inherit the canonical brain-first Convention callout by
|
||||
// default; authors of pure-infra skills can delete this line and add
|
||||
// `brain_first: exempt` to frontmatter instead. Both are zero-friction
|
||||
// one-line operations. The skillify-check item 12 (brain-first
|
||||
// compliance, required) catches the no-callout / no-exempt case at
|
||||
// audit time if the author removes this without opting out.
|
||||
lines.push(
|
||||
'> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) ' +
|
||||
'for the lookup chain (search → query → get_page → external).',
|
||||
);
|
||||
lines.push('');
|
||||
lines.push('## The rule');
|
||||
lines.push('');
|
||||
lines.push(`<!-- ${SKILLIFY_STUB_MARKER} -->`);
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
/**
|
||||
* E2E for the v0.36.x skill_brain_first wave (T10 from /plan-eng-review).
|
||||
*
|
||||
* Two layers:
|
||||
* 1. **Fixture-corpus shape assertions (always runs in CI):** assemble a
|
||||
* synthetic skills dir under tempdir using the fixture corpus and the
|
||||
* `manifest.json` shape `loadOrDeriveManifest` accepts. Run the
|
||||
* `skillBrainFirstCheck()` doctor check + the auto-fix INSERT path
|
||||
* and assert structural invariants (T1 — shape, not count):
|
||||
* - structurally-exempt skills MUST be ok
|
||||
* - canonical-callout skills MUST be ok
|
||||
* - skills with `brain_first: exempt` MUST be ok
|
||||
* - external + no compliance + no exempt MUST be warn
|
||||
* - --fix materializes the callout at the correct insertion site
|
||||
* - re-running doctor after --fix reports 0 violations
|
||||
*
|
||||
* 2. **Live OpenClaw shape (opt-in, $OPENCLAW_WORKSPACE-gated):** lives
|
||||
* at `scripts/live-brain-first-check.ts`. NOT in `bun run verify`.
|
||||
* Manual run during dev/QA to validate the wave against the real
|
||||
* deployment.
|
||||
*
|
||||
* No DATABASE_URL needed — entirely filesystem. Auto-fix path requires a
|
||||
* git repo (the safety gate refuses writes outside one), so the test
|
||||
* inits a throwaway git repo in the tempdir and commits the fixtures
|
||||
* before exercising --fix.
|
||||
*/
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { execFileSync } from 'child_process';
|
||||
|
||||
import { skillBrainFirstCheck } from '../../src/commands/doctor.ts';
|
||||
import { autoFixDryViolations } from '../../src/core/dry-fix.ts';
|
||||
|
||||
const FIXTURE_SOURCE = join(import.meta.dir, '..', 'fixtures', 'brain-first-skills');
|
||||
|
||||
interface Workspace {
|
||||
dir: string;
|
||||
skillsDir: string;
|
||||
cleanup: () => void;
|
||||
}
|
||||
|
||||
function copyFixturesIntoTempWorkspace(): Workspace {
|
||||
const root = join(
|
||||
tmpdir(),
|
||||
`brain-first-e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
);
|
||||
const skills = join(root, 'skills');
|
||||
mkdirSync(skills, { recursive: true });
|
||||
|
||||
// Mirror every fixture skill into the tempdir.
|
||||
for (const name of readdirSync(FIXTURE_SOURCE)) {
|
||||
const srcDir = join(FIXTURE_SOURCE, name);
|
||||
const dstDir = join(skills, name);
|
||||
mkdirSync(dstDir, { recursive: true });
|
||||
const src = readFileSync(join(srcDir, 'SKILL.md'), 'utf-8');
|
||||
writeFileSync(join(dstDir, 'SKILL.md'), src);
|
||||
}
|
||||
|
||||
// Init git so the dry-fix safety gate sees "clean tracked file" not
|
||||
// "not a repo." The auto-fix REFUSES writes when the file isn't under
|
||||
// git (would destroy the only copy with no rollback).
|
||||
execFileSync('git', ['init', '-q'], { cwd: root });
|
||||
execFileSync('git', ['add', '-A'], { cwd: root });
|
||||
execFileSync(
|
||||
'git',
|
||||
['-c', 'user.email=e2e@test', '-c', 'user.name=e2e', 'commit', '-q', '-m', 'fixtures'],
|
||||
{ cwd: root },
|
||||
);
|
||||
|
||||
return {
|
||||
dir: root,
|
||||
skillsDir: skills,
|
||||
cleanup: () => {
|
||||
try { rmSync(root, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
let workspace: Workspace;
|
||||
|
||||
beforeEach(() => {
|
||||
workspace = copyFixturesIntoTempWorkspace();
|
||||
// Redirect audit dir to the tempdir so the snapshot file doesn't pollute
|
||||
// the real $GBRAIN_AUDIT_DIR.
|
||||
process.env.GBRAIN_AUDIT_DIR = join(workspace.dir, 'audit');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env.GBRAIN_AUDIT_DIR;
|
||||
workspace.cleanup();
|
||||
});
|
||||
|
||||
describe('skill_brain_first E2E (fixture corpus)', () => {
|
||||
test('SHAPE: warn surfaces non-compliant fixtures by name', () => {
|
||||
const result = skillBrainFirstCheck(workspace.skillsDir);
|
||||
expect(result.status).toBe('warn');
|
||||
const violatorSlugs = (result.issues ?? []).map(i => i.skill).sort();
|
||||
// SHAPE: exactly the deliberately-bad fixtures must be in violators.
|
||||
expect(violatorSlugs).toContain('missing-brain-first');
|
||||
expect(violatorSlugs).toContain('multi-pattern');
|
||||
expect(violatorSlugs).toContain('typo-frontmatter');
|
||||
});
|
||||
|
||||
test('SHAPE: compliant fixtures MUST NOT appear in violators', () => {
|
||||
const result = skillBrainFirstCheck(workspace.skillsDir);
|
||||
const violatorSlugs = new Set((result.issues ?? []).map(i => i.skill));
|
||||
expect(violatorSlugs.has('compliant-callout')).toBe(false);
|
||||
expect(violatorSlugs.has('compliant-phase')).toBe(false);
|
||||
expect(violatorSlugs.has('compliant-position')).toBe(false);
|
||||
expect(violatorSlugs.has('exempt-frontmatter')).toBe(false);
|
||||
expect(violatorSlugs.has('no-external')).toBe(false);
|
||||
expect(violatorSlugs.has('negation-prose')).toBe(false);
|
||||
});
|
||||
|
||||
test('SHAPE: typo-frontmatter surfaces typo_hint in fix payload', () => {
|
||||
const result = skillBrainFirstCheck(workspace.skillsDir);
|
||||
const typoIssue = (result.issues ?? []).find(i => i.skill === 'typo-frontmatter');
|
||||
expect(typoIssue).toBeDefined();
|
||||
expect(typoIssue!.fix.typo_hint).toBeDefined();
|
||||
expect(typoIssue!.fix.typo_hint).toContain('snake_case');
|
||||
});
|
||||
|
||||
test('--fix INSERT path: dry-run reports proposed callouts without writing', () => {
|
||||
const initialContent = readFileSync(
|
||||
join(workspace.skillsDir, 'missing-brain-first', 'SKILL.md'),
|
||||
'utf-8',
|
||||
);
|
||||
const report = autoFixDryViolations(workspace.skillsDir, { dryRun: true });
|
||||
const proposed = report.fixed.filter(f => f.status === 'proposed' && f.patternLabel === 'brain-first compliance');
|
||||
expect(proposed.length).toBeGreaterThanOrEqual(2); // missing-brain-first + multi-pattern at least
|
||||
// File MUST NOT have been written during dry-run.
|
||||
const afterDryRun = readFileSync(
|
||||
join(workspace.skillsDir, 'missing-brain-first', 'SKILL.md'),
|
||||
'utf-8',
|
||||
);
|
||||
expect(afterDryRun).toBe(initialContent);
|
||||
});
|
||||
|
||||
test('--fix INSERT path: applied write inserts canonical callout at correct site', () => {
|
||||
const before = readFileSync(
|
||||
join(workspace.skillsDir, 'missing-brain-first', 'SKILL.md'),
|
||||
'utf-8',
|
||||
);
|
||||
const report = autoFixDryViolations(workspace.skillsDir);
|
||||
const applied = report.fixed.filter(f => f.status === 'applied' && f.patternLabel === 'brain-first compliance');
|
||||
expect(applied.length).toBeGreaterThanOrEqual(2);
|
||||
const after = readFileSync(
|
||||
join(workspace.skillsDir, 'missing-brain-first', 'SKILL.md'),
|
||||
'utf-8',
|
||||
);
|
||||
expect(after).not.toBe(before);
|
||||
// The callout must be present.
|
||||
expect(after).toMatch(/^>\s*\*\*Convention:\*\*[^\n]*brain-first/im);
|
||||
// It must land AFTER frontmatter close + AFTER first H1.
|
||||
const frontmatterClose = after.indexOf('---\n', 4); // second `---`
|
||||
const h1Index = after.indexOf('# missing-brain-first');
|
||||
const calloutIndex = after.search(/^>\s*\*\*Convention:\*\*[^\n]*brain-first/im);
|
||||
expect(calloutIndex).toBeGreaterThan(frontmatterClose);
|
||||
expect(calloutIndex).toBeGreaterThan(h1Index);
|
||||
});
|
||||
|
||||
test('--fix INSERT path: second run is idempotent (already_delegated)', () => {
|
||||
autoFixDryViolations(workspace.skillsDir);
|
||||
const report2 = autoFixDryViolations(workspace.skillsDir);
|
||||
const applied = report2.fixed.filter(f => f.status === 'applied' && f.patternLabel === 'brain-first compliance');
|
||||
expect(applied.length).toBe(0);
|
||||
// The skipped reasons for any brain-first attempts should be `already_delegated`.
|
||||
const skippedBf = report2.skipped.filter(f => f.patternLabel === 'brain-first compliance');
|
||||
for (const sk of skippedBf) {
|
||||
expect(sk.reason).toBe('already_delegated');
|
||||
}
|
||||
});
|
||||
|
||||
test('--fix INSERT path: re-running doctor after fix reports 0 brain-first violators (excluding typo case which still has invalid frontmatter)', () => {
|
||||
autoFixDryViolations(workspace.skillsDir);
|
||||
const result = skillBrainFirstCheck(workspace.skillsDir);
|
||||
const violatorSlugs = (result.issues ?? []).map(i => i.skill).sort();
|
||||
// After --fix, missing-brain-first and multi-pattern should be resolved
|
||||
// via the inserted callout. typo-frontmatter ALSO gets the callout
|
||||
// inserted (because the typo'd brain_first didn't exempt it, so the
|
||||
// analyzer flagged it, so --fix inserted). So it should also flip to ok.
|
||||
// (The typo hint goes away too because it's only surfaced for warn cases.)
|
||||
expect(violatorSlugs).not.toContain('missing-brain-first');
|
||||
expect(violatorSlugs).not.toContain('multi-pattern');
|
||||
expect(violatorSlugs).not.toContain('typo-frontmatter');
|
||||
});
|
||||
|
||||
test('SHAPE: structural-exemption-by-no-external-pattern is honored (no-external fixture)', () => {
|
||||
const result = skillBrainFirstCheck(workspace.skillsDir);
|
||||
const issue = (result.issues ?? []).find(i => i.skill === 'no-external');
|
||||
expect(issue).toBeUndefined();
|
||||
});
|
||||
|
||||
test('SHAPE: declarative opt-out is honored (exempt-frontmatter fixture)', () => {
|
||||
const result = skillBrainFirstCheck(workspace.skillsDir);
|
||||
const issue = (result.issues ?? []).find(i => i.skill === 'exempt-frontmatter');
|
||||
expect(issue).toBeUndefined();
|
||||
});
|
||||
|
||||
test('AUDIT: first doctor run bootstraps snapshot + writes detected events for current violators', () => {
|
||||
skillBrainFirstCheck(workspace.skillsDir);
|
||||
const auditDir = process.env.GBRAIN_AUDIT_DIR!;
|
||||
expect(existsSync(join(auditDir, 'skill-brain-first-snapshot.json'))).toBe(true);
|
||||
// Read the audit file and confirm detected events are present.
|
||||
const files = readdirSync(auditDir).filter(f => f.startsWith('skill-brain-first-') && f.endsWith('.jsonl'));
|
||||
expect(files.length).toBe(1);
|
||||
const audit = readFileSync(join(auditDir, files[0]), 'utf-8');
|
||||
expect(audit).toContain('"event":"detected"');
|
||||
expect(audit).toContain('missing-brain-first');
|
||||
});
|
||||
|
||||
test('AUDIT: second doctor run with no transitions produces zero new audit lines (A2 contract)', () => {
|
||||
skillBrainFirstCheck(workspace.skillsDir); // first run bootstraps
|
||||
const auditDir = process.env.GBRAIN_AUDIT_DIR!;
|
||||
const files = readdirSync(auditDir).filter(f => f.endsWith('.jsonl'));
|
||||
const initialLength = files.length > 0
|
||||
? readFileSync(join(auditDir, files[0]), 'utf-8').split('\n').filter(l => l.length > 0).length
|
||||
: 0;
|
||||
skillBrainFirstCheck(workspace.skillsDir); // second run — no transitions
|
||||
const afterLength = files.length > 0
|
||||
? readFileSync(join(auditDir, files[0]), 'utf-8').split('\n').filter(l => l.length > 0).length
|
||||
: 0;
|
||||
expect(afterLength).toBe(initialLength);
|
||||
});
|
||||
|
||||
test('AUDIT: applied --fix shifts skills from warn → ok across runs (transition signal)', () => {
|
||||
skillBrainFirstCheck(workspace.skillsDir); // bootstrap
|
||||
autoFixDryViolations(workspace.skillsDir); // resolve violations
|
||||
skillBrainFirstCheck(workspace.skillsDir); // post-fix scan
|
||||
const auditDir = process.env.GBRAIN_AUDIT_DIR!;
|
||||
const files = readdirSync(auditDir).filter(f => f.endsWith('.jsonl'));
|
||||
const audit = readFileSync(join(auditDir, files[0]), 'utf-8');
|
||||
// The post-fix run should have written resolved events for the
|
||||
// skills that flipped from warn → ok.
|
||||
expect(audit).toContain('"event":"resolved"');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: compliant-callout
|
||||
description: External-lookup skill with canonical Convention callout
|
||||
triggers:
|
||||
- "research a person"
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# compliant-callout
|
||||
|
||||
A skill that researches people via Perplexity but properly delegates to
|
||||
the brain-first convention.
|
||||
|
||||
> **Convention:** see conventions/brain-first.md for the lookup chain (search → query → get_page → external).
|
||||
|
||||
## Phase 1: Research
|
||||
|
||||
Use Perplexity to find recent news about the person; cross-reference web_search
|
||||
for primary sources.
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
name: compliant-phase
|
||||
description: External-lookup skill with explicit Phase 1 brain heading
|
||||
triggers:
|
||||
- "enrich entity"
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# compliant-phase
|
||||
|
||||
A skill that enriches entities via web_search but starts with an explicit
|
||||
Phase 1 brain-first lookup section.
|
||||
|
||||
## Phase 1: Brain-First Lookup
|
||||
|
||||
Before reaching for external sources, check what the brain already knows.
|
||||
|
||||
## Phase 2: External Enrichment
|
||||
|
||||
If the brain answer is thin, run web_search for missing context, then
|
||||
cross-reference with exa.api for citations.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: compliant-position
|
||||
description: Skill that runs gbrain search before external lookup
|
||||
triggers:
|
||||
- "look up a topic"
|
||||
---
|
||||
|
||||
# compliant-position
|
||||
|
||||
This skill demonstrates position-relative compliance: the first brain
|
||||
reference (gbrain search) appears strictly before the first external
|
||||
reference (web_search), so the analyzer accepts it without requiring
|
||||
the canonical callout.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Run `gbrain search "topic"` to find existing brain pages.
|
||||
2. If brain answer is thin, fall back to web_search for fresh data.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
name: exempt-frontmatter
|
||||
description: Pure-infra skill that opts out via frontmatter
|
||||
triggers:
|
||||
- "schedule a cron job"
|
||||
mutating: true
|
||||
brain_first: exempt
|
||||
---
|
||||
|
||||
# exempt-frontmatter
|
||||
|
||||
This skill manages cron schedules. It does call web_search for time-zone
|
||||
data and perplexity for cron syntax help — but the maintainer declared
|
||||
`brain_first: exempt` because the skill is pure infrastructure that
|
||||
doesn't consult brain knowledge.
|
||||
|
||||
## How
|
||||
|
||||
Use web_search for tz data, perplexity for cron syntax. Update the
|
||||
crontab via the host system call.
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
name: missing-brain-first
|
||||
description: External-lookup skill with NO brain-first compliance
|
||||
triggers:
|
||||
- "research thing"
|
||||
---
|
||||
|
||||
# missing-brain-first
|
||||
|
||||
A skill that goes straight to external APIs with no brain check first.
|
||||
This SHOULD be flagged by the analyzer.
|
||||
|
||||
## How
|
||||
|
||||
Call web_search to find information. Hit perplexity for synthesis.
|
||||
No brain consultation at all.
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
name: multi-pattern
|
||||
description: Skill that uses multiple external tools without compliance
|
||||
triggers:
|
||||
- "deep research"
|
||||
---
|
||||
|
||||
# multi-pattern
|
||||
|
||||
A skill that fans out across exa.search, perplexity, and crustdata for
|
||||
comprehensive lookups. No brain-first compliance declared. Should be
|
||||
flagged with all three external patterns reported.
|
||||
|
||||
## How
|
||||
|
||||
Query exa.search for general matches. Cross-reference perplexity for
|
||||
deeper synthesis. Pull crustdata for LinkedIn signal.
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: negation-prose
|
||||
description: Skill where brain reference precedes a negation-prose mention
|
||||
triggers:
|
||||
- "do research"
|
||||
---
|
||||
|
||||
# negation-prose
|
||||
|
||||
This skill starts with `gbrain search`, then talks about NOT using
|
||||
web_search before the brain — testing that the position-relative check
|
||||
correctly resolves to compliant (brain ref appears first in body).
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Always start with `gbrain search "topic"` for existing context.
|
||||
2. Do NOT use web_search before checking the brain. The brain has the
|
||||
answer 90% of the time.
|
||||
3. Only after step 1 is empty, fall back to perplexity for fresh data.
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
name: no-external
|
||||
description: Skill that operates purely on local state
|
||||
triggers:
|
||||
- "rotate the log file"
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# no-external
|
||||
|
||||
This skill rotates log files locally. No external APIs, no brain queries.
|
||||
Trivially exempt from brain-first compliance because there's nothing
|
||||
to consult.
|
||||
|
||||
## How
|
||||
|
||||
Read the log path from config, rename to .log.1, truncate the active file.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: typo-frontmatter
|
||||
description: Skill with typo in brain_first declaration
|
||||
triggers:
|
||||
- "do a thing"
|
||||
brain-first: exempt
|
||||
---
|
||||
|
||||
# typo-frontmatter
|
||||
|
||||
The maintainer tried to opt out but used kebab-case `brain-first` instead
|
||||
of canonical snake_case `brain_first`. The analyzer should surface a
|
||||
typo hint AND still flag the skill (because the exempt declaration
|
||||
didn't land).
|
||||
|
||||
## How
|
||||
|
||||
Call web_search and perplexity for fresh data.
|
||||
@@ -0,0 +1,620 @@
|
||||
/**
|
||||
* Unit suite for the v0.36.x skill_brain_first analyzer (T9 from
|
||||
* /plan-eng-review).
|
||||
*
|
||||
* Drives the fixture corpus at `test/fixtures/brain-first-skills/*`.
|
||||
* Absorbs PR #1206's 10 inline-string cases (IRON-RULE regression
|
||||
* preservation: every behavior the PR pinned must still pass here).
|
||||
*
|
||||
* Coverage targets:
|
||||
* - parseSkillFrontmatter: canonical / typo variants / array fields
|
||||
* - analyzeSkillBrainFirst: all 6 BrainFirstReason values × edge cases
|
||||
* - Position-relative gate body-only semantics (F6 regression)
|
||||
* - Canonical callout regex shape variations (F7 regression)
|
||||
* - FORMERLY_HARDCODED_EXEMPT membership preserved
|
||||
* - buildBrainFirstSummaryLine output shape (doctor + skillify consume it)
|
||||
* - Snapshot+diff audit logic (A2 contract: zero writes on no-transition runs)
|
||||
*
|
||||
* Hermetic: no DATABASE_URL, no network, no real audit dir. The audit
|
||||
* snapshot tests redirect `GBRAIN_AUDIT_DIR` to a tempdir.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { existsSync, mkdirSync, readdirSync, readFileSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import {
|
||||
parseSkillFrontmatter,
|
||||
formatBrainFirstTypoHint,
|
||||
} from '../src/core/skill-frontmatter.ts';
|
||||
import {
|
||||
analyzeSkillBrainFirst,
|
||||
buildBrainFirstSummaryLine,
|
||||
findFirstBrainRefOffset,
|
||||
findFirstExternalRefOffset,
|
||||
stripFrontmatter,
|
||||
CONVENTION_CALLOUT_RE,
|
||||
PHASE_HEADING_RE,
|
||||
EXTERNAL_LOOKUP_PATTERNS,
|
||||
FORMERLY_HARDCODED_EXEMPT,
|
||||
} from '../src/core/skill-brain-first.ts';
|
||||
import {
|
||||
diffAgainstSnapshot,
|
||||
loadSnapshot,
|
||||
writeSnapshotAtomically,
|
||||
computeBrainFirstAuditFilename,
|
||||
logBrainFirstEvent,
|
||||
readRecentBrainFirstEvents,
|
||||
appendAuditEventsForTransitions,
|
||||
_resetWarnedSetForTests,
|
||||
} from '../src/core/audit-skill-brain-first.ts';
|
||||
|
||||
const FIXTURE_DIR = join(import.meta.dir, 'fixtures', 'brain-first-skills');
|
||||
|
||||
function loadFixture(name: string): { content: string; skillName: string } {
|
||||
const skillMd = join(FIXTURE_DIR, name, 'SKILL.md');
|
||||
return { content: readFileSync(skillMd, 'utf-8'), skillName: name };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// parseSkillFrontmatter
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('parseSkillFrontmatter', () => {
|
||||
test('returns null when no frontmatter fence is present', () => {
|
||||
expect(parseSkillFrontmatter('# Just a heading\nNo frontmatter here.')).toBeNull();
|
||||
});
|
||||
|
||||
test('parses name, mutating, writes_pages, writes_to', () => {
|
||||
const content = [
|
||||
'---',
|
||||
'name: thing',
|
||||
'mutating: true',
|
||||
'writes_pages: true',
|
||||
'writes_to:',
|
||||
' - people/',
|
||||
' - companies/',
|
||||
'---',
|
||||
'',
|
||||
'# thing',
|
||||
].join('\n');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm).not.toBeNull();
|
||||
expect(fm!.name).toBe('thing');
|
||||
expect(fm!.mutating).toBe(true);
|
||||
expect(fm!.writes_pages).toBe(true);
|
||||
expect(fm!.writes_to).toEqual(['people/', 'companies/']);
|
||||
});
|
||||
|
||||
test('parses inline array forms for writes_to / tools / triggers', () => {
|
||||
const content = [
|
||||
'---',
|
||||
'name: x',
|
||||
'writes_to: [people/, companies/]',
|
||||
'tools: [search, query, put_page]',
|
||||
'triggers: [foo, bar]',
|
||||
'---',
|
||||
].join('\n');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm!.writes_to).toEqual(['people/', 'companies/']);
|
||||
expect(fm!.tools).toEqual(['search', 'query', 'put_page']);
|
||||
expect(fm!.triggers).toEqual(['foo', 'bar']);
|
||||
});
|
||||
|
||||
test('canonical brain_first: exempt populates the typed field', () => {
|
||||
const content = '---\nname: x\nbrain_first: exempt\n---\n';
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm!.brain_first).toBe('exempt');
|
||||
expect(fm!.brain_first_typo).toBeUndefined();
|
||||
});
|
||||
|
||||
test('kebab-case brain-first triggers noncanonical_key typo', () => {
|
||||
const content = '---\nname: x\nbrain-first: exempt\n---\n';
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm!.brain_first).toBeUndefined();
|
||||
expect(fm!.brain_first_typo).toBeDefined();
|
||||
expect(fm!.brain_first_typo!.reason).toBe('noncanonical_key');
|
||||
expect(fm!.brain_first_typo!.key).toBe('brain-first');
|
||||
});
|
||||
|
||||
test('CamelCase BrainFirst triggers noncanonical_key typo', () => {
|
||||
const content = '---\nname: x\nBrainFirst: exempt\n---\n';
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm!.brain_first).toBeUndefined();
|
||||
expect(fm!.brain_first_typo!.reason).toBe('noncanonical_key');
|
||||
});
|
||||
|
||||
test('quoted value triggers quoted_value typo', () => {
|
||||
const content = "---\nname: x\nbrain_first: 'exempt'\n---\n";
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm!.brain_first).toBeUndefined();
|
||||
expect(fm!.brain_first_typo!.reason).toBe('quoted_value');
|
||||
});
|
||||
|
||||
test('capitalized value triggers capitalized_value typo', () => {
|
||||
const content = '---\nname: x\nbrain_first: Exempt\n---\n';
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm!.brain_first).toBeUndefined();
|
||||
expect(fm!.brain_first_typo!.reason).toBe('capitalized_value');
|
||||
});
|
||||
|
||||
test('unknown value triggers unknown_value typo', () => {
|
||||
const content = '---\nname: x\nbrain_first: required\n---\n';
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
expect(fm!.brain_first).toBeUndefined();
|
||||
expect(fm!.brain_first_typo!.reason).toBe('unknown_value');
|
||||
});
|
||||
|
||||
test('formatBrainFirstTypoHint produces paste-ready strings for every reason', () => {
|
||||
expect(
|
||||
formatBrainFirstTypoHint({ key: 'brain-first', value: 'exempt', reason: 'noncanonical_key' }),
|
||||
).toContain('snake_case');
|
||||
expect(
|
||||
formatBrainFirstTypoHint({ key: 'brain_first', value: "'exempt'", reason: 'quoted_value' }),
|
||||
).toContain('drop the quotes');
|
||||
expect(
|
||||
formatBrainFirstTypoHint({ key: 'brain_first', value: 'Exempt', reason: 'capitalized_value' }),
|
||||
).toContain('lowercase');
|
||||
expect(
|
||||
formatBrainFirstTypoHint({ key: 'brain_first', value: 'required', reason: 'unknown_value' }),
|
||||
).toContain("only 'brain_first: exempt'");
|
||||
expect(formatBrainFirstTypoHint(undefined)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// stripFrontmatter + offset helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('stripFrontmatter', () => {
|
||||
test('removes leading YAML fence', () => {
|
||||
const content = '---\nname: x\n---\n# Body\nrest';
|
||||
expect(stripFrontmatter(content)).toBe('# Body\nrest');
|
||||
});
|
||||
|
||||
test('leaves content unchanged when no fence is present', () => {
|
||||
expect(stripFrontmatter('# No fence')).toBe('# No fence');
|
||||
});
|
||||
|
||||
test('CRITICAL F6 — frontmatter exclusion prevents tools: [web_search] false-positive', () => {
|
||||
const content = [
|
||||
'---',
|
||||
'name: x',
|
||||
'tools: [web_search]',
|
||||
'---',
|
||||
'',
|
||||
'# x',
|
||||
'',
|
||||
'Body says gbrain search comes first.',
|
||||
'Then perplexity for follow-up.',
|
||||
].join('\n');
|
||||
const body = stripFrontmatter(content);
|
||||
// `web_search` appears ONLY in frontmatter — body has no external pattern
|
||||
// before the gbrain reference.
|
||||
expect(findFirstBrainRefOffset(body)).toBeGreaterThanOrEqual(0);
|
||||
// Body should NOT contain `web_search` (it was in the stripped frontmatter).
|
||||
expect(body.includes('web_search')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('offset helpers', () => {
|
||||
test('findFirstBrainRefOffset finds earliest gbrain ref', () => {
|
||||
const body = 'Some preamble. gbrain search and later gbrain query.';
|
||||
const o = findFirstBrainRefOffset(body);
|
||||
expect(o).toBe(body.indexOf('gbrain search'));
|
||||
});
|
||||
|
||||
test('findFirstExternalRefOffset finds earliest external pattern', () => {
|
||||
const body = 'First call perplexity, then later web_search.';
|
||||
const o = findFirstExternalRefOffset(body);
|
||||
expect(o).toBe(body.indexOf('perplexity'));
|
||||
});
|
||||
|
||||
test('returns -1 when no match', () => {
|
||||
expect(findFirstBrainRefOffset('no brain ref here')).toBe(-1);
|
||||
expect(findFirstExternalRefOffset('no external ref here')).toBe(-1);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// CONVENTION_CALLOUT_RE (F7 — path-syntax-agnostic)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('CONVENTION_CALLOUT_RE', () => {
|
||||
test('matches plain-text path form (brain-ops shape)', () => {
|
||||
const line = '> **Convention:** See skills/conventions/brain-first.md for the 5-step lookup protocol.';
|
||||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(true);
|
||||
});
|
||||
|
||||
test('matches relative plain-text form (perplexity-research shape)', () => {
|
||||
const line = '> **Convention:** see conventions/brain-first.md for the lookup chain.';
|
||||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(true);
|
||||
});
|
||||
|
||||
test('matches markdown-link form (auto-fix output shape)', () => {
|
||||
const line = '> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) for the lookup chain.';
|
||||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(true);
|
||||
});
|
||||
|
||||
test('does NOT match a non-blockquote mention of brain-first', () => {
|
||||
const line = 'In a paragraph: see conventions/brain-first.md for the chain.';
|
||||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT match an unrelated Convention callout', () => {
|
||||
const line = '> **Convention:** see conventions/quality.md for citation format.';
|
||||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// PHASE_HEADING_RE
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('PHASE_HEADING_RE', () => {
|
||||
test('matches ## Phase 1: Brain-First Lookup', () => {
|
||||
expect(PHASE_HEADING_RE.test('## Phase 1: Brain-First Lookup')).toBe(true);
|
||||
});
|
||||
|
||||
test('matches ### Step 0: Brain Context', () => {
|
||||
expect(PHASE_HEADING_RE.test('### Step 0: Brain Context')).toBe(true);
|
||||
});
|
||||
|
||||
test('does NOT match # Phase 1 (H1, not H2+)', () => {
|
||||
expect(PHASE_HEADING_RE.test('# Phase 1: Brain Lookup')).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT match Phase 2 etc.', () => {
|
||||
expect(PHASE_HEADING_RE.test('## Phase 2: Synthesis')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// EXTERNAL_LOOKUP_PATTERNS coverage matrix
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('EXTERNAL_LOOKUP_PATTERNS', () => {
|
||||
test('all 8 patterns are present and named', () => {
|
||||
const names = EXTERNAL_LOOKUP_PATTERNS.map(p => p.name).sort();
|
||||
expect(names).toEqual([
|
||||
'captain_api',
|
||||
'crustdata',
|
||||
'exa',
|
||||
'firecrawl',
|
||||
'happenstance',
|
||||
'perplexity',
|
||||
'web_fetch',
|
||||
'web_search',
|
||||
]);
|
||||
});
|
||||
|
||||
test('captain_api regex handles all four shapes', () => {
|
||||
const re = EXTERNAL_LOOKUP_PATTERNS.find(p => p.name === 'captain_api')!.re;
|
||||
expect(re.test('use captain api')).toBe(true);
|
||||
expect(re.test('use captain_api')).toBe(true);
|
||||
expect(re.test('use captain-api')).toBe(true);
|
||||
expect(re.test('use captainapi')).toBe(true);
|
||||
expect(re.test('captain is not an api ref')).toBe(false);
|
||||
});
|
||||
|
||||
test('exa requires a separator (avoids matching exam, exalt)', () => {
|
||||
const re = EXTERNAL_LOOKUP_PATTERNS.find(p => p.name === 'exa')!.re;
|
||||
expect(re.test('use exa.search')).toBe(true);
|
||||
expect(re.test('use exa_lookup')).toBe(true);
|
||||
expect(re.test('use exa-api')).toBe(true);
|
||||
expect(re.test('do not use exam')).toBe(false);
|
||||
expect(re.test('exalt the brain')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// FORMERLY_HARDCODED_EXEMPT preservation
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('FORMERLY_HARDCODED_EXEMPT', () => {
|
||||
test('preserves all 40+ entries from PR #1206 allowlist', () => {
|
||||
// Spot-check across the three categories from PR #1206.
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.has('brain-ops')).toBe(true);
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.has('gbrain')).toBe(true);
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.has('exa')).toBe(true);
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.has('perplexity-research')).toBe(false); // not in PR allowlist
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.has('browser')).toBe(true);
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.has('cron-scheduler')).toBe(true);
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.has('ask-user')).toBe(true);
|
||||
});
|
||||
|
||||
test('contains at least 40 entries (preserves PR #1206 intent)', () => {
|
||||
expect(FORMERLY_HARDCODED_EXEMPT.size).toBeGreaterThanOrEqual(40);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// analyzeSkillBrainFirst — fixture corpus drive (the IRON-RULE regression set)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('analyzeSkillBrainFirst (fixture corpus)', () => {
|
||||
test('compliant-callout → ok via compliant_callout reason', () => {
|
||||
const { content, skillName } = loadFixture('compliant-callout');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.reason).toBe('compliant_callout');
|
||||
expect(result.external_patterns_matched.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('compliant-phase → ok via compliant_phase reason', () => {
|
||||
const { content, skillName } = loadFixture('compliant-phase');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.reason).toBe('compliant_phase');
|
||||
});
|
||||
|
||||
test('compliant-position → ok via compliant_position reason', () => {
|
||||
const { content, skillName } = loadFixture('compliant-position');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.reason).toBe('compliant_position');
|
||||
});
|
||||
|
||||
test('missing-brain-first → warn via missing_brain_first', () => {
|
||||
const { content, skillName } = loadFixture('missing-brain-first');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('warn');
|
||||
expect(result.reason).toBe('missing_brain_first');
|
||||
expect(result.external_patterns_matched).toContain('web_search');
|
||||
expect(result.external_patterns_matched).toContain('perplexity');
|
||||
});
|
||||
|
||||
test('exempt-frontmatter → ok via exempt_explicit', () => {
|
||||
const { content, skillName } = loadFixture('exempt-frontmatter');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.reason).toBe('exempt_explicit');
|
||||
});
|
||||
|
||||
test('no-external → ok via exempt_no_external', () => {
|
||||
const { content, skillName } = loadFixture('no-external');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.reason).toBe('exempt_no_external');
|
||||
expect(result.external_patterns_matched).toEqual([]);
|
||||
});
|
||||
|
||||
test('multi-pattern → warn with all 3 external patterns matched', () => {
|
||||
const { content, skillName } = loadFixture('multi-pattern');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('warn');
|
||||
expect(result.reason).toBe('missing_brain_first');
|
||||
expect(result.external_patterns_matched).toContain('exa');
|
||||
expect(result.external_patterns_matched).toContain('perplexity');
|
||||
expect(result.external_patterns_matched).toContain('crustdata');
|
||||
});
|
||||
|
||||
test('typo-frontmatter → warn with typo_hint surfaced', () => {
|
||||
const { content, skillName } = loadFixture('typo-frontmatter');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('warn');
|
||||
expect(result.typo_hint).toBeDefined();
|
||||
expect(result.typo_hint).toContain('brain-first');
|
||||
expect(result.typo_hint).toContain('snake_case');
|
||||
});
|
||||
|
||||
test('negation-prose → ok via compliant_position (brain ref appears first)', () => {
|
||||
const { content, skillName } = loadFixture('negation-prose');
|
||||
const fm = parseSkillFrontmatter(content);
|
||||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.reason).toBe('compliant_position');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// analyzeSkillBrainFirst — direct-input regression (PR #1206 absorption)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('analyzeSkillBrainFirst (PR #1206 regression preservation)', () => {
|
||||
test('PR test case: good skill with Step 0 brain context passes', () => {
|
||||
// Direct copy of PR #1206 `good-skill` test case shape.
|
||||
const content = `---\nname: good-skill\n---\n# Good Skill\n\n## Step 0: Brain Context\nSearch the brain first with \`gbrain search\` for relevant context.\n\n## Step 1: External Lookup\nUse web_search to find additional information.\n`;
|
||||
const result = analyzeSkillBrainFirst(content, 'good-skill', parseSkillFrontmatter(content));
|
||||
expect(result.status).toBe('ok');
|
||||
});
|
||||
|
||||
test('PR test case: bad skill with web_search but no brain-first → warn', () => {
|
||||
const content = `---\nname: bad-skill\n---\n# Bad Skill\n\n## Step 1: Research\nUse web_search to find information about the entity.\nThen use Perplexity for deeper research.\n`;
|
||||
const result = analyzeSkillBrainFirst(content, 'bad-skill', parseSkillFrontmatter(content));
|
||||
expect(result.status).toBe('warn');
|
||||
});
|
||||
|
||||
test('PR test case: skill with brain search reference is not flagged', () => {
|
||||
const content = `---\nname: bf\n---\n# Brain-First\n\nFirst, search the brain for existing context.\nThen use web_search for anything missing. Use gbrain search.\n`;
|
||||
const result = analyzeSkillBrainFirst(content, 'bf', parseSkillFrontmatter(content));
|
||||
expect(result.status).toBe('ok');
|
||||
});
|
||||
|
||||
test('PR test case: skill without external lookups not flagged', () => {
|
||||
const content = `---\nname: internal\n---\n# Internal Skill\nThis skill only operates on local files. No external lookups.\n`;
|
||||
const result = analyzeSkillBrainFirst(content, 'internal', parseSkillFrontmatter(content));
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.reason).toBe('exempt_no_external');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// buildBrainFirstSummaryLine
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('buildBrainFirstSummaryLine', () => {
|
||||
test('ok results render with reason tag', () => {
|
||||
const line = buildBrainFirstSummaryLine({
|
||||
skill: 'x',
|
||||
status: 'ok',
|
||||
reason: 'compliant_callout',
|
||||
external_patterns_matched: [],
|
||||
formerly_hardcoded_exempt: false,
|
||||
});
|
||||
expect(line).toBe('x: ok (compliant_callout)');
|
||||
});
|
||||
|
||||
test('warn results include external patterns + hint', () => {
|
||||
const line = buildBrainFirstSummaryLine({
|
||||
skill: 'bad',
|
||||
status: 'warn',
|
||||
reason: 'missing_brain_first',
|
||||
external_patterns_matched: ['perplexity', 'exa'],
|
||||
formerly_hardcoded_exempt: false,
|
||||
});
|
||||
expect(line).toContain('bad');
|
||||
expect(line).toContain('perplexity, exa');
|
||||
});
|
||||
|
||||
test('warn results with formerly_hardcoded_exempt include the PR #1206 hint', () => {
|
||||
const line = buildBrainFirstSummaryLine({
|
||||
skill: 'browser',
|
||||
status: 'warn',
|
||||
reason: 'missing_brain_first',
|
||||
external_patterns_matched: ['web_fetch'],
|
||||
formerly_hardcoded_exempt: true,
|
||||
});
|
||||
expect(line).toContain('hardcoded-exempt in PR #1206');
|
||||
expect(line).toContain('brain_first: exempt');
|
||||
});
|
||||
|
||||
test('warn results carry typo hint through to message', () => {
|
||||
const line = buildBrainFirstSummaryLine({
|
||||
skill: 'typo',
|
||||
status: 'warn',
|
||||
reason: 'missing_brain_first',
|
||||
external_patterns_matched: ['perplexity'],
|
||||
typo_hint: 'Found brain-first: exempt — did you mean brain_first?',
|
||||
formerly_hardcoded_exempt: false,
|
||||
});
|
||||
expect(line).toContain('typo');
|
||||
expect(line).toContain('brain-first');
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Snapshot+diff audit (A2 contract — zero writes on no-transition runs)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Helper: provision an isolated audit tempdir for one test body and tear
|
||||
* it down via try/finally. Wraps the body in `withEnv()` so the
|
||||
* GBRAIN_AUDIT_DIR mutation is scoped to this test only — cross-test-
|
||||
* safe (no leak to other tests in the same shard) per the test-
|
||||
* isolation lint (R1).
|
||||
*/
|
||||
async function withAuditDir<T>(fn: (auditDir: string) => Promise<T> | T): Promise<T> {
|
||||
_resetWarnedSetForTests();
|
||||
const auditDir = join(
|
||||
tmpdir(),
|
||||
`brain-first-audit-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
);
|
||||
mkdirSync(auditDir, { recursive: true });
|
||||
try {
|
||||
return await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, () => fn(auditDir));
|
||||
} finally {
|
||||
try { rmSync(auditDir, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
|
||||
describe('audit-skill-brain-first (snapshot+diff)', () => {
|
||||
test('loadSnapshot returns present:false when file missing', async () => {
|
||||
await withAuditDir(() => {
|
||||
const r = loadSnapshot();
|
||||
expect(r.present).toBe(false);
|
||||
expect(r.violators.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('writeSnapshotAtomically + loadSnapshot round-trip', async () => {
|
||||
await withAuditDir(() => {
|
||||
writeSnapshotAtomically(new Set(['a', 'b', 'c']));
|
||||
const r = loadSnapshot();
|
||||
expect(r.present).toBe(true);
|
||||
expect(r.violators.has('a')).toBe(true);
|
||||
expect(r.violators.has('b')).toBe(true);
|
||||
expect(r.violators.has('c')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
test('diffAgainstSnapshot detects added/removed/unchanged', () => {
|
||||
// Pure function — no audit dir needed.
|
||||
const prev = new Set(['a', 'b', 'c']);
|
||||
const curr = new Set(['b', 'c', 'd']);
|
||||
const diff = diffAgainstSnapshot(curr, prev);
|
||||
expect(diff.added).toEqual(['d']);
|
||||
expect(diff.removed).toEqual(['a']);
|
||||
expect(diff.unchanged).toEqual(['b', 'c']);
|
||||
});
|
||||
|
||||
test('diff result is sorted for determinism', () => {
|
||||
const prev = new Set(['c', 'a', 'd']);
|
||||
const curr = new Set(['a', 'b', 'e']);
|
||||
const diff = diffAgainstSnapshot(curr, prev);
|
||||
expect(diff.added).toEqual(['b', 'e']);
|
||||
expect(diff.removed).toEqual(['c', 'd']);
|
||||
expect(diff.unchanged).toEqual(['a']);
|
||||
});
|
||||
|
||||
test('corrupt snapshot JSON treated as missing with warn-once', async () => {
|
||||
await withAuditDir(auditDir => {
|
||||
const file = join(auditDir, 'skill-brain-first-snapshot.json');
|
||||
require('fs').writeFileSync(file, 'not-json-at-all');
|
||||
const r = loadSnapshot();
|
||||
expect(r.present).toBe(false);
|
||||
expect(r.violators.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('appendAuditEventsForTransitions writes one line per added/removed', async () => {
|
||||
await withAuditDir(() => {
|
||||
const diff = { added: ['skill-a'], removed: ['skill-b'], unchanged: ['skill-c'] };
|
||||
const patterns = new Map([['skill-a', ['web_search']]]);
|
||||
appendAuditEventsForTransitions(diff, patterns, 'test-run-1');
|
||||
const events = readRecentBrainFirstEvents(7);
|
||||
expect(events.length).toBe(2);
|
||||
const detected = events.find(e => e.event === 'detected');
|
||||
const resolved = events.find(e => e.event === 'resolved');
|
||||
expect(detected?.skill).toBe('skill-a');
|
||||
expect(detected?.external_patterns).toEqual(['web_search']);
|
||||
expect(resolved?.skill).toBe('skill-b');
|
||||
});
|
||||
});
|
||||
|
||||
test('no-transition diff produces zero audit writes (A2 contract)', async () => {
|
||||
await withAuditDir(() => {
|
||||
const diff = { added: [], removed: [], unchanged: ['skill-a', 'skill-b'] };
|
||||
appendAuditEventsForTransitions(diff, new Map(), 'test-run-2');
|
||||
const events = readRecentBrainFirstEvents(7);
|
||||
expect(events.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('logBrainFirstEvent writes a fixed event', async () => {
|
||||
await withAuditDir(() => {
|
||||
logBrainFirstEvent({ event: 'fixed', skill: 'browser' });
|
||||
const events = readRecentBrainFirstEvents(7);
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0].event).toBe('fixed');
|
||||
expect(events[0].skill).toBe('browser');
|
||||
expect(events[0].code).toBe('SKILL_BRAIN_FIRST');
|
||||
expect(events[0].severity).toBe('info');
|
||||
});
|
||||
});
|
||||
|
||||
test('computeBrainFirstAuditFilename produces ISO-week format', () => {
|
||||
const name = computeBrainFirstAuditFilename(new Date('2026-05-19T10:00:00Z'));
|
||||
expect(name).toMatch(/^skill-brain-first-2026-W\d{2}\.jsonl$/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user