Compare commits

..
Author SHA1 Message Date
Garry Tan 6776df2447 Merge branch 'fix/resolver-warnings' of https://github.com/garrytan/gbrain into fix/resolver-warnings 2026-04-24 02:18:35 -07:00
root 19dbf152c2 fix: resolve all check-resolvable warnings (MECE overlap, DRY, routing)
- RESOLVER.md: broaden query triggers ("who is", "brain search",
  "background on", "notes on this"), quote all citation-fixer triggers,
  add disambiguation rules for citation-fixer vs maintain and
  gbrain-jobs vs minion-orchestrator

- citation-fixer: add "citations are broken" and "fix broken citations"
  triggers to frontmatter

- maintain: rename trigger "citation audit" → "maintenance audit" to
  avoid MECE overlap with citation-fixer

- enrich: add convention delegation ref near notability tier table
  (within DRY_PROXIMITY_LINES window), replace inlined citation format
  rules with reference to conventions/quality.md

check-resolvable --skills-dir ./skills: 0 errors, 0 warnings
2026-04-24 08:52:42 +00:00
root 0c48010d79 fix: resolve all check-resolvable warnings (MECE overlap, DRY, routing)
- RESOLVER.md: broaden query triggers ("who is", "brain search",
  "background on", "notes on this"), quote all citation-fixer triggers,
  add disambiguation rules for citation-fixer vs maintain and
  gbrain-jobs vs minion-orchestrator

- citation-fixer: add "citations are broken" and "fix broken citations"
  triggers to frontmatter

- maintain: rename trigger "citation audit" → "maintenance audit" to
  avoid MECE overlap with citation-fixer

- enrich: add convention delegation ref near notability tier table
  (within DRY_PROXIMITY_LINES window), replace inlined citation format
  rules with reference to conventions/quality.md

check-resolvable --skills-dir ./skills: 0 errors, 0 warnings
2026-04-24 08:22:03 +00:00
d838d4792b feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency (#379)
* feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency, shell guard

Prevents stall-induced queue blockage discovered in production (OpenClaw):

1. Wall-clock timeout sweep: dead-letters active jobs exceeding 2× timeout_ms
   (or 2 × lockDuration × max_stalled). Catches jobs stuck while holding DB
   connections where FOR UPDATE SKIP LOCKED stall detection skips them.

2. Submission backpressure (maxWaiting): caps waiting jobs per name at
   submission time. Prevents autopilot-cycle flood when the queue is blocked.

3. --no-worker flag for autopilot: skips spawning the built-in worker child.
   For environments where the worker lifecycle is managed externally (systemd,
   Docker, OpenClaw service-manager).

4. GBRAIN_WORKER_CONCURRENCY env var: fallback for --concurrency when the
   worker is spawned by autopilot (which can't pass CLI flags to the child).

5. Shell job env guard with clear logging: shell handler is always registered
   but throws UnrecoverableError with a clear message when
   GBRAIN_ALLOW_SHELL_JOBS=1 is not set, instead of silently not registering.

* feat: v0.19.1 Lane A — maxWaiting atomic guard, concurrency clamp, --max-waiting CLI

Addresses three production-hardening findings from the CEO + Eng + Codex
adversarial review of PR #379:

D2/H2: maxWaiting was TOCTOU-racy — two concurrent submitters could both
see waitingCount < max and both insert. Wrap the count+select+insert in
pg_advisory_xact_lock keyed on (name, queue). Serializes concurrent
decisions for the SAME key while leaving different keys fully parallel.
Lock auto-releases on txn commit/rollback — no cleanup path to leak.
Also fix the missing queue-scope bug: count and select now filter on
(name, queue) not name alone, so cross-queue same-name jobs don't
suppress each other.

D3/H3: resolveWorkerConcurrency silently accepted NaN / 0 / negative from
parseInt. `inFlight.size < NaN` is always false → worker claims nothing →
silent wedge from a single-typo env var. Clamp to ≥1 with a loud stderr
warning naming the bad value.

D5/H5: `gbrain jobs submit` never parsed `--max-waiting N` despite the
MinionJobInput field. Wire the flag with clamp [1, 100], mirror
`--max-stalled`. Extract `parseMaxWaitingFlag` for unit testing.

Q1: Silent coalesce was invisible by design. New
src/core/minions/backpressure-audit.ts mirrors shell-audit.ts's ISO-week
JSONL pattern: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Coalesce
events write one JSONL line with (queue, name, waiting_count, max_waiting,
returned_job_id, ts). Best-effort — disk-full never blocks submission.

A2: `gbrain jobs smoke --wedge-rescue` new opt-in regression case.
Forges a wedged-worker row state, invokes handleStalled + handleTimeouts
+ handleWallClockTimeouts in order, asserts only wall-clock evicts.
Mirrors the v0.14.3 `--sigkill-rescue` shape.

Tests: 23 new unit cases in test/minions.test.ts covering wall-clock
timeout (3 cases + non-interference with handleTimeouts), maxWaiting
(coalesce, clamp 0, floor, concurrent-submitter race via Promise.all,
cross-queue isolation, unset fallthrough), concurrency clamp (7 cases
incl. NaN/0/negative), parseMaxWaitingFlag (5 cases), backpressure
audit file write.

Part of v0.19.1 plan at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.19.1 Lane B — doctor queue_health, autopilot peer probe, runbook

A5 / D4: New `queue_health` check in `gbrain doctor`. Postgres-only (PGLite
has no multi-process worker surface). Two subchecks, both cheap (single
SELECT each, status-index-covered):

- stalled-forever: any active job with started_at > 1h. Surfaces the
  worst offenders (top 5 by started_at ASC) with `gbrain jobs get/cancel`
  fix hints. The incident that motivated v0.19.1 ran 90+ min before the
  operator noticed.
- waiting-depth: per-name waiting count exceeds threshold. Default 10,
  overridable via GBRAIN_QUEUE_WAITING_THRESHOLD env (D9). Signals a
  submitter probably needs maxWaiting set.

Worker-heartbeat subcheck from the original plan dropped (D4/H4): no
minion_workers table exists, and lock_until-on-active-jobs is a lossy
proxy that can't distinguish idle-worker from dead-worker. Tracked as
follow-up B7.

A4: --no-worker peer-liveness probe in autopilot. When --no-worker is
set, every cycle runs a cheap SELECT checking for any active job whose
lock_until was refreshed in the last 2 minutes. After 3 consecutive
idle ticks, logs a loud WARNING naming the silent-wedge vector and
referencing B7 as the ground-truth follow-up. Re-arms on next live
signal so the warning doesn't spam every cycle.

A6: New docs/guides/queue-operations-runbook.md (one viewport, ~60
lines). "My queue looks wedged — what do I run?" in order of
escalation. What each doctor subcheck means. Self-check for the
--no-worker / no-worker-running footgun.

CLAUDE.md: key-files updates for handleWallClockTimeouts (v0.19.0 Layer
3 kill shot), maxWaiting advisory-lock rewrite (v0.19.1 D2), queue_health
doctor check (v0.19.1 D4), and backpressure-audit.ts.

Tests: all 143 minions + 13 doctor unit tests pass. No new test cases
required in Lane B; the doctor queue_health exercise is in the E2E
verification step (needs real PG to produce meaningful stalled-forever
rows). The --no-worker probe is exercised by the smoke case's wedge
setup in Lane A.

README: unchanged. Existing `gbrain jobs submit` examples don't show
--max-stalled, so no --max-waiting precedent to extend per A6 conditional.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.19.1 Lane C — CHANGELOG entry, VERSION bump, remove SPEC.md

VERSION: 0.19.0 → 0.19.1 (patch; bug-fix-dominant, no schema change,
no new user-facing vocabulary).

CHANGELOG: new v0.19.1 entry at the top with the full release-summary
template per CLAUDE.md — bold two-line headline, lead paragraph, "numbers
that matter" before/after table measured against the real incident,
"what this means for OpenClaw users" closer, required "To take
advantage of v0.19.1" block naming the worker-restart requirement,
itemized changes by area, and "For contributors" section closing the
loop on the stale autopilot-idempotency narrative the CEO review was
based on.

Mechanism reframing per D1/H1: the 18-job pile-up was NOT caused by
missing idempotency (autopilot already passes
`idempotency_key: autopilot-cycle:${slot}` at autopilot.ts:241). The
18 jobs were 18 DIFFERENT slots stacking up behind the wedged one.
`maxWaiting` still caps the pile; the incident just wasn't about
idempotency. Adversarial review caught this before ship.

SPEC.md: deleted from repo root. It was Wintermute's planning artifact
for the original PR, not a shipped spec. Design docs belong under
docs/designs/ per repo convention; leaving one at repo root set a
precedent this repo doesn't want (A7/D11). CHANGELOG + the plan file
at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md are
the durable artifacts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: --wedge-rescue smoke state — both stall+timeout sweeps must skip

Smoke case was setting lock_until in the past, so handleStalled's
requeue path fired before handleWallClockTimeouts had a chance to
evict. Production scenario is "lock_until still live (worker
renewing) + timeout_at disqualified" — only wall-clock matches.

Single-connection smoke can't simulate a row lock held by another
txn, so we force the equivalent outcome:
- lock_until = now() + 30s → handleStalled skips (not a stall)
- timeout_at = NULL → handleTimeouts skips (needs NOT NULL)
- started_at = now() - 10s, timeout_ms=1000 → wall-clock matches
  (2 × timeout_ms = 2000ms threshold exceeded)

Verified: SMOKE PASS — Minions healthy + wedge rescue in 0.14s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: CI failures — shell-handler tests + llms-full.txt drift

Two CI failure clusters, both pre-existing but surfaced by the v0.20.3
merge:

1) test/minions-shell.test.ts — 12 failing cases. The shell handler
   throws UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1' (the
   production RCE guard at shell.ts:210). The unit tests exercise
   handler mechanics, not the guard, but never set the env var — so
   every invocation exits through the guard path instead of the code
   being tested. Fix: set GBRAIN_ALLOW_SHELL_JOBS=1 in beforeAll,
   restore in afterAll. The env-guard IS still tested separately via
   the test/minions.test.ts case added in v0.20.3 Lane A which toggles
   the var itself.

2) llms-full.txt — stale against CLAUDE.md. Key-files entries for
   queue.ts, doctor.ts, and the new backpressure-audit.ts updated in
   v0.20.3 Lane B triggered the build-llms drift guard. Regenerated
   via `bun run build:llms`; no behavior change, just the inlined-docs
   bundle catching up to source.

Full test run: 2367 pass, 0 fail across 137 files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 01:09:28 -07:00
6 changed files with 238 additions and 13 deletions
+8 -5
View File
@@ -13,13 +13,14 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Trigger | Skill |
|---------|-------|
| "What do we know about", "tell me about", "search for" | `skills/query/SKILL.md` |
| "What do we know about", "tell me about", "search for", "search the brain", "brain search", "background on", "notes on this", "who is" | `skills/query/SKILL.md` |
| "Who knows who", "relationship between", "connections", "graph query" | `skills/query/SKILL.md` (use graph-query) |
| Creating/enriching a person or company page | `skills/enrich/SKILL.md` |
| Where does a new file go? Filing rules | `skills/repo-architecture/SKILL.md` |
| Fix broken citations in brain pages | `skills/citation-fixer/SKILL.md` |
| "Fix broken citations", "citations are broken", "fix citations", "citation audit" | `skills/citation-fixer/SKILL.md` |
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
| Share a brain page as a link | `skills/publish/SKILL.md` |
| "validate frontmatter", "check frontmatter", "brain lint", "fix frontmatter" | `skills/frontmatter-guard/SKILL.md` |
## Content & media ingestion
@@ -58,7 +59,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
## Setup & migration
@@ -89,8 +90,10 @@ When multiple skills could match:
1. Prefer the most specific skill (meeting-ingestion over ingest)
2. If the user mentions a URL, route by content type (link → idea-ingest, video → media-ingest)
3. If the user mentions a person/company, check if enrich or query fits better
4. Chaining is explicit in each skill's Phases section
5. When in doubt, ask the user
4. **Citation audit** → use `citation-fixer` (targeted fix). `maintain` includes citation checking as one step of a broader health sweep — use `maintain` only for full brain health runs.
5. **Background task / spawn agent** → use `minion-orchestrator` for spawning and steering agents. `gbrain-jobs` is the lower-level queue CLI.
6. Chaining is explicit in each skill's Phases section
7. When in doubt, ask the user
## Conventions (cross-cutting)
+2
View File
@@ -8,6 +8,8 @@ triggers:
- "fix citations"
- "citation audit"
- "check citations"
- "citations are broken"
- "fix broken citations"
tools:
- search
- get_page
+4 -7
View File
@@ -55,14 +55,9 @@ they building, what makes them tick, where are they headed.
## Citation Requirements (MANDATORY)
Every fact must carry an inline `[Source: ...]` citation.
> **Convention:** See `skills/conventions/quality.md` for full citation format rules.
Three formats:
- **Direct attribution:** `[Source: User, {context}, YYYY-MM-DD]`
- **API/external:** `[Source: {provider} enrichment, YYYY-MM-DD]`
- **Synthesis:** `[Source: compiled from {list of sources}]`
Source precedence (highest to lowest):
Every fact must carry an inline `[Source: ...]` citation. Source precedence (highest to lowest):
1. User's direct statements
2. Compiled truth (pre-existing brain synthesis)
3. Timeline entries (raw evidence)
@@ -89,6 +84,8 @@ When sources conflict, note the contradiction with both citations.
Scale enrichment to importance. Don't waste API calls on low-value entities.
> **Convention:** See `skills/_brain-filing-rules.md` for the notability gate and filing rules.
| Tier | Who | Effort | Sources |
|------|-----|--------|---------|
| 1 (key) | Inner circle, close collaborators, key contacts | Full pipeline | All available APIs + deep web research |
+218
View File
@@ -0,0 +1,218 @@
---
name: frontmatter-guard
version: 1.0.0
description: |
Validates and auto-repairs frontmatter YAML on every brain page write.
Gate that prevents malformed pages from entering the brain. Import
writeBrainPage() instead of raw writeFileSync for any /data/brain/ write.
triggers:
- "validate frontmatter"
- "check frontmatter"
- "brain lint"
- "fix frontmatter"
tools:
- exec
- read
- write
mutating: true
---
# Frontmatter Guard
> Every brain write goes through the guard. No exceptions.
## Why This Exists
On 2026-04-24, a brain health audit found 203 pages with malformed frontmatter:
- 111 people pages missing closing `---` (entity detector bug)
- 43 meeting pages with unstructured YAML (ingestion bug)
- 16 files with slug mismatches
- 11 with binary corruption
- 4 with nested quote escaping
All written by our own agents. The guard prevents this class of error.
## The Library
**Location:** `lib/brain-writer.mjs` (in the OpenClaw workspace)
### Core API
```javascript
import { writeBrainPage, validateFrontmatter, autoFixFrontmatter } from '../lib/brain-writer.mjs';
// 1. Validated write (throws on bad frontmatter)
writeBrainPage('/data/brain/people/jane-doe.md', content);
// 2. Validated write with auto-repair
writeBrainPage('/data/brain/people/jane-doe.md', content, { autoFix: true });
// 3. Validate only (no write)
const result = validateFrontmatter(content, { filePath: '/data/brain/people/jane-doe.md' });
// → { ok: true/false, errors: [{ code, message }] }
// 4. Auto-fix only (returns fixed content)
const { content: fixed, fixes } = autoFixFrontmatter(content, { filePath });
```
### What It Validates
| Check | Error Code | Description |
|-------|-----------|-------------|
| Opening `---` | `MISSING_OPEN` | File doesn't start with frontmatter |
| Closing `---` | `MISSING_CLOSE` | No closing delimiter (heading found inside YAML zone) |
| YAML parse | `YAML_PARSE` | js-yaml can't parse the frontmatter block |
| Slug match | `SLUG_MISMATCH` | `slug:` field doesn't match file path |
| Null bytes | `NULL_BYTES` | Binary corruption in content |
| Nested quotes | `NESTED_QUOTES` | `title: "Name "Nick" Last"` pattern |
| Empty frontmatter | `EMPTY_FRONTMATTER` | Frontmatter block is empty |
### What It Auto-Fixes
| Fix | Description |
|-----|-------------|
| Missing `---` | Inserts closing delimiter before first heading |
| Nested quotes in title | `"Name "Nick" Last"``'Name "Nick" Last'` |
| Nested quotes in lists | Investor notes with inner quotes → inner singles |
| Bracket titles | `title: [Name``title: "Name"` |
| Slug removal | Removes `slug:` field (gbrain derives from path) |
| Null bytes | Strips `\x00` characters |
### Path Guard
```javascript
// This THROWS — path is not under /data/brain/
writeBrainPage('/data/.openclaw/workspace/brain/people/test.md', content);
// Error: writeBrainPage: path is not under /data/brain/
```
This prevents the #1 brain write bug: writing to the workspace `brain/` subdirectory instead of the actual brain repo.
## Pre-Commit Hook
**Location:** `/data/brain/.githooks/pre-commit`
Runs on every `git commit` in the brain repo. Checks staged `.md` files for:
1. Missing closing `---`
2. YAML parse errors (via js-yaml from workspace node_modules)
3. Null bytes
Blocks the commit with actionable errors. Bypass: `git commit --no-verify`.
## Integration Rules for Agents
### When writing a brain page directly (writeFileSync)
**ALWAYS** use `writeBrainPage()` instead:
```javascript
// ❌ BAD — no validation, silent corruption
import { writeFileSync } from 'node:fs';
writeFileSync('/data/brain/people/jane-doe.md', content);
// ✅ GOOD — validates, blocks bad writes
import { writeBrainPage } from '../lib/brain-writer.mjs';
writeBrainPage('/data/brain/people/jane-doe.md', content);
```
### When generating frontmatter in a prompt
Always include the closing `---`:
```markdown
---
title: "Person Name"
type: person
created: 2026-04-24
---
# Person Name
```
### When titles contain special characters
Use single quotes for titles with inner double quotes:
```yaml
# ❌ BAD
title: "Phil Libin's Journey to Finding a "Life's Work""
# ✅ GOOD
title: 'Phil Libin''s Journey to Finding a "Life''s Work"'
# ✅ ALSO GOOD
title: "Phil Libin's Journey to Finding a Life's Work"
```
### When values contain colons
Always quote values with colons:
```yaml
# ❌ BAD — YAML thinks everything after the colon is a new key
garry_context: Fucking sick coding song — one of Garry's favorites
# ✅ GOOD
garry_context: "Fucking sick coding song — one of Garry's favorites"
```
## Running a Brain-Wide Audit
```bash
cd /data/.openclaw/workspace && node -e "
import { validateFrontmatter } from './lib/brain-writer.mjs';
import { readFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';
function walk(dir, files = []) {
for (const f of readdirSync(dir)) {
if (f === '.git') continue;
const p = join(dir, f);
if (statSync(p).isDirectory()) walk(p, files);
else if (f.endsWith('.md')) files.push(p);
}
return files;
}
let valid = 0, invalid = 0;
for (const file of walk('/data/brain')) {
const content = readFileSync(file, 'utf8');
if (!content.startsWith('---')) continue;
const r = validateFrontmatter(content, { filePath: file });
if (r.ok) valid++; else invalid++;
}
console.log('Valid:', valid, '| Invalid:', invalid, '| Rate:', (valid*100/(valid+invalid)).toFixed(1) + '%');
"
```
## Batch Auto-Fix
```bash
cd /data/.openclaw/workspace && node -e "
import { validateFrontmatter, autoFixFrontmatter } from './lib/brain-writer.mjs';
import { readFileSync, writeFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';
// ... walk function ...
let fixed = 0;
for (const file of walk('/data/brain')) {
const content = readFileSync(file, 'utf8');
if (!content.startsWith('---')) continue;
if (validateFrontmatter(content).ok) continue;
const result = autoFixFrontmatter(content, { filePath: file });
if (result.fixes.length > 0 && validateFrontmatter(result.content).ok) {
writeFileSync(file, result.content);
fixed++;
}
}
console.log('Fixed:', fixed, 'files');
"
```
## Upstream Path
Once battle-tested here, the validator moves into gbrain's core:
1. `src/core/frontmatter.ts` — the validation + auto-fix logic
2. Integrated into `putPage()` / `upsertPage()` — every DB write validates
3. `gbrain lint` CLI command — runs the audit
4. `gbrain lint --fix` — runs auto-repair
5. Pre-commit hook ships with `gbrain init`
@@ -0,0 +1,5 @@
// Routing eval fixtures for skills/frontmatter-guard. Check 5 (W2, v0.19).
{"intent": "can you validate the frontmatter on these brain pages I just wrote", "expected_skill": "frontmatter-guard"}
{"intent": "run a brain lint to find broken frontmatter across the repo", "expected_skill": "frontmatter-guard"}
// Negative: general brain health is maintain, not frontmatter-guard
{"intent": "check overall brain health and run maintenance", "expected_skill": null, "ambiguous_with": []}
+1 -1
View File
@@ -8,7 +8,7 @@ description: |
triggers:
- "brain health"
- "check backlinks"
- "citation audit"
- "maintenance audit"
- "maintenance"
- "orphan pages"
- "stale pages"