mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
feat: v0.41.19.0 Supavisor Retry Cathedral (#1537)
Engine-level retry primitive that closes the v0.41.17 production incident
where ~3,000 wiki links + timeline entries were silently lost per dream
cycle on a 16K-page brain. Supavisor's circuit-breaker takes 5-10s to
recover; the prior single-500ms-retry shape couldn't survive it.
ARCHITECTURE
============
Retry becomes a data-primitive contract, not a caller responsibility.
postgres-engine.ts + pglite-engine.ts now self-retry inside addLinksBatch,
addTimelineEntriesBatch, and upsertChunks. Every caller — current AND
future — inherits retry-for-free. CI lint guard `scripts/check-no-double-retry.sh`
fails the build if anyone re-wraps an engine batch method (preventing
3×3=9 retry amplification on incomplete reverts).
CODEX-HARDENED DEFAULTS
=======================
BULK_RETRY_OPTS = {maxRetries:3, delayMs:1000, delayMaxMs:10000,
jitter:'decorrelated'}. Total worst-case wait ≈12s covers full Supavisor
recovery window. Decorrelated jitter (AWS-style uniform(base, prevDelay*3)
capped at maxDelay) replaces 'full' which allowed near-zero retries that
re-hit the still-recovering breaker.
AbortSignal threading from MinionWorker.shutdownAbort.signal through
engine method opts → withRetry → abortableSleep. SIGTERM aborts sleeping
retries instead of blocking deploys for up to delayMaxMs.
OBSERVABILITY
=============
`~/.gbrain/audit/batch-retry-YYYY-Www.jsonl` records every retry event
(success-after-blip AND exhausted-retries). Built on the v0.40.4.0
audit-writer cathedral. Privacy posture: never logs slugs / page IDs /
content (mirrors shell-audit.ts).
`gbrain doctor` learns `batch_retry_health` check. Reads last 24h
(not 7d — codex H-9: avoid permanent noise from one historical blip).
Thresholds: ok (zero or <3 same-site), warn (>=3 same-site OR >=5
cross-site), fail (>=20 sustained breaker). Surfaces bad GBRAIN_BULK_*
env at startup (codex M-10). Corrupt-JSONL tolerant.
30-day audit pruning hooked into the dream cycle's purge phase (codex H-8
— implements the 'pruning convention' for real).
OPERATOR TUNING
===============
GBRAIN_BULK_MAX_RETRIES (int >= 0; 0 disables retries for debugging)
GBRAIN_BULK_RETRY_BASE_MS (int > 0)
GBRAIN_BULK_RETRY_MAX_MS (int >= base)
Bad values throw GBrainError with paste-ready fix hints at doctor startup,
not at first-retry mid-cycle.
VERIFICATION
============
- bun run verify: 28/28 checks green (includes 2 new lint guards:
check-no-double-retry, check-batch-audit-site)
- bun run test: 11453 pass / 1 pre-existing flake (schema-cli.test.ts —
confirmed by running on clean master, NOT introduced by this wave)
- bun run test:slow: 40/40 including new test/core/retry-stress.slow.test.ts
(100 batches × 30% blip rate × decorrelated jitter, zero row loss)
- bunx tsc --noEmit: 0 errors
REVIEWS
=======
- CEO review (SELECTIVE EXPANSION): 4 cherry-picks proposed, 4 accepted
- Eng review (2 passes): 10 findings, 0 critical gaps, architectural
pivot from per-site to engine-level wrap
- Codex independent review: 23 findings; 10 critical/high absorbed
(decorrelated jitter, 12s backoff window, AbortSignal, idempotency
proof, backfill unification, typed audit-site enum, doctor expiry
thresholds, audit pruning, env validation at doctor startup)
PR #1523 closed and absorbed (@garrytan-agents original extract.ts fix
preserved via co-author trailer; 5 test cases moved to test/core/retry.test.ts
with assertions adjusted for the v0.41.19.0 BULK_RETRY_OPTS defaults).
Co-authored-by: garrytan-agents <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
garrytan-agents
garrytan-agents
parent
10816cba38
commit
a7b79b66d4
Executable
+79
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.41.18.0 — CI guard against batch-audit-site typo drift (codex H-7).
|
||||
#
|
||||
# auditSite labels flow from call sites into the batch-retry audit JSONL
|
||||
# and from there into `gbrain doctor`'s batch_retry_health check. A typo
|
||||
# like `'extract.lnks_inc'` doesn't break compilation (TypeScript narrows
|
||||
# string literals only via the BatchAuditSite type, but external string
|
||||
# values escape this — e.g. config, environment, dynamic dispatch).
|
||||
#
|
||||
# This script extracts every string-literal `auditSite: '...'` value from
|
||||
# src/ and validates it appears in the BATCH_AUDIT_SITES const list in
|
||||
# src/core/retry.ts. Fails the build on mismatch.
|
||||
#
|
||||
# Usage: scripts/check-batch-audit-site.sh
|
||||
# Exit: 0 when every literal matches the enum, 1 otherwise.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
RETRY_FILE="src/core/retry.ts"
|
||||
if [ ! -f "$RETRY_FILE" ]; then
|
||||
echo "ERROR: $RETRY_FILE missing — cannot validate audit sites."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Extract every entry inside BATCH_AUDIT_SITES = [ ... ] as const.
|
||||
# Strips quotes + trailing commas + whitespace. Strict awk window between
|
||||
# the array open and closing `] as const`.
|
||||
KNOWN_SITES=$(awk '
|
||||
/BATCH_AUDIT_SITES = \[/ { capture = 1; next }
|
||||
capture && /\] as const/ { capture = 0; exit }
|
||||
capture {
|
||||
# Pull out '\''xyz'\'' or "xyz" string literals on the line.
|
||||
while (match($0, /['\''"]([^'\''"]+)['\''"]/)) {
|
||||
print substr($0, RSTART + 1, RLENGTH - 2)
|
||||
$0 = substr($0, RSTART + RLENGTH)
|
||||
}
|
||||
}
|
||||
' "$RETRY_FILE" | sort -u)
|
||||
|
||||
if [ -z "$KNOWN_SITES" ]; then
|
||||
echo "ERROR: Could not extract BATCH_AUDIT_SITES from $RETRY_FILE."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Extract every `auditSite: '...'` literal from src/ (excluding retry.ts
|
||||
# itself which contains the enum definition, and test files which are
|
||||
# allowed to use synthetic sites for assertion scaffolding).
|
||||
USED_SITES=$(
|
||||
grep -rEh "auditSite:[[:space:]]*['\"][^'\"]+['\"]" src/ \
|
||||
--include='*.ts' \
|
||||
--exclude-dir=core/audit \
|
||||
--exclude='retry.ts' \
|
||||
| sed -E "s/.*auditSite:[[:space:]]*['\"]([^'\"]+)['\"].*/\1/" \
|
||||
| sort -u
|
||||
)
|
||||
|
||||
if [ -z "$USED_SITES" ]; then
|
||||
echo "OK: no auditSite literals found in src/ (engines use defaults)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
UNKNOWN_SITES=$(comm -23 <(echo "$USED_SITES") <(echo "$KNOWN_SITES") || true)
|
||||
|
||||
if [ -n "$UNKNOWN_SITES" ]; then
|
||||
echo "ERROR: Unknown auditSite literal(s) found in src/:"
|
||||
echo "$UNKNOWN_SITES" | sed 's/^/ /'
|
||||
echo
|
||||
echo "Fix: add the value to BATCH_AUDIT_SITES in src/core/retry.ts."
|
||||
echo " The enum is the closed list of known sites."
|
||||
echo
|
||||
echo "Known sites:"
|
||||
echo "$KNOWN_SITES" | sed 's/^/ /'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: all auditSite literals match BATCH_AUDIT_SITES enum"
|
||||
Executable
+63
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.41.18.0 — CI guard against double-retry hazard.
|
||||
#
|
||||
# Engine batch methods (addLinksBatch / addTimelineEntriesBatch /
|
||||
# upsertChunks) self-retry via withRetry(BULK_RETRY_OPTS) inside the engine
|
||||
# implementation. Wrapping them ALSO at the call site produces 3×3=9 retry
|
||||
# attempts under failure, amplifying load on a recovering circuit breaker
|
||||
# and worsening the very incident the wave was designed to fix.
|
||||
#
|
||||
# This script greps src/ for the pattern and fails the build if found.
|
||||
# Catches the migration-ordering hazard from the v0.41.18.0 eng review (D6)
|
||||
# AND prevents future refactors from re-introducing the bug class.
|
||||
#
|
||||
# Usage: scripts/check-no-double-retry.sh
|
||||
# Exit: 0 when no matches, 1 when matches found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Match: withRetry(...) wrapping any of the 3 engine batch methods.
|
||||
# The greedy `.*` between `withRetry(` and `engine.` covers both the
|
||||
# arrow-fn form and any direct invocation. (gbrain-allow-direct-insert: doc comment)
|
||||
# Multi-line wraps are caught by `grep -E` per file (line-wise) for the
|
||||
# common single-line case; multi-line wraps still get caught by a separate
|
||||
# multi-line pass below.
|
||||
PATTERN='withRetry\([^)]*engine\.(addLinksBatch|addTimelineEntriesBatch|upsertChunks)'
|
||||
|
||||
# Single-line scan (covers ~95% of real cases).
|
||||
if grep -rEn "$PATTERN" src/ --include='*.ts' 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: Found withRetry(...engine.{addLinksBatch|addTimelineEntriesBatch|upsertChunks})"
|
||||
echo " pattern in src/."
|
||||
echo
|
||||
echo " Engine batch methods self-retry via withRetry(BULK_RETRY_OPTS) in"
|
||||
echo " postgres-engine.ts + pglite-engine.ts. Wrapping AGAIN at the call site"
|
||||
echo " produces 3×3=9 retry attempts under failure, amplifying load on a"
|
||||
echo " recovering circuit breaker."
|
||||
echo
|
||||
echo " Fix: delete the outer withRetry wrap. Pass auditSite as a kwarg:"
|
||||
echo " await engine.addLinks(batch, { auditSite: 'extract.links_inc' }); // example"
|
||||
echo
|
||||
echo " Audit JSONL records the retries silently at "
|
||||
echo " ~/.gbrain/audit/batch-retry-YYYY-Www.jsonl; check"
|
||||
echo " \`gbrain doctor\` for the batch_retry_health surface."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Multi-line scan: a withRetry( on one line and the engine call on the next
|
||||
# few. Bounded to 3-line window so we don't flag distant unrelated calls.
|
||||
# Uses pcregrep if available, else falls back to a simple awk window.
|
||||
if command -v pcregrep >/dev/null 2>&1; then
|
||||
if pcregrep -r -M -n --include='\.ts$' \
|
||||
'withRetry\([^)]*\n\s*\(?[^)]*=>\s*engine\.(addLinksBatch|addTimelineEntriesBatch|upsertChunks)' \
|
||||
src/ 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: Multi-line withRetry(...engine.batch...) wrap found in src/. See above."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "OK: no withRetry(...engine.batch...) double-retry patterns in src/"
|
||||
@@ -59,6 +59,8 @@ CHECKS=(
|
||||
"check:conversation-parser"
|
||||
"check:resolver"
|
||||
"check:source-scope-onboard"
|
||||
"check:no-double-retry"
|
||||
"check:batch-audit-site"
|
||||
"typecheck"
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user