mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
v0.42.76.0 fix(upgrade,security,cli): strict flag validation, bootstrap-wedge class kill, federated chunk scope (#3902)
* fix(schema): bootstrap timeline_entries.event_page_id forward reference — un-wedge pre-v121 upgrades (#2626 #2594 #2579 #2537 #2536) v0.42.56.0 (Chronicle, migration v121) added timeline_entries.event_page_id and two partial indexes in the embedded schema blobs without extending applyForwardReferenceBootstrap — any brain whose timeline_entries predates v121 wedged initSchema at blob replay ("column event_page_id does not exist") before runMigrations could apply v121, with no in-band recovery. - Add the timeline_entries.event_page_id probe + column-only ALTER to applyForwardReferenceBootstrap in BOTH engines; FK + partial indexes land via the idempotent v121 / blob replay afterwards. Stays in the always-run bootstrap (never a migration hook — those skip oddly-stamped brains). - REQUIRED_BOOTSTRAP_COVERAGE entry + strip blocks in both runtime tests. - e2e: pre-v121 rewind → full initSchema converges (indexes re-created); wedged-brain recovery — a brain that already FAILED the upgrade attempt converges on retry with full final shape (column + FK + both partial indexes) and no ledger residue. Absorbs PR #2548 (@chetan-guevara) and the e2e test from PR #2623 (@colinagent) — thank you both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(schema): coverage guard cross-references migration-added columns — close the scan hole that shipped the v121 wedge The A2 static check treated a column as covered when the current CREATE TABLE body declared it. But a column that is BOTH in the blob's CREATE TABLE AND added by a migration is a forward reference by definition — on pre-existing tables CREATE TABLE IF NOT EXISTS no-ops and the blob's CREATE INDEX crashes initSchema before runMigrations can help. That mask is exactly how timeline_entries.event_page_id passed the guard while wedging every pre-v121 brain. - buildIndexRefCoveragePredicate: migration-added columns (from extractAddedColumnsFromMigrations over the MIGRATIONS array) require a bootstrap ALTER; CREATE TABLE presence no longer counts for them. - Unit test pins the v121 regression shape red/green with synthetic inputs; the A2 test pins the incident triple directly (migration-added + blob-indexed + bootstrap-covered). - The strengthened predicate immediately surfaced two more latent wedges of the same class: minion_jobs.timeout_at + minion_jobs.idempotency_key (migration v7, blob-indexed, unprobed). Added probes in both engines + coverage entries + runtime strip blocks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): serve --http hides the generated admin token from non-TTY output by default (#2624) Generated admin bootstrap tokens printed into container/log-aggregator stdout on every headless start. shouldSuppressBootstrapPrint now defaults to hidden unless stderr is an interactive TTY; env-sourced tokens are never printed; --print-admin-token is the explicit escape hatch for capturing the value on a trusted non-TTY start; --suppress-bootstrap-token still overrides everything. Unit-tested across all five postures. Absorbs PR #2625 (@irresi) — thank you. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): legacy bearer tokens honor permissions.takes_holders over serve --http (#2529) GBrainOAuthProvider.verifyAccessToken never returned takesHoldersAllowList, so the serve --http dispatch site always fell back to ['world'] — remote MCP callers with an operator-configured takes_holders grant saw only public takes. The legacy branch now extracts permissions.takes_holders exactly like src/mcp/http-transport.ts (fail-safe ['world'] default, non-string entries dropped, malformed permissions JSON fails closed without throwing), and AuthInfo carries the field as a typed contract. OAuth-registered clients have no takes_holders storage on oauth_clients; that lane is design work tracked in TODOS (column migration + DCR/ registration surface), not part of this hotfix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): get_chunks honors the federated source grant + stops shipping embedding vectors (#2555, getChunks half of #2544) The get_chunks op still used the pre-#2200 scalar pattern (ctx.sourceId ? {sourceId} : {}) and engine.getChunks had no sourceIds[] support — a federated client that could read a page via get_page got [] from get_chunks. The op now routes through sourceScopeOpts (canonical ladder: federated array > scalar floor > nothing) and both engines gain the getPage-style sourceIds[] precedence branch; the unset-opts 'default' floor is preserved for local callers (importCodeFile contract). While in the function: SELECT cc.* pulled every embedding vector over the wire per chunk only for rowToChunk to discard them — replaced with the explicit non-vector column list in both engines (the getChunks half of #2544; the per-put_page getAllSlugs half is tracked separately). getChunksWithEmbeddings stays scalar-only by design (engine-internal, zero remote-reachable callers — documented at the interface). Tests: op-level federated repro + isolation + default-floor bleed guard (PGLite), engine precedence + Chunk-shape pin, and a DATABASE_URL-gated engine-parity test covering all three scope shapes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(auth): /admin/api/register-client accepts source + federatedRead bindings (#2143 enabler) The HTTP register endpoint hardcoded source_id='default' and federated_read=undefined — only the CLI could mint a client bound to a non-default source, so HTTP-registered MCP clients wrote into 'default' regardless of intent. The endpoint now accepts optional source / federatedRead body fields, validated via assertValidSourceId with a structured 400 on bad input; omitting both preserves the historical default. The admin-UI form layer is a tracked follow-up. Absorbs PR #2016 — thank you. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(scope,calibration): think reads over MCP; source-scoped takes reads; calibration CLI reachability + model resolution (#2078-class #2451) - think op → scope:'read' for OAuth/MCP clients: the handler already forces save/take off for remote callers before persistence, so a read-scoped token can think without a write grant; local CLI persistence unchanged. Scope-annotation test carries an explicit remote-gated allowlist. - takes_list / takes_search / takes_scorecard / takes_calibration route through sourceScopeOpts (the #2200 class on the takes read lane) with engine support in BOTH engines + tests. - 'calibration' added to CLI_ONLY (the command was registered but unreachable — dispatch-gap class) and calibration_profile/voice-gate resolve models through the canonical gateway tier resolver instead of bare ids that parseModelId rejects. - BigInt-safe local-op output normalization (bigintToStringReplacer, postgres.js wire parity) — first half of the #2450 fix; the formatResult default case lands with the cli-output commit. Absorbs PR #2598 (@colinagent) and PR #2452 (@spinsirr) — thank you both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): output correctness — BigInt-safe rendering, search --json, files bigint, and 5 unreachable commands (#2450 #2527 #2042 #2035-class) - normalizeLocalResult wraps the local-op output round-trip with the bigint→string replacer (postgres.js wire parity; a bare stringify THROWS on BIGSERIAL keys); formatResult's default renderer gets the same replacer so nothing upstream can crash it. - search/query --json: CLI-local formatter flag threaded through the shared formatter — stdout is a parseable result array, never human text on the --json path (the #2042 residual). - file_list normalizes size_bytes (Postgres BIGINT → Number) so MCP serialization and the CLI KB math survive; null preserved. - NEW dispatch-gap guard: every handleCliOnly top-level case label must be reachable via CLI_ONLY. It immediately caught FIVE live unreachable commands: pages, backfill, reconcile-links, notability-eval (added to CLI_ONLY), and the documented 'gbrain search modes|stats|tune' dashboards (pre-fix, 'search modes' silently keyword-searched the word "modes") — now routed via a pre-dispatch subcommand gate. 'whoknows' stays on its op-alias route (collision guard); tracked with PR #2509. Absorbs PR #2494 and PR #2531 (@javieraldape) and adapts PR #472 (@vinsew) — thank you. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(db): updateSourceConfig survives mixed-array config rows + repair/doctor cover the subagent jsonb columns (#2251) The array-coercion branch called jsonb_each(elem) bare — a mixed array (e.g. '["x", {"last_full_cycle_at": ...}]') threw 'cannot call jsonb_each on a non-object' DURING row production, permanently failing every subsequent updateSourceConfig (last_full_cycle_at could never be written again). Non-object elements are now neutralized inline via a CASE-guarded jsonb_each; the row self-heals to a flat object on the next write, object elements' keys recovered. Pinned ungated on PGLite (real Postgres semantics) and via a DATABASE_URL-gated e2e on the real engine. repair-jsonb + doctor's jsonb_integrity check extend from 5 to 8 columns (subagent_messages.content_blocks, subagent_tool_executions.input/output — historical damage rows from the pre-v0.42.53.0 positional double-encode; the write paths themselves were fixed in #2375) with a to_regclass skip for brains predating those tables. Adapts the repair/doctor extension from PR #597 (@vinsew) — thank you. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): dry-run honesty — strict unknown-flag rejection CLI-wide + unify-types worker defaults to dry-run (#2185 #1575) #2185: 'gbrain init --migrate-only --dry-run' applied REAL migrations — flags are read ad hoc (args.includes) so anything a handler doesn't look for was silently ignored, including intent-bearing safety flags. The CLI now validates every flag pre-dispatch and pre-engine: - Op commands validate against the operation contract (op.params + CLI-local --json/--explain), mirroring parseOpArgs so flag values that begin with '--' are never misread. parseOpArgs also gains the --key=value inline form (previously parsed as a junk key that consumed the NEXT token). - CLI_ONLY commands validate against a GENERATED per-command registry (scripts/generate-flag-registry.ts scans each command's case block + imported modules + one level of relative imports; deliberately over-inclusive so a missed flag can't break a working invocation). Committed as src/core/cli-flag-registry.generated.ts; 'bun run build:flag-registry' regenerates. - Passthrough by construction: everything after '--', plus call / config / 'jobs submit' payloads (handler-defined params are their contract). - Guards: sweep test (every command × nonsense flag → error), acceptance tests (real flags, --no- negation, = form, -- passthrough), drift guard (every CLI_ONLY member has an entry), freshness guard (committed registry == fresh generator run), subprocess smokes incl. the literal #2185 repro failing loud with zero engine work. BREAKING: scripts passing stray flags now fail loud with "Unknown flag --x for 'gbrain <cmd>'" — that is the point. #1575: the unify-types worker registration passed apply ?? true while the handler documents 'Default false (dry-run)' — the canonical operator invocation destructively retyped 25K+ pages by default. Now ?? false with a structural test; explicit --params '{"apply":true}' is the only way to mutate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(todos): file fix-wave 1 follow-ups (OAuth takes_holders design, parseFlags end-state, whoknows routing, #2544 half, #1558 UI, #2536 diagnostics) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): memory-safe unit runner — adaptive concurrency + serial OOM rescue pass A default run (4 shards × 4 intra-shard files) holds up to 16 concurrent PGLite WASM instances (~1.5GB each). With sibling Conductor workspaces running their own suites, PGLite connect failed with 'Out of memory' across every shard at once — 369 phantom test failures on a healthy branch, indistinguishable from real breakage at a glance. Two default-on layers in scripts/run-unit-parallel.sh: 1. Memory-aware sizing: total concurrency is capped to available memory (vm_stat on macOS, MemAvailable on Linux) at GBRAIN_TEST_MEM_PER_FILE_MB (default 1536) per concurrent file, shedding shards before intra-shard width. Quiet machines are unaffected (banner: mem-ok); pressured ones degrade instead of OOMing (banner: mem-adapted AxB→CxD). 2. Serial OOM rescue: failures whose shard log carries the WASM out-of-memory signature are re-run at --max-concurrency 1 after the fan-out drains. Phantoms pass serially → run goes green with an oom_rescued note and the failure blocks marked superseded; real failures fail again and stay red. Plain assertion failures never match the signature and never enter the rescue lane (existing exit-code and failure-log contract tests unchanged). Escape hatches: GBRAIN_TEST_NO_MEM_ADAPT=1, GBRAIN_TEST_NO_OOM_FALLBACK=1. Tests: OOM-once fixture rescued to exit 0; kill-switch stays red; banner advertises the sizing verdict. Documented in docs/TESTING.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test,cli): merge-seam repairs — shard timeout for the tripled suite, init flag-error contract, think scope exemption Three post-merge repairs surfaced by the first full-suite run: - Shard timeout 1500s -> 3000s: the suite roughly tripled since the cap was sized (~3900 -> 11k+ tests; PGLite inits replay 120 migrations, was 92). Two shards were killed mid-progress at 1500s. - The #2185 pre-dispatch validator now emits the same error contract as init.ts's in-handler check it preempts: lowercase 'unknown flag' on stderr + structured {status:'error', reason:'invalid_flag'} on stdout for --json callers (pinned by test/init-migrate-only.test.ts). - test/operations-trust-boundary.test.ts gets the same documented remote-gated allowlist for think's read scope (#2598) that test/oauth.test.ts already carries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): rescue lane also covers externally-killed shards (sibling-workspace pkill / memory jetsam) Second phantom class observed on a multi-workspace Conductor machine: 3 shards SIGTERM'd + 1 SIGKILL'd at ~700s under a 3000s cap, all mid-progress — an external killer, not a wedge. The dead shards then poisoned the serial pass (lock/state residue → 18 more phantoms), and every one of the 18 passed standalone. The runner now stamps per-shard start/end epochs; a shard dying on 143/137 before 80% of SHARD_TIMEOUT is classified externally-killed and its file list joins the serial rescue queue (real wedges die AT the cap and stay red). Serial-pass failures that occur while any shard was externally killed are treated as suspect residue and rescued too. Structural tests pin the detector, threshold, and routing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(auth): keep tokenEndpointAuthMethod terminal in the register-client destructure (PKCE structural contract) The #2016 absorb appended source/federatedRead after tokenEndpointAuthMethod; test/fix-wave-structural.test.ts pins tokenEndpointAuthMethod as the final destructured field (v0.36.1.x #1077 PKCE regression contract). The added fields move into the regex's optional-middle slot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): re-apply the #2598 remote-gated scope exemption to master's oauth scope-annotation test Taking master's v0.42.74.0 oauth.test.ts (its #2529 implementation) dropped the think read-scope allowlist that PR #2598 carries; re-applied to match test/operations-trust-boundary.test.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(review): pre-landing review batch — CLI-local flag acceptance, json= coherence, re-landed getChunks trim, payload-aware jsonb repair, rescue-lane tightening Ship Step 9 findings (checklist pass + 4 specialists), all verified before fixing: - CRITICAL: the #2185 validator rejected --source/--dry-run on op commands (makeContext CLI-locals consumed outside the op contract) — 'gbrain search "x" --source y' exited 1. Exempted with parser-mirroring value consumption + unit/subprocess tests incl. global-flag acceptance. - CRITICAL: --json=<v> diverged between validator (accepted) and parseOpArgs (junk-key path consumed the NEXT token, corrupting positionals). Parser now handles --json=true|false; =-forms of bare-only CLI-locals reject loud. parseOpArgs inline-= suite added (regression rule). - CRITICAL: the master merge silently restored SELECT cc.* in both engines' getChunks while docs claimed the #2544 trim. Re-landed the explicit non-vector column list + a source-level structural pin so a merge can't silently undo it again. - repair-jsonb/doctor: the subagent columns legitimately hold jsonb string scalars (persistToolExec binds pre-serialized strings) — unconditional unwrap would abort the repair run or corrupt legit values. jsonPayloadOnly predicate (JSON-container content only) on those 3 targets, mirrored in doctor + parameterized to_regclass + behavioral test (damage flagged, legit string ignored, absent table skipped). - runner rescue-lane tightening: serial failures rescue-eligible only with their own OOM signature or after an external shard kill (residue), never because a sibling shard OOM'd — flaky serial tests stay red. Rescue passes no longer double-count into TOTAL_PASS; shard timeout scales when mem-adaptation sheds shards; negative-path tests (mixed run stays red, deterministic OOM-signature failure stays red). - fail-closed remote spelling at 2 forward sites (ctx.remote !== false per the CLAUDE.md invariant); registry generator drops template-literal flag prefixes; real-PG e2es for the v121 + minion_jobs wedge classes; stale comments corrected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: regenerate flag registry after master merge (#3864 added extract help flags) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(review): red-team batch — dispatch-order validation, real --dry-run boolean, rescue-lane timeout/isolation/cap, parseable-payload repair gate Red-team pass (post-specialist) findings, all verified before fixing: - CRITICAL: validateCommandFlags checked the op lane before CLI_ONLY while dispatch runs CLI_ONLY first — dual-lane commands (think/salience/ anomalies) were validated against the WRONG contract, rejecting documented invocations ('salience --kind entity'). Lane order now mirrors dispatch. - CRITICAL: --dry-run was blessed as legal on op commands but parseOpArgs never SET it (trailing → nothing → ctx.dryRun false → the REAL destructive action ran; leading → consumed the next token). Now a CLI-local boolean exactly like --json, with --dry-run=false support and regression tests. - CRITICAL: the rescue lane ran bun test WITHOUT --timeout=60000 (bun default 5s) — PGLite phantoms re-failed on timeout and were mislabeled 'confirmed real'. Both rescue invocations now mirror the shard flags; serial files re-run one process per file (run-serial-tests.sh isolation contract); rescue wallclock capped at 2x the shard timeout. - CRITICAL: the jsonPayloadOnly probe matched container-LOOKING invalid JSON ('[INFO] fetch complete') whose repair cast would throw and abort the run mid-loop. Predicate now gates on pg_input_is_valid (PG16+, same floor as the existing IS JSON usage) + per-target catch records and continues; doctor mirrors; behavioral test covers the lookalike row. - Registry generator bounded at handleCliOnly's closing brace (the LAST case block absorbed ~100 junk flags from the rest of cli.ts, neutering strict validation for it); uppercase flag typos reject loudly in both lanes (handlers are lowercase-sensitive); --json=true spelling gets the structured invalid_flag envelope; get_chunks __all__ narrowing filed as a Wave 3 TODO. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(engines): getChunks trimmed SELECT must carry modality (codex P1-B) The #2544 egress trim replaced SELECT cc.* with an explicit column list but omitted cc.modality — every rowToChunk field except the vector must survive the trim, or the embed round-trip (getChunks -> upsertChunks) rewrites image chunks as text. Both engines; the structural pin now iterates the full rowToChunk field list instead of spot-checking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): safety flags require consumption evidence in the flag registry (codex P1-A) upgrade.ts prints a help hint naming another command's --dry-run; that literal is depth-0 text for post-upgrade, so the generator allowlisted --dry-run there — recreating the exact #2185 repro this wave kills (post-upgrade --dry-run accepted, ignored, migrations run for real). Safety flags now need a tight-quoted standalone literal (an args read like has('--dry-run')) before the registry grants them; prose bleed embeds the flag inside a longer string and never qualifies. Regenerated registry drops --dry-run from post-upgrade, keeps genuine consumers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * v0.42.76.0 fix(upgrade,security,cli): strict flag validation, bootstrap-wedge class kill, federated chunk scope Version bump + CHANGELOG for the fix wave: CLI-wide unknown-flag rejection with a generated per-command registry (#2185, #1575 class), minion_jobs bootstrap probes + migration-aware coverage guard (v121 wedge class), get_chunks federated scope + egress trim (#2555, half of #2544), think read-scope over MCP (#2598), register-client source bindings (#2016, #2143 enabler), repair-jsonb/doctor subagent columns with a parse-validated damage predicate, memory-safe unit-test runner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update project documentation for v0.42.76.0 CONTRIBUTING.md: test-runner claims match the memory-safe 4-shard default (was 8-shard) and the CLI-only command recipe now includes the build:flag-registry regen step. KEY_FILES.md: repair-jsonb entry updated to the 8-column parse-gated current state; new entries for the strict flag-validation subsystem and src/core/source-id.ts; operations/engine/ serve-http entries updated for get_chunks scope ladder + trimmed SELECT and the register-client HTTP source bindings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply doc-review fixes for v0.42.76.0 Cross-model doc review caught stragglers: docs/TESTING.md still said 8-shard in the file taxonomy, carried a two-generations-stale shard timeout default (600s -> 3000s), and didn't name the new run-unit-parallel regression test or the remaining runner knobs; CONTRIBUTING.md's fast-loop file count predated the tripled suite (92+ -> 1000+); test-count claims unified at 3700+; the CLIENT_FENCED_WRITE_OPS comment in operations.ts still described think as scope write; KEY_FILES names the exported findUnknownFlag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): rescue lane survives CI — strip ::group:: prefixes, satisfy the bun-test-timeout guard Two CI-only breaks from the master merge: (1) under GITHUB_ACTIONS the shard wraps file sections as ::group::path.test.ts, so the rescue pass extracted literal ::group:: non-paths that matched zero test files — failing_files_in_log now strips the prefix; (2) master's new check-bun-test-timeout guard greps for bare 'bun test' and tripped on run_rescue's comment text (the invocations themselves carry --timeout=60000) — comment reworded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): shard-mechanics tests disable mem-adaptation — CI's 7GB runner collapsed explicit 2 shards to 1 The runner deliberately adapts even explicit --shards to available memory (GBRAIN_TEST_NO_MEM_ADAPT=1 is the escape hatch); on GitHub's ~7GB runners that collapsed the tests' 2-shard sandbox runs to 1 shard, breaking every 'shard 1/2:' expectation while passing locally. The tests pin shard MECHANICS with tiny synthetic files, so they now set the escape hatch; the one test that checks the mem banner overrides it back on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Fable 5
Claude Fable 5
parent
485c4773ca
commit
130d321d23
@@ -0,0 +1,177 @@
|
||||
/**
|
||||
* #2185 — known-flags registry generator for CLI_ONLY commands.
|
||||
*
|
||||
* gbrain's CLI_ONLY commands read flags ad hoc (`args.includes('--force')`,
|
||||
* per-command parseFlags helpers), so there is no parser to make strict. The
|
||||
* pre-dispatch validator in src/cli.ts needs to know each command's legal
|
||||
* flags; this script derives them from the source instead of a hand-typed
|
||||
* list that would rot.
|
||||
*
|
||||
* How: parse handleCliOnly's top-level `case 'X': {` blocks out of src/cli.ts,
|
||||
* collect every `import('./commands/Y.ts')` inside each block, then scan the
|
||||
* case-block text plus each imported module (plus one level of that module's
|
||||
* ./relative same-directory imports) for `--flag` string literals — including
|
||||
* help text, which deliberately over-includes: accepting a flag the handler
|
||||
* ignores is the pre-#2185 status quo for that flag, while missing a real
|
||||
* flag would break working invocations on upgrade.
|
||||
*
|
||||
* Output: src/core/cli-flag-registry.generated.ts (committed; freshness is
|
||||
* pinned by test/cli-flag-validation.test.ts the same way build:llms pins the
|
||||
* llms bundles). Regenerate: bun run build:flag-registry
|
||||
*
|
||||
* Hand-tuning lane: EXTRA_FLAGS below, for flags that live deeper than the
|
||||
* one-level scan (add with a comment naming the deep module).
|
||||
*/
|
||||
import { readFileSync, writeFileSync, existsSync } from 'fs';
|
||||
import { dirname, resolve as resolvePath, join } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
|
||||
const ROOT = resolvePath(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
|
||||
/** Flags that live deeper than the one-level module scan. Keep commented. */
|
||||
const EXTRA_FLAGS: Record<string, string[]> = {
|
||||
// embed's pace knobs resolve inside src/core/pace-mode.ts (two levels deep).
|
||||
embed: ['--pace', '--pace-max-concurrency'],
|
||||
// sync shares the same pace surface via env/config plus CLI passthrough.
|
||||
sync: ['--pace', '--pace-max-concurrency'],
|
||||
};
|
||||
|
||||
/** Universal helper flags every command may see (parsed or short-circuited upstream). */
|
||||
const UNIVERSAL_FLAGS = ['--help', '--json', '--brain', '--source'];
|
||||
|
||||
const FLAG_RE = /--[a-z0-9][a-z0-9-]*/g;
|
||||
|
||||
function flagsInText(text: string): Set<string> {
|
||||
const out = new Set<string>();
|
||||
for (const m of text.matchAll(FLAG_RE)) {
|
||||
// Template-literal prefixes (`--bound-${key}` scans as `--bound-`) are
|
||||
// not real flags — a trailing hyphen would make the validator accept
|
||||
// every typo sharing the prefix.
|
||||
if (!m[0].endsWith('-')) out.add(m[0]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** One level of ./relative imports (static or dynamic) from a module's source. */
|
||||
function relativeImports(src: string, fromDir: string): string[] {
|
||||
const paths = new Set<string>();
|
||||
for (const m of src.matchAll(/from\s+'(\.\.?\/[^']+\.ts)'/g)) paths.add(m[1]);
|
||||
for (const m of src.matchAll(/import\('(\.\.?\/[^']+\.ts)'\)/g)) paths.add(m[1]);
|
||||
return [...paths]
|
||||
.map(p => resolvePath(fromDir, p))
|
||||
.filter(p => existsSync(p));
|
||||
}
|
||||
|
||||
export function buildFlagRegistry(): Record<string, string[]> {
|
||||
const cliSource = readFileSync(join(ROOT, 'src/cli.ts'), 'utf-8');
|
||||
|
||||
// CLI_ONLY membership (the single source of truth in src/cli.ts). Strip
|
||||
// line comments first — the set literal carries commentary whose quoted
|
||||
// words ('Unknown command', 'pages') must not parse as members.
|
||||
const onlyMatch = cliSource.match(/const CLI_ONLY = new Set(?:<string>)?\(\[([\s\S]*?)\]\)/);
|
||||
if (!onlyMatch) throw new Error('CLI_ONLY set not found in src/cli.ts');
|
||||
const onlyBody = onlyMatch[1].replace(/\/\/[^\n]*/g, '');
|
||||
const commands = [...onlyBody.matchAll(/'([^']+)'/g)].map(m => m[1]);
|
||||
|
||||
// handleCliOnly body — bounded at the function's closing brace (column 0).
|
||||
// Unbounded, the LAST case block absorbed every --flag literal in the rest
|
||||
// of cli.ts (printHelp's full flag surface included), handing whichever
|
||||
// command sits last in the switch a ~100-flag junk allowlist that made
|
||||
// strict validation a no-op for it.
|
||||
const fnStart = cliSource.indexOf('async function handleCliOnly');
|
||||
if (fnStart < 0) throw new Error('handleCliOnly not found in src/cli.ts');
|
||||
const fnTail = cliSource.slice(fnStart);
|
||||
const fnEndRel = fnTail.search(/\n\}\n/);
|
||||
const fnSrc = fnEndRel > 0 ? fnTail.slice(0, fnEndRel) : fnTail;
|
||||
|
||||
// handleCliOnly dispatches through TWO styles: an `if (command === 'X')`
|
||||
// chain (DB-free commands like init/auth/schema) and a switch with
|
||||
// `case 'X':` labels. Segment on BOTH marker kinds; the text between a
|
||||
// marker and the next marker belongs to that label.
|
||||
const markRe = /(?:^\s*if \(command === '([a-z0-9-]+)'\)|^ case '([a-z0-9-]+)':)/gm;
|
||||
const marks: Array<{ label: string; start: number }> = [];
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = markRe.exec(fnSrc)) !== null) {
|
||||
marks.push({ label: (m[1] ?? m[2])!, start: m.index });
|
||||
}
|
||||
|
||||
const blocks = new Map<string, string>();
|
||||
for (let i = 0; i < marks.length; i++) {
|
||||
const end = i + 1 < marks.length ? marks[i + 1].start : fnSrc.length;
|
||||
const body = fnSrc.slice(marks[i].start, end);
|
||||
// Fall-through labels share the following block.
|
||||
blocks.set(marks[i].label, (blocks.get(marks[i].label) ?? '') + body);
|
||||
}
|
||||
|
||||
// Safety flags carry destructive-bypass semantics: allowlisting one that
|
||||
// the handler never reads recreates the #2185 repro (`post-upgrade
|
||||
// --dry-run` accepted, ignored, migrations run for real). Presence isn't
|
||||
// enough — upgrade.ts prints a HINT naming another command's --dry-run,
|
||||
// which is depth-0 text for post-upgrade. These flags are only legal with
|
||||
// CONSUMPTION evidence in the command's own code: the flag as a TIGHT-QUOTED
|
||||
// standalone literal (`includes('--dry-run')`, `has('--dry-run')`,
|
||||
// `=== '--dry-run'`). Prose bleed embeds the flag inside a longer string, so
|
||||
// it never has quotes on both sides of the bare flag.
|
||||
const SAFETY_FLAGS = new Set(['--dry-run']);
|
||||
const consumes = (text: string, flag: string): boolean =>
|
||||
new RegExp(`['"\`]${flag}['"\`]`).test(text);
|
||||
|
||||
const registry: Record<string, string[]> = {};
|
||||
for (const command of commands) {
|
||||
const block = blocks.get(command) ?? '';
|
||||
const flags = new Set<string>(UNIVERSAL_FLAGS);
|
||||
const depthZero = new Set<string>();
|
||||
let depthZeroText = block;
|
||||
for (const f of flagsInText(block)) { flags.add(f); depthZero.add(f); }
|
||||
|
||||
// Modules imported inside the case block, plus one level of each module's
|
||||
// own ./relative imports.
|
||||
const commandModules = [...block.matchAll(/import\('(\.\/[^']+\.ts)'\)/g)]
|
||||
.map(mm => resolvePath(join(ROOT, 'src'), mm[1]))
|
||||
.filter(p => existsSync(p));
|
||||
for (const modPath of commandModules) {
|
||||
const modSrc = readFileSync(modPath, 'utf-8');
|
||||
depthZeroText += modSrc;
|
||||
for (const f of flagsInText(modSrc)) { flags.add(f); depthZero.add(f); }
|
||||
for (const dep of relativeImports(modSrc, dirname(modPath))) {
|
||||
for (const f of flagsInText(readFileSync(dep, 'utf-8'))) flags.add(f);
|
||||
}
|
||||
}
|
||||
|
||||
for (const f of EXTRA_FLAGS[command] ?? []) { flags.add(f); depthZero.add(f); }
|
||||
for (const f of SAFETY_FLAGS) {
|
||||
if (flags.has(f) && !consumes(depthZeroText, f)) flags.delete(f);
|
||||
}
|
||||
registry[command] = [...flags].sort();
|
||||
}
|
||||
return registry;
|
||||
}
|
||||
|
||||
export function renderRegistryModule(registry: Record<string, string[]>): string {
|
||||
const entries = Object.keys(registry)
|
||||
.sort()
|
||||
.map(cmd => ` '${cmd}': [${registry[cmd].map(f => `'${f}'`).join(', ')}],`)
|
||||
.join('\n');
|
||||
return `// AUTO-GENERATED by scripts/generate-flag-registry.ts — do not edit by hand.
|
||||
// Regenerate: bun run build:flag-registry
|
||||
// Freshness + drift pinned by test/cli-flag-validation.test.ts (#2185).
|
||||
//
|
||||
// Per-command legal flags for CLI_ONLY commands, derived from each command's
|
||||
// source (case block + imported modules + one level of relative imports +
|
||||
// scripts/generate-flag-registry.ts EXTRA_FLAGS). Deliberately over-inclusive
|
||||
// (help-text mentions count): accepting an ignored flag is the pre-#2185
|
||||
// status quo; missing a real one breaks working invocations.
|
||||
export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
|
||||
${entries}
|
||||
};
|
||||
`;
|
||||
}
|
||||
|
||||
if (import.meta.main) {
|
||||
const registry = buildFlagRegistry();
|
||||
const outPath = join(ROOT, 'src/core/cli-flag-registry.generated.ts');
|
||||
writeFileSync(outPath, renderRegistryModule(registry));
|
||||
const n = Object.keys(registry).length;
|
||||
const total = Object.values(registry).reduce((a, v) => a + v.length, 0);
|
||||
console.log(`wrote ${outPath} (${n} commands, ${total} flag entries)`);
|
||||
}
|
||||
@@ -13,9 +13,29 @@
|
||||
#
|
||||
# Env overrides:
|
||||
# SHARDS=N same as --shards
|
||||
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 1500)
|
||||
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 3000)
|
||||
# GBRAIN_TEST_SHARD_KILL_AFTER grace after TERM before KILL (default 30)
|
||||
# GBRAIN_TEST_MAX_CONCURRENCY passed through to bun test (default 4)
|
||||
# GBRAIN_TEST_MEM_PER_FILE_MB memory budget per concurrent test file used by
|
||||
# the adaptive sizing below (default 1536 — a
|
||||
# PGLite WASM instance reserves ~1-1.5GB)
|
||||
# GBRAIN_TEST_NO_MEM_ADAPT=1 disable memory-aware concurrency reduction
|
||||
# GBRAIN_TEST_NO_OOM_FALLBACK=1 disable the serial OOM-rescue pass
|
||||
#
|
||||
# Memory safety (two layers; both default-on):
|
||||
# 1. ADAPTIVE SIZING — before spawning, total concurrency (shards ×
|
||||
# intra-shard --max-concurrency) is capped to what available memory can
|
||||
# hold at GBRAIN_TEST_MEM_PER_FILE_MB per concurrent file. Concurrent
|
||||
# Conductor workspaces running their own suites shrink the budget
|
||||
# automatically instead of OOMing each other.
|
||||
# 2. SERIAL PHANTOM RESCUE — two phantom classes are re-run serially
|
||||
# (--max-concurrency 1) after the parallel pass: (a) failures whose
|
||||
# shard log carries the PGLite WASM out-of-memory signature, and
|
||||
# (b) shards killed EXTERNALLY (SIGTERM/SIGKILL well before the shard
|
||||
# timeout — sibling Conductor workspaces' process cleanup, macOS memory
|
||||
# jetsam). Phantoms pass serially and the run goes green with an
|
||||
# oom_rescued note; real failures fail again and stay red. Plain
|
||||
# assertion failures never match either signature.
|
||||
#
|
||||
# Output files (workspace-local; falls back to /tmp if .context/ unwritable):
|
||||
# .context/test-failures.log failure blocks (cleared at start)
|
||||
@@ -38,6 +58,35 @@ detect_cpus() {
|
||||
echo 4
|
||||
}
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# Available-memory detection (MB). macOS: vm_stat free + inactive +
|
||||
# speculative + purgeable pages (inactive/purgeable are reclaimable on
|
||||
# pressure, which is exactly the scenario we size for). Linux: MemAvailable.
|
||||
# Unknown platform → 0, and the caller skips adaptation entirely.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
detect_available_mem_mb() {
|
||||
if command -v vm_stat >/dev/null 2>&1; then
|
||||
vm_stat 2>/dev/null | awk '
|
||||
/page size of/ { psize = $8 }
|
||||
/Pages free/ { free = $NF }
|
||||
/Pages inactive/ { inactive = $NF }
|
||||
/Pages speculative/ { spec = $NF }
|
||||
/Pages purgeable/ { purge = $NF }
|
||||
END {
|
||||
gsub(/\./, "", free); gsub(/\./, "", inactive)
|
||||
gsub(/\./, "", spec); gsub(/\./, "", purge)
|
||||
if (psize == 0) psize = 16384
|
||||
printf "%d\n", (free + inactive + spec + purge) * psize / 1048576
|
||||
}'
|
||||
return
|
||||
fi
|
||||
if [ -r /proc/meminfo ]; then
|
||||
awk '/MemAvailable/ { printf "%d\n", $2 / 1024; found = 1 } END { if (!found) print 0 }' /proc/meminfo
|
||||
return
|
||||
fi
|
||||
echo 0
|
||||
}
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# Argument parsing. --shards N override wins over $SHARDS; both are clamped.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
@@ -79,12 +128,60 @@ INTRA_CONC="${MAX_CONCURRENCY_OVERRIDE:-${GBRAIN_TEST_MAX_CONCURRENCY:-4}}"
|
||||
# had completed in 968s. 1500s cap gives ~55% headroom over observed
|
||||
# 4-shard wallclock; real hangs still hit it. Override via
|
||||
# GBRAIN_TEST_SHARD_TIMEOUT=N.
|
||||
SHARD_TIMEOUT="${GBRAIN_TEST_SHARD_TIMEOUT:-1500}"
|
||||
# v0.42.74 sizing: 1500 -> 3000. The suite roughly tripled since the 1500s
|
||||
# cap was set (June: ~3900 tests, 92-migration PGLite replay; now: 11k+
|
||||
# tests, 120-migration replay per PGLite init). At 4 shards, two shards were
|
||||
# killed at 1500s while making steady per-test progress. 3000s keeps the
|
||||
# same ~55%-headroom doctrine over observed wallclock; real hangs still die.
|
||||
SHARD_TIMEOUT="${GBRAIN_TEST_SHARD_TIMEOUT:-3000}"
|
||||
SHARD_KILL_AFTER="${GBRAIN_TEST_SHARD_KILL_AFTER:-30}"
|
||||
if ! printf '%s' "$SHARD_KILL_AFTER" | grep -qE '^[0-9]+$' || [ "$SHARD_KILL_AFTER" -lt 1 ]; then
|
||||
echo "ERROR: invalid shard kill-after: $SHARD_KILL_AFTER" >&2; exit 2
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# Memory-aware concurrency (layer 1). Total concurrent test files =
|
||||
# N shards × INTRA_CONC; each concurrent file can hold a PGLite WASM
|
||||
# instance (~1-1.5GB reserved). 4×4 = 16 concurrent instances OOM'd on a
|
||||
# 128GB machine when other Conductor workspaces ran their suites at the
|
||||
# same time — every PGLite connect across every shard failed at once
|
||||
# ("Out of memory" at PGlite.create). Cap total concurrency to what's
|
||||
# actually available, keeping a 4GB reserve for the OS + bun itself.
|
||||
# Applies to explicit --shards overrides too (an operator who wants an
|
||||
# over-committed run sets GBRAIN_TEST_NO_MEM_ADAPT=1).
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
MEM_PER_FILE_MB="${GBRAIN_TEST_MEM_PER_FILE_MB:-1536}"
|
||||
MEM_NOTE=""
|
||||
if [ "${GBRAIN_TEST_NO_MEM_ADAPT:-0}" != "1" ]; then
|
||||
AVAIL_MB=$(detect_available_mem_mb)
|
||||
if [ "${AVAIL_MB:-0}" -gt 0 ] 2>/dev/null; then
|
||||
BUDGET_MB=$((AVAIL_MB - 4096))
|
||||
[ "$BUDGET_MB" -lt "$MEM_PER_FILE_MB" ] && BUDGET_MB="$MEM_PER_FILE_MB"
|
||||
MAX_TOTAL=$((BUDGET_MB / MEM_PER_FILE_MB))
|
||||
[ "$MAX_TOTAL" -lt 1 ] && MAX_TOTAL=1
|
||||
ORIG_N="$N"; ORIG_INTRA="$INTRA_CONC"
|
||||
# Shed shards before intra-shard concurrency: fewer bun processes frees
|
||||
# more than narrower ones (each process carries its own heap + WASM).
|
||||
while [ $((N * INTRA_CONC)) -gt "$MAX_TOTAL" ]; do
|
||||
if [ "$N" -gt 1 ]; then N=$((N - 1))
|
||||
elif [ "$INTRA_CONC" -gt 1 ]; then INTRA_CONC=$((INTRA_CONC - 1))
|
||||
else break
|
||||
fi
|
||||
done
|
||||
if [ "$N" != "$ORIG_N" ] || [ "$INTRA_CONC" != "$ORIG_INTRA" ]; then
|
||||
# Fewer shards → more files per shard → each shard legitimately runs
|
||||
# longer. Scale the per-shard cap by the shed ratio so adaptation
|
||||
# doesn't convert memory safety into false WEDGED verdicts.
|
||||
if [ "$N" -lt "$ORIG_N" ]; then
|
||||
SHARD_TIMEOUT=$((SHARD_TIMEOUT * ORIG_N / N))
|
||||
fi
|
||||
MEM_NOTE=" | mem-adapted ${ORIG_N}x${ORIG_INTRA}→${N}x${INTRA_CONC} (avail=${AVAIL_MB}MB, ${MEM_PER_FILE_MB}MB/file, timeout→${SHARD_TIMEOUT}s)"
|
||||
else
|
||||
MEM_NOTE=" | mem-ok (avail=${AVAIL_MB}MB)"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# Output directories. Prefer workspace-local .context/, fall back to /tmp.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
@@ -100,7 +197,7 @@ else
|
||||
mkdir -p "$LOG_DIR" || { echo "ERROR: cannot create log dir" >&2; exit 2; }
|
||||
fi
|
||||
# Clear from prior run.
|
||||
rm -f "$LOG_DIR"/shard-*.log "$LOG_DIR"/shard-*.exit "$LOG_DIR"/shard-*.wedged 2>/dev/null
|
||||
rm -f "$LOG_DIR"/shard-*.log "$LOG_DIR"/shard-*.exit "$LOG_DIR"/shard-*.wedged "$LOG_DIR"/shard-*.start "$LOG_DIR"/shard-*.end 2>/dev/null
|
||||
: > "$FAILURES_LOG"
|
||||
: > "$SUMMARY_FILE"
|
||||
|
||||
@@ -114,7 +211,7 @@ elif command -v timeout >/dev/null 2>&1; then TIMEOUT_BIN="timeout"
|
||||
fi
|
||||
|
||||
START_TS=$(date +%s)
|
||||
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | kill-after=${SHARD_KILL_AFTER}s | logs=$LOG_DIR" >&2
|
||||
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | kill-after=${SHARD_KILL_AFTER}s | logs=$LOG_DIR${MEM_NOTE}" >&2
|
||||
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "[unit-parallel] dry-run: would spawn $N shards with the above settings."
|
||||
@@ -133,6 +230,7 @@ SHARD_PIDS=()
|
||||
for i in $(seq 1 "$N"); do
|
||||
(
|
||||
SHARD_LOG="$LOG_DIR/shard-$i.log"
|
||||
date +%s > "$LOG_DIR/shard-$i.start"
|
||||
if [ -n "$TIMEOUT_BIN" ]; then
|
||||
"$TIMEOUT_BIN" --signal=TERM --kill-after="${SHARD_KILL_AFTER}s" "${SHARD_TIMEOUT}s" \
|
||||
env SHARD="$i/$N" \
|
||||
@@ -162,6 +260,7 @@ for i in $(seq 1 "$N"); do
|
||||
kill "$cap_pid" 2>/dev/null
|
||||
wait "$cap_pid" 2>/dev/null
|
||||
fi
|
||||
date +%s > "$LOG_DIR/shard-$i.end"
|
||||
echo "$rc" > "$LOG_DIR/shard-$i.exit"
|
||||
{ [ "$rc" = "124" ] || [ "$rc" = "137" ]; } && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
|
||||
) &
|
||||
@@ -316,6 +415,40 @@ TOTAL_FAILURES=0
|
||||
TOTAL_PASS=0
|
||||
TOTAL_SKIP=0
|
||||
TOTAL_RC=0
|
||||
|
||||
# Layer 2 state (serial OOM rescue). A shard whose log carries the WASM
|
||||
# out-of-memory signature gets its failing files queued for a serial re-run;
|
||||
# NON_OOM_FAIL records that at least one failure exists that the rescue lane
|
||||
# must NOT absolve (plain assertion failures, wedges without the signature).
|
||||
OOM_RE='Out of memory|WebAssembly\.Memory|RuntimeError: [Aa]borted|Aborted\(\)'
|
||||
OOM_RESCUE_LIST="$LOG_DIR/oom-rescue-files.txt"
|
||||
: > "$OOM_RESCUE_LIST"
|
||||
NON_OOM_FAIL=0
|
||||
# Set when any shard was killed externally — killed-midrun shards leave lock/
|
||||
# state residue that can poison the LATER serial pass, so serial failures are
|
||||
# only rescue-eligible under this flag (or their own OOM signature). A flaky
|
||||
# serial test in an otherwise-clean run must stay red.
|
||||
EXTERNAL_KILL_ANY=0
|
||||
|
||||
# failing_files_in_log: attribute each `(fail)` block to the test file whose
|
||||
# `path.test.ts:` header most recently preceded it in bun's output. Under
|
||||
# GITHUB_ACTIONS the shard wraps each file section as `::group::path.test.ts:`
|
||||
# — strip that prefix or the rescue pass feeds bun literal `::group::...`
|
||||
# non-paths that match zero test files (CI-only; local runs have no groups).
|
||||
failing_files_in_log() {
|
||||
local file="$1"
|
||||
[ -f "$file" ] || return 0
|
||||
awk '
|
||||
/^(::group::)?[^ ].*\.test\.ts:$/ {
|
||||
current = $0
|
||||
sub(/^::group::/, "", current)
|
||||
current = substr(current, 1, length(current) - 1)
|
||||
next
|
||||
}
|
||||
/^\(fail\) / && current != "" { print current }
|
||||
' "$file" | sort -u
|
||||
}
|
||||
|
||||
for i in $(seq 1 "$N"); do
|
||||
SHARD_LOG="$LOG_DIR/shard-$i.log"
|
||||
EXIT_FILE="$LOG_DIR/shard-$i.exit"
|
||||
@@ -330,17 +463,74 @@ for i in $(seq 1 "$N"); do
|
||||
TOTAL_FAILURES=$((TOTAL_FAILURES + fail_count))
|
||||
TOTAL_SKIP=$((TOTAL_SKIP + skip_count))
|
||||
|
||||
shard_oom=0
|
||||
if [ "$rc" != "0" ] && [ "${GBRAIN_TEST_NO_OOM_FALLBACK:-0}" != "1" ] \
|
||||
&& [ -f "$SHARD_LOG" ] && grep -qE "$OOM_RE" "$SHARD_LOG"; then
|
||||
shard_oom=1
|
||||
fi
|
||||
|
||||
# External-kill detection: rc 143 (SIGTERM) / 137 (SIGKILL) with the shard
|
||||
# dying before 80% of the shard timeout means something OUTSIDE the runner
|
||||
# killed it — sibling Conductor workspaces' process cleanup and macOS
|
||||
# memory jetsam both present exactly this way (observed: 3 shards TERM'd +
|
||||
# 1 KILL'd at ~700s under a 3000s cap, all mid-progress). A REAL wedge is
|
||||
# killed BY the runner at ~SHARD_TIMEOUT and stays red. Externally-killed
|
||||
# shards are phantoms: queue for the serial rescue lane like OOM.
|
||||
shard_external_kill=0
|
||||
if [ "$shard_oom" = "0" ] && [ "${GBRAIN_TEST_NO_OOM_FALLBACK:-0}" != "1" ] \
|
||||
&& { [ "$rc" = "143" ] || [ "$rc" = "137" ]; }; then
|
||||
s_start=$(cat "$LOG_DIR/shard-$i.start" 2>/dev/null) || s_start=""
|
||||
s_end=$(cat "$LOG_DIR/shard-$i.end" 2>/dev/null) || s_end=""
|
||||
if [ -n "$s_start" ] && [ -n "$s_end" ]; then
|
||||
s_elapsed=$((s_end - s_start))
|
||||
if [ "$s_elapsed" -lt $((SHARD_TIMEOUT * 80 / 100)) ]; then
|
||||
shard_external_kill=1
|
||||
EXTERNAL_KILL_ANY=1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -f "$WEDGED_FILE" ]; then
|
||||
TOTAL_RC=1
|
||||
if [ "$shard_external_kill" = "1" ]; then
|
||||
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
|
||||
echo "shard $i/$N: KILLED externally after ${s_elapsed}s (rc=$rc, well before ${SHARD_TIMEOUT}s cap — queued for serial rescue)" >> "$SUMMARY_FILE"
|
||||
elif [ "$shard_oom" = "1" ]; then
|
||||
# Wedged UNDER memory pressure: we can't attribute failures, so queue
|
||||
# the shard's entire file list for the serial rescue pass.
|
||||
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
|
||||
echo "shard $i/$N: WEDGED after ${SHARD_TIMEOUT}s (rc=$rc, OOM signature — queued for serial rescue)" >> "$SUMMARY_FILE"
|
||||
else
|
||||
NON_OOM_FAIL=1
|
||||
echo "shard $i/$N: WEDGED after ${SHARD_TIMEOUT}s (rc=$rc)" >> "$SUMMARY_FILE"
|
||||
fi
|
||||
{
|
||||
echo "--- shard $i: WEDGED after ${SHARD_TIMEOUT}s ---"
|
||||
[ -f "$SHARD_LOG" ] && tail -50 "$SHARD_LOG"
|
||||
echo ""
|
||||
} >> "$FAILURES_LOG"
|
||||
echo "shard $i/$N: WEDGED after ${SHARD_TIMEOUT}s (rc=$rc)" >> "$SUMMARY_FILE"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [ "$rc" != "0" ]; then
|
||||
if [ "$shard_oom" = "1" ]; then
|
||||
# One scan, reused for both the queue append and the emptiness check.
|
||||
shard_failing_files=$(failing_files_in_log "$SHARD_LOG")
|
||||
if [ -n "$shard_failing_files" ]; then
|
||||
printf '%s\n' "$shard_failing_files" >> "$OOM_RESCUE_LIST"
|
||||
else
|
||||
# OOM signature but no attributable files (e.g. bun died before any
|
||||
# file header) → rescue the whole shard.
|
||||
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
|
||||
fi
|
||||
elif [ "$shard_external_kill" = "1" ]; then
|
||||
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
|
||||
echo "shard $i/$N: KILLED externally after ${s_elapsed}s (rc=$rc — queued for serial rescue)" >> "$SUMMARY_FILE"
|
||||
else
|
||||
NON_OOM_FAIL=1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "shard $i/$N: pass=$pass_count fail=$fail_count skip=$skip_count rc=$rc" >> "$SUMMARY_FILE"
|
||||
|
||||
if [ "$rc" != "0" ]; then
|
||||
@@ -395,6 +585,17 @@ if [ "$SERIAL_FILES_COUNT" -gt 0 ]; then
|
||||
cat "$LOG_DIR/serial.log"
|
||||
if [ "$SERIAL_RC" != "0" ]; then
|
||||
TOTAL_RC=1
|
||||
if [ "${GBRAIN_TEST_NO_OOM_FALLBACK:-0}" != "1" ] \
|
||||
&& { grep -qE "$OOM_RE" "$LOG_DIR/serial.log" || [ "$EXTERNAL_KILL_ANY" = "1" ]; }; then
|
||||
# Serial failures are rescue-eligible ONLY with their own OOM signature
|
||||
# or when an externally-killed shard ran earlier in this invocation
|
||||
# (killed-midrun shards leave lock/state residue that poisons the serial
|
||||
# pass). A merely-OOM'd sibling shard is NOT grounds — a flaky serial
|
||||
# test must stay red rather than get silently absolved.
|
||||
failing_files_in_log "$LOG_DIR/serial.log" >> "$OOM_RESCUE_LIST"
|
||||
else
|
||||
NON_OOM_FAIL=1
|
||||
fi
|
||||
s_fail=$(bun_summary_count "fail" "$LOG_DIR/serial.log")
|
||||
TOTAL_FAILURES=$((TOTAL_FAILURES + s_fail))
|
||||
if [ "$s_fail" -gt 0 ]; then
|
||||
@@ -420,6 +621,92 @@ if [ "$SERIAL_FILES_COUNT" -gt 0 ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# Layer 2: serial OOM rescue. Re-run every file that failed inside an
|
||||
# OOM-signature shard, one at a time (1 shard, --max-concurrency 1), after
|
||||
# the parallel fan-out has fully drained. Phantom failures (the WASM ran out
|
||||
# of memory because 16 instances were up at once) pass here and the run goes
|
||||
# green with an oom_rescued note; real failures fail again and stay red.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
OOM_RESCUED=0
|
||||
OOM_RESCUE_NOTE=""
|
||||
sort -u "$OOM_RESCUE_LIST" -o "$OOM_RESCUE_LIST" 2>/dev/null
|
||||
# grep -c exits 1 on zero matches — assign in two steps so an empty rescue
|
||||
# list yields a single "0" (the grep_count double-output bug, same class).
|
||||
RESCUE_COUNT=$(grep -c . "$OOM_RESCUE_LIST" 2>/dev/null) || RESCUE_COUNT=0
|
||||
if [ "$TOTAL_RC" != "0" ] && [ "${RESCUE_COUNT:-0}" -gt 0 ]; then
|
||||
echo "════════════ OOM rescue pass ($RESCUE_COUNT files, serial) ════════════"
|
||||
echo "[unit-parallel] OOM signature detected — re-running $RESCUE_COUNT failing file(s) at --max-concurrency 1" >&2
|
||||
RESCUE_LOG="$LOG_DIR/oom-rescue.log"
|
||||
# 60s-per-file floor with the shard cap as a minimum, and 2x the shard cap
|
||||
# as a CEILING: a wedged shard queueing its whole file list must not turn
|
||||
# `bun run test` into an unbounded multi-hour serial re-run — hitting the
|
||||
# ceiling reads as a red rescue, not silence.
|
||||
RESCUE_TIMEOUT=$((RESCUE_COUNT * 60))
|
||||
[ "$RESCUE_TIMEOUT" -lt "$SHARD_TIMEOUT" ] && RESCUE_TIMEOUT="$SHARD_TIMEOUT"
|
||||
[ "$RESCUE_TIMEOUT" -gt $((SHARD_TIMEOUT * 2)) ] && RESCUE_TIMEOUT=$((SHARD_TIMEOUT * 2))
|
||||
# Split the queue: *.serial.test.ts files require one bun PROCESS per file
|
||||
# (run-serial-tests.sh's isolation contract — top-level mock.module leaks
|
||||
# across files in a shared registry); the remainder batches in one process.
|
||||
# Both lanes mirror the shard invocation's --timeout=60000 — bun's default
|
||||
# 5s per-test timeout would re-fail PGLite phantoms (120-migration replay)
|
||||
# and mislabel them 'confirmed real'.
|
||||
grep -v '\.serial\.test\.ts$' "$OOM_RESCUE_LIST" > "$LOG_DIR/oom-rescue-batch.txt" || true
|
||||
grep '\.serial\.test\.ts$' "$OOM_RESCUE_LIST" > "$LOG_DIR/oom-rescue-serial.txt" || true
|
||||
RESCUE_RC=0
|
||||
: > "$RESCUE_LOG"
|
||||
run_rescue() { # $1 = per-invocation timeout seconds; rest = test-file args
|
||||
local t="$1"; shift
|
||||
if [ -n "$TIMEOUT_BIN" ]; then
|
||||
"$TIMEOUT_BIN" --signal=TERM --kill-after="${SHARD_KILL_AFTER}s" "${t}s" \
|
||||
bun test --max-concurrency 1 --timeout=60000 "$@" >> "$RESCUE_LOG" 2>&1
|
||||
else
|
||||
bun test --max-concurrency 1 --timeout=60000 "$@" >> "$RESCUE_LOG" 2>&1
|
||||
fi
|
||||
}
|
||||
if [ -s "$LOG_DIR/oom-rescue-batch.txt" ]; then
|
||||
# shellcheck disable=SC2046
|
||||
run_rescue "$RESCUE_TIMEOUT" $(cat "$LOG_DIR/oom-rescue-batch.txt") || RESCUE_RC=1
|
||||
fi
|
||||
if [ -s "$LOG_DIR/oom-rescue-serial.txt" ]; then
|
||||
while IFS= read -r serial_file; do
|
||||
[ -n "$serial_file" ] || continue
|
||||
run_rescue 300 "$serial_file" || RESCUE_RC=1
|
||||
done < "$LOG_DIR/oom-rescue-serial.txt"
|
||||
fi
|
||||
cat "$RESCUE_LOG"
|
||||
r_pass=$(bun_summary_count "pass" "$RESCUE_LOG")
|
||||
r_fail=$(bun_summary_count "fail" "$RESCUE_LOG")
|
||||
if [ "$RESCUE_RC" = "0" ] && [ "$NON_OOM_FAIL" = "0" ]; then
|
||||
# Every failure in the run was OOM-phantom and every rescued file passed
|
||||
# serially: the run is green. Adjust the headline numbers so they reflect
|
||||
# the rescue verdict, and mark the earlier failure blocks superseded.
|
||||
TOTAL_RC=0
|
||||
OOM_RESCUED=1
|
||||
# Do NOT fold r_pass into TOTAL_PASS — the failing shard's own summary
|
||||
# already counted the rescued files' passing tests, so folding would
|
||||
# double-count. Rescue results ride in the note instead.
|
||||
TOTAL_FAILURES=0
|
||||
OOM_RESCUE_NOTE=" | oom_rescued=${RESCUE_COUNT}files(${r_pass}p serial)"
|
||||
{
|
||||
echo "--- OOM rescue: all $RESCUE_COUNT file(s) passed serially (${r_pass} tests) ---"
|
||||
echo "--- failure blocks above were WASM out-of-memory phantoms, superseded ---"
|
||||
} >> "$FAILURES_LOG"
|
||||
echo "oom-rescue: $RESCUE_COUNT files pass=$r_pass rc=0 (phantom OOM failures superseded)" >> "$SUMMARY_FILE"
|
||||
else
|
||||
# Real failures confirmed serially (or a non-OOM failure exists anyway).
|
||||
OOM_RESCUE_NOTE=" | oom_rescue_failed=${r_fail}real"
|
||||
awk '
|
||||
/^\(fail\) / { in_block=1; print "--- oom-rescue (serial, confirmed real): " $0; next }
|
||||
in_block {
|
||||
if (/^\(pass\)/ || /^\(skip\)/ || /^[[:space:]]*$/ || /__bun_test_summary__/) { in_block=0; print ""; next }
|
||||
print $0
|
||||
}
|
||||
' "$RESCUE_LOG" >> "$FAILURES_LOG"
|
||||
echo "oom-rescue: $RESCUE_COUNT files pass=$r_pass fail=$r_fail rc=$RESCUE_RC (real failures confirmed)" >> "$SUMMARY_FILE"
|
||||
fi
|
||||
fi
|
||||
|
||||
END_TS=$(date +%s)
|
||||
ELAPSED=$((END_TS - START_TS))
|
||||
|
||||
@@ -436,10 +723,10 @@ if [ "$TOTAL_RC" != "0" ]; then
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
tail -30 "$FAILURES_LOG"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP"
|
||||
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP${OOM_RESCUE_NOTE}"
|
||||
} >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP" >&2
|
||||
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP${OOM_RESCUE_NOTE}" >&2
|
||||
exit 0
|
||||
|
||||
Reference in New Issue
Block a user