Compare commits

..
Author SHA1 Message Date
Garry TanandClaude Fable 5 60fb33c0d9 fix(embed): stop worker pool from dispatching new slices after a sub-batch failure
Review finding on #3130: when one sub-batch rejected, the surviving pool
workers kept draining ALL remaining slices in the background after
embedBatch had already rejected — real provider spend post-failure,
onBatchComplete firing after the caller handled the error, and stacked
429 pressure when embedBatchWithBackoff retried while the failed run was
still draining. A shared failed flag now stops further dispatch (in-flight
sibling calls still settle, bounded by concurrency-1) and suppresses
post-failure progress callbacks. Pinned by a new test: 10 slices /
concurrency 2 / first call fails → no calls after rejection, no
completions reported.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:52:30 -07:00
Garry TanandClaude Fable 5 11ed0871c2 test: fix CI red on #3130 — withEnv for batch-concurrency env + close resetGateway shard-order poison window
Two real failures surfaced by this PR's re-sharding:

1. verify/check-test-isolation: embed-batch-concurrency.test.ts mutated
   process.env directly (R1). Now uses withEnv().

2. test (9) source-health "expected 1280 dimensions, not 1536": a file
   whose last afterEach calls resetGateway() leaves the gateway slot
   empty during the NEXT file's beforeAll (which runs before any
   beforeEach can restore the legacy 1536 pin), so initSchema() sizes
   the embedding column from the zembed-1/1280 defaults and every
   1536-d fixture in that file fails. Which pair collides depends on
   shard composition, so adding test files (as this PR does) can
   surface it anywhere. The legacy-embedding preload now also repairs
   the empty slot in a global afterEach (preload after-hooks run after
   file-local ones), closing the window at the root instead of
   patching one victim file.

Reproduced locally with a poison/afterEach-reset file followed by a
schema-creating file: embedding column typmod 1280 before the fix,
1536 after. check-test-isolation, typecheck, and the affected suites
all pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:08:07 -07:00
Garry TanandClaude Fable 5 595eeb7d6f fix(embed): per-request batch caps (google/dashscope) + parallel batch dispatch (#970 #1199 #1207 #1818)
Four embedding-throughput/correctness fixes:

- #970: google recipe now declares max_batch_tokens (204,800 — derived
  from Gemini's real limits: 100 inputs per batchEmbedContents × 2048
  tokens per input) + max_batch_count 100 + chars_per_token, silencing
  the missing-cap startup warning and enabling the gateway pre-split.
  Deliberately NOT the 2048 per-input limit, which would over-split 50x.

- #1199: new optional EmbeddingTouchpoint.max_batch_count enforced in
  splitByTokenBudget (flush at N inputs even when the token budget has
  room); dashscope sets 10 (provider hard-caps embeddings at 10 inputs
  per request). isTokenLimitError also learns DashScope's
  "batch size is invalid" message so recursive halving backstops it.

- #1207: gbrain import without --workers now resolves through the shared
  autoConcurrency policy (PGLite → 1, >100 files on Postgres → 4)
  instead of hardcoding serial; explicit --workers still wins.

- #1818: embedBatch dispatches its 100-input sub-batches through a
  bounded worker pool (default 4; EmbedBatchOptions.concurrency /
  GBRAIN_EMBED_BATCH_CONCURRENCY override) with index-addressed results
  so output order is preserved; single-batch fast path unchanged.

Also: listRecipes() now reads the exported RECIPES map instead of the
private ALL array (one source of truth; lets tests inject a synthetic
capless recipe to keep the startup-warning path covered now that every
real recipe declares a cap).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 14:39:33 -07:00
31 changed files with 549 additions and 766 deletions
+4 -82
View File
@@ -527,9 +527,6 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
process.on('SIGINT', () => { void shutdown('SIGINT'); });
let consecutiveErrors = 0;
// Parser-probe fixture warning is once-per-process, not once-per-cycle
// (compiled-binary installs have no source tree; don't spam the log).
let parserProbeFixtureWarned = false;
// v0.37.7.0 #1162 — counter for consecutive reconnect failures.
// Reset on every successful health probe or reconnect. Threshold
// controlled by GBRAIN_AUTOPILOT_MAX_RECONNECT_FAILS env (default 30).
@@ -1076,36 +1073,17 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// loop. Probe runs even when cycleOk=false (probe may surface signal
// explaining why the cycle is failing).
try {
const { resolveProbeEnabled, resolveProbeMaxUsd, runNightlyQualityProbe } = await import('../core/cycle/nightly-quality-probe.ts');
// Dual-plane read: `gbrain config set` (what the doctor enable hint
// prints) writes the DB plane; ~/.gbrain/config.json is the fallback.
let dbEnabled: string | null = null;
let dbMaxUsd: string | null = null;
try {
dbEnabled = await engine.getConfig('autopilot.nightly_quality_probe.enabled');
dbMaxUsd = await engine.getConfig('autopilot.nightly_quality_probe.max_usd');
} catch { /* DB unavailable → file plane only */ }
const probeEnabled = resolveProbeEnabled(dbEnabled, cfg?.autopilot?.nightly_quality_probe?.enabled);
const probeEnabled = cfg?.autopilot?.nightly_quality_probe?.enabled === true;
if (probeEnabled) {
const { runNightlyQualityProbe } = await import('../core/cycle/nightly-quality-probe.ts');
const { runLongMemEvalForProbe, runCrossModalBatchForProbe } = await import('../core/cycle/nightly-probe-adapters.ts');
const { isAvailable } = await import('../core/ai/gateway.ts');
const { existsSync } = await import('node:fs');
const { fileURLToPath } = await import('node:url');
const { join } = await import('node:path');
const maxUsd = resolveProbeMaxUsd(dbMaxUsd, cfg?.autopilot?.nightly_quality_probe?.max_usd);
// The committed fixture (test/fixtures/longmemeval-nightly.jsonl)
// lives in the gbrain PACKAGE, not the brain repo — repoPath is
// sync.repo_path (the user's brain), where the fixture never
// exists, so the probe error'd on every real install. Resolve the
// package root from the module location; keep repoPath as the
// fallback for setups that vendor the fixture into the brain repo.
const pkgRoot = fileURLToPath(new URL('../..', import.meta.url));
const fixtureAtPkgRoot = existsSync(join(pkgRoot, 'test', 'fixtures', 'longmemeval-nightly.jsonl'));
const maxUsd = Number(cfg?.autopilot?.nightly_quality_probe?.max_usd ?? 5);
await runNightlyQualityProbe({
isEnabled: () => true, // already gated above; phase re-checks for defense-in-depth
hasEmbeddingProvider: () => isAvailable('embedding'),
resolveMaxUsd: () => maxUsd,
resolveRepoRoot: () => (fixtureAtPkgRoot ? pkgRoot : repoPath ?? gbrainHomePath('.')),
resolveRepoRoot: () => repoPath ?? gbrainHomePath('.'),
runLongMemEval: runLongMemEvalForProbe,
runCrossModalBatch: runCrossModalBatchForProbe,
now: () => new Date(),
@@ -1117,62 +1095,6 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// informational; autopilot loop continues.
}
// 4.6 — Nightly conversation-parser probe (v0.41.16.0 phase module;
// the scheduler wire-up was deferred at ship and is added here). Same
// posture as 4.5: the phase owns its gates (enabled/mode-gate, LLM
// key), the wiring owns invocation + the audit row, and a probe
// failure NEVER crashes the autopilot loop. Per D10 the probe is
// default-ON for search.mode=tokenmax, opt-in otherwise.
try {
const { runConversationParserNightlyProbe } = await import('../core/conversation-parser/nightly-probe.ts');
const { logParserProbeEvent, parserProbeRanWithin } = await import('../core/audit-parser-probe.ts');
const { isAvailable } = await import('../core/ai/gateway.ts');
const { existsSync } = await import('node:fs');
const { fileURLToPath } = await import('node:url');
const { join } = await import('node:path');
// Flag reads dual-plane: the DB row (`gbrain config set …`) wins,
// ~/.gbrain/config.json is the fallback. search.mode lives on the
// DB plane only (mode.ts owns it).
let parserDbEnabled: string | null = null;
let dbSearchMode: string | null = null;
try {
parserDbEnabled = await engine.getConfig('autopilot.conversation_parser_probe.enabled');
dbSearchMode = await engine.getConfig('search.mode');
} catch { /* DB unavailable → file plane only */ }
const parserEnabled = parserDbEnabled != null
? parserDbEnabled === 'true'
: cfg?.autopilot?.conversation_parser_probe?.enabled === true;
const searchMode = dbSearchMode ?? '';
// Fixtures are committed in the gbrain package (test/fixtures/…),
// NOT the brain repo — resolve from the module location. Compiled
// binaries carry no source tree: skip quietly instead of writing
// failure rows that would flip doctor to WARN on every binary install.
const pkgRoot = fileURLToPath(new URL('../..', import.meta.url));
const fixturePath = join(pkgRoot, 'test', 'fixtures', 'conversation-formats', 'all.jsonl');
const adversarialPath = join(pkgRoot, 'test', 'fixtures', 'conversation-formats', 'adversarial.jsonl');
const shouldInvoke = parserEnabled || searchMode === 'tokenmax';
if (shouldInvoke && existsSync(fixturePath) && existsSync(adversarialPath)) {
const result = await runConversationParserNightlyProbe({
isEnabled: () => parserEnabled,
searchMode: () => searchMode,
hasLlmKey: () => isAvailable('chat'),
resolveFixturePath: () => fixturePath,
resolveAdversarialPath: () => adversarialPath,
now: () => new Date(),
shouldSkipForRateLimit: () => parserProbeRanWithin(24 * 60 * 60 * 1000),
});
// rate_limited is a non-run: the loop ticks every few minutes, so
// logging every skip would flood the audit file with no-signal rows.
if (result.outcome !== 'rate_limited') logParserProbeEvent(result);
} else if (shouldInvoke && !parserProbeFixtureWarned) {
parserProbeFixtureWarned = true;
console.error(`[parser-probe] fixtures not found under ${pkgRoot}; skipping (probe needs a source-checkout install)`);
}
} catch (e) {
logError('autopilot.parser_probe', e);
// Informational, like 4.5: do NOT bump consecutiveErrors.
}
// Wait for next cycle
await new Promise(r => setTimeout(r, interval * 1000));
}
+14 -79
View File
@@ -2960,54 +2960,6 @@ function _resolveSyncFreshnessHours(varName: string, fallback: number): number {
* branch (disabled / enabled-no-events / enabled-all-pass / enabled-with-failures)
* without spinning up the audit JSONL or a real config file.
*/
/**
* Pure function form of the conversation_parser_probe_health check.
* Mirrors computeNightlyQualityProbeHealthCheck: skip-with-hint when the
* probe is off and silent, surface the last 7 days of audit events when
* it has run, WARN on any non-pass outcome.
*
* `effectiveEnabled` folds the D10 mode-gate in: explicitly enabled OR
* search.mode=tokenmax (where the probe is default-on).
*/
export function computeConversationParserProbeHealthCheck(
effectiveEnabled: boolean,
events: ReadonlyArray<{ outcome: string; ts: string; reason?: string }>,
): Check {
const name = 'conversation_parser_probe_health';
if (!effectiveEnabled && events.length === 0) {
return {
name,
status: 'ok',
message:
'disabled (opt-in; default-on only for search.mode=tokenmax). Enable with: ' +
'`gbrain config set autopilot.conversation_parser_probe.enabled true`',
};
}
if (events.length === 0) {
return {
name,
status: 'ok',
message: 'enabled but no probe events in the last 7 days (next run by autopilot; fixtures require a source-checkout install).',
};
}
const bad = events.filter(e => e.outcome !== 'pass');
const latest = events[events.length - 1]!;
if (bad.length > 0) {
return {
name,
status: 'warn',
message:
`${bad.length}/${events.length} probe run(s) in the last 7 days did not pass; ` +
`latest: ${latest.outcome}${latest.reason ? ` (${latest.reason})` : ''}`,
};
}
return {
name,
status: 'ok',
message: `${events.length} probe run(s) in the last 7 days, all pass (latest ${latest.ts}).`,
};
}
export function computeNightlyQualityProbeHealthCheck(
probeEnabled: boolean,
events: ReadonlyArray<{ outcome: string; ts: string; detail?: string }>,
@@ -4891,17 +4843,10 @@ export async function buildChecks(
try {
const { readRecentQualityProbeEvents } = await import('../core/audit-quality-probe.ts');
const { loadConfig } = await import('../core/config.ts');
const { resolveProbeEnabled } = await import('../core/cycle/nightly-quality-probe.ts');
let probeEnabled = false;
try {
// Dual-plane read, matching the autopilot gate: the DB row (what the
// enable hint's `gbrain config set` writes) wins; file plane fallback.
let dbVal: string | null = null;
try {
dbVal = engine ? await engine.getConfig('autopilot.nightly_quality_probe.enabled') : null;
} catch { /* DB unavailable → file plane only */ }
const cfg = loadConfig();
probeEnabled = resolveProbeEnabled(dbVal, (cfg as any)?.autopilot?.nightly_quality_probe?.enabled);
probeEnabled = Boolean((cfg as any)?.autopilot?.nightly_quality_probe?.enabled);
} catch { /* config unavailable → treat as disabled */ }
const events = readRecentQualityProbeEvents(7);
const check = computeNightlyQualityProbeHealthCheck(probeEnabled, events);
@@ -5085,29 +5030,19 @@ export async function buildChecks(
// 3d.5 v0.41.13.0 — conversation_parser_probe_health. Mode-gated
// per D10: ON when search.mode=tokenmax, opt-in for other modes.
// Surfaces the last 7 days of nightly-probe audit events; warn on any
// non-pass outcome (fail / budget_exceeded / adversarial_false_positive).
// (Until the autopilot wire-up this was a hardcoded "Skipped" stub.)
try {
const { readRecentParserProbeEvents } = await import('../core/audit-parser-probe.ts');
let parserProbeEnabled = false;
try {
let dbVal: string | null = null;
let dbMode: string | null = null;
try {
dbVal = engine ? await engine.getConfig('autopilot.conversation_parser_probe.enabled') : null;
dbMode = engine ? await engine.getConfig('search.mode') : null;
} catch { /* DB unavailable → file plane only */ }
const { loadConfig } = await import('../core/config.ts');
const fileVal = (loadConfig() as any)?.autopilot?.conversation_parser_probe?.enabled;
const flagOn = dbVal != null ? dbVal === 'true' : fileVal === true;
parserProbeEnabled = flagOn || dbMode === 'tokenmax';
} catch { /* config unavailable → treat as disabled */ }
const parserEvents = readRecentParserProbeEvents(7);
checks.push(computeConversationParserProbeHealthCheck(parserProbeEnabled, parserEvents));
} catch {
// Best-effort; audit-log read failure shouldn't stop doctor.
}
// Surface the last 7 days of nightly-probe events; warn on FAIL /
// BUDGET_EXCEEDED / adversarial_false_positive.
//
// v0.41.13.0 ships the probe as opt-in (autopilot wiring deferred
// to T7 in the cathedral plan); this check skips with an enable
// hint until the probe has at least one audit event written.
checks.push({
name: 'conversation_parser_probe_health',
status: 'ok',
message:
'Skipped (nightly probe is opt-in; enable with ' +
'`gbrain config set autopilot.conversation_parser_probe.enabled true`)',
});
// 3e. home_dir_in_worktree (v0.35.8.0). Walks up from `gbrainPath()`
// looking for a `.git` directory OR file. If found, warns: `~/.gbrain/`
+2 -15
View File
@@ -76,7 +76,7 @@ FLAGS:
dimensions (goal, depth, sourcing, specificity, useful).
--cycles N 1-3. Default: 3 in TTY, 1 in non-TTY (T11). Each
cycle is 3 model calls; verdict aggregates over them.
--slot-a-model <id> Override default 'openai:gpt-5.2'.
--slot-a-model <id> Override default 'openai:gpt-4o'.
--slot-b-model <id> Override default 'anthropic:claude-opus-4-7'.
--slot-c-model <id> Override default 'google:gemini-1.5-pro'.
--receipt-dir <path> Default: gbrainPath('eval-receipts').
@@ -468,14 +468,6 @@ interface BatchRow {
question_id: string;
question: string;
hypothesis: string;
/**
* Gold answer from the benchmark dataset, when the upstream eval emits
* it (eval-longmemeval does). Folded into the judge task so CORRECTNESS
* is verifiable — without it a judge panel that sees only
* {question, hypothesis} cannot validate a terse factual answer against
* a haystack it never saw.
*/
answer?: string;
}
/**
@@ -589,7 +581,6 @@ function readBatchRows(path: string): BatchReadResult {
question_id: typeof obj.question_id === 'string' ? obj.question_id : `line-${lineNo}`,
question: obj.question,
hypothesis: obj.hypothesis,
...(typeof obj.answer === 'string' && obj.answer.length > 0 ? { answer: obj.answer } : {}),
});
}
if (summarySkipped > 0) {
@@ -706,11 +697,7 @@ async function runBatchMode(parsed: ParsedArgs, opts: RunCrossModalOpts): Promis
fn: async (row, idx) => {
process.stderr.write(`[eval cross-modal batch] ${idx + 1}/${rows.length} ${row.question_id} starting...\n`);
return await runEvalFn({
// With a gold answer the judges can actually verify correctness;
// without one they see only {question, hypothesis} and cannot.
task: row.answer
? `${row.question}\n\nExpected answer (gold label from the benchmark dataset): ${row.answer}`
: row.question,
task: row.question,
output: row.hypothesis,
slug: row.question_id,
dimensions,
+3 -17
View File
@@ -33,7 +33,6 @@ import {
type AliasMap,
} from '../eval/longmemeval/extract.ts';
import { extractCandidateEntities } from '../core/think/entity-extract.ts';
import { splitProviderModelId } from '../core/model-id.ts';
import { resolveEntitySlugWithSource, type ResolutionSource } from '../core/entities/resolve.ts';
import { formatTrajectoryBlock } from '../core/trajectory-format.ts';
@@ -470,22 +469,14 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
});
// Wrap Anthropic SDK so its `.messages.create` shape matches ThinkLLMClient.
// Same pattern as src/core/think/index.ts:247-249 — EXCEPT think's default
// client routes through the gateway, which parses `provider:model` recipe
// ids. This eval's client is a raw SDK by design (hermetic, no gateway
// dependency), and resolveModel returns RECIPE ids (`anthropic:claude-…`);
// passing one through unstripped 404s every answer/extractor call, which
// surfaces downstream as all-upstream_error batches in the nightly probe.
const toSdkModel = (m: string): string => splitProviderModelId(m).model || m;
// Same pattern as src/core/think/index.ts:247-249.
const realClient = new Anthropic();
const client: ThinkLLMClient = runOpts.client ?? {
create: (params, callOpts) =>
realClient.messages.create({ ...params, model: toSdkModel(params.model) }, callOpts),
create: (params, callOpts) => realClient.messages.create(params, callOpts),
};
// v0.40.2.0 — separate extractor client (defaults to same SDK).
const extractorClient: ThinkLLMClient = runOpts.extractorClient ?? {
create: (params, callOpts) =>
realClient.messages.create({ ...params, model: toSdkModel(params.model) }, callOpts),
create: (params, callOpts) => realClient.messages.create(params, callOpts),
};
const trajectoryEnabled = !opts.noTrajectory;
const extractorModel = trajectoryEnabled
@@ -760,11 +751,6 @@ async function runOneQuestion(
// v0.40.1.0 (Track D / T2) — copy question_type into the row so the
// by_type_summary can be rebuilt from the file on resume runs.
question_type: q.question_type,
// Gold answer for downstream consumers that verify correctness (the
// cross-modal --batch judge folds it into the task; evaluate_qa.py
// ignores unknown fields). Without it a judge can't validate a terse
// factual hypothesis against a haystack it never saw.
...(q.answer !== undefined ? { answer: q.answer } : {}),
hypothesis,
retrieved_session_ids: retrievedSessionIds,
...(recallHit !== undefined ? { recall_hit: recallHit } : {}),
+10 -5
View File
@@ -170,10 +170,14 @@ export async function runImport(
// v0.22.13 (PR #490 Q2): shared parseWorkers helper rejects bad input
// (--workers 0, -3, "foo") with a loud error instead of silently falling
// through to 1. Mirrors sync.ts's flag handling.
const { parseWorkers } = await import('../core/sync-concurrency.ts');
let workerCount: number;
const { parseWorkers, autoConcurrency } = await import('../core/sync-concurrency.ts');
// #1207: undefined (no --workers flag) defers to autoConcurrency below —
// the shared sync/import policy (PGLite → 1, >100 files → 4) — instead of
// hardcoding serial. Large Postgres imports stop paying one embedding
// round-trip per file in sequence.
let workerCount: number | undefined;
try {
workerCount = parseWorkers(workersArg ?? undefined) ?? 1;
workerCount = parseWorkers(workersArg ?? undefined);
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
@@ -252,8 +256,9 @@ export async function runImport(
}
const files = resumeFilter(allFiles, dir, completed);
// Determine actual worker count
const actualWorkers = workerCount > 1 ? workerCount : 1;
// Determine actual worker count. Explicit --workers wins; otherwise the
// shared autoConcurrency policy decides from engine kind + file count.
const actualWorkers = autoConcurrency(engine, files.length, workerCount);
if (actualWorkers > 1) {
console.log(`Using ${actualWorkers} parallel workers`);
}
+24 -6
View File
@@ -1513,12 +1513,21 @@ export async function embed(texts: string[], opts?: EmbedOpts): Promise<Float32A
const embedding = recipe.touchpoints?.embedding;
const maxBatchTokens = embedding?.max_batch_tokens;
const maxBatchCount = embedding?.max_batch_count;
const charsPerToken = embedding?.chars_per_token ?? DEFAULT_CHARS_PER_TOKEN;
// Pre-split is gated on max_batch_tokens. Recipes without it (e.g. OpenAI)
// ride the fast path: one embedMany call, no recursion safety net.
const batches = maxBatchTokens
? splitByTokenBudget(truncated, Math.floor(maxBatchTokens * effectiveSafetyFactor(recipe)), charsPerToken)
// Pre-split is gated on max_batch_tokens / max_batch_count. Recipes with
// neither (e.g. OpenAI) ride the fast path: one embedMany call, no
// recursion safety net.
const batches = (maxBatchTokens || maxBatchCount)
? splitByTokenBudget(
truncated,
maxBatchTokens
? Math.floor(maxBatchTokens * effectiveSafetyFactor(recipe))
: Number.MAX_SAFE_INTEGER,
charsPerToken,
maxBatchCount,
)
: [truncated];
const allEmbeddings: Float32Array[] = [];
@@ -1568,6 +1577,9 @@ export async function embed(texts: string[], opts?: EmbedOpts): Promise<Float32A
* responsible for applying any safety-factor shrink before passing in.
* @param charsPerToken - Provider-specific character density. Defaults to
* `DEFAULT_CHARS_PER_TOKEN` (4) when omitted, matching OpenAI tiktoken.
* @param maxBatchCount - #1199: optional cap on INPUTS per sub-batch, for
* providers that reject batches by count (DashScope: 10). When omitted,
* only the token budget governs.
*
* @internal exported for tests; not part of the public gateway API.
*/
@@ -1575,15 +1587,17 @@ export function splitByTokenBudget(
texts: string[],
budgetTokens: number,
charsPerToken: number = DEFAULT_CHARS_PER_TOKEN,
maxBatchCount?: number,
): string[][] {
const ratio = charsPerToken > 0 ? charsPerToken : DEFAULT_CHARS_PER_TOKEN;
const maxCount = maxBatchCount !== undefined && maxBatchCount > 0 ? maxBatchCount : Infinity;
const batches: string[][] = [];
let current: string[] = [];
let currentTokens = 0;
for (const text of texts) {
const estTokens = Math.ceil(text.length / ratio);
if (current.length > 0 && currentTokens + estTokens > budgetTokens) {
if (current.length > 0 && (currentTokens + estTokens > budgetTokens || current.length >= maxCount)) {
batches.push(current);
current = [];
currentTokens = 0;
@@ -1609,7 +1623,11 @@ export function isTokenLimitError(err: unknown): boolean {
/token.*limit.*exceeded/i.test(msg) ||
// OpenAI embeddings: "Invalid 'input': maximum request size is 300000 tokens per request."
/maximum request size.*tokens/i.test(msg) ||
/max.*tokens.*per.*request/i.test(msg)
/max.*tokens.*per.*request/i.test(msg) ||
// DashScope: "batch size is invalid, it should not be larger than 10." (#1199)
// Count-cap error, but recursive halving shrinks count too, so the same
// safety net converges.
/batch size is invalid/i.test(msg)
);
}
+4
View File
@@ -31,6 +31,10 @@ export const dashscope: Recipe = {
// path. Conservative declaration so the gateway pre-splits before
// hitting whatever undocumented server-side limit exists.
max_batch_tokens: 8192,
// #1199: DashScope hard-caps embeddings at 10 inputs per request
// ("batch size is invalid, it should not be larger than 10"). The
// token budget alone admits far more than 10 short chunks per batch.
max_batch_count: 10,
// text-embedding-v3 mixes English + CJK heavily; the tokenizer is
// closer to Voyage density than OpenAI tiktoken for CJK-dominant
// content. Conservative chars_per_token=2 leaves headroom.
+9
View File
@@ -16,6 +16,15 @@ export const google: Recipe = {
dims_options: [768, 1536, 3072],
cost_per_1m_tokens_usd: 0.15,
price_last_verified: '2026-04-20',
// #970: Gemini's documented limits are per-INPUT (2048 tokens,
// silently truncated beyond) and per-REQUEST count (batchEmbedContents
// caps at 100 inputs). There is no separate per-request token cap, so
// the token budget is derived: 100 inputs × 2048 tokens. The count cap
// binds first for typical chunk sizes. Do NOT copy the 2048 per-input
// limit into max_batch_tokens — that would over-split 50×.
max_batch_tokens: 204_800,
chars_per_token: 4,
max_batch_count: 100,
},
expansion: {
models: ['gemini-2.0-flash', 'gemini-2.0-flash-lite'],
+4 -1
View File
@@ -58,5 +58,8 @@ export function getRecipe(id: string): Recipe | undefined {
}
export function listRecipes(): Recipe[] {
return [...ALL];
// Read the map (not ALL) so there is one source of truth — getRecipe,
// model-resolver, and listRecipes all see the same registry, and tests
// can inject a synthetic recipe via RECIPES to exercise registry walks.
return [...RECIPES.values()];
}
+10
View File
@@ -46,6 +46,16 @@ export interface EmbeddingTouchpoint {
* Only consulted when `max_batch_tokens` is also set.
*/
chars_per_token?: number;
/**
* #1199: maximum number of INPUTS per embedding request, for providers
* that hard-cap batch size by count rather than (or in addition to)
* tokens DashScope text-embedding-v3 rejects batches > 10 with
* `InvalidParameter`, Gemini batchEmbedContents caps at 100 requests.
* When set, the gateway's pre-split flushes a sub-batch at this count
* even if the token budget still has room. Independent of
* `max_batch_tokens`; either alone triggers the pre-split.
*/
max_batch_count?: number;
/**
* Budget-utilization ceiling in (0, 1]. The gateway pre-splits at
* `safety_factor × max_batch_tokens` to leave headroom for tokenizer
-63
View File
@@ -1,63 +0,0 @@
/**
* Nightly conversation-parser probe audit trail.
*
* One event per REAL probe run lands in
* `~/.gbrain/audit/parser-probe-YYYY-Www.jsonl` (ISO-week rotation via the
* shared audit-writer primitive; honors `GBRAIN_AUDIT_DIR`).
* Scheduler-cadence skips (`rate_limited`) are NOT logged the autopilot
* loop ticks every few minutes, so logging every skip would flood the
* audit file with rows that carry no signal.
*
* Read by `gbrain doctor`'s `conversation_parser_probe_health` check and
* by the autopilot wiring's 24h rate-limit gate (`parserProbeRanWithin`).
*/
import { createAuditWriter } from './audit/audit-writer.ts';
import type { NightlyProbeResult } from './conversation-parser/nightly-probe.ts';
export type ParserProbeAuditEvent = NightlyProbeResult;
const writer = createAuditWriter<ParserProbeAuditEvent>({
featureName: 'parser-probe',
errorLabel: 'gbrain',
errorMessagePrefix: 'parser-probe audit ',
errorTrailer: '; probe continues',
});
/** Append one parser-probe event. Best-effort; never throws. */
export function logParserProbeEvent(event: ParserProbeAuditEvent): void {
writer.log(event);
}
/**
* Read recent parser-probe events (current + previous ISO week, filtered
* to the window). Missing files and corrupt rows are skipped silently.
*/
export function readRecentParserProbeEvents(
days = 7,
now: Date = new Date(),
): ParserProbeAuditEvent[] {
return writer.readRecent(days, now);
}
/** Exposed for tests pinning the rotation edge cases. */
export function computeParserProbeAuditFilename(now: Date = new Date()): string {
return writer.computeFilename(now);
}
/**
* 24h rate-limit gate for the autopilot wiring: true when any audited run
* happened within `windowMs` of `now`. Only REAL outcomes are audited (see
* module header), so a pass/fail today blocks re-runs until tomorrow while
* scheduler-cadence skips never extend the window.
*/
export function parserProbeRanWithin(
windowMs: number,
now: Date = new Date(),
): boolean {
const cutoff = now.getTime() - windowMs;
return readRecentParserProbeEvents(2, now).some((ev) => {
const ts = Date.parse(ev.ts);
return Number.isFinite(ts) && ts >= cutoff;
});
}
-10
View File
@@ -105,16 +105,6 @@ export interface GBrainConfig {
*/
max_usd?: number;
};
/**
* v0.41.16.0 nightly conversation-parser probe. Per D10: default ON
* for `search.mode=tokenmax` brains, opt-in for conservative/balanced.
* ~$0.05/night with the committed fixtures × Haiku polish. Gated
* INSIDE the autopilot tick body, like nightly_quality_probe.
*/
conversation_parser_probe?: {
/** Enable for non-tokenmax modes. Defaults to false. */
enabled?: boolean;
};
/**
* v0.42.x (#1685 GAP D) extract_atoms backlog auto-drain. Default ON so a
* pack-gated silent backlog never piles up unseen; daily-spend-capped so the
@@ -17,11 +17,11 @@
* Cost: ~$0.05/night with default fixtures × Haiku polish. Bounded
* by the active BudgetTracker the autopilot loop creates per-tick.
*
* Wired into the autopilot loop (step 4.6 in autopilot.ts), following
* the same shape as `src/core/cycle/nightly-quality-probe.ts`
* (v0.40.1.0 Track D / T6): the wiring resolves fixtures from the
* gbrain package root, writes real outcomes to the parser-probe audit
* trail (`audit-parser-probe.ts`), and never crashes the loop.
* **Wiring into the autopilot loop is deferred to a follow-up**
* (filed in TODOS.md). v0.41.16.0 ships the phase as a callable
* module so doctor + future cron drivers can invoke it; the
* scheduler wire-up follows the same shape as
* `src/core/cycle/nightly-quality-probe.ts` (v0.40.1.0 Track D / T6).
*
* Test seam: all dependencies are injected via NightlyProbeDeps so
* unit tests don't touch real LLMs or real fixtures.
+1 -6
View File
@@ -44,12 +44,7 @@ export const DEFAULT_DIMENSIONS: string[] = [
* `--slot-a-model`, `--slot-b-model`, `--slot-c-model` on the CLI.
*/
export const DEFAULT_SLOTS: SlotConfig[] = [
// Every default MUST be listed in its recipe's chat touchpoint (pinned by
// test/cross-modal-default-slots.test.ts) — `openai:gpt-4o` sat here after
// the OpenAI recipe dropped it, so slot A errored "not listed for OpenAI
// chat" on every install and the 3-slot panel could never reach its
// 2-model quorum without a Google key (verdict: permanently inconclusive).
{ id: 'A', model: 'openai:gpt-5.2' },
{ id: 'A', model: 'openai:gpt-4o' },
{ id: 'B', model: 'anthropic:claude-opus-4-7' },
{ id: 'C', model: 'google:gemini-1.5-pro' },
];
-24
View File
@@ -70,28 +70,6 @@ export async function runLongMemEvalForProbe(args: LongMemEvalProbeArgs): Promis
* the batch input) or unparseable (cross-modal wrote garbage). Both
* cases are paste-ready in the error message.
*/
/**
* QA-shaped judge dimensions for the nightly probe. The batch judge's
* DEFAULT_DIMENSIONS rubric (DEPTH / SOURCING / SPECIFICITY / ) is built
* for rich agent responses; LongMemEval hypotheses are deliberately terse
* factual answers ("in widget-co") that can never score 7 on DEPTH or
* SOURCING so with the default rubric the probe FAILs every night even
* when retrieval + answering are perfectly healthy. The probe owns its
* invocation of the eval tool and passes dimensions matching the
* fixture's QA shape instead.
*
* NOTE: the `--dimensions` CLI flag splits on commas, so these dimension
* descriptions must stay comma-free.
*/
export const PROBE_QA_DIMENSIONS: string[] = [
// No faithfulness/grounding dimension on purpose: the judge never sees
// the haystack, so any accurate detail beyond the terse gold label reads
// as "invented" and correct answers fail (verified empirically — a
// correct "before + dates" answer scored 4/10 on such a dimension).
'CORRECTNESS — Does the hypothesis state the same fact as the expected answer? A terse direct answer is ideal.',
'DIRECTNESS — Does it answer THIS question without hedging or padding or answering something else?',
];
export async function runCrossModalBatchForProbe(
args: CrossModalProbeArgs,
): Promise<{ exitCode: number; summary: CrossModalBatchSummary }> {
@@ -103,8 +81,6 @@ export async function runCrossModalBatchForProbe(
args.summaryPath,
'--max-usd',
String(args.maxUsd),
'--dimensions',
PROBE_QA_DIMENSIONS.join(','),
'--yes',
'--json',
]);
+11 -43
View File
@@ -62,42 +62,6 @@ export interface NightlyProbeDeps {
now: () => Date;
}
/**
* Dual-plane flag resolution (same precedent as `mcp.publish_skills` in
* serve-http.ts): the DB config row what `gbrain config set` writes
* wins when present; the file plane (~/.gbrain/config.json) is the
* fallback. Doctor's paste-ready enable hint says `gbrain config set
* autopilot.nightly_quality_probe.enabled true`, so the gate MUST read
* the DB plane a file-only read turns that hint into a silent no-op.
*/
export function resolveProbeEnabled(
dbVal: string | null | undefined,
fileVal: unknown,
): boolean {
if (dbVal != null) return dbVal === 'true';
return fileVal === true;
}
/**
* Same dual-plane rule for the per-run cost cap. Malformed or negative
* values on either plane fall through to the next plane / the default.
*/
export function resolveProbeMaxUsd(
dbVal: string | null | undefined,
fileVal: unknown,
fallback: number = DEFAULT_MAX_USD,
): number {
if (dbVal != null) {
const n = Number(dbVal);
if (Number.isFinite(n) && n >= 0) return n;
}
if (fileVal != null) {
const n = Number(fileVal);
if (Number.isFinite(n) && n >= 0) return n;
}
return fallback;
}
/**
* Pure function: decide whether the probe should run given the audit
* history. Returns reason when skipping.
@@ -137,17 +101,21 @@ export async function runNightlyQualityProbe(deps: NightlyProbeDeps): Promise<Ni
return { outcome: 'disabled', exit_code: 0, detail: 'feature flag off' };
}
// 24h rate limit — skip WITHOUT an audit row. The autopilot loop invokes
// the probe every cycle (~5-10 min), so all but one invocation per day
// lands here; logging each skip floods the audit file (~hundreds of
// rows/day) and — because doctor treats any non-pass outcome as bad
// signal — flips nightly_quality_probe_health to a permanent WARN the
// moment the probe is enabled. A skip is a non-event: the real runs are
// the signal, and their rows are what gates the next 24h window.
// 24h rate limit — skip + audit "rate_limited".
const now = deps.now();
const recent = readRecentQualityProbeEvents(2, now); // 2-day window is enough for 24h check
const decision = shouldRunNightly(now, recent);
if (!decision.run) {
logQualityProbeEvent({
outcome: 'rate_limited',
exit_code: 0,
pass_count: 0,
fail_count: 0,
inconclusive_count: 0,
error_count: 0,
est_cost_usd: 0,
detail: 'already ran within 24h window',
});
return { outcome: 'rate_limited', exit_code: 0, detail: 'already ran within 24h' };
}
+56 -6
View File
@@ -79,15 +79,34 @@ export interface EmbedBatchOptions {
* and amplify rate-limit pressure.
*/
maxRetries?: number;
/**
* #1818: bounded parallelism across BATCH_SIZE sub-batches. Defaults to
* `GBRAIN_EMBED_BATCH_CONCURRENCY` env, else 4. Results are
* index-addressed so output order always matches input order. Set 1 to
* force the pre-v0.42 serial dispatch.
*/
concurrency?: number;
}
/**
* Embed a batch of texts via the gateway. Sub-batches of 100 so upstream
* progress callbacks fire incrementally on large imports. The gateway owns
* adaptive batch splitting and per-recipe token-budget logic; this paginator
* is purely about progress-callback granularity.
* owns progress-callback granularity and (#1818) bounded parallel dispatch
* of the sub-batches the embed-stale.ts worker-pool pattern, scoped down.
*/
const BATCH_SIZE = 100;
const DEFAULT_EMBED_BATCH_CONCURRENCY = 4;
function resolveEmbedBatchConcurrency(options: EmbedBatchOptions): number {
if (options.concurrency !== undefined) {
return Math.max(1, Math.floor(options.concurrency));
}
const env = Number(process.env.GBRAIN_EMBED_BATCH_CONCURRENCY);
if (Number.isFinite(env) && env >= 1) return Math.floor(env);
return DEFAULT_EMBED_BATCH_CONCURRENCY;
}
export async function embedBatch(
texts: string[],
options: EmbedBatchOptions = {},
@@ -103,13 +122,44 @@ export async function embedBatch(
if (texts.length <= BATCH_SIZE && !options.onBatchComplete) {
return gatewayEmbed(texts, gwOpts);
}
const results: Float32Array[] = [];
// #1818: dispatch sub-batches through a bounded worker pool instead of a
// serial loop. Results are written into a preallocated index-addressed
// array so output order matches input order regardless of completion
// order; onBatchComplete reports a monotonic completed-embedding count.
const slices: Array<{ start: number; texts: string[] }> = [];
for (let i = 0; i < texts.length; i += BATCH_SIZE) {
const slice = texts.slice(i, i + BATCH_SIZE);
const out = await gatewayEmbed(slice, gwOpts);
results.push(...out);
options.onBatchComplete?.(results.length, texts.length);
slices.push({ start: i, texts: texts.slice(i, i + BATCH_SIZE) });
}
const results = new Array<Float32Array>(texts.length);
let next = 0;
let done = 0;
const numWorkers = Math.min(resolveEmbedBatchConcurrency(options), slices.length);
// Once any sub-batch fails, `failed` stops the surviving workers from
// dispatching FURTHER slices — the whole call is rejecting anyway, so
// continuing would burn real provider spend in the background and fire
// onBatchComplete after the caller already saw the failure (worst with
// embedBatchWithBackoff, whose 429 backoff assumes nothing is in flight).
// In-flight sibling calls still run to completion (bounded by numWorkers-1).
let failed = false;
const worker = async (): Promise<void> => {
while (!failed && next < slices.length) {
// NOTE: no local aborted-check here — an aborted signal makes the next
// gatewayEmbed call throw (SDK-side), which rejects the pool. Returning
// silently instead would resolve with holes in `results`.
const slice = slices[next++];
let out: Float32Array[];
try {
out = await gatewayEmbed(slice.texts, gwOpts);
} catch (err) {
failed = true;
throw err;
}
for (let j = 0; j < out.length; j++) results[slice.start + j] = out[j];
done += out.length;
if (!failed) options.onBatchComplete?.(done, texts.length);
}
};
await Promise.all(Array.from({ length: numWorkers }, () => worker()));
return results;
}
-5
View File
@@ -75,11 +75,6 @@ export const CANONICAL_PRICING: Record<string, ModelPricing> = {
'openai:gpt-4o': { input: 2.50, output: 10.00 },
'openai:gpt-4o-mini': { input: 0.15, output: 0.60 },
'openai:gpt-5': { input: 5.00, output: 20.00 },
// gpt-5.2: rates from the OpenAI recipe chat touchpoint (verified
// 2026-04-20). Needed here because it's the cross-modal DEFAULT_SLOTS
// slot-A model — without a canonical entry estimateCost silently drops
// slot A from the --max-usd pre-flight and est_cost_usd audit rows.
'openai:gpt-5.2': { input: 1.25, output: 10.00 },
'openai:gpt-5.5': { input: 4.00, output: 16.00 },
// ── Google ─────────────────────────────────────────────────────────────
+109 -5
View File
@@ -39,6 +39,8 @@ import {
__getShrinkStateForTests,
} from '../../src/core/ai/gateway.ts';
import { AIConfigError, AITransientError } from '../../src/core/ai/errors.ts';
import { RECIPES } from '../../src/core/ai/recipes/index.ts';
import type { Recipe } from '../../src/core/ai/types.ts';
// The last test in this file leaves the gateway configured with a remote
// provider + fake key and a REAL embed transport. Without a final reset,
@@ -93,6 +95,14 @@ function configureGoogle(): void {
});
}
function configureDashscope(): void {
configureGateway({
embedding_model: 'dashscope:text-embedding-v3',
embedding_dimensions: 1024,
env: { DASHSCOPE_API_KEY: 'sk-fake' },
});
}
// --------- 1. Pure helpers ---------
describe('splitByTokenBudget (pure helper)', () => {
@@ -149,6 +159,27 @@ describe('splitByTokenBudget (pure helper)', () => {
expect(splitByTokenBudget(texts, 96_000, 0)).toEqual(splitByTokenBudget(texts, 96_000, 4));
expect(splitByTokenBudget(texts, 96_000, -1)).toEqual(splitByTokenBudget(texts, 96_000, 4));
});
// #1199: count cap for providers that reject batches by input count.
test('max_batch_count flushes even when token budget has room', () => {
const texts = Array.from({ length: 25 }, (_, i) => `t${i}`);
const result = splitByTokenBudget(texts, 1_000_000, 4, 10);
expect(result.map(b => b.length)).toEqual([10, 10, 5]);
expect(result.flat()).toEqual(texts);
});
test('token budget still governs alongside max_batch_count', () => {
const texts = ['a'.repeat(50_000), 'b'.repeat(50_000), 'c'.repeat(50_000)];
const result = splitByTokenBudget(texts, 96_000, 1, 10);
expect(result).toHaveLength(3);
});
test('undefined / zero / negative max_batch_count is ignored', () => {
const texts = Array.from({ length: 25 }, () => 'x');
expect(splitByTokenBudget(texts, 1_000_000, 4, undefined)).toHaveLength(1);
expect(splitByTokenBudget(texts, 1_000_000, 4, 0)).toHaveLength(1);
expect(splitByTokenBudget(texts, 1_000_000, 4, -5)).toHaveLength(1);
});
});
describe('isTokenLimitError (pure helper)', () => {
@@ -179,6 +210,12 @@ describe('isTokenLimitError (pure helper)', () => {
expect(isTokenLimitError(new Error('Exceeded 300000 max tokens per request'))).toBe(true);
});
test('matches DashScope batch-count error (#1199)', () => {
expect(isTokenLimitError(new Error(
'InvalidParameter: batch size is invalid, it should not be larger than 10.',
))).toBe(true);
});
test('does not match unrelated errors', () => {
expect(isTokenLimitError(new Error('Connection refused'))).toBe(false);
expect(isTokenLimitError(new Error('Invalid API key'))).toBe(false);
@@ -387,26 +424,92 @@ describe('shrink-on-miss adaptive cache', () => {
});
});
// --------- 8. Pre-split count cap through public embed() (#1199 / #970) ---------
describe('embed() pre-split honors max_batch_count', () => {
beforeEach(() => resetGateway());
afterEach(() => __setEmbedTransportForTests(null));
test('dashscope never dispatches more than 10 inputs per call (#1199)', async () => {
configureDashscope();
const stub = mock(async ({ values }: { values: string[] }) => fakeEmbeddings(values, 1024));
__setEmbedTransportForTests(stub as any);
// 25 short texts fit trivially in the 8192-token budget; without the
// count cap they'd ship as ONE batch and DashScope would reject it.
const texts = Array.from({ length: 25 }, (_, i) => `short-${i}`);
const result = await embed(texts);
expect(result).toHaveLength(25);
const callLengths = stub.mock.calls.map(([arg]) => (arg as { values: string[] }).values.length);
expect(Math.max(...callLengths)).toBeLessThanOrEqual(10);
expect(callLengths.reduce((a, b) => a + b, 0)).toBe(25);
// Order preserved across sub-batches.
expect((stub.mock.calls[0][0] as { values: string[] }).values[0]).toBe('short-0');
});
test('google pre-splits at 100 inputs per batchEmbedContents call (#970)', async () => {
configureGoogle();
const stub = mock(async ({ values }: { values: string[] }) => fakeEmbeddings(values, 768));
__setEmbedTransportForTests(stub as any);
const texts = Array.from({ length: 250 }, (_, i) => `g${i}`);
const result = await embed(texts);
expect(result).toHaveLength(250);
const callLengths = stub.mock.calls.map(([arg]) => (arg as { values: string[] }).values.length);
expect(callLengths).toEqual([100, 100, 50]);
});
});
// --------- 7. Startup warning (D9-B) ---------
describe('startup warning for recipes missing max_batch_tokens', () => {
beforeEach(() => resetGateway());
// #970 closed google's missing cap, so no registered recipe is capless
// anymore. Inject a synthetic capless recipe to keep the warning path
// covered for the NEXT recipe that forgets the field.
const caplessRecipe: Recipe = {
id: 'capless-test',
name: 'Capless Test Provider',
tier: 'openai-compat',
implementation: 'openai-compatible',
base_url_default: 'https://example.invalid/v1',
auth_env: { required: [] },
touchpoints: {
embedding: { models: ['capless-embed-1'], default_dims: 768 },
},
};
function configureCapless(): void {
configureGateway({
embedding_model: 'capless-test:capless-embed-1',
embedding_dimensions: 768,
env: {},
});
}
test('configured missing-cap recipe warns once; unrelated recipes stay quiet', () => {
const warnings: string[] = [];
const original = console.warn;
console.warn = (msg: string) => warnings.push(String(msg));
RECIPES.set(caplessRecipe.id, caplessRecipe);
try {
configureOpenAI();
expect(warnings.length).toBe(0);
// #970 regression: google now declares max_batch_tokens → quiet.
configureGoogle();
expect(warnings.length).toBe(0);
configureCapless();
const firstCallCount = warnings.length;
// Reconfigure: the warning should NOT re-fire for the same recipes
// within one process (we already told the operator).
configureGoogle();
configureCapless();
expect(warnings.length).toBe(firstCallCount);
} finally {
console.warn = original;
RECIPES.delete(caplessRecipe.id);
}
// The warning text should match the documented contract.
@@ -415,11 +518,12 @@ describe('startup warning for recipes missing max_batch_tokens', () => {
);
expect(contractMatch.length).toBe(1);
// Voyage declares max_batch_tokens → suppressed. OpenAI is the
// canonical fast-path recipe → also suppressed by id. Both must be
// absent from the warnings.
// Voyage + google declare max_batch_tokens → suppressed. OpenAI is the
// canonical fast-path recipe → also suppressed by id. All must be
// absent from the warnings; only the synthetic capless recipe fires.
expect(warnings.find(w => w.includes('"voyage"'))).toBeUndefined();
expect(warnings.find(w => w.includes('"openai"'))).toBeUndefined();
expect(warnings.find(w => w.includes('"google"'))).toBeDefined();
expect(warnings.find(w => w.includes('"google"'))).toBeUndefined();
expect(warnings.find(w => w.includes('"capless-test"'))).toBeDefined();
});
});
+13 -13
View File
@@ -52,16 +52,7 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
}
});
test('configureGateway warns for google only when google embedding is configured', () => {
warnSpy.mockClear();
resetGateway();
configureGateway({ env: {} });
let messages = warnSpy.mock.calls.map(c => String(c[0] ?? ''));
expect(
messages.some(m => m.includes('"google"') && m.includes('without max_batch_tokens')),
'google should not warn while OpenAI default is configured',
).toBe(false);
test('configureGateway does NOT warn for google now that it declares batch caps (#970)', () => {
warnSpy.mockClear();
resetGateway();
configureGateway({
@@ -69,11 +60,20 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
embedding_dimensions: 768,
env: { GOOGLE_GENERATIVE_AI_API_KEY: 'fake' },
});
messages = warnSpy.mock.calls.map(c => String(c[0] ?? ''));
const messages = warnSpy.mock.calls.map(c => String(c[0] ?? ''));
expect(
messages.some(m => m.includes('"google"') && m.includes('without max_batch_tokens')),
'google should warn when configured because it has fixed-cap models',
).toBe(true);
'google declares max_batch_tokens/max_batch_count since #970 — no warning',
).toBe(false);
});
test('google recipe declares its derived batch caps (#970)', () => {
const e = getRecipe('google')!.touchpoints.embedding!;
// Count cap is the REAL Gemini limit (batchEmbedContents: 100 inputs);
// the token budget is derived (100 × 2048 per-input tokens), NOT the
// 2048 per-input limit — copying that verbatim would over-split 50×.
expect(e.max_batch_count).toBe(100);
expect(e.max_batch_tokens).toBe(204_800);
});
test('every recipe with empty models[] declares user_provided_models OR has openai-fast-path', () => {
+5
View File
@@ -55,6 +55,11 @@ describe('recipe: dashscope', () => {
expect(r.touchpoints.embedding!.chars_per_token).toBeGreaterThan(0);
});
test('declares max_batch_count: 10 — DashScope rejects larger batches (#1199)', () => {
const r = getRecipe('dashscope')!;
expect(r.touchpoints.embedding!.max_batch_count).toBe(10);
});
test('dimsProviderOptions threads dimensions for text-embedding-v3 (Matryoshka)', async () => {
// Codex finding #1: DashScope text-embedding-v3 is Matryoshka 64-1024.
// Without `dimensions` on the wire, user-selected non-default dims are
-102
View File
@@ -1,102 +0,0 @@
/**
* Tests for the parser-probe audit trail + the 24h rate-limit gate.
*
* Uses GBRAIN_AUDIT_DIR override pointed at a tmpdir for hermeticity
* (same pattern as audit-slug-fallback.serial.test.ts). Serial because
* the env override is process-global.
*/
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
import { mkdtempSync, rmSync, readdirSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import {
computeParserProbeAuditFilename,
logParserProbeEvent,
parserProbeRanWithin,
readRecentParserProbeEvents,
type ParserProbeAuditEvent,
} from '../src/core/audit-parser-probe.ts';
let auditDir: string;
let savedEnv: string | undefined;
beforeEach(() => {
auditDir = mkdtempSync(join(tmpdir(), 'parser-probe-audit-'));
savedEnv = process.env.GBRAIN_AUDIT_DIR;
process.env.GBRAIN_AUDIT_DIR = auditDir;
});
afterEach(() => {
if (savedEnv === undefined) delete process.env.GBRAIN_AUDIT_DIR;
else process.env.GBRAIN_AUDIT_DIR = savedEnv;
rmSync(auditDir, { recursive: true, force: true });
});
function makeEvent(overrides: Partial<ParserProbeAuditEvent> = {}): ParserProbeAuditEvent {
return {
schema_version: 1,
ts: new Date().toISOString(),
outcome: 'pass',
fixtures_total: 12,
fixtures_passed: 12,
recall_mean: 0.98,
participants_recall_mean: 0.97,
adversarial_false_positives: 0,
failed_fixture_ids: [],
...overrides,
};
}
describe('parser-probe audit trail', () => {
test('log + readRecent round-trip', () => {
logParserProbeEvent(makeEvent({ outcome: 'fail', reason: '2 fixture(s) failed' }));
const events = readRecentParserProbeEvents(7);
expect(events.length).toBe(1);
expect(events[0]!.outcome).toBe('fail');
expect(events[0]!.reason).toBe('2 fixture(s) failed');
const files = readdirSync(auditDir);
expect(files.length).toBe(1);
expect(files[0]).toMatch(/^parser-probe-\d{4}-W\d{2}\.jsonl$/);
});
test('filename uses ISO-week rotation with the parser-probe prefix', () => {
// Year-boundary edge pinned by the shared writer's own tests; here we
// pin the prefix wiring.
expect(computeParserProbeAuditFilename(new Date('2026-07-06T12:00:00Z'))).toBe(
'parser-probe-2026-W28.jsonl',
);
});
test('readRecent filters by window', () => {
const old = new Date(Date.now() - 10 * 86400000).toISOString();
logParserProbeEvent(makeEvent({ ts: old }));
expect(readRecentParserProbeEvents(7).length).toBe(0);
});
});
describe('parserProbeRanWithin — 24h rate-limit gate', () => {
const DAY_MS = 24 * 60 * 60 * 1000;
test('false when no runs are audited', () => {
expect(parserProbeRanWithin(DAY_MS)).toBe(false);
});
test('true when a run landed within the window', () => {
logParserProbeEvent(makeEvent({ ts: new Date(Date.now() - 60_000).toISOString() }));
expect(parserProbeRanWithin(DAY_MS)).toBe(true);
});
test('false when the last run is older than the window', () => {
logParserProbeEvent(makeEvent({ ts: new Date(Date.now() - 25 * 3600_000).toISOString() }));
expect(parserProbeRanWithin(DAY_MS)).toBe(false);
});
test('non-pass outcomes also hold the window (mirrors quality-probe semantics)', () => {
logParserProbeEvent(makeEvent({
outcome: 'no_embedding_key',
ts: new Date(Date.now() - 3600_000).toISOString(),
}));
expect(parserProbeRanWithin(DAY_MS)).toBe(true);
});
});
+4 -20
View File
@@ -31,15 +31,10 @@ describe('autopilot wiring: nightly quality probe', () => {
expect(SOURCE).toContain(`runCrossModalBatchForProbe`);
});
test('feature flag gate present: dual-plane read (DB row wins, file plane fallback)', () => {
test('feature flag gate present: cfg.autopilot.nightly_quality_probe.enabled', () => {
// Per D10: the scheduler ONLY checks the feature flag. The 24h rate-limit
// lives inside runNightlyQualityProbe itself (no scheduler-side precheck).
// The flag resolves through resolveProbeEnabled so `gbrain config set
// autopilot.nightly_quality_probe.enabled true` (the doctor hint, DB
// plane) and ~/.gbrain/config.json (file plane) BOTH work — a file-only
// read made the printed hint a silent no-op.
expect(SOURCE).toContain(`getConfig('autopilot.nightly_quality_probe.enabled')`);
expect(SOURCE).toMatch(/resolveProbeEnabled\(dbEnabled,\s*cfg\?\.autopilot\?\.nightly_quality_probe\?\.enabled\)/);
expect(SOURCE).toContain(`nightly_quality_probe?.enabled === true`);
});
test('NO scheduler-side rate-limit check (D10 simplification)', () => {
@@ -69,23 +64,12 @@ describe('autopilot wiring: nightly quality probe', () => {
expect(SOURCE).toContain(`now:`);
});
test('resolveRepoRoot prefers the gbrain package root (committed fixture home), not the brain repoPath', () => {
// The DI harness in nightly-quality-probe.test.ts passes process.cwd()
// (= the gbrain repo in CI), which papered over the wiring passing
// repoPath (= sync.repo_path, the user's BRAIN repo, where the fixture
// never exists). Pin the package-root resolution + existence check.
expect(SOURCE).toMatch(/fileURLToPath\(new URL\('\.\.\/\.\.', import\.meta\.url\)\)/);
expect(SOURCE).toContain(`'longmemeval-nightly.jsonl'`);
expect(SOURCE).toMatch(/fixtureAtPkgRoot \? pkgRoot : repoPath/);
});
test('hasEmbeddingProvider reads from gateway.isAvailable("embedding") (codex round-2 #12 — in-process, not subprocess)', () => {
expect(SOURCE).toContain(`isAvailable('embedding')`);
expect(SOURCE).toContain(`gateway`);
});
test('max_usd resolves dual-plane (default = 5 pinned by resolveProbeMaxUsd unit tests)', () => {
expect(SOURCE).toContain(`getConfig('autopilot.nightly_quality_probe.max_usd')`);
expect(SOURCE).toMatch(/resolveProbeMaxUsd\(dbMaxUsd,\s*cfg\?\.autopilot\?\.nightly_quality_probe\?\.max_usd\)/);
test('max_usd default = 5 when config unset (matches plan default per D10)', () => {
expect(SOURCE).toMatch(/max_usd\s*\?\?\s*5/);
});
});
@@ -1,77 +0,0 @@
/**
* Source-shape regression tests for the autopilot wiring of
* `runConversationParserNightlyProbe` (step 4.6).
*
* Same rationale as autopilot-nightly-probe-wiring.test.ts: the loop is
* hard to drive end-to-end, so these pin the structural protections
* the dual-plane flag read, the D10 tokenmax mode-gate, the package-root
* fixture resolution, the audit-flood guard, and the try/catch posture.
*
* The probe's own gate/scoring logic is pinned by the module's unit
* tests; the audit trail by audit-parser-probe.serial.test.ts.
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'node:fs';
import { resolve } from 'node:path';
const AUTOPILOT_SRC = resolve('src/commands/autopilot.ts');
const SOURCE = readFileSync(AUTOPILOT_SRC, 'utf-8');
describe('autopilot wiring: conversation-parser probe', () => {
test('invokes the phase module and the audit trail', () => {
expect(SOURCE).toContain(`runConversationParserNightlyProbe`);
expect(SOURCE).toContain(`conversation-parser/nightly-probe`);
expect(SOURCE).toContain(`logParserProbeEvent`);
expect(SOURCE).toContain(`audit-parser-probe`);
});
test('flag reads dual-plane: DB row (gbrain config set) wins, file plane fallback', () => {
expect(SOURCE).toContain(`getConfig('autopilot.conversation_parser_probe.enabled')`);
expect(SOURCE).toContain(`cfg?.autopilot?.conversation_parser_probe?.enabled === true`);
});
test('D10 mode-gate present: tokenmax brains run the probe by default', () => {
expect(SOURCE).toMatch(/parserEnabled \|\| searchMode === 'tokenmax'/);
});
test('fixtures resolve from the gbrain package root, NOT the brain repoPath', () => {
// The committed fixtures live in the gbrain source tree; resolving
// them against sync.repo_path would point into the user's brain repo.
expect(SOURCE).toMatch(/fileURLToPath\(new URL\('\.\.\/\.\.', import\.meta\.url\)\)/);
expect(SOURCE).toContain(`'conversation-formats', 'all.jsonl'`);
expect(SOURCE).toContain(`'conversation-formats', 'adversarial.jsonl'`);
});
test('missing fixtures skip quietly (no audit row, once-per-process stderr note)', () => {
// Compiled-binary installs carry no source tree; writing failure rows
// would flip doctor to WARN on every binary install.
expect(SOURCE).toContain(`parserProbeFixtureWarned`);
});
test('rate_limited outcomes are NOT audit-logged (flood guard)', () => {
expect(SOURCE).toMatch(/outcome !== 'rate_limited'\) logParserProbeEvent\(result\)/);
});
test('rate-limit gate delegates to the audit module, not inline event reads', () => {
expect(SOURCE).toContain(`parserProbeRanWithin(24 * 60 * 60 * 1000)`);
});
test('LLM-key gate reads gateway.isAvailable("chat") in-process', () => {
expect(SOURCE).toContain(`isAvailable('chat')`);
});
test('probe call wrapped in try/catch that does NOT bump consecutiveErrors', () => {
expect(SOURCE).toMatch(/catch[\s\S]*?autopilot\.parser_probe[\s\S]*?do NOT bump consecutiveErrors/);
});
test('DI shape: the exact 7 fields of the parser probe NightlyProbeDeps', () => {
expect(SOURCE).toContain(`isEnabled:`);
expect(SOURCE).toContain(`searchMode:`);
expect(SOURCE).toContain(`hasLlmKey:`);
expect(SOURCE).toContain(`resolveFixturePath:`);
expect(SOURCE).toContain(`resolveAdversarialPath:`);
expect(SOURCE).toContain(`shouldSkipForRateLimit:`);
expect(SOURCE).toContain(`now:`);
});
});
-47
View File
@@ -1,47 +0,0 @@
/**
* Consistency guard: every cross-modal DEFAULT_SLOTS model must be listed
* in its recipe's chat touchpoint. `openai:gpt-4o` drifted out of the
* OpenAI recipe while remaining the slot-A default the gateway then
* rejected slot A ("not listed for OpenAI chat") on every install, and the
* 3-slot judge panel could never reach its 2-model quorum without a Google
* key, pinning every batch verdict at inconclusive (which the nightly
* quality probe surfaces as a doctor WARN).
*/
import { describe, expect, test } from 'bun:test';
import { DEFAULT_SLOTS } from '../src/core/cross-modal-eval/runner.ts';
import { getRecipe } from '../src/core/ai/recipes/index.ts';
import { splitProviderModelId } from '../src/core/model-id.ts';
import { canonicalLookup } from '../src/core/model-pricing.ts';
describe('cross-modal DEFAULT_SLOTS ↔ recipe consistency', () => {
test('every default slot model is listed in its recipe chat touchpoint', () => {
for (const slot of DEFAULT_SLOTS) {
const { provider, model } = splitProviderModelId(slot.model);
expect(provider).not.toBeNull();
const recipe = getRecipe(provider!);
expect(recipe, `slot ${slot.id}: unknown recipe "${provider}"`).toBeDefined();
const chatModels = recipe!.touchpoints.chat?.models ?? [];
expect(
chatModels,
`slot ${slot.id}: "${model}" not listed for ${provider} chat — the judge slot can never run`,
).toContain(model);
}
});
test('every default slot model has a canonical pricing entry', () => {
// Without one, estimateCost silently drops the slot from the
// --max-usd pre-flight and est_cost_usd audit rows (~1/3 under-count).
for (const slot of DEFAULT_SLOTS) {
expect(
canonicalLookup(slot.model),
`slot ${slot.id}: "${slot.model}" missing from CANONICAL_PRICING`,
).toBeDefined();
}
});
test('slots span three distinct providers (uncorrelated blind spots)', () => {
const providers = new Set(DEFAULT_SLOTS.map(s => splitProviderModelId(s.model).provider));
expect(providers.size).toBe(3);
});
});
-51
View File
@@ -1,51 +0,0 @@
/**
* Tests for computeConversationParserProbeHealthCheck the pure function
* behind doctor's conversation_parser_probe_health check, which replaced
* the v0.41.13.0 hardcoded "Skipped" stub when the autopilot wiring
* landed. Mirrors the branch coverage style of the quality-probe check.
*/
import { describe, expect, test } from 'bun:test';
import { computeConversationParserProbeHealthCheck } from '../src/commands/doctor.ts';
const ev = (outcome: string, reason?: string, ts = new Date().toISOString()) => ({
outcome,
ts,
...(reason !== undefined ? { reason } : {}),
});
describe('computeConversationParserProbeHealthCheck', () => {
test('disabled + no events → ok with paste-ready enable hint', () => {
const check = computeConversationParserProbeHealthCheck(false, []);
expect(check.status).toBe('ok');
expect(check.message).toContain('gbrain config set autopilot.conversation_parser_probe.enabled true');
});
test('enabled + no events yet → ok, next run by autopilot', () => {
const check = computeConversationParserProbeHealthCheck(true, []);
expect(check.status).toBe('ok');
expect(check.message).toContain('no probe events');
});
test('disabled flag but events exist (tokenmax mode-gate ran it) → events win over the hint', () => {
const check = computeConversationParserProbeHealthCheck(false, [ev('pass')]);
expect(check.status).toBe('ok');
expect(check.message).toContain('all pass');
});
test('any non-pass outcome in the window → warn, latest surfaced with reason', () => {
const check = computeConversationParserProbeHealthCheck(true, [
ev('pass'),
ev('adversarial_false_positive', '1 adversarial fixture(s) parsed to non-empty'),
]);
expect(check.status).toBe('warn');
expect(check.message).toContain('adversarial_false_positive');
expect(check.message).toContain('parsed to non-empty');
});
test('all pass → ok with run count', () => {
const check = computeConversationParserProbeHealthCheck(true, [ev('pass'), ev('pass')]);
expect(check.status).toBe('ok');
expect(check.message).toContain('2 probe run(s)');
});
});
+161
View File
@@ -0,0 +1,161 @@
/**
* #1818: embedBatch dispatches its 100-input sub-batches through a bounded
* worker pool (the embed-stale.ts concurrency pattern) instead of a serial
* `for` loop. This file pins:
*
* - output order matches input order regardless of completion order
* (index-addressed results)
* - parallelism actually happens (max in-flight > 1) and stays bounded
* (max in-flight <= configured concurrency)
* - concurrency: 1 restores the serial pre-#1818 dispatch
* - GBRAIN_EMBED_BATCH_CONCURRENCY env is honored when the option is unset
* - onBatchComplete reports a monotonic completed count ending at total
*
* Transport is stubbed via the gateway's __setEmbedTransportForTests seam
* (same pattern as test/ai/adaptive-embed-batch.test.ts). OpenAI recipe =
* fast path (no pre-split), so each embedBatch sub-batch is exactly one
* transport call.
*/
import { afterAll, afterEach, beforeEach, describe, expect, test } from 'bun:test';
import {
configureGateway,
resetGateway,
__setEmbedTransportForTests,
} from '../src/core/ai/gateway.ts';
import { embedBatch } from '../src/core/embedding.ts';
import { withEnv } from './helpers/with-env.ts';
const DIMS = 1536;
function configureOpenAI(): void {
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: DIMS,
env: { OPENAI_API_KEY: 'sk-fake' },
});
}
/**
* Install a transport whose returned embedding encodes the GLOBAL input
* index in dim 0 (texts are `t<N>`), so order can be asserted end-to-end.
* Tracks the max number of concurrently in-flight transport calls.
*/
function installTrackingTransport(delayMs = 5): { maxInFlight: () => number } {
let inFlight = 0;
let maxInFlight = 0;
__setEmbedTransportForTests((async ({ values }: { values: string[] }) => {
inFlight++;
maxInFlight = Math.max(maxInFlight, inFlight);
await new Promise(r => setTimeout(r, delayMs));
inFlight--;
return {
embeddings: values.map(v => {
const idx = Number(v.slice(1));
return Array.from({ length: DIMS }, (_, j) => (j === 0 ? idx : 0.1));
}),
};
}) as any);
return { maxInFlight: () => maxInFlight };
}
const texts = Array.from({ length: 250 }, (_, i) => `t${i}`);
afterAll(() => resetGateway());
describe('embedBatch bounded parallelism (#1818)', () => {
beforeEach(() => {
resetGateway();
configureOpenAI();
});
afterEach(() => {
__setEmbedTransportForTests(null);
});
test('default pool dispatches sub-batches in parallel, order preserved', async () => {
const tracker = installTrackingTransport();
const result = await embedBatch(texts, { onBatchComplete: () => {} });
expect(result).toHaveLength(250);
for (let i = 0; i < 250; i++) {
expect(result[i][0]).toBe(i);
}
// 250 texts → 3 sub-batches; default concurrency 4 → all 3 in flight.
expect(tracker.maxInFlight()).toBeGreaterThan(1);
expect(tracker.maxInFlight()).toBeLessThanOrEqual(4);
});
test('concurrency: 1 keeps the serial dispatch', async () => {
const tracker = installTrackingTransport();
const result = await embedBatch(texts, { concurrency: 1, onBatchComplete: () => {} });
expect(result).toHaveLength(250);
expect(tracker.maxInFlight()).toBe(1);
});
test('GBRAIN_EMBED_BATCH_CONCURRENCY env bounds the pool when option unset', async () => {
const tracker = installTrackingTransport();
await withEnv({ GBRAIN_EMBED_BATCH_CONCURRENCY: '2' }, async () => {
await embedBatch(texts, { onBatchComplete: () => {} });
});
expect(tracker.maxInFlight()).toBeGreaterThan(1);
expect(tracker.maxInFlight()).toBeLessThanOrEqual(2);
});
test('onBatchComplete reports a monotonic count ending at total', async () => {
installTrackingTransport();
const seen: number[] = [];
await embedBatch(texts, {
onBatchComplete: (done, total) => {
expect(total).toBe(250);
seen.push(done);
},
});
expect(seen).toHaveLength(3); // 100 + 100 + 50 sub-batches
for (let i = 1; i < seen.length; i++) {
expect(seen[i]).toBeGreaterThan(seen[i - 1]);
}
expect(seen[seen.length - 1]).toBe(250);
});
test('a failing sub-batch rejects the whole call', async () => {
let call = 0;
__setEmbedTransportForTests((async ({ values }: { values: string[] }) => {
call++;
if (call === 2) throw new Error('boom');
await new Promise(r => setTimeout(r, 2));
return { embeddings: values.map(() => Array.from({ length: DIMS }, () => 0.1)) };
}) as any);
await expect(embedBatch(texts, { onBatchComplete: () => {} })).rejects.toThrow();
});
test('after a failure, surviving workers stop dispatching new slices', async () => {
// 1000 texts → 10 slices, concurrency 2. First call fails immediately;
// without the `failed` flag the second worker would keep draining all
// 10 slices in the background AFTER embedBatch already rejected —
// burning provider spend and firing onBatchComplete post-rejection.
let calls = 0;
const completions: number[] = [];
__setEmbedTransportForTests((async ({ values }: { values: string[] }) => {
calls++;
if (calls === 1) throw new Error('boom');
await new Promise(r => setTimeout(r, 5));
return { embeddings: values.map(() => Array.from({ length: DIMS }, () => 0.1)) };
}) as any);
const many = Array.from({ length: 1000 }, (_, i) => `t${i}`);
await expect(
embedBatch(many, { concurrency: 2, onBatchComplete: d => completions.push(d) }),
).rejects.toThrow('boom');
const callsAtRejection = calls;
await new Promise(r => setTimeout(r, 50)); // would-be background drain window
expect(calls).toBe(callsAtRejection); // no new dispatch after rejection
expect(calls).toBeLessThanOrEqual(2); // only the in-flight sibling ran
expect(completions).toHaveLength(0); // no progress reported after failure
});
test('single small batch without callback stays on the one-call fast path', async () => {
const tracker = installTrackingTransport(1);
const result = await embedBatch(['t0', 't1', 't2']);
expect(result).toHaveLength(3);
expect(result[1][0]).toBe(1);
expect(tracker.maxInFlight()).toBe(1);
});
});
+27 -3
View File
@@ -19,7 +19,7 @@
* overwrites this preload.
*/
import { configureGateway, getEmbeddingDimensions } from '../../src/core/ai/gateway.ts';
import { beforeEach } from 'bun:test';
import { afterEach, beforeEach } from 'bun:test';
const LEGACY_CONFIG = {
embedding_model: 'openai:text-embedding-3-large',
@@ -52,7 +52,7 @@ applyLegacy();
// 2. file-local beforeAll → may overwrite to ZE/1280
// Since beforeAll runs once per file BEFORE the first beforeEach,
// file-local beforeAll wins for that file's tests. ✓
beforeEach(() => {
function applyLegacyIfEmpty() {
try {
// Only re-apply if the gateway was reset (or never configured).
// Tests that explicitly configured a different model in their
@@ -62,4 +62,28 @@ beforeEach(() => {
} catch {
applyLegacy();
}
});
}
beforeEach(applyLegacyIfEmpty);
// PR #3130 shard-order fix: beforeEach alone leaves ONE window open — a file
// whose LAST afterEach calls resetGateway() poisons the NEXT file's
// beforeAll, which runs BEFORE any beforeEach fires. A beforeAll there that
// does engine.initSchema() then sizes the embedding column from the gateway
// DEFAULTS (zembed-1/1280d) instead of the pinned legacy 1536, and every
// 1536-d Float32Array fixture in that file dies with
// "expected 1280 dimensions, not 1536". Which file pair collides is a
// function of shard composition, so adding/removing ANY test file can
// surface it (that is exactly how it bit shard 9).
//
// Preload hooks are registered before any file-local hooks, and bun runs
// after-hooks inside-out (file-local afterEach first, then this one), so
// this repairs the empty slot immediately after the poisoning reset —
// before the next file's beforeAll can observe it.
//
// Known remaining window: a file whose afterAll() resets the gateway (no
// hook runs between its afterAll and the next file's beforeAll). Files
// that reset in afterAll and can precede a schema-creating file should
// re-apply their own config, or the victim file should configureGateway()
// explicitly in its beforeAll.
afterEach(applyLegacyIfEmpty);
+69
View File
@@ -0,0 +1,69 @@
/**
* #1207: `gbrain import` without `--workers` used to hardcode workerCount=1,
* so a large Postgres import paid one serial embedding round-trip per file.
* runImport now routes the default through the shared autoConcurrency policy
* (PGLite 1, >100 files on Postgres DEFAULT_PARALLEL_WORKERS), while an
* explicit `--workers N` still wins.
*
* The engine here is a minimal postgres-kind stub with no database_url in
* config runImport's parallel branch then falls back to serial processing
* (its PR #490 guard) but the WORKER-COUNT DECISION (the thing #1207 fixes)
* is still observable via the "Using N parallel workers" log line. Per-file
* imports fail against the stub engine and are swallowed by runImport's
* per-file catch; that's fine this test pins the policy, not the import.
*/
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync, realpathSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { withEnv } from './helpers/with-env.ts';
import { runImport } from '../src/commands/import.ts';
const fakePostgresEngine = {
kind: 'postgres',
executeRaw: async () => [],
logIngest: async () => {},
setConfig: async () => {},
getConfig: async () => null,
} as any;
let workspace: string;
let brainDir: string;
let logs: string[];
const realLog = console.log;
beforeEach(() => {
workspace = mkdtempSync(join(tmpdir(), 'gbrain-import-workers-home-'));
mkdirSync(join(workspace, '.gbrain'), { recursive: true });
brainDir = realpathSync(mkdtempSync(join(tmpdir(), 'gbrain-import-workers-brain-')));
// 101 files: one past AUTO_CONCURRENCY_FILE_THRESHOLD (100).
for (let i = 0; i < 101; i++) {
writeFileSync(join(brainDir, `page-${i}.md`), `# Page ${i}\n\nbody ${i}\n`);
}
logs = [];
console.log = (msg?: unknown) => logs.push(String(msg));
});
afterEach(() => {
console.log = realLog;
rmSync(workspace, { recursive: true, force: true });
rmSync(brainDir, { recursive: true, force: true });
});
describe('import default worker count (#1207)', () => {
test('no --workers flag → autoConcurrency picks 4 for >100 files on Postgres', async () => {
await withEnv({ GBRAIN_HOME: join(workspace, '.gbrain'), GBRAIN_SOURCE: undefined }, async () => {
await runImport(fakePostgresEngine, [brainDir, '--no-embed'], { sourceId: 'default' });
});
expect(logs.some(l => l.includes('Using 4 parallel workers'))).toBe(true);
});
test('explicit --workers 2 still wins over the auto policy', async () => {
await withEnv({ GBRAIN_HOME: join(workspace, '.gbrain'), GBRAIN_SOURCE: undefined }, async () => {
await runImport(fakePostgresEngine, [brainDir, '--no-embed', '--workers', '2'], { sourceId: 'default' });
});
expect(logs.some(l => l.includes('Using 2 parallel workers'))).toBe(true);
expect(logs.some(l => l.includes('Using 4 parallel workers'))).toBe(false);
});
});
-74
View File
@@ -1,74 +0,0 @@
// Regression test for the nightly-quality-probe config-plane split-brain.
//
// The doctor check prints a paste-ready enable hint — `gbrain config set
// autopilot.nightly_quality_probe.enabled true` — which writes the DB config
// plane. But both the autopilot gate and the doctor check used to read ONLY
// the file plane (~/.gbrain/config.json via loadConfig), so following the
// hint was a silent no-op: the probe never ran and doctor kept reporting
// "disabled (opt-in)".
//
// resolveProbeEnabled / resolveProbeMaxUsd pin the dual-plane rule (same
// precedent as `mcp.publish_skills` in serve-http.ts): DB row wins when
// present, file plane is the fallback.
import { describe, expect, test } from 'bun:test';
import {
resolveProbeEnabled,
resolveProbeMaxUsd,
} from '../src/core/cycle/nightly-quality-probe.ts';
describe('resolveProbeEnabled — dual-plane flag resolution', () => {
test('DB plane "true" enables regardless of file plane (the doctor hint path)', () => {
expect(resolveProbeEnabled('true', undefined)).toBe(true);
expect(resolveProbeEnabled('true', false)).toBe(true);
});
test('explicit DB "false" wins over file-plane true (config set off sticks)', () => {
expect(resolveProbeEnabled('false', true)).toBe(false);
});
test('file plane is the fallback when no DB row exists', () => {
expect(resolveProbeEnabled(null, true)).toBe(true);
expect(resolveProbeEnabled(undefined, true)).toBe(true);
expect(resolveProbeEnabled(null, undefined)).toBe(false);
expect(resolveProbeEnabled(null, false)).toBe(false);
});
test('file plane stays strict boolean — string "true" in config.json does not enable', () => {
// Matches the pre-fix autopilot gate (`=== true`); the doctor check used
// Boolean(...) and could disagree with autopilot on a string value.
// Both call sites now share this helper, so they can no longer diverge.
expect(resolveProbeEnabled(null, 'true')).toBe(false);
expect(resolveProbeEnabled(null, 1)).toBe(false);
});
test('non-"true" DB strings are off (mcp.publish_skills semantics)', () => {
expect(resolveProbeEnabled('1', true)).toBe(false);
expect(resolveProbeEnabled('yes', true)).toBe(false);
expect(resolveProbeEnabled('', true)).toBe(false);
});
});
describe('resolveProbeMaxUsd — dual-plane cost cap resolution', () => {
test('DB plane wins when parseable', () => {
expect(resolveProbeMaxUsd('2.5', 10)).toBe(2.5);
expect(resolveProbeMaxUsd('0', 10)).toBe(0);
});
test('malformed or negative DB value falls through to file plane', () => {
expect(resolveProbeMaxUsd('banana', 3)).toBe(3);
expect(resolveProbeMaxUsd('-1', 3)).toBe(3);
});
test('file plane used when no DB row; default when both absent/invalid', () => {
expect(resolveProbeMaxUsd(null, 7)).toBe(7);
expect(resolveProbeMaxUsd(null, '4')).toBe(4);
expect(resolveProbeMaxUsd(null, undefined)).toBe(5);
expect(resolveProbeMaxUsd(null, 'banana')).toBe(5);
expect(resolveProbeMaxUsd(undefined, -2)).toBe(5);
});
test('explicit fallback override is honored', () => {
expect(resolveProbeMaxUsd(null, undefined, 12)).toBe(12);
});
});
+4 -7
View File
@@ -132,20 +132,17 @@ describe('runNightlyQualityProbe (DI stub harness)', () => {
});
});
test('enabled + recent run within 24h → outcome: rate_limited, NO audit row', async () => {
test('enabled + recent run within 24h → outcome: rate_limited', async () => {
// Pre-seed a recent audit event by running the probe once first.
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
// First run succeeds.
await runNightlyQualityProbe(makeDeps());
// Second run, same hour → rate_limited. A skip is a non-event: the
// autopilot loop invokes the probe every cycle (~5-10 min), so
// logging each skip would flood the audit file and flip doctor's
// any-non-pass-is-bad filter to a permanent WARN.
// Second run, same hour → rate_limited.
const r2 = await runNightlyQualityProbe(makeDeps());
expect(r2.outcome).toBe('rate_limited');
const events = await readEvents();
expect(events.length).toBe(1);
expect(events[0].outcome).toBe('pass');
expect(events.length).toBe(2);
expect(events[1].outcome).toBe('rate_limited');
});
});