mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
22ed50c8a5 | ||
|
|
5a570f6cd1 | ||
|
|
2943a57bb6 | ||
|
|
f099227862 | ||
|
|
fe5ad147cb | ||
|
|
32a03dd2f7 | ||
|
|
98a94ee558 | ||
|
|
9ee4513b69 | ||
|
|
459f0a0609 | ||
|
|
d1cc3a56b5 | ||
|
|
d738d28811 | ||
|
|
aa60b64989 | ||
|
|
0906ab0abd |
+104
@@ -2,6 +2,110 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.31.0] - 2026-05-30
|
||||
|
||||
**Your nightly `gbrain sync --all` cron stops getting blocked. It used to
|
||||
demand `--yes` on every non-interactive run even when nothing needed
|
||||
embedding. Now it just runs. And when you change your embedding model,
|
||||
`gbrain embed --stale` actually re-embeds the pages still on the old model
|
||||
instead of silently leaving them stale.**
|
||||
|
||||
Before this release, `gbrain sync --all` always stopped and asked for
|
||||
confirmation in any non-interactive run (cron, script, piped output) based on
|
||||
a guess of your whole corpus's embedding cost. On an already-synced brain that
|
||||
guess was large and the real work was zero, so the cron exited with an error
|
||||
every night and you had to wire in `--yes` permanently, which defeats the
|
||||
safety check.
|
||||
|
||||
Here's what changed. On the default fast sync path, embedding doesn't run
|
||||
during the sync itself; it's handed to background jobs that already cap their
|
||||
own spend at $25 per source per day. So the sync command now prints what's
|
||||
queued and proceeds. It never blocks. You'll only see a confirmation prompt
|
||||
when sync embeds inline (the older non-parallel mode), and even then only when
|
||||
the new content this sync embeds crosses a dollar threshold you control.
|
||||
|
||||
**The threshold knob:**
|
||||
```
|
||||
gbrain config set sync.cost_gate_min_usd 0.50 # default; raise or lower to taste
|
||||
```
|
||||
At or below this estimate an inline sync proceeds silently. Set it to 0 to
|
||||
confirm on any nonzero cost.
|
||||
|
||||
**Real stale detection.** gbrain now records which model and dimensions
|
||||
produced each page's vectors. Change your embedding model (OpenAI to Voyage,
|
||||
or a different dimension) and `gbrain embed --stale` finds and re-embeds every
|
||||
page still on the old model. Before, "stale" only meant "never embedded," so a
|
||||
model swap was invisible and search quietly mixed vector spaces.
|
||||
|
||||
This does NOT trigger a surprise re-embed of your whole brain on upgrade. Pages
|
||||
that predate this release have no recorded signature, and gbrain treats "no
|
||||
signature" as "leave it alone." Only pages embedded after you upgrade carry a
|
||||
signature, and only an actual model change marks them stale.
|
||||
|
||||
**See your embedding backlog.** `gbrain sources status` gains a BACKFILL column
|
||||
showing whether each source has embed jobs queued, running, or idle, plus when
|
||||
the last one finished. After a `sync --all` you can tell at a glance whether
|
||||
embeddings are caught up or still trickling in.
|
||||
|
||||
## To take advantage of v0.41.31.0
|
||||
|
||||
`gbrain upgrade` does this automatically (it applies migration v108, which adds
|
||||
the per-page embedding-signature column). If `gbrain doctor` warns about a
|
||||
partial migration:
|
||||
|
||||
1. **Apply migrations manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Verify the cron no longer blocks:**
|
||||
```bash
|
||||
gbrain sync --all --dry-run # previews, exits 0, no confirmation demanded
|
||||
gbrain sources status # BACKFILL column present
|
||||
```
|
||||
3. **After a future model swap:** `gbrain embed --stale` re-embeds the drifted
|
||||
pages. (Pages embedded before this upgrade keep their old vectors until they
|
||||
are next re-embedded for some other reason — the no-surprise-cost
|
||||
grandfather behavior.)
|
||||
4. **If anything looks wrong,** file an issue at
|
||||
https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Sync cost gate
|
||||
- `gbrain sync --all` is mode-aware. When embedding is deferred to backfill
|
||||
jobs (the federated_v2 default), the gate is informational only: it prints a
|
||||
deferred notice (cap-aware, "not charged by this sync") plus the current
|
||||
backlog and never exits 2. The blocking confirmation gate fires only when
|
||||
sync embeds inline AND the new-content estimate exceeds
|
||||
`sync.cost_gate_min_usd` (default $0.50).
|
||||
- The inline estimate is delta-aware: sources unchanged since their last sync
|
||||
(HEAD == last_commit, clean tree, chunker version current) contribute 0;
|
||||
changed sources contribute the full-tree ceiling. The pre-existing stale
|
||||
backlog is shown informationally but does not gate inline sync (that backlog
|
||||
is `gbrain embed --stale`'s job, not sync's).
|
||||
- The cost preview prices against the configured model's real rate rather than
|
||||
a hardcoded OpenAI rate.
|
||||
|
||||
#### Real stale semantics (migration v108)
|
||||
- New `pages.embedding_signature` column records `<provider:model>:<dims>` when
|
||||
a page's chunks are embedded. `gbrain embed --stale`, the embed-backfill
|
||||
jobs, and the sync cost preview treat a page as stale when its signature
|
||||
differs from the current model's. NULL signature is grandfathered (never
|
||||
stale) so upgrading does not mass-re-embed.
|
||||
- All embed-write paths stamp the signature: `gbrain embed`/`--all`/`--stale`,
|
||||
the embed-backfill minion, `gbrain sync`'s inline import, and `gbrain import`.
|
||||
- New engine methods on both Postgres and PGLite: `sumStaleChunkChars`,
|
||||
`setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, plus an
|
||||
optional `signature` filter on `countStaleChunks` / `sumStaleChunkChars`.
|
||||
|
||||
#### Backfill visibility
|
||||
- `gbrain sources status` shows per-source embed-backfill state
|
||||
(queued/active/idle + last completion). The deferred sync notice appends the
|
||||
queued-job count so a cron operator sees work is enqueued, not lost.
|
||||
|
||||
#### Configuration
|
||||
- `sync.cost_gate_min_usd` (default 0.50) sets the inline-sync confirmation
|
||||
floor.
|
||||
## [0.41.30.0] - 2026-05-30
|
||||
|
||||
**`gbrain lsd --save` (and `brainstorm --save`) now actually writes the
|
||||
|
||||
@@ -10,6 +10,34 @@ change automatically.
|
||||
this mismatch and refuse to silently proceed. This doc is the recipe
|
||||
they point at.
|
||||
|
||||
## Same-dimension model swaps (v0.41.31.0 — automatic)
|
||||
|
||||
If you switch to a different model at the **same** dimension count
|
||||
(e.g. one 1536-dim provider to another, or a re-tuned model that keeps
|
||||
its width), the column type doesn't change, so no `ALTER`/wipe recipe
|
||||
is needed. As of v0.41.31.0, gbrain stamps an embedding-provenance
|
||||
signature (`<provider:model>:<dims>`) onto each page when its chunks are
|
||||
embedded. After you point the config at the new model, the stored
|
||||
signatures differ from the current one, and `gbrain embed --stale`
|
||||
re-embeds exactly those pages:
|
||||
|
||||
```bash
|
||||
# After switching to the new same-dim model in your config:
|
||||
gbrain embed --stale # re-embeds signature-drifted pages
|
||||
gbrain embed --stale --dry-run # preview the count without re-embedding
|
||||
```
|
||||
|
||||
Under federated_v2, the same drift is picked up by the per-source
|
||||
`embed-backfill` jobs that `gbrain sync --all` enqueues (capped
|
||||
`$X/source/24h`). **Grandfather:** pages embedded before v0.41.31.0
|
||||
carry a NULL signature and are NEVER flagged stale, so upgrading to
|
||||
v0.41.31.0 does NOT trigger a whole-corpus re-embed. Signatures only
|
||||
get stamped going forward.
|
||||
|
||||
A **dimension** change still requires the wipe-and-reinit (PGLite) or
|
||||
column-alter (Postgres) recipe below — the on-disk `vector(N)` width
|
||||
genuinely has to change.
|
||||
|
||||
## Why we don't do this automatically
|
||||
|
||||
Switching dimensions requires:
|
||||
|
||||
+15
-9
File diff suppressed because one or more lines are too long
+1
-1
@@ -141,5 +141,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.41.30.0"
|
||||
"version": "0.41.31.0"
|
||||
}
|
||||
|
||||
+46
-3
@@ -1,5 +1,5 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { embedBatch } from '../core/embedding.ts';
|
||||
import { embedBatch, currentEmbeddingSignature } from '../core/embedding.ts';
|
||||
import type { ChunkInput } from '../core/types.ts';
|
||||
import { chunkText } from '../core/chunkers/recursive.ts';
|
||||
import { createProgress, type ProgressReporter } from '../core/progress.ts';
|
||||
@@ -378,6 +378,16 @@ async function embedPage(
|
||||
}));
|
||||
|
||||
await engine.upsertChunks(slug, updated, opts);
|
||||
// v0.41.31: stamp provenance so a later model/dims swap is detectable as
|
||||
// stale. embedPage is the per-slug path used by `gbrain embed <slug>` AND
|
||||
// by `gbrain sync`'s post-import embed step (runEmbedCore({slugs})).
|
||||
// Guard: only stamp when EVERY chunk was (re)embedded this pass. If some
|
||||
// chunks were preserved from a prior embed (unknown/old provenance), the
|
||||
// page is mixed — don't claim it's current. `embed --all` fully re-embeds
|
||||
// such a page and then stamps it.
|
||||
if (toEmbed.length === chunks.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
}
|
||||
result.embedded += toEmbed.length;
|
||||
result.pages_processed++;
|
||||
slog(`${slug}: embedded ${toEmbed.length} chunks`);
|
||||
@@ -396,6 +406,10 @@ async function embedAll(
|
||||
catchUp?: boolean;
|
||||
},
|
||||
) {
|
||||
// v0.41.31: current embedding provenance signature. Stamped onto pages
|
||||
// when their chunks are (re)embedded so a later model/dimension swap is
|
||||
// detectable as stale.
|
||||
const signature = currentEmbeddingSignature();
|
||||
// ─────────────────────────────────────────────────────────────
|
||||
// Stale-only fast path: avoid the listPages + per-page getChunks
|
||||
// bomb that pulled every page row + every chunk's embedding column
|
||||
@@ -412,7 +426,7 @@ async function embedAll(
|
||||
if (staleOnly) {
|
||||
// D7: thread sourceId so `gbrain embed --stale --source X` actually scopes.
|
||||
// v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path.
|
||||
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts);
|
||||
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature);
|
||||
}
|
||||
|
||||
// v0.31.12: when sourceId is set, scope listPages to that source.
|
||||
@@ -482,6 +496,9 @@ async function embedAll(
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(page.slug, updated, pageOpts);
|
||||
// v0.41.31: stamp embedding provenance so a later model swap is
|
||||
// detectable as stale.
|
||||
await engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature });
|
||||
result.embedded += toEmbed.length;
|
||||
} catch (e: unknown) {
|
||||
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
|
||||
@@ -543,13 +560,31 @@ async function embedAllStale(
|
||||
priority?: 'recent';
|
||||
catchUp?: boolean;
|
||||
},
|
||||
signature?: string,
|
||||
) {
|
||||
// D7: thread sourceId so source-scoped runs only count + visit
|
||||
// that source's NULL embeddings.
|
||||
const sourceOpt = sourceId ? { sourceId } : undefined;
|
||||
|
||||
// v0.41.31: re-embed pages whose embedding_signature drifted (model/dims
|
||||
// swap). dry-run must NOT mutate, so it counts signature-stale via the
|
||||
// widened predicate; a live run NULLs them first so the existing
|
||||
// NULL-embedding cursor (listStaleChunks) picks them up unchanged.
|
||||
if (!dryRun && signature) {
|
||||
const invalidated = await engine.invalidateStaleSignatureEmbeddings({
|
||||
signature,
|
||||
...(sourceId && { sourceId }),
|
||||
});
|
||||
if (invalidated > 0) {
|
||||
slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`);
|
||||
}
|
||||
}
|
||||
|
||||
// Pre-flight: 0 stale chunks → nothing to do, no further DB reads.
|
||||
const staleCount = await engine.countStaleChunks(sourceOpt);
|
||||
// dry-run includes signature-drift in the count without mutating.
|
||||
const staleCount = await engine.countStaleChunks(
|
||||
dryRun && signature ? { ...sourceOpt, signature } : sourceOpt,
|
||||
);
|
||||
if (staleCount === 0) {
|
||||
if (dryRun) {
|
||||
slog('[dry-run] Would embed 0 chunks (0 stale found)');
|
||||
@@ -671,6 +706,14 @@ async function embedAllStale(
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(slug, merged, { sourceId: keySourceId });
|
||||
// v0.41.31: stamp provenance after the page's chunks are embedded —
|
||||
// but only when EVERY chunk was stale (fully re-embedded this pass).
|
||||
// A partially-stale page keeps preserved chunks of unknown/old
|
||||
// provenance, so don't claim it's current. (After invalidate, a
|
||||
// signature-drifted page IS fully stale → this stamps it.)
|
||||
if (signature && stale.length === existing.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature });
|
||||
}
|
||||
result.embedded += stale.length;
|
||||
} catch (e: unknown) {
|
||||
// Budget-fired aborts are expected on the way out; don't spam
|
||||
|
||||
+10
-3
@@ -599,22 +599,29 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
// Human-readable table: SOURCE | LAG | EMBED | FAILS | QUEUE | PAGES | LAST SYNC
|
||||
// Human-readable table: SOURCE | LAG | EMBED | BACKFILL | FAILS | QUEUE | PAGES | LAST SYNC
|
||||
console.log('SOURCES — health');
|
||||
console.log('────────────────');
|
||||
console.log(
|
||||
` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`,
|
||||
` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'BACKFILL'.padEnd(9)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`,
|
||||
);
|
||||
for (const m of metrics) {
|
||||
const lag = m.lag_seconds === null
|
||||
? 'never'
|
||||
: formatLag(m.lag_seconds);
|
||||
const embed = `${m.embed_coverage_pct.toFixed(0)}%`;
|
||||
// v0.41.31: embed-backfill state (active beats queued beats idle) so a
|
||||
// cron operator sees deferred embedding work after `sync --all`.
|
||||
const backfill = m.backfill_active > 0
|
||||
? `active(${m.backfill_active})`
|
||||
: m.backfill_queued > 0
|
||||
? `queued(${m.backfill_queued})`
|
||||
: 'idle';
|
||||
const fails = String(m.failed_jobs_24h);
|
||||
const queue = String(m.queue_depth);
|
||||
const pages = m.total_pages.toLocaleString();
|
||||
const sync = m.last_sync_at ? new Date(m.last_sync_at).toISOString().slice(0, 19).replace('T', ' ') : 'never';
|
||||
console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`);
|
||||
console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${backfill.padEnd(9)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`);
|
||||
}
|
||||
console.log('');
|
||||
for (const m of metrics) {
|
||||
|
||||
+300
-100
@@ -17,7 +17,17 @@ import {
|
||||
formatCodeBreakdown,
|
||||
} from '../core/sync.ts';
|
||||
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts';
|
||||
import {
|
||||
estimateEmbeddingCostUsd,
|
||||
getEmbeddingModelName,
|
||||
currentEmbeddingPricePerMTok,
|
||||
currentEmbeddingSignature,
|
||||
willEmbedSynchronously,
|
||||
shouldBlockSync,
|
||||
} from '../core/embedding.ts';
|
||||
import { estimateCostFromChars } from '../core/embedding-pricing.ts';
|
||||
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
|
||||
import { SPEND_CAP_CONFIG_KEY } from '../core/embed-backfill-submit.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
@@ -78,64 +88,114 @@ export interface SyncResult {
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 8 (D1) — walk each source's working tree and
|
||||
* sum tokens for every syncable file. This is a conservative overestimate
|
||||
* (full file content, not just the incremental diff) because `sync --all`
|
||||
* on a source that hasn't been synced yet WILL embed every file in the
|
||||
* working tree. For already-synced sources with only incremental changes,
|
||||
* the overestimate is the ceiling, not the floor — users never get
|
||||
* surprised by MORE cost than the preview claims. The false-high bias is
|
||||
* intentional: a lower estimate that undersells the real bill would be
|
||||
* worse than one that oversells.
|
||||
* Walk ONE source's working tree and sum tokens for every syncable file.
|
||||
* Conservative full-tree ceiling (full file content, not the incremental
|
||||
* diff) — over-counts, never under-counts, and matches the filesystem set
|
||||
* `sync` actually imports (collectSyncableFiles + content_hash, NOT a git
|
||||
* commit diff). Best-effort per file and per source: anything unreadable
|
||||
* contributes 0 rather than blocking the preview.
|
||||
*
|
||||
* v0.31.2: routed through collectSyncableFiles (lstat + inode-cycle +
|
||||
* max-depth) so the preview walks exactly what the real sync walks.
|
||||
*/
|
||||
function estimateSyncAllCost(sources: Array<{ local_path: string | null; config: Record<string, unknown> }>): {
|
||||
totalTokens: number;
|
||||
totalFiles: number;
|
||||
activeSources: number;
|
||||
perSource: Array<{ path: string; tokens: number; files: number }>;
|
||||
} {
|
||||
let totalTokens = 0;
|
||||
let totalFiles = 0;
|
||||
let activeSources = 0;
|
||||
const perSource: Array<{ path: string; tokens: number; files: number }> = [];
|
||||
function estimateSourceTreeTokens(
|
||||
localPath: string,
|
||||
strategy: 'markdown' | 'code' | 'auto',
|
||||
): { tokens: number; files: number } {
|
||||
let tokens = 0;
|
||||
let files = 0;
|
||||
try {
|
||||
const fileList = collectSyncableFiles(localPath, { strategy });
|
||||
for (const fullPath of fileList) {
|
||||
try {
|
||||
const stat = statSync(fullPath);
|
||||
if (stat.size > 5_000_000) continue; // skip large binaries
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
tokens += estimateTokens(content);
|
||||
files++;
|
||||
} catch {
|
||||
// Best-effort per file; sync itself tolerates the same.
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Best-effort: a source whose local_path is gone/unreadable contributes 0.
|
||||
}
|
||||
return { tokens, files };
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.41.31 — INLINE-path new-content estimate. Per source, contribute ZERO
|
||||
* when the source is provably unchanged since its last sync (HEAD ==
|
||||
* last_commit AND clean working tree AND chunker_version matches CURRENT) —
|
||||
* `content_hash` short-circuits every file so nothing re-embeds. Otherwise
|
||||
* contribute the full-tree ceiling. The unchanged predicate mirrors
|
||||
* doctor's `sync_freshness` and sync's own "do work?" gate (sync.ts:1057+
|
||||
* 1075). `isSourceUnchangedSinceSync` is fail-open (probe error → false), so
|
||||
* a source we can't prove unchanged is conservatively re-estimated rather
|
||||
* than silently priced at $0.
|
||||
*/
|
||||
function estimateInlineNewTokens(
|
||||
sources: Array<{
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
last_commit: string | null;
|
||||
chunker_version: string | null;
|
||||
}>,
|
||||
currentChunkerVersion: string,
|
||||
): { tokens: number; changedSources: number; unchangedSources: number } {
|
||||
let tokens = 0;
|
||||
let changedSources = 0;
|
||||
let unchangedSources = 0;
|
||||
for (const src of sources) {
|
||||
if (!src.local_path) continue;
|
||||
const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' };
|
||||
if (cfg.syncEnabled === false) continue;
|
||||
activeSources++;
|
||||
let sourceTokens = 0;
|
||||
let sourceFiles = 0;
|
||||
try {
|
||||
// v0.31.2: cost preview routed through collectSyncableFiles
|
||||
// (single hardened walker; see import.ts). Previously
|
||||
// walkSyncableFiles used statSync (followed symlinks). New walker
|
||||
// uses lstat + inode-cycle + max-depth so the preview matches
|
||||
// what the real sync will actually walk.
|
||||
const files = collectSyncableFiles(src.local_path, { strategy: cfg.strategy ?? 'markdown' });
|
||||
for (const fullPath of files) {
|
||||
try {
|
||||
const stat = statSync(fullPath);
|
||||
if (stat.size > 5_000_000) continue; // skip large binaries
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
sourceTokens += estimateTokens(content);
|
||||
sourceFiles++;
|
||||
} catch {
|
||||
// Best-effort per file. Skip unreadable files silently;
|
||||
// sync itself tolerates the same.
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Best-effort: a source whose local_path is gone or unreadable just
|
||||
// contributes 0. The sync itself would have failed anyway; no point
|
||||
// blocking the preview on a pre-existing fault.
|
||||
const unchanged =
|
||||
isSourceUnchangedSinceSync(src.local_path, src.last_commit, { requireCleanWorkingTree: true }) &&
|
||||
src.chunker_version === currentChunkerVersion;
|
||||
if (unchanged) {
|
||||
unchangedSources++;
|
||||
continue;
|
||||
}
|
||||
totalTokens += sourceTokens;
|
||||
totalFiles += sourceFiles;
|
||||
perSource.push({ path: src.local_path, tokens: sourceTokens, files: sourceFiles });
|
||||
changedSources++;
|
||||
tokens += estimateSourceTreeTokens(src.local_path, cfg.strategy ?? 'markdown').tokens;
|
||||
}
|
||||
return { tokens, changedSources, unchangedSources };
|
||||
}
|
||||
|
||||
return { totalTokens, totalFiles, activeSources, perSource };
|
||||
/**
|
||||
* Resolve the inline-path cost-gate floor in USD. Config key
|
||||
* `sync.cost_gate_min_usd` (DB plane), default $0.50. Below this estimate
|
||||
* the inline gate proceeds without blocking. Fail-open to the default on a
|
||||
* missing/invalid value or a config-read error (the gate must never crash
|
||||
* the sync). Accepts 0 (an operator can set the floor to $0 to make the
|
||||
* gate block on any nonzero inline cost).
|
||||
*/
|
||||
async function resolveCostGateFloorUsd(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const raw = await engine.getConfig('sync.cost_gate_min_usd');
|
||||
if (raw === null || raw === undefined) return 0.5;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n >= 0 ? n : 0.5;
|
||||
} catch {
|
||||
return 0.5;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the per-source embed-backfill 24h spend cap (USD) for the deferred
|
||||
* notice. Mirrors embed-backfill-submit.ts's own resolution of
|
||||
* SPEND_CAP_CONFIG_KEY (default 25). Fail-open to the default.
|
||||
*/
|
||||
async function resolveBackfillCapUsd(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const raw = await engine.getConfig(SPEND_CAP_CONFIG_KEY);
|
||||
if (raw === null || raw === undefined) return 25;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n > 0 ? n : 25;
|
||||
} catch {
|
||||
return 25;
|
||||
}
|
||||
}
|
||||
|
||||
/** Interactive [y/N] prompt. Resolves false on non-y answers or EOF. */
|
||||
@@ -2231,62 +2291,149 @@ See also:
|
||||
// source (no checkout) has nothing for `sync` to pull. Sources with
|
||||
// syncEnabled=false in config.jsonb are skipped too.
|
||||
if (syncAll) {
|
||||
const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record<string, unknown> }>(
|
||||
`SELECT id, name, local_path, config FROM sources WHERE local_path IS NOT NULL`,
|
||||
// v0.41.31: SELECT carries last_commit + chunker_version so the inline
|
||||
// cost preview's "unchanged source → 0" short-circuit can mirror sync's
|
||||
// own "do work?" gate (sync.ts:1057+1075) + doctor's sync_freshness.
|
||||
// Both columns predate v0.41 (writeSyncAnchor / writeChunkerVersion); no
|
||||
// schema migration needed.
|
||||
const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record<string, unknown>; last_commit: string | null; chunker_version: string | null }>(
|
||||
`SELECT id, name, local_path, config, last_commit, chunker_version FROM sources WHERE local_path IS NOT NULL`,
|
||||
);
|
||||
if (!sources || sources.length === 0) {
|
||||
console.log('No sources with local_path configured. Use `gbrain sources add <id> --path <path>` first.');
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 8 D1 — cost preview + ConfirmationRequired
|
||||
// gate. Before kicking off a multi-source sync that may embed tens of
|
||||
// thousands of chunks (real money), walk the sync-diff set(s), sum
|
||||
// tokens, compute USD estimate, and gate:
|
||||
// - TTY + !json + !yes → interactive [y/N] prompt
|
||||
// - non-TTY OR --json OR piped → emit ConfirmationRequired envelope,
|
||||
// exit 2 (reserve 1 for runtime errors)
|
||||
// - --yes → skip prompt entirely
|
||||
// - --dry-run → preview + exit 0
|
||||
// Skipped entirely when --no-embed is set (user already opted out of
|
||||
// the cost and will run `embed --stale` later).
|
||||
// v0.41.31 — mode-aware cost gate. Resolve federated_v2 ONCE here so both
|
||||
// the gate (below) and the fan-out (further down) share it.
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
const v2Enabled = await isFederatedV2Enabled(engine);
|
||||
|
||||
// v0.41.31 cost gate (supersedes the v0.20.0 unconditional gate). Under
|
||||
// federated_v2 sync DEFERS embedding to per-source embed-backfill jobs
|
||||
// that carry their own $X/source/24h spend cap, so sync itself spends
|
||||
// nothing synchronously — the gate is INFORMATIONAL (never exit 2) on
|
||||
// that path. The blocking ConfirmationRequired gate fires ONLY when embed
|
||||
// runs INLINE (v2 off, or --serial without --no-embed) AND the estimated
|
||||
// spend exceeds `sync.cost_gate_min_usd` (default $0.50). Skipped entirely
|
||||
// when --no-embed is set (user opted out; will run `embed --stale` later).
|
||||
if (!noEmbed) {
|
||||
const preview = estimateSyncAllCost(sources);
|
||||
const costUsd = estimateEmbeddingCostUsd(preview.totalTokens);
|
||||
const previewMsg =
|
||||
`sync --all preview: ${preview.totalFiles} files across ${preview.activeSources} source(s), ` +
|
||||
`~${preview.totalTokens.toLocaleString()} tokens, est. $${costUsd.toFixed(2)} on ${EMBEDDING_MODEL}.`;
|
||||
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', preview, costUsd, model: EMBEDDING_MODEL }));
|
||||
} else {
|
||||
console.log(previewMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed });
|
||||
// Stale backlog: cheap single SQL; fail-open to 0 so a transient DB
|
||||
// hiccup never blocks the sync.
|
||||
let staleChars = 0;
|
||||
try {
|
||||
// v0.41.31: signature-aware so a model/dims swap surfaces in the
|
||||
// backlog estimate (NULL signature grandfathered → not counted).
|
||||
staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() });
|
||||
} catch {
|
||||
staleChars = 0;
|
||||
}
|
||||
const rate = currentEmbeddingPricePerMTok();
|
||||
const staleCostUsd = estimateCostFromChars(staleChars, rate);
|
||||
const embeddingModelName = getEmbeddingModelName();
|
||||
const floorUsd = await resolveCostGateFloorUsd(engine);
|
||||
|
||||
if (!yesFlag) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || jsonOut) {
|
||||
// Agent-facing path: emit structured envelope, exit 2.
|
||||
const envelope = serializeError(errorFor({
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'cost_preview_requires_yes',
|
||||
message: previewMsg,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
|
||||
}));
|
||||
console.log(JSON.stringify({ error: envelope, preview, costUsd, model: EMBEDDING_MODEL }));
|
||||
process.exit(2);
|
||||
if (mode === 'deferred') {
|
||||
// Deferred path: print an FYI, NEVER exit 2. The backfill cap is the
|
||||
// real money gate (D1/D4).
|
||||
const capUsd = await resolveBackfillCapUsd(engine);
|
||||
// v0.41.31 (TODO-2): surface already-queued backfill jobs so a cron
|
||||
// operator sees work is enqueued, not lost. Best-effort — minion_jobs
|
||||
// may not exist on a brain that never ran a worker.
|
||||
let queuedBackfills = 0;
|
||||
try {
|
||||
const r = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM minion_jobs
|
||||
WHERE name = 'embed-backfill'
|
||||
AND status IN ('waiting','active','delayed','waiting-children')`,
|
||||
);
|
||||
queuedBackfills = Number(r[0]?.n) || 0;
|
||||
} catch {
|
||||
queuedBackfills = 0;
|
||||
}
|
||||
// Interactive TTY path: prompt [y/N].
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
const deferredMsg =
|
||||
`sync --all: embedding deferred to backfill jobs ` +
|
||||
`(capped $${capUsd}/source/24h, not charged by this sync). ` +
|
||||
`Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` +
|
||||
`${embeddingModelName}) across ${sources.length} source(s); ` +
|
||||
`${queuedBackfills} backfill job(s) queued.`;
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
}
|
||||
// fall through to sync — no exit 2.
|
||||
} else {
|
||||
// Inline path: sync embeds synchronously with no backfill cap to
|
||||
// protect it, so the blocking gate applies. The BLOCKING cost is the
|
||||
// new-content estimate ONLY (full-tree ceiling for changed sources;
|
||||
// unchanged contribute 0) — that's what this sync actually embeds.
|
||||
// The pre-existing stale backlog (NULL embeddings + signature drift)
|
||||
// is NOT swept by sync; `gbrain embed --stale` clears it. So we show
|
||||
// it informationally but never gate on cost this sync won't incur
|
||||
// (else a model swap would block the next inline cron — F2).
|
||||
const currentChunkerVersion = String(CHUNKER_VERSION);
|
||||
const inline = estimateInlineNewTokens(sources, currentChunkerVersion);
|
||||
const newCostUsd = estimateEmbeddingCostUsd(inline.tokens);
|
||||
const costUsd = newCostUsd;
|
||||
const staleNote = staleChars > 0
|
||||
? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)`
|
||||
: '';
|
||||
const previewMsg =
|
||||
`sync --all preview (inline embed): ${inline.changedSources} changed source(s), ` +
|
||||
`${inline.unchangedSources} unchanged; ~${inline.tokens.toLocaleString()} new tokens, ` +
|
||||
`est. $${costUsd.toFixed(2)} on ${embeddingModelName}${staleNote}.`;
|
||||
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(previewMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (!yesFlag) {
|
||||
if (shouldBlockSync(costUsd, floorUsd, mode)) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || jsonOut) {
|
||||
// Agent-facing path: emit structured envelope, exit 2.
|
||||
const envelope = serializeError(errorFor({
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'cost_preview_requires_yes',
|
||||
message: previewMsg,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
|
||||
}));
|
||||
console.log(JSON.stringify({ error: envelope, mode, gate: 'confirmation_required', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
process.exit(2);
|
||||
}
|
||||
// Interactive TTY path: prompt [y/N].
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
// Below floor → proceed without blocking (kills inline-cron noise).
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(`${previewMsg} Below cost gate floor ($${floorUsd.toFixed(2)}), proceeding.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2302,8 +2449,7 @@ See also:
|
||||
// - withSourcePrefix wrap inside runOne so slog/serr lines from
|
||||
// performSync get the [<source-id>] prefix under parallel mode (D6)
|
||||
// - stable JSON envelope {schema_version:1, sources, ...} when --json
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
const v2Enabled = await isFederatedV2Enabled(engine);
|
||||
// v0.41.31: v2Enabled resolved once above (cost gate). Reused here.
|
||||
const activeSources = sources.filter((s) => {
|
||||
const cfg = (s.config || {}) as { syncEnabled?: boolean };
|
||||
return cfg.syncEnabled !== false;
|
||||
@@ -2791,6 +2937,13 @@ export interface SyncStatusReportSource {
|
||||
chunks_total: number;
|
||||
chunks_unembedded: number;
|
||||
embedding_coverage_pct: number;
|
||||
// v0.41.31: embed-backfill job visibility (federated_v2 defers embedding
|
||||
// to these jobs; without this an operator can't see queued/lagging work
|
||||
// after `sync --all` exits 0). Best-effort — all 0 / null on brains
|
||||
// without the minion_jobs table.
|
||||
backfill_queued: number;
|
||||
backfill_active: number;
|
||||
backfill_last_completed_at: string | null;
|
||||
}
|
||||
|
||||
export interface SyncStatusReport {
|
||||
@@ -2892,6 +3045,43 @@ export async function buildSyncStatusReport(
|
||||
});
|
||||
}
|
||||
|
||||
// v0.41.31: per-source embed-backfill job state. Best-effort — the
|
||||
// minion_jobs table doesn't exist on every brain (a brain that never ran
|
||||
// a worker has the pre-minions schema), and the dashboard must not crash
|
||||
// for that. A failure → empty map → all sources report 0/null.
|
||||
type BackfillRow = {
|
||||
source_id: string | null;
|
||||
queued: string | number;
|
||||
active: string | number;
|
||||
last_completed_at: string | Date | null;
|
||||
};
|
||||
const backfillMap = new Map<string, { queued: number; active: number; last_completed_at: string | null }>();
|
||||
if (sourceIds.length > 0) {
|
||||
try {
|
||||
const backfillRows = await engine.executeRaw<BackfillRow>(
|
||||
`SELECT data->>'sourceId' AS source_id,
|
||||
COUNT(*) FILTER (WHERE status IN ('waiting','delayed','waiting-children'))::int AS queued,
|
||||
COUNT(*) FILTER (WHERE status = 'active')::int AS active,
|
||||
MAX(finished_at) FILTER (WHERE status = 'completed') AS last_completed_at
|
||||
FROM minion_jobs
|
||||
WHERE name = 'embed-backfill' AND data->>'sourceId' = ANY($1::text[])
|
||||
GROUP BY data->>'sourceId'`,
|
||||
[sourceIds],
|
||||
);
|
||||
for (const r of backfillRows) {
|
||||
if (!r.source_id) continue;
|
||||
const last = r.last_completed_at;
|
||||
backfillMap.set(r.source_id, {
|
||||
queued: Number(r.queued) || 0,
|
||||
active: Number(r.active) || 0,
|
||||
last_completed_at: last == null ? null : (last instanceof Date ? last.toISOString() : last),
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// minion_jobs absent / unreadable → leave backfillMap empty.
|
||||
}
|
||||
}
|
||||
|
||||
const now = Date.now();
|
||||
const out: SyncStatusReportSource[] = sources.map((src) => {
|
||||
const cfgEntry = (src.config || {}) as { syncEnabled?: boolean };
|
||||
@@ -2928,6 +3118,9 @@ export async function buildSyncStatusReport(
|
||||
chunks_total: counts.chunks_total,
|
||||
chunks_unembedded: counts.chunks_unembedded,
|
||||
embedding_coverage_pct: embeddingCoveragePct,
|
||||
backfill_queued: backfillMap.get(src.id)?.queued ?? 0,
|
||||
backfill_active: backfillMap.get(src.id)?.active ?? 0,
|
||||
backfill_last_completed_at: backfillMap.get(src.id)?.last_completed_at ?? null,
|
||||
};
|
||||
});
|
||||
|
||||
@@ -2968,7 +3161,7 @@ export function printSyncStatusReport(
|
||||
write(' (no sources registered)');
|
||||
return;
|
||||
}
|
||||
const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'LAST SYNC'];
|
||||
const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'BACKFILL', 'LAST SYNC'];
|
||||
const rows = report.sources.map((s) => {
|
||||
const stale = s.staleness_hours === null
|
||||
? 'never'
|
||||
@@ -2976,21 +3169,28 @@ export function printSyncStatusReport(
|
||||
const stateBits: string[] = [];
|
||||
if (!s.sync_enabled) stateBits.push('disabled');
|
||||
stateBits.push(s.staleness_class);
|
||||
// BACKFILL: active beats queued beats idle for the at-a-glance cell.
|
||||
const backfill = s.backfill_active > 0
|
||||
? `active(${s.backfill_active})`
|
||||
: s.backfill_queued > 0
|
||||
? `queued(${s.backfill_queued})`
|
||||
: 'idle';
|
||||
return [
|
||||
s.name,
|
||||
stateBits.join(','),
|
||||
stale,
|
||||
String(s.pages),
|
||||
`${s.embedding_coverage_pct}%`,
|
||||
backfill,
|
||||
s.last_sync_at ?? '(never)',
|
||||
];
|
||||
});
|
||||
const widths = headers.map((h, i) =>
|
||||
Math.max(h.length, ...rows.map((r) => r[i].length)),
|
||||
);
|
||||
// Numeric columns (PAGES at index 3, EMBEDDED at index 4, STALENESS
|
||||
// at index 2) right-pad-left so digits align cleanly. Text columns
|
||||
// left-pad-right per the existing `sources list` convention.
|
||||
// Numeric columns (STALENESS=2, PAGES=3, EMBEDDED=4) right-pad-left so
|
||||
// digits align cleanly. Text columns (incl. BACKFILL=5) left-pad-right
|
||||
// per the existing `sources list` convention.
|
||||
const NUMERIC_COLS = new Set([2, 3, 4]);
|
||||
const fmt = (cells: string[]) =>
|
||||
cells.map((c, i) => (NUMERIC_COLS.has(i) ? c.padStart(widths[i]) : c.padEnd(widths[i]))).join(' ');
|
||||
|
||||
@@ -51,6 +51,14 @@ export interface EmbedStaleOpts {
|
||||
* the gateway. Production callers leave it unset.
|
||||
*/
|
||||
embedFn?: (texts: string[], opts: { abortSignal?: AbortSignal }) => Promise<Float32Array[]>;
|
||||
/**
|
||||
* v0.41.31: current embedding provenance signature (`<provider:model>:<dims>`).
|
||||
* When set, embeddings stamped under a DIFFERENT signature are invalidated
|
||||
* (NULLed) at the start so they flow through the NULL cursor and get
|
||||
* re-embedded; each page's signature is stamped after its chunks land.
|
||||
* Omit to keep the legacy `embedding IS NULL`-only behavior.
|
||||
*/
|
||||
embeddingSignature?: string;
|
||||
}
|
||||
|
||||
export interface EmbedStaleResult {
|
||||
@@ -109,6 +117,18 @@ export async function embedStaleForSource(
|
||||
done: false,
|
||||
aborted: false,
|
||||
};
|
||||
const signature = opts.embeddingSignature;
|
||||
|
||||
// v0.41.31: invalidate embeddings stamped under a prior model signature so
|
||||
// the NULL cursor below re-embeds them. GRANDFATHER: NULL signature
|
||||
// untouched. Best-effort — a failure here must not abort the backfill.
|
||||
if (signature) {
|
||||
try {
|
||||
await engine.invalidateStaleSignatureEmbeddings({ signature, sourceId });
|
||||
} catch {
|
||||
// Non-fatal: fall through to the NULL-only stale loop.
|
||||
}
|
||||
}
|
||||
|
||||
for (;;) {
|
||||
if (signal?.aborted) {
|
||||
@@ -170,6 +190,13 @@ export async function embedStaleForSource(
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(slug, merged, { sourceId: keySourceId });
|
||||
// v0.41.31: stamp provenance only when EVERY chunk was stale (fully
|
||||
// re-embedded this pass) — a partially-stale page keeps preserved
|
||||
// chunks of unknown provenance, so don't claim current. After the
|
||||
// invalidate pass above, signature-drifted pages ARE fully stale.
|
||||
if (signature && stale.length === existing.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature });
|
||||
}
|
||||
result.embedded += stale.length;
|
||||
result.pagesProcessed += 1;
|
||||
} catch (e: unknown) {
|
||||
|
||||
+99
-5
@@ -12,6 +12,7 @@ import {
|
||||
getEmbeddingModel as gatewayGetModel,
|
||||
getEmbeddingDimensions as gatewayGetDims,
|
||||
} from './ai/gateway.ts';
|
||||
import { lookupEmbeddingPrice } from './embedding-pricing.ts';
|
||||
|
||||
// v0.27.1: re-export multimodal embedding so callers can pull both text and
|
||||
// image embedding APIs from `src/core/embedding`. import-image-file consumes
|
||||
@@ -127,13 +128,106 @@ export const EMBEDDING_MODEL = 'text-embedding-3-large';
|
||||
export const EMBEDDING_DIMENSIONS = 1536;
|
||||
|
||||
/**
|
||||
* USD cost per 1k tokens for text-embedding-3-large. Used by
|
||||
* `gbrain sync --all` cost preview and `reindex-code` to surface
|
||||
* expected spend before accepting expensive operations.
|
||||
* USD cost per 1k tokens for text-embedding-3-large. Retained for back-compat
|
||||
* with callers/tests that import it directly; new cost math resolves the
|
||||
* ACTUAL configured model's rate via embedding-pricing.ts instead of assuming
|
||||
* OpenAI. (Hardcoding this rate produced cost previews that named the wrong
|
||||
* provider and over-stated spend ~2.6x when the brain ran on a cheaper model.)
|
||||
*/
|
||||
export const EMBEDDING_COST_PER_1K_TOKENS = 0.00013;
|
||||
|
||||
/** Compute USD cost estimate for embedding `tokens` at current model rate. */
|
||||
/**
|
||||
* Resolve the price-per-1M-tokens for the currently-configured embedding
|
||||
* model. Falls back to the OpenAI text-embedding-3-large rate only when the
|
||||
* model is unknown to the pricing table.
|
||||
*/
|
||||
export function currentEmbeddingPricePerMTok(): number {
|
||||
let modelString: string;
|
||||
try {
|
||||
modelString = gatewayGetModel(); // e.g. 'zeroentropyai:zembed-1'
|
||||
} catch {
|
||||
// Gateway not configured (e.g. unit tests, cost preview before connect).
|
||||
// Fall back to the OpenAI text-embedding-3-large default rate.
|
||||
return 0.13;
|
||||
}
|
||||
const hit = lookupEmbeddingPrice(modelString);
|
||||
return hit.kind === 'known' ? hit.pricePerMTok : 0.13;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute USD cost estimate for embedding `tokens` at the CURRENT configured
|
||||
* model's rate (not a hardcoded OpenAI rate).
|
||||
*/
|
||||
export function estimateEmbeddingCostUsd(tokens: number): number {
|
||||
return (tokens / 1000) * EMBEDDING_COST_PER_1K_TOKENS;
|
||||
return (tokens / 1_000_000) * currentEmbeddingPricePerMTok();
|
||||
}
|
||||
|
||||
/**
|
||||
* Embedding provenance signature for the currently-configured model:
|
||||
* `<provider:model>:<dims>` (e.g. `openai:text-embedding-3-large:1536`).
|
||||
* Stamped onto `pages.embedding_signature` when a page's chunks are
|
||||
* embedded so a later model/dimension swap can be detected as stale.
|
||||
*
|
||||
* Deliberately does NOT include the chunker version — chunker drift is
|
||||
* already tracked per-page via `pages.chunker_version` (used by sync +
|
||||
* doctor). This signature is strictly about the EMBEDDING space.
|
||||
*
|
||||
* Falls back to the OpenAI default signature when the gateway is
|
||||
* unconfigured (unit-test context), matching the other estimator fallbacks.
|
||||
*/
|
||||
export function currentEmbeddingSignature(): string {
|
||||
try {
|
||||
return `${gatewayGetModel()}:${gatewayGetDims()}`;
|
||||
} catch {
|
||||
return `${EMBEDDING_MODEL}:${EMBEDDING_DIMENSIONS}`;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a `gbrain sync --all` invocation will embed at sync time
|
||||
* ('inline') or defer embedding to per-source `embed-backfill` minion jobs
|
||||
* ('deferred'). Under federated_v2 the default path defers; the backfill
|
||||
* jobs carry their own 10-min cooldown + $25/source/24h spend cap, so the
|
||||
* sync-time cost gate only BLOCKS on the inline path. See sync.ts:2346
|
||||
* (`effectiveNoEmbed`) — this mirrors that resolution exactly.
|
||||
*/
|
||||
export type SyncEmbedMode = 'deferred' | 'inline';
|
||||
|
||||
/**
|
||||
* Resolve the embed mode from the same three signals sync.ts uses to
|
||||
* compute `effectiveNoEmbed`. Single source of truth so the cost gate and
|
||||
* the actual embed decision can never drift.
|
||||
*
|
||||
* effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed
|
||||
*
|
||||
* Embed runs INLINE iff that resolves to false:
|
||||
* - v2 off → inline (legacy synchronous embed)
|
||||
* - v2 on + --serial + !--no-embed → inline
|
||||
* - v2 on (parallel) → deferred (backfill jobs)
|
||||
* - --no-embed (any path) → the caller skips the gate entirely;
|
||||
* we report 'deferred' for completeness.
|
||||
*/
|
||||
export function willEmbedSynchronously(opts: {
|
||||
v2Enabled: boolean;
|
||||
serialFlag: boolean;
|
||||
noEmbed: boolean;
|
||||
}): SyncEmbedMode {
|
||||
const effectiveNoEmbed =
|
||||
opts.v2Enabled && !opts.serialFlag && !opts.noEmbed ? true : opts.noEmbed;
|
||||
return effectiveNoEmbed ? 'deferred' : 'inline';
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure cost-gate decision. The gate BLOCKS (prompt in TTY, exit 2 envelope
|
||||
* in non-TTY) only when embed runs inline AND the estimated spend exceeds
|
||||
* the floor. Deferred mode NEVER blocks — the backfill cap is the real
|
||||
* money gate, and blocking the cheap markdown import for cost the import
|
||||
* doesn't synchronously incur is the bug this fix removes.
|
||||
*/
|
||||
export function shouldBlockSync(
|
||||
costUsd: number,
|
||||
floorUsd: number,
|
||||
mode: SyncEmbedMode,
|
||||
): boolean {
|
||||
return mode === 'inline' && costUsd > floorUsd;
|
||||
}
|
||||
|
||||
+32
-1
@@ -956,7 +956,38 @@ export interface BrainEngine {
|
||||
* `gbrain embed --stale --source media-corpus` expect only that
|
||||
* source's NULLs touched; the caller threads `sourceId` here.
|
||||
*/
|
||||
countStaleChunks(opts?: { sourceId?: string }): Promise<number>;
|
||||
countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number>;
|
||||
/**
|
||||
* Sum of LENGTH(chunk_text) over stale chunks — the character-count
|
||||
* backlog the embed phase / embed-backfill will process. Sibling of
|
||||
* countStaleChunks (same stale predicate + embed_skip filter + optional
|
||||
* sourceId scope); used by the `gbrain sync --all` cost preview to price
|
||||
* the embedding backlog via estimateCostFromChars. Returns 0 on an
|
||||
* empty/fully-embedded brain.
|
||||
*
|
||||
* v0.41.31: `signature` (optional) widens "stale" to ALSO include chunks
|
||||
* whose page `embedding_signature` is set AND differs from the current
|
||||
* model signature (a model/dims swap). NULL signature is GRANDFATHERED
|
||||
* (never counted) so the post-migration corpus isn't flagged en masse.
|
||||
* Omit `signature` for the legacy `embedding IS NULL`-only count.
|
||||
*/
|
||||
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number>;
|
||||
/**
|
||||
* Stamp `pages.embedding_signature = signature` for one page. Called after
|
||||
* a page's chunks are (re)embedded so a later model swap can detect it as
|
||||
* stale. Idempotent. No-op if the page doesn't exist.
|
||||
*/
|
||||
setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise<void>;
|
||||
/**
|
||||
* NULL out the embeddings (and embedded_at) of every chunk whose page
|
||||
* `embedding_signature` is set AND differs from `signature` — i.e. pages
|
||||
* embedded under a now-stale model. Returns the chunk count invalidated.
|
||||
* The embed-stale loop calls this BEFORE listStaleChunks so signature-
|
||||
* drift pages flow through the existing NULL-embedding cursor (keeps
|
||||
* listStaleChunks's keyset pagination untouched). GRANDFATHER: NULL
|
||||
* signature is never invalidated. `sourceId` scopes the sweep.
|
||||
*/
|
||||
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number>;
|
||||
/**
|
||||
* Return every chunk where embedding IS NULL, with the metadata needed
|
||||
* to call embedBatch + upsertChunks. The `embedding` column is omitted
|
||||
|
||||
+16
-1
@@ -8,7 +8,7 @@ import { chunkText } from './chunkers/recursive.ts';
|
||||
import { chunkCodeText, chunkCodeTextFull, detectCodeLanguage, CHUNKER_VERSION } from './chunkers/code.ts';
|
||||
import { findChunkForOffset } from './chunkers/edge-extractor.ts';
|
||||
import { extractCodeRefs, imageOfCandidates } from './link-extraction.ts';
|
||||
import { embedBatch, embedMultimodal } from './embedding.ts';
|
||||
import { embedBatch, embedMultimodal, currentEmbeddingSignature } from './embedding.ts';
|
||||
import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts';
|
||||
import type { ChunkInput, PageInput, PageType } from './types.ts';
|
||||
import { computeEffectiveDate } from './effective-date.ts';
|
||||
@@ -658,6 +658,13 @@ export async function importFromContent(
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await tx.upsertChunks(slug, chunks, txOpts);
|
||||
// v0.41.31: stamp embedding provenance when this import actually
|
||||
// embedded (not --no-embed), so a later model/dims swap is detectable
|
||||
// as stale via embed --stale. The deferred/backfill + per-slug embed
|
||||
// paths stamp too; this covers the inline import/sync path.
|
||||
if (!opts.noEmbed) {
|
||||
await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
}
|
||||
} else {
|
||||
// Content is empty — delete stale chunks so they don't ghost in search results
|
||||
await tx.deleteChunks(slug, txOpts);
|
||||
@@ -968,6 +975,14 @@ export async function importCodeFile(
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await tx.upsertChunks(slug, chunks, txOpts);
|
||||
// v0.41.31: stamp embedding provenance ONLY when every chunk was
|
||||
// freshly embedded with the current model this call (no reuse-by-hash
|
||||
// carrying old-model vectors). Mixed pages stay unstamped rather than
|
||||
// falsely marked current; `reindex --code --force` / `embed --stale`
|
||||
// handle the swap for those.
|
||||
if (!opts.noEmbed && needsEmbedIndexes.length === chunks.length) {
|
||||
await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
}
|
||||
} else {
|
||||
await tx.deleteChunks(slug, txOpts);
|
||||
}
|
||||
|
||||
@@ -4920,6 +4920,32 @@ export const MIGRATIONS: Migration[] = [
|
||||
EXECUTE FUNCTION bump_page_generation_clock_fn();
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 108,
|
||||
name: 'pages_embedding_signature',
|
||||
// v0.41.31 — embedding provenance for real stale semantics.
|
||||
//
|
||||
// Adds `pages.embedding_signature TEXT NULL` = `<provider:model>:<dims>`
|
||||
// stamped when a page's chunks are embedded (setPageEmbeddingSignature).
|
||||
// A later model/dimension swap makes the stored signature differ from
|
||||
// the current one, so countStaleChunks/sumStaleChunkChars (with the
|
||||
// `signature` opt) and invalidateStaleSignatureEmbeddings can detect and
|
||||
// re-embed those pages.
|
||||
//
|
||||
// GRANDFATHER (critical): the stale predicate is
|
||||
// `embedding_signature IS NOT NULL AND embedding_signature <> $current`
|
||||
// so a NULL signature is NEVER stale. After this migration every existing
|
||||
// page has NULL — none are flagged — so the next `embed --stale` does NOT
|
||||
// re-embed the whole corpus. Signatures only get stamped going forward.
|
||||
//
|
||||
// No index: the column is read only via a JOINed pages row in the
|
||||
// chunk-grain stale queries; no standalone lookup hot path. ADD COLUMN
|
||||
// with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT NULL;
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -35,6 +35,7 @@ import { tryAcquireDbLock } from '../../db-lock.ts';
|
||||
import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts';
|
||||
import { withBudgetTracker } from '../../ai/gateway.ts';
|
||||
import { embedStaleForSource } from '../../embed-stale.ts';
|
||||
import { currentEmbeddingSignature } from '../../embedding.ts';
|
||||
import type { BrainEngine } from '../../engine.ts';
|
||||
import type { MinionJobContext } from '../types.ts';
|
||||
|
||||
@@ -123,6 +124,9 @@ export function makeEmbedBackfillHandler(engine: BrainEngine) {
|
||||
embedStaleForSource(engine, sourceId, {
|
||||
batchSize,
|
||||
signal: job.signal,
|
||||
// v0.41.31: re-embed pages whose model signature drifted + stamp
|
||||
// provenance as chunks land.
|
||||
embeddingSignature: currentEmbeddingSignature(),
|
||||
onProgress: ({ embedded, chunksProcessed, cursor }) => {
|
||||
// Fire-and-forget; updateProgress returns a Promise but the
|
||||
// handler is sync inside the loop.
|
||||
|
||||
+94
-21
@@ -422,7 +422,9 @@ export class PGLiteEngine implements BrainEngine {
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='sources' AND column_name='trust_frontmatter_overrides') AS sources_trust_fm_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='embedding_signature') AS pages_embedding_signature_exists
|
||||
`);
|
||||
const probe = rows[0] as {
|
||||
pages_exists: boolean;
|
||||
@@ -463,6 +465,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
sources_cr_mode_exists: boolean;
|
||||
sources_trust_fm_exists: boolean;
|
||||
pages_generation_exists: boolean;
|
||||
pages_embedding_signature_exists: boolean;
|
||||
};
|
||||
|
||||
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
|
||||
@@ -530,6 +533,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// body; bootstrap only needs to add the column on pre-v91 brains so
|
||||
// the CREATE INDEX doesn't crash.
|
||||
const needsPagesGeneration = probe.pages_exists && !probe.pages_generation_exists;
|
||||
// v0.41.31 (v108): pages.embedding_signature for real stale semantics.
|
||||
// No SCHEMA_SQL index references it today; bootstrap is defense-in-depth
|
||||
// so future schema work doesn't wedge pre-v108 brains.
|
||||
const needsPagesEmbeddingSignature = probe.pages_exists && !probe.pages_embedding_signature_exists;
|
||||
|
||||
// Fresh installs (no tables yet) and modern brains both no-op.
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap
|
||||
@@ -539,7 +546,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
&& !needsFilesBootstrap && !needsOauthClientsBootstrap
|
||||
&& !needsSourcesArchive && !needsPagesLastRetrievedAt
|
||||
&& !needsPagesProvenance
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration) return;
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration
|
||||
&& !needsPagesEmbeddingSignature) return;
|
||||
|
||||
process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n');
|
||||
|
||||
@@ -767,6 +775,15 @@ export class PGLiteEngine implements BrainEngine {
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsPagesEmbeddingSignature) {
|
||||
// v108 (pages_embedding_signature): embedding provenance for real
|
||||
// stale semantics. NULL grandfathered (never stale). v108 runs later
|
||||
// via runMigrations and is idempotent.
|
||||
await this.db.exec(`
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
||||
@@ -2061,35 +2078,91 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r));
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string }): Promise<number> {
|
||||
// D7: source-scoped count for `gbrain embed --stale --source X`.
|
||||
// v0.41 (D4+D8+Codex r2 #11): always JOIN pages so embed-skip filter
|
||||
// applies via `NOT (frontmatter ? 'embed_skip')`. PGLite is
|
||||
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`,
|
||||
);
|
||||
const count = (rows[0] as { count: number } | undefined)?.count ?? 0;
|
||||
return Number(count);
|
||||
/**
|
||||
* Build the stale-chunk WHERE clause + positional params. embed_skip is
|
||||
* always excluded. `signature` widens "stale" to include embedding_signature
|
||||
* drift (NULL grandfathered → never stale). Shared by countStaleChunks +
|
||||
* sumStaleChunkChars so they can't drift.
|
||||
*/
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
|
||||
const params: unknown[] = [];
|
||||
const conds: string[] = [];
|
||||
if (opts?.signature !== undefined) {
|
||||
params.push(opts.signature);
|
||||
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
|
||||
} else {
|
||||
conds.push(`cc.embedding IS NULL`);
|
||||
}
|
||||
conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`);
|
||||
if (opts?.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
conds.push(`p.source_id = $${params.length}`);
|
||||
}
|
||||
return { where: conds.join(' AND '), params };
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// D7: source-scoped count for `gbrain embed --stale --source X`. Always
|
||||
// JOIN pages so embed-skip + signature predicates apply. PGLite is
|
||||
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = $1
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`,
|
||||
[opts.sourceId],
|
||||
WHERE ${where}`,
|
||||
params,
|
||||
);
|
||||
const count = (rows[0] as { count: number } | undefined)?.count ?? 0;
|
||||
return Number(count);
|
||||
}
|
||||
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
|
||||
// length for the sync cost preview. ::bigint guards int4 overflow.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params,
|
||||
);
|
||||
const chars = (rows[0] as { chars: number | string } | undefined)?.chars ?? 0;
|
||||
return Number(chars);
|
||||
}
|
||||
|
||||
async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise<void> {
|
||||
await this.db.query(
|
||||
`UPDATE pages SET embedding_signature = $1 WHERE slug = $2 AND source_id = $3`,
|
||||
[opts.signature, slug, opts.sourceId ?? 'default'],
|
||||
);
|
||||
}
|
||||
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
|
||||
// NULL out embeddings whose page signature is set AND differs from the
|
||||
// current model signature. GRANDFATHER: NULL signature untouched. Feeds
|
||||
// the existing NULL-embedding cursor so listStaleChunks stays unchanged.
|
||||
const params: unknown[] = [opts.signature];
|
||||
let srcClause = '';
|
||||
if (opts.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
srcClause = ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`UPDATE content_chunks cc
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.embedding IS NOT NULL
|
||||
AND p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1${srcClause}
|
||||
RETURNING cc.page_id`,
|
||||
params,
|
||||
);
|
||||
return (rows as unknown[]).length;
|
||||
}
|
||||
|
||||
async listStaleChunks(opts?: {
|
||||
batchSize?: number;
|
||||
afterPageId?: number;
|
||||
|
||||
+98
-30
@@ -455,7 +455,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'trust_frontmatter_overrides') AS sources_trust_fm_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'embedding_signature') AS pages_embedding_signature_exists
|
||||
`;
|
||||
const probe = probeRows[0]!;
|
||||
|
||||
@@ -530,6 +532,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
sources_cr_mode_exists?: boolean;
|
||||
sources_trust_fm_exists?: boolean;
|
||||
pages_generation_exists?: boolean;
|
||||
pages_embedding_signature_exists?: boolean;
|
||||
};
|
||||
const needsContextualRetrievalColumns = (probe.pages_exists
|
||||
&& (!probeCr.pages_cr_mode_exists || !probeCr.pages_corpus_generation_exists))
|
||||
@@ -540,6 +543,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
// it. Pre-v91 brains crash without the column; bootstrap adds it before
|
||||
// SCHEMA_SQL replay creates the index.
|
||||
const needsPagesGeneration = probe.pages_exists && !probeCr.pages_generation_exists;
|
||||
// v0.41.31 (v108): pages.embedding_signature for real stale semantics.
|
||||
// No SCHEMA_SQL index references it; bootstrap is defense-in-depth.
|
||||
const needsPagesEmbeddingSignature = probe.pages_exists && !probeCr.pages_embedding_signature_exists;
|
||||
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap
|
||||
&& !needsPagesDeletedAt && !needsMcpLogBootstrap && !needsSubagentProviderId
|
||||
@@ -548,7 +554,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
&& !needsOauthClientsBootstrap && !needsSourcesArchive
|
||||
&& !needsPagesLastRetrievedAt
|
||||
&& !needsPagesProvenance
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration) return;
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration
|
||||
&& !needsPagesEmbeddingSignature) return;
|
||||
|
||||
process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n');
|
||||
|
||||
@@ -774,6 +781,15 @@ export class PostgresEngine implements BrainEngine {
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsPagesEmbeddingSignature) {
|
||||
// v108 (pages_embedding_signature): embedding provenance for real stale
|
||||
// semantics. NULL grandfathered. v108 runs later via runMigrations and
|
||||
// is idempotent.
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
||||
@@ -2093,36 +2109,88 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows.map((r) => rowToChunk(r as Record<string, unknown>));
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string }): Promise<number> {
|
||||
const sql = this.sql;
|
||||
// v0.41 (D4+D8+Codex r2 #11): the embed-skip filter requires JOIN
|
||||
// pages so we always join — the pre-v0.41 "fast path" without join
|
||||
// is gone. JSONB `?` existence check is cheap on the small set of
|
||||
// skipped pages; full-scan benefits from the partial index on
|
||||
// embedding IS NULL regardless.
|
||||
//
|
||||
// D7: source_id scoping. NULL/undefined = scan all sources;
|
||||
// a value scopes to that source so `gbrain embed --stale --source X`
|
||||
// does what it says.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const [row] = await sql`
|
||||
SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
/**
|
||||
* Build the stale-chunk WHERE clause + positional params for sql.unsafe.
|
||||
* embed_skip always excluded. `signature` widens "stale" to include
|
||||
* embedding_signature drift (NULL grandfathered). Shared by
|
||||
* countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling).
|
||||
*/
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
|
||||
const params: unknown[] = [];
|
||||
const conds: string[] = [];
|
||||
if (opts?.signature !== undefined) {
|
||||
params.push(opts.signature);
|
||||
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
|
||||
} else {
|
||||
conds.push(`cc.embedding IS NULL`);
|
||||
}
|
||||
const [row] = await sql`
|
||||
SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`);
|
||||
if (opts?.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
conds.push(`p.source_id = $${params.length}`);
|
||||
}
|
||||
return { where: conds.join(' AND '), params };
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// Always JOIN pages so the embed_skip + signature predicates apply.
|
||||
// D7: source_id scoping. v0.41.31: optional signature widens staleness
|
||||
// to embedding_signature drift (NULL grandfathered).
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const rows = await this.sql.unsafe(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
|
||||
// length for the sync cost preview. ::bigint guards int4 overflow.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const rows = await this.sql.unsafe(
|
||||
`SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { chars?: number | string } | undefined)?.chars ?? 0);
|
||||
}
|
||||
|
||||
async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise<void> {
|
||||
const sql = this.sql;
|
||||
await sql`
|
||||
UPDATE pages SET embedding_signature = ${opts.signature}
|
||||
WHERE slug = ${slug} AND source_id = ${opts.sourceId ?? 'default'}
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
|
||||
// NULL embeddings whose page signature is set AND differs from current.
|
||||
// GRANDFATHER: NULL signature untouched. Feeds the NULL-embedding cursor
|
||||
// so listStaleChunks stays unchanged. RETURNING → row count.
|
||||
const params: unknown[] = [opts.signature];
|
||||
let srcClause = '';
|
||||
if (opts.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
srcClause = ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const rows = await this.sql.unsafe(
|
||||
`UPDATE content_chunks cc
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.embedding IS NOT NULL
|
||||
AND p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1${srcClause}
|
||||
RETURNING cc.page_id`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return (rows as unknown[]).length;
|
||||
}
|
||||
|
||||
async listStaleChunks(opts?: {
|
||||
|
||||
@@ -33,6 +33,10 @@ export interface SourceMetrics {
|
||||
failed_jobs_24h: number;
|
||||
/** Waiting + active + delayed jobs (sync OR embed-backfill) for this source. */
|
||||
queue_depth: number;
|
||||
/** v0.41.31: embed-backfill jobs specifically active right now. */
|
||||
backfill_active: number;
|
||||
/** v0.41.31: embed-backfill jobs queued (waiting/delayed/waiting-children). */
|
||||
backfill_queued: number;
|
||||
tracked_branch: string | null;
|
||||
priority_label: PriorityLabel;
|
||||
/** Webhook configured? (true iff config.webhook_secret is set.) */
|
||||
@@ -131,7 +135,7 @@ export async function computeAllSourceMetrics(
|
||||
const cfg = parseSourceConfig(src.config);
|
||||
const pages = pageCounts.get(src.id) ?? 0;
|
||||
const chunkStats = chunkCounts.get(src.id) ?? { total: 0, embedded: 0 };
|
||||
const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0 };
|
||||
const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0, backfill_active: 0, backfill_queued: 0 };
|
||||
|
||||
const embedCoverage = chunkStats.total === 0
|
||||
? 100
|
||||
@@ -155,6 +159,8 @@ export async function computeAllSourceMetrics(
|
||||
lag_seconds: lagSeconds,
|
||||
failed_jobs_24h: jobStats.failed_24h,
|
||||
queue_depth: jobStats.queue_depth,
|
||||
backfill_active: jobStats.backfill_active,
|
||||
backfill_queued: jobStats.backfill_queued,
|
||||
tracked_branch: typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : null,
|
||||
priority_label: resolvePriorityLabel(src.id, src.config),
|
||||
webhook_configured: typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0,
|
||||
@@ -189,21 +195,30 @@ async function chunkCountsBySource(engine: BrainEngine): Promise<Map<string, { t
|
||||
return m;
|
||||
}
|
||||
|
||||
async function jobCountsBySource(engine: BrainEngine): Promise<Map<string, { failed_24h: number; queue_depth: number }>> {
|
||||
type JobStats = { failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number };
|
||||
|
||||
async function jobCountsBySource(engine: BrainEngine): Promise<Map<string, JobStats>> {
|
||||
// Pre-v0.11 brains don't have minion_jobs; return empty map.
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number }>(
|
||||
const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number }>(
|
||||
`SELECT data->>'sourceId' AS source_id,
|
||||
COUNT(*) FILTER (WHERE status IN ('failed','dead') AND created_at > NOW() - INTERVAL '24 hours')::int AS failed_24h,
|
||||
COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth
|
||||
COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth,
|
||||
COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status = 'active')::int AS backfill_active,
|
||||
COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status IN ('waiting','delayed','waiting-children'))::int AS backfill_queued
|
||||
FROM minion_jobs
|
||||
WHERE name IN ('sync','embed-backfill')
|
||||
AND data->>'sourceId' IS NOT NULL
|
||||
GROUP BY data->>'sourceId'`,
|
||||
);
|
||||
const m = new Map<string, { failed_24h: number; queue_depth: number }>();
|
||||
const m = new Map<string, JobStats>();
|
||||
for (const r of rows) {
|
||||
m.set(r.source_id, { failed_24h: Number(r.failed_24h), queue_depth: Number(r.queue_depth) });
|
||||
m.set(r.source_id, {
|
||||
failed_24h: Number(r.failed_24h),
|
||||
queue_depth: Number(r.queue_depth),
|
||||
backfill_active: Number(r.backfill_active),
|
||||
backfill_queued: Number(r.backfill_queued),
|
||||
});
|
||||
}
|
||||
return m;
|
||||
} catch {
|
||||
|
||||
@@ -19,6 +19,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from '../src/core/search/embedding-column.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import type { PageInput, ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -26,6 +27,17 @@ let chunkId: number;
|
||||
let chunkId2: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Pin the embedding dim to 1536 BEFORE initSchema. initSchema sizes the
|
||||
// `embedding` column from getEmbeddingDimensions() (default 1280 =
|
||||
// zeroentropyai). This test hardcodes 1536-dim vectors + asserts 1536, so
|
||||
// it must NOT inherit ambient/leaked gateway state (which is 1536 from a
|
||||
// local ~/.gbrain config but 1280 in CI → vector(1280) → insert fails).
|
||||
// Pinning here makes the column deterministically 1536 regardless of order.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test-cosine-rescore' },
|
||||
});
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
@@ -87,6 +99,7 @@ beforeAll(async () => {
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('getEmbeddingsByChunkIds — column parameter (D9)', () => {
|
||||
|
||||
@@ -25,6 +25,8 @@ mock.module('../../src/core/embedding.ts', () => ({
|
||||
// Deterministic fake vector for each chunk.
|
||||
return texts.map(() => new Float32Array(1536));
|
||||
},
|
||||
// v0.41.31: embed phase reads the current signature to stamp provenance.
|
||||
currentEmbeddingSignature: () => 'test:model:1536',
|
||||
}));
|
||||
|
||||
const { runCycle } = await import('../../src/core/cycle.ts');
|
||||
|
||||
@@ -46,6 +46,8 @@ mock.module('../../src/core/embedding.ts', () => ({
|
||||
EMBEDDING_DIMENSIONS: 1536,
|
||||
EMBEDDING_COST_PER_1K_TOKENS: 0.00013,
|
||||
estimateEmbeddingCostUsd: (tokens: number) => (tokens / 1000) * 0.00013,
|
||||
// v0.41.31: embed phase reads the current signature to stamp provenance.
|
||||
currentEmbeddingSignature: () => 'text-embedding-3-large:1536',
|
||||
}));
|
||||
|
||||
const { runCycle, ALL_PHASES } = await import('../../src/core/cycle.ts');
|
||||
@@ -106,6 +108,9 @@ async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
|
||||
// v0.31 — added `consolidate` between recompute_emotional_weight and embed
|
||||
// v0.33 — added `resolve_symbol_edges` between extract and patterns
|
||||
type CyclePhase = (typeof ALL_PHASES)[number];
|
||||
// Mirrors src/core/cycle.ts ALL_PHASES order exactly. v0.41.31: synced the
|
||||
// three phases that drifted in after this test was last touched (v0.41.0.0):
|
||||
// extract_atoms (v0.41 T9), synthesize_concepts, conversation_facts_backfill.
|
||||
const EXPECTED_PHASES: CyclePhase[] = [
|
||||
'lint',
|
||||
'backlinks',
|
||||
@@ -113,13 +118,16 @@ const EXPECTED_PHASES: CyclePhase[] = [
|
||||
'synthesize',
|
||||
'extract',
|
||||
'extract_facts', // v0.32.2 — reconcile fence → DB facts index
|
||||
'extract_atoms', // v0.41 T9 — pack-gated atom extraction
|
||||
'resolve_symbol_edges', // v0.33.3 — within-file symbol resolution
|
||||
'patterns',
|
||||
'synthesize_concepts', // v0.41 — concept synthesis after patterns
|
||||
'recompute_emotional_weight', // v0.29
|
||||
'consolidate', // v0.31
|
||||
'propose_takes', // v0.36.1.0 — hindsight calibration wave
|
||||
'grade_takes', // v0.36.1.0
|
||||
'calibration_profile', // v0.36.1.0
|
||||
'conversation_facts_backfill', // v0.41.11.0 — config-gated (default off)
|
||||
'embed',
|
||||
'orphans',
|
||||
'schema-suggest', // v0.39.0.0 — passive schema-suggest after orphans
|
||||
|
||||
@@ -19,6 +19,8 @@ import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.
|
||||
// Mock embedBatch so embed phase doesn't call OpenAI.
|
||||
mock.module('../../src/core/embedding.ts', () => ({
|
||||
embedBatch: async (texts: string[]) => texts.map(() => new Float32Array(1536)),
|
||||
// v0.41.31: embed phase reads the current signature to stamp provenance.
|
||||
currentEmbeddingSignature: () => 'test:model:1536',
|
||||
}));
|
||||
|
||||
const { runDream } = await import('../../src/commands/dream.ts');
|
||||
|
||||
@@ -197,6 +197,43 @@ describe('buildSyncStatusReport against real PGLite (IRON RULE regression for Bl
|
||||
expect(b.embedding_coverage_pct).toBe(100);
|
||||
});
|
||||
|
||||
test('v0.41.31: embed-backfill job state surfaced per source (TODO-2)', async () => {
|
||||
// Seed embed-backfill minion jobs for source-a: 2 queued + 1 active +
|
||||
// 1 completed. source-b has none → idle.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data) VALUES
|
||||
('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb),
|
||||
('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb),
|
||||
('embed-backfill', 'active', '{"sourceId":"source-a"}'::jsonb)`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data, finished_at)
|
||||
VALUES ('embed-backfill', 'completed', '{"sourceId":"source-a"}'::jsonb, now())`,
|
||||
);
|
||||
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string; name: string; local_path: string | null; config: Record<string, unknown>;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources
|
||||
WHERE local_path IS NOT NULL AND archived IS NOT TRUE ORDER BY id`,
|
||||
);
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
const byId = new Map(report.sources.map((s) => [s.source_id, s]));
|
||||
|
||||
const a = byId.get('source-a')!;
|
||||
expect(a.backfill_queued).toBe(2);
|
||||
expect(a.backfill_active).toBe(1);
|
||||
expect(a.backfill_last_completed_at).not.toBeNull();
|
||||
|
||||
const b = byId.get('source-b')!;
|
||||
expect(b.backfill_queued).toBe(0);
|
||||
expect(b.backfill_active).toBe(0);
|
||||
expect(b.backfill_last_completed_at).toBeNull();
|
||||
|
||||
// Clean up so sibling tests in this describe see a clean minion_jobs.
|
||||
await engine.executeRaw(`DELETE FROM minion_jobs WHERE name = 'embed-backfill'`);
|
||||
});
|
||||
|
||||
test('soft-deleted pages excluded from pages count (v0.26.5 regression)', async () => {
|
||||
// Verifies the `WHERE pg.deleted_at IS NULL` clause in BOTH subqueries
|
||||
// of the dashboard SQL. Pre-fix the original PR query would have
|
||||
|
||||
@@ -32,6 +32,11 @@ mock.module('../src/core/embedding.ts', () => ({
|
||||
activeEmbedCalls--;
|
||||
}
|
||||
},
|
||||
// v0.41.31: embedAll/embedAllStale read the current embedding signature to
|
||||
// stamp provenance. The mock returns a stable value; the mock engine's
|
||||
// setPageEmbeddingSignature / invalidateStaleSignatureEmbeddings resolve to
|
||||
// null via the Proxy default, so the signature value is inert here.
|
||||
currentEmbeddingSignature: () => 'test:model:1536',
|
||||
}));
|
||||
|
||||
// Import AFTER mocking.
|
||||
@@ -105,6 +110,29 @@ describe('runEmbed --all (parallel)', () => {
|
||||
expect(maxConcurrentEmbedCalls).toBeLessThanOrEqual(10);
|
||||
});
|
||||
|
||||
test('v0.41.31: stamps embedding_signature after embedding each page (--all)', async () => {
|
||||
const pages = [{ slug: 'a', source_id: 'default' }, { slug: 'b', source_id: 'default' }];
|
||||
const chunksBySlug = new Map(
|
||||
pages.map(p => [
|
||||
p.slug,
|
||||
[{ chunk_index: 0, chunk_text: `text ${p.slug}`, chunk_source: 'compiled_truth', embedded_at: null, token_count: 4 }],
|
||||
]),
|
||||
);
|
||||
const engine = mockEngine({
|
||||
listPages: async () => pages,
|
||||
getChunks: async (slug: string) => chunksBySlug.get(slug) || [],
|
||||
upsertChunks: async () => {},
|
||||
});
|
||||
|
||||
await runEmbed(engine, ['--all']);
|
||||
|
||||
// The wiring gap this pins: embedAll must CALL setPageEmbeddingSignature
|
||||
// after upsertChunks, with the current signature (mocked to test:model:1536).
|
||||
const stampCalls = (engine as any)._calls.filter((c: any) => c.method === 'setPageEmbeddingSignature');
|
||||
expect(stampCalls.length).toBe(2); // one per page
|
||||
expect(stampCalls[0].args[1]).toEqual({ sourceId: 'default', signature: 'test:model:1536' });
|
||||
});
|
||||
|
||||
test('respects GBRAIN_EMBED_CONCURRENCY=1 (serial)', async () => {
|
||||
const pages = Array.from({ length: 5 }, (_, i) => ({ slug: `page-${i}` }));
|
||||
const chunksBySlug = new Map(
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* v0.41.31 — real stale semantics via pages.embedding_signature (PGLite).
|
||||
*
|
||||
* Pins the commit-3 contract:
|
||||
* - R-4 (grandfather, CRITICAL): a page embedded under a NULL signature is
|
||||
* NEVER stale. After the v108 migration every existing page has NULL, so
|
||||
* the next embed --stale must NOT re-embed the whole corpus.
|
||||
* - signature mismatch (model/dims swap) → counted as stale.
|
||||
* - matching signature → not stale.
|
||||
* - invalidateStaleSignatureEmbeddings NULLs only mismatched (grandfathered
|
||||
* NULL + matching untouched) and returns the count.
|
||||
* - setPageEmbeddingSignature stamps.
|
||||
*
|
||||
* Canonical PGLite block (CLAUDE.md R3+R4).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let colDim: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`,
|
||||
);
|
||||
colDim = Number(rows[0]?.dim);
|
||||
});
|
||||
|
||||
/**
|
||||
* Seed a page with one EMBEDDED chunk (non-null vector) and a given
|
||||
* embedding_signature (null → grandfathered legacy state).
|
||||
*/
|
||||
async function seedEmbedded(slug: string, text: string, signature: string | null, sourceId?: string): Promise<void> {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }, sourceId ? { sourceId } : undefined);
|
||||
const chunks: ChunkInput[] = [
|
||||
{ chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined },
|
||||
];
|
||||
await engine.upsertChunks(slug, chunks, sourceId ? { sourceId } : undefined);
|
||||
// Flip the chunk to a non-null vector sized to the actual column dim.
|
||||
await engine.executeRaw(
|
||||
`UPDATE content_chunks
|
||||
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector
|
||||
WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = $3)`,
|
||||
[colDim, slug, sourceId ?? 'default'],
|
||||
);
|
||||
if (signature !== null) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId, signature });
|
||||
}
|
||||
}
|
||||
|
||||
describe('embedding_signature stale semantics', () => {
|
||||
test('R-4 GRANDFATHER: NULL signature is never stale', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null); // embedded, NULL signature
|
||||
// No NULL embeddings, NULL signature → not stale under any signature.
|
||||
expect(await engine.countStaleChunks({ signature: 'openai:m:1536' })).toBe(0);
|
||||
expect(await engine.sumStaleChunkChars({ signature: 'openai:m:1536' })).toBe(0);
|
||||
});
|
||||
|
||||
test('signature MISMATCH (model swap) is counted as stale', async () => {
|
||||
await seedEmbedded('drifted', 'abcde', 'openai:old:1536'); // 5 chars
|
||||
expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(1);
|
||||
expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(5);
|
||||
// Without the signature opt, the legacy NULL-only predicate ignores it.
|
||||
expect(await engine.countStaleChunks()).toBe(0);
|
||||
});
|
||||
|
||||
test('MATCHING signature is not stale', async () => {
|
||||
await seedEmbedded('fresh', 'abcde', 'voyage:new:1024');
|
||||
expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(0);
|
||||
expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(0);
|
||||
});
|
||||
|
||||
test('invalidateStaleSignatureEmbeddings NULLs only mismatched; grandfathered + matching untouched', async () => {
|
||||
await seedEmbedded('old', 'abcde', 'openai:old:1536'); // mismatched → invalidate
|
||||
await seedEmbedded('legacy', 'fghij', null); // grandfathered → keep
|
||||
await seedEmbedded('new', 'klmno', 'voyage:new:1024'); // matching → keep
|
||||
|
||||
const invalidated = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' });
|
||||
expect(invalidated).toBe(1); // only 'old'
|
||||
|
||||
// Now exactly the 'old' page's chunk is NULL → legacy stale count = 1.
|
||||
expect(await engine.countStaleChunks()).toBe(1);
|
||||
// Re-running is idempotent (nothing left to invalidate).
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' })).toBe(0);
|
||||
});
|
||||
|
||||
test('invalidate is sourceId-scoped', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{}'::jsonb) ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
await seedEmbedded('a', 'abcde', 'openai:old:1536'); // default
|
||||
await seedEmbedded('b', 'fghij', 'openai:old:1536', 'other'); // other
|
||||
const n = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024', sourceId: 'default' });
|
||||
expect(n).toBe(1);
|
||||
expect(await engine.countStaleChunks({ sourceId: 'default' })).toBe(1);
|
||||
expect(await engine.countStaleChunks({ sourceId: 'other' })).toBe(0); // untouched
|
||||
});
|
||||
|
||||
test('setPageEmbeddingSignature stamps the page', async () => {
|
||||
await engine.putPage('p', { type: 'note', title: 'p', compiled_truth: '# p' });
|
||||
await engine.setPageEmbeddingSignature('p', { signature: 'openai:m:1536' });
|
||||
const rows = await engine.executeRaw<{ embedding_signature: string | null }>(
|
||||
`SELECT embedding_signature FROM pages WHERE slug = 'p' AND source_id = 'default'`,
|
||||
);
|
||||
expect(rows[0]?.embedding_signature).toBe('openai:m:1536');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* v0.41.31 (F1) — importFromContent stamps pages.embedding_signature when it
|
||||
* embeds inline.
|
||||
*
|
||||
* The inline import/sync path (importFromContent) writes embeddings without
|
||||
* going through embed.ts/embed-stale.ts. Before the F1 fix it never stamped
|
||||
* the signature, so non-federated/inline brains kept NULL signatures forever
|
||||
* → grandfathered → `embed --stale` never re-embedded them after a model swap
|
||||
* (the headline feature silently inert). This pins the stamp.
|
||||
*
|
||||
* Serial: stubs the gateway embed transport (process-global module state).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { importFromContent } from '../src/core/import-file.ts';
|
||||
import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';
|
||||
import { currentEmbeddingSignature } from '../src/core/embedding.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let colDim: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`,
|
||||
);
|
||||
colDim = Number(rows[0]?.dim);
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: colDim,
|
||||
env: { OPENAI_API_KEY: 'sk-test-import-stamp' },
|
||||
});
|
||||
// Fake transport (AI SDK embedMany shape): receives { values }, returns
|
||||
// one zero-vector (sized to the column) per input value.
|
||||
__setEmbedTransportForTests(async ({ values }: { values: string[] }) => ({
|
||||
embeddings: values.map(() => Array(colDim).fill(0)),
|
||||
usage: { tokens: 0 },
|
||||
}) as any);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
async function signatureOf(slug: string): Promise<string | null> {
|
||||
const rows = await engine.executeRaw<{ embedding_signature: string | null }>(
|
||||
`SELECT embedding_signature FROM pages WHERE slug = $1 AND source_id = 'default'`,
|
||||
[slug],
|
||||
);
|
||||
return rows[0]?.embedding_signature ?? null;
|
||||
}
|
||||
|
||||
describe('importFromContent embedding_signature stamping (F1)', () => {
|
||||
test('embeds inline → stamps the current signature', async () => {
|
||||
await importFromContent(engine, 'concepts/stamped', '# Stamped\n\nsome body content to chunk and embed.', {});
|
||||
expect(await signatureOf('concepts/stamped')).toBe(currentEmbeddingSignature());
|
||||
});
|
||||
|
||||
test('--no-embed → leaves signature NULL (grandfathered, not stale)', async () => {
|
||||
await importFromContent(engine, 'concepts/unstamped', '# Unstamped\n\nbody content.', { noEmbed: true });
|
||||
expect(await signatureOf('concepts/unstamped')).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -158,6 +158,10 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [
|
||||
// pages_generation_idx (CREATE INDEX ON pages (generation)) so bootstrap
|
||||
// probes guard pre-v91 brains.
|
||||
{ kind: 'column', table: 'pages', column: 'generation' },
|
||||
// v0.41.31 (v108) — pages.embedding_signature TEXT for real stale
|
||||
// semantics. No SCHEMA_SQL index references it; bootstrap probe is
|
||||
// defense-in-depth (and satisfies the MIGRATIONS ADD COLUMN coverage gate).
|
||||
{ kind: 'column', table: 'pages', column: 'embedding_signature' },
|
||||
];
|
||||
|
||||
test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => {
|
||||
|
||||
@@ -167,6 +167,24 @@ describe('computeAllSourceMetrics', () => {
|
||||
expect(result.find((m) => m.source_id === 'other')!.total_pages).toBe(1);
|
||||
});
|
||||
|
||||
test('v0.41.31: embed-backfill active/queued counts per source (CLI BACKFILL column)', async () => {
|
||||
// 2 queued + 1 active embed-backfill for default; a non-backfill 'sync'
|
||||
// job must NOT inflate the backfill counts (only the generic queue_depth).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data) VALUES
|
||||
('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb),
|
||||
('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb),
|
||||
('embed-backfill', 'active', '{"sourceId":"default"}'::jsonb),
|
||||
('sync', 'waiting', '{"sourceId":"default"}'::jsonb)`,
|
||||
);
|
||||
const sources = await loadAllSources(engine);
|
||||
const dflt = (await computeAllSourceMetrics(engine, sources)).find((m) => m.source_id === 'default')!;
|
||||
expect(dflt.backfill_active).toBe(1);
|
||||
expect(dflt.backfill_queued).toBe(2);
|
||||
// generic queue_depth includes the sync job too (4 total waiting/active).
|
||||
expect(dflt.queue_depth).toBe(4);
|
||||
});
|
||||
|
||||
test('webhook_configured reflects config.webhook_secret presence', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('webhooky', 'webhooky', '{"federated":true,"webhook_secret":"x","github_repo":"a/b"}'::jsonb)`,
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
/**
|
||||
* v0.41.31 — BrainEngine.sumStaleChunkChars tests (PGLite).
|
||||
*
|
||||
* Sibling of countStaleChunks: sums LENGTH(chunk_text) over stale chunks so
|
||||
* the `gbrain sync --all` cost preview can price the embedding backlog.
|
||||
* Validates: empty brain → 0, exact char sum, sourceId scope, embed_skip
|
||||
* exclusion, and that non-null (already-embedded) chunks are NOT counted.
|
||||
*
|
||||
* Uses the canonical PGLite block (one engine per file, resetPgliteState in
|
||||
* beforeEach) per CLAUDE.md test-isolation rules R3+R4.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
/** Seed a page + stale (NULL-embedding) chunks with explicit chunk_text. */
|
||||
async function seedStale(
|
||||
slug: string,
|
||||
texts: string[],
|
||||
opts?: { sourceId?: string; frontmatter?: Record<string, unknown> },
|
||||
): Promise<void> {
|
||||
await engine.putPage(
|
||||
slug,
|
||||
{
|
||||
type: 'note',
|
||||
title: slug,
|
||||
compiled_truth: `# ${slug}`,
|
||||
...(opts?.frontmatter ? { frontmatter: opts.frontmatter } : {}),
|
||||
},
|
||||
opts?.sourceId ? { sourceId: opts.sourceId } : undefined,
|
||||
);
|
||||
const chunks: ChunkInput[] = texts.map((t, i) => ({
|
||||
chunk_index: i,
|
||||
chunk_text: t,
|
||||
chunk_source: 'compiled_truth',
|
||||
token_count: 4,
|
||||
embedding: undefined, // NULL = stale
|
||||
}));
|
||||
await engine.upsertChunks(slug, chunks, opts?.sourceId ? { sourceId: opts.sourceId } : undefined);
|
||||
}
|
||||
|
||||
describe('sumStaleChunkChars', () => {
|
||||
test('empty brain returns 0', async () => {
|
||||
expect(await engine.sumStaleChunkChars()).toBe(0);
|
||||
});
|
||||
|
||||
test('sums LENGTH(chunk_text) across stale chunks', async () => {
|
||||
await seedStale('a', ['abcde', 'fghij']); // 5 + 5
|
||||
await seedStale('b', ['xyz']); // 3
|
||||
expect(await engine.sumStaleChunkChars()).toBe(13);
|
||||
});
|
||||
|
||||
test('scopes to sourceId when provided', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}'::jsonb) ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
await seedStale('a', ['abcde']); // default, 5
|
||||
await seedStale('b', ['xyzwv'], { sourceId: 'other' }); // other, 5
|
||||
expect(await engine.sumStaleChunkChars({ sourceId: 'default' })).toBe(5);
|
||||
expect(await engine.sumStaleChunkChars({ sourceId: 'other' })).toBe(5);
|
||||
expect(await engine.sumStaleChunkChars()).toBe(10); // all sources
|
||||
});
|
||||
|
||||
test('excludes pages with embed_skip frontmatter', async () => {
|
||||
await seedStale('keep', ['abcde']); // 5, counted
|
||||
await seedStale('skip', ['1234567890'], {
|
||||
frontmatter: { embed_skip: { at: '2026-01-01T00:00:00Z', reason: 'oversize' } },
|
||||
});
|
||||
expect(await engine.sumStaleChunkChars()).toBe(5);
|
||||
});
|
||||
|
||||
test('does NOT count already-embedded (non-null) chunks', async () => {
|
||||
// Stale page → counted.
|
||||
await seedStale('stale', ['abcde']); // 5
|
||||
// "Embedded" page: seed as stale, then flip its chunk's embedding to a
|
||||
// non-null vector via raw SQL. We bypass upsertChunks so the test doesn't
|
||||
// depend on gateway-configured dimensions (keeps it robust regardless of
|
||||
// any leaked gateway state from sibling files).
|
||||
await seedStale('done', ['this is embedded and should not count']);
|
||||
// Size the embedding to the ACTUAL column dimension (pgvector stores it
|
||||
// in atttypmod) so the test is robust to whatever dim initSchema chose.
|
||||
const dimRows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`,
|
||||
);
|
||||
const dim = Number(dimRows[0]?.dim);
|
||||
expect(dim).toBeGreaterThan(0);
|
||||
await engine.executeRaw(
|
||||
`UPDATE content_chunks
|
||||
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector
|
||||
WHERE page_id = (SELECT id FROM pages WHERE slug = 'done' AND source_id = 'default')`,
|
||||
[dim],
|
||||
);
|
||||
expect(await engine.sumStaleChunkChars()).toBe(5);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,160 @@
|
||||
/**
|
||||
* v0.41.31 — `gbrain sync --all` cost-gate wiring regressions (PGLite).
|
||||
*
|
||||
* Pure shouldBlockSync / willEmbedSynchronously logic is pinned in
|
||||
* test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in
|
||||
* runSync's --all path:
|
||||
*
|
||||
* R-1 (headline): deferred-embed sync --all, non-TTY, with backlog →
|
||||
* emits gate:'deferred_notice' and NEVER exit 2 (the cron-blocking
|
||||
* bug this release fixes).
|
||||
* R-2 (protection): inline-embed sync --all (--serial), non-TTY, above
|
||||
* floor → still exit 2 with gate:'confirmation_required'.
|
||||
*
|
||||
* Serial-quarantined: stubs process.exit + console.log (process-global).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runSources } from '../src/commands/sources.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
let headSha: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
// Configure the gateway with a dummy key so the pre-gate embedding-creds
|
||||
// preflight passes (diagnoseEmbedding reads gateway configure-time state,
|
||||
// not live env). The gate runs before any real embed call, so no network
|
||||
// request is made.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test-costgate' },
|
||||
});
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-costgate-'));
|
||||
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.com"', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repoPath, stdio: 'pipe' });
|
||||
mkdirSync(join(repoPath, 'topics'), { recursive: true });
|
||||
writeFileSync(
|
||||
join(repoPath, 'topics/foo.md'),
|
||||
['---', 'type: concept', 'title: Foo', '---', '', 'some body content to estimate.'].join('\n'),
|
||||
);
|
||||
execSync('git add -A && git commit -m initial', { cwd: repoPath, stdio: 'pipe' });
|
||||
headSha = execSync('git rev-parse HEAD', { cwd: repoPath, stdio: 'pipe' }).toString().trim();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
resetGateway();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
/** Run runSync(args) with process.exit + console.log captured. */
|
||||
async function runSyncCaptured(args: string[]): Promise<{ exitCode: number | undefined; stdout: string }> {
|
||||
const { runSync } = await import('../src/commands/sync.ts');
|
||||
const origExit = process.exit;
|
||||
const origLog = console.log.bind(console);
|
||||
const out: string[] = [];
|
||||
let exitCode: number | undefined;
|
||||
console.log = (...a: unknown[]) => {
|
||||
out.push(a.map((x) => (typeof x === 'string' ? x : JSON.stringify(x))).join(' '));
|
||||
};
|
||||
process.exit = ((code?: number) => {
|
||||
exitCode = code;
|
||||
throw new Error('__exit__');
|
||||
}) as typeof process.exit;
|
||||
try {
|
||||
await runSync(engine, args);
|
||||
} catch (e) {
|
||||
if ((e as Error).message !== '__exit__') throw e;
|
||||
} finally {
|
||||
process.exit = origExit;
|
||||
console.log = origLog;
|
||||
}
|
||||
return { exitCode, stdout: out.join('\n') };
|
||||
}
|
||||
|
||||
describe('v0.41.31 — sync --all cost gate wiring', () => {
|
||||
test('R-1: deferred sync --all (non-TTY) emits deferred_notice and never exit 2', async () => {
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
// Make the fan-out a clean no-op: last_commit == HEAD so performSync
|
||||
// reports up_to_date (no git pull, no backfill submit).
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1 WHERE id = 'vault'`, [headSha]);
|
||||
// Seed a stale backlog so the deferred notice has a non-zero figure.
|
||||
await engine.putPage('vault/note', { type: 'note', title: 'note', compiled_truth: '# note' }, { sourceId: 'vault' });
|
||||
const chunks: ChunkInput[] = [
|
||||
{ chunk_index: 0, chunk_text: 'x'.repeat(500), chunk_source: 'compiled_truth', token_count: 4, embedding: undefined },
|
||||
];
|
||||
await engine.upsertChunks('vault/note', chunks, { sourceId: 'vault' });
|
||||
|
||||
// v2 default ON → deferred. --json → agent path. NO --yes. --no-pull
|
||||
// because the synthetic repo has no remote.
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--json', '--no-pull']);
|
||||
|
||||
// The headline assertion: NOT blocked.
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).toContain('"gate":"deferred_notice"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-2: inline sync --all (--serial) above floor still exit 2', async () => {
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
// Floor 0 → any nonzero inline cost blocks. Source is unsynced
|
||||
// (last_commit NULL) so estimateInlineNewTokens sees it as changed →
|
||||
// full-tree tokens > 0 → costUsd > 0 > floor.
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
// --serial forces inline even with v2 on. --json → non-TTY exit-2 path.
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stdout).toContain('"gate":"confirmation_required"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-3: inline, git-unchanged source but STALE chunker_version still estimates (not $0)', async () => {
|
||||
// The unchanged-source short-circuit requires HEAD==last_commit AND clean
|
||||
// tree AND chunker_version == current. Here git is unchanged but the
|
||||
// chunker drifted, so the source must NOT be treated as 0 — sync would
|
||||
// re-chunk + re-embed everything. floor=0 so any nonzero cost blocks.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, 'STALE-0']);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stdout).toContain('"gate":"confirmation_required"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-3 control: inline, git-unchanged + CURRENT chunker_version short-circuits to $0 (no exit 2)', async () => {
|
||||
// Same setup but chunker_version matches current → the source IS unchanged
|
||||
// → contributes 0 new-content tokens → below floor → proceeds (no block).
|
||||
// Proves the short-circuit fires when (and only when) everything matches.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).not.toContain('"gate":"confirmation_required"');
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -14,23 +14,54 @@
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { EMBEDDING_COST_PER_1K_TOKENS, estimateEmbeddingCostUsd } from '../src/core/embedding.ts';
|
||||
import {
|
||||
EMBEDDING_COST_PER_1K_TOKENS,
|
||||
estimateEmbeddingCostUsd,
|
||||
willEmbedSynchronously,
|
||||
shouldBlockSync,
|
||||
} from '../src/core/embedding.ts';
|
||||
import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts';
|
||||
import { estimateTokens } from '../src/core/chunkers/code.ts';
|
||||
|
||||
describe('Layer 8 D1 — embedding cost model', () => {
|
||||
test('EMBEDDING_COST_PER_1K_TOKENS is text-embedding-3-large pricing', () => {
|
||||
// Update this when OpenAI changes text-embedding-3-large pricing.
|
||||
// As of 2026-04-24: $0.00013 / 1k tokens.
|
||||
test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => {
|
||||
// Retained only for back-compat imports. Live cost math now resolves the
|
||||
// CONFIGURED model's rate via embedding-pricing.ts (see model-aware test
|
||||
// below). As of 2026-04-24: $0.00013 / 1k tokens.
|
||||
expect(EMBEDDING_COST_PER_1K_TOKENS).toBe(0.00013);
|
||||
});
|
||||
|
||||
test('estimateEmbeddingCostUsd scales linearly with tokens', () => {
|
||||
test('estimateEmbeddingCostUsd scales linearly (gateway-unconfigured fallback = OpenAI rate)', () => {
|
||||
// With no gateway configured (unit-test context) the estimator falls back
|
||||
// to the OpenAI text-embedding-3-large rate ($0.13/Mtok = $0.00013/1k).
|
||||
expect(estimateEmbeddingCostUsd(0)).toBe(0);
|
||||
expect(estimateEmbeddingCostUsd(1000)).toBeCloseTo(0.00013, 5);
|
||||
expect(estimateEmbeddingCostUsd(10_000)).toBeCloseTo(0.0013, 4);
|
||||
expect(estimateEmbeddingCostUsd(1_000_000)).toBeCloseTo(0.13, 4);
|
||||
});
|
||||
|
||||
test('cost preview uses the CONFIGURED model rate, not a hardcoded OpenAI rate', () => {
|
||||
// Regression: the cost gate previously hardcoded $0.00013/1k (OpenAI
|
||||
// text-embedding-3-large) regardless of the configured embedding model,
|
||||
// so a brain on a cheaper model (e.g. zeroentropyai:zembed-1 @ $0.05/Mtok)
|
||||
// saw a preview that named the wrong provider and over-stated spend ~2.6x.
|
||||
// The pricing table is the single source of truth per provider:model.
|
||||
const TOKENS = 2_590_710_262; // a real large-brain sync preview
|
||||
const openai = lookupEmbeddingPrice('openai:text-embedding-3-large');
|
||||
const zeroentropy = lookupEmbeddingPrice('zeroentropyai:zembed-1');
|
||||
expect(openai.kind).toBe('known');
|
||||
expect(zeroentropy.kind).toBe('known');
|
||||
if (openai.kind === 'known' && zeroentropy.kind === 'known') {
|
||||
const openaiCost = (TOKENS / 1_000_000) * openai.pricePerMTok;
|
||||
const zeCost = (TOKENS / 1_000_000) * zeroentropy.pricePerMTok;
|
||||
// The two models must produce materially different previews; a fix that
|
||||
// collapses both to the OpenAI number would regress this assertion.
|
||||
expect(openaiCost).toBeCloseTo(336.79, 1);
|
||||
expect(zeCost).toBeCloseTo(129.54, 1);
|
||||
expect(zeCost).toBeLessThan(openaiCost);
|
||||
}
|
||||
});
|
||||
|
||||
test('5K-file TS repo sanity check: ~$5 at ~400k tokens', () => {
|
||||
// A 5K-file TS repo at ~80 tokens/file averages ~400k tokens. Cost:
|
||||
// 400_000 / 1000 * 0.00013 = $0.052 ≈ $0.05. Not $5. The CHANGELOG
|
||||
@@ -43,6 +74,50 @@ describe('Layer 8 D1 — embedding cost model', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.41.31 — willEmbedSynchronously (embed-mode resolver)', () => {
|
||||
// Mirrors sync.ts:2346 effectiveNoEmbed = v2 && !serial && !noEmbed ? true : noEmbed.
|
||||
// Embed runs INLINE iff that resolves to false.
|
||||
test('v2 off → inline (legacy synchronous embed)', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: false })).toBe('inline');
|
||||
});
|
||||
test('v2 on + parallel → deferred (backfill jobs)', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: false, noEmbed: false })).toBe('deferred');
|
||||
});
|
||||
test('v2 on + --serial → inline', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: false })).toBe('inline');
|
||||
});
|
||||
test('--no-embed forces deferred regardless of v2/serial', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: true })).toBe('deferred');
|
||||
expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: true })).toBe('deferred');
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.41.31 — shouldBlockSync (cost-gate decision)', () => {
|
||||
// R-1: deferred NEVER blocks, even at absurd cost (the headline fix — a
|
||||
// nightly cron over a synced corpus must not exit 2).
|
||||
test('R-1: deferred never blocks, even at $999', () => {
|
||||
expect(shouldBlockSync(999, 0.5, 'deferred')).toBe(false);
|
||||
expect(shouldBlockSync(0, 0.5, 'deferred')).toBe(false);
|
||||
});
|
||||
// R-2: inline still blocks above the floor (protection preserved where
|
||||
// sync actually spends synchronously).
|
||||
test('R-2: inline blocks above floor', () => {
|
||||
expect(shouldBlockSync(0.51, 0.5, 'inline')).toBe(true);
|
||||
expect(shouldBlockSync(130, 0.5, 'inline')).toBe(true);
|
||||
});
|
||||
test('inline at exactly the floor does NOT block (boundary)', () => {
|
||||
expect(shouldBlockSync(0.5, 0.5, 'inline')).toBe(false);
|
||||
});
|
||||
test('inline below floor does not block (kills cents-level cron noise)', () => {
|
||||
expect(shouldBlockSync(0.03, 0.5, 'inline')).toBe(false);
|
||||
expect(shouldBlockSync(0, 0.5, 'inline')).toBe(false);
|
||||
});
|
||||
test('floor of 0 makes inline block on any nonzero cost', () => {
|
||||
expect(shouldBlockSync(0.0001, 0, 'inline')).toBe(true);
|
||||
expect(shouldBlockSync(0, 0, 'inline')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Layer 8 D1 — estimateTokens (exported from chunkers/code.ts)', () => {
|
||||
test('empty string is 0 tokens', () => {
|
||||
expect(estimateTokens('')).toBe(0);
|
||||
|
||||
Reference in New Issue
Block a user