mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
60
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
92656a221b | ||
|
|
6ec762cbcd | ||
|
|
4a81c017a0 | ||
|
|
0612b0daa8 | ||
|
|
f529eaa231 | ||
|
|
447e57ec41 | ||
|
|
d61808d806 | ||
|
|
60125ee626 | ||
|
|
e320ad71b3 | ||
|
|
b6dd3e1121 | ||
|
|
3cc34c92ee | ||
|
|
a93fcf504f | ||
|
|
84fad4738d | ||
|
|
f815246eef | ||
|
|
42c4ea929f | ||
|
|
6370ce3d7e | ||
|
|
c21d7b253a | ||
|
|
4df7796061 | ||
|
|
6db4cea2e4 | ||
|
|
11eebc3605 | ||
|
|
ee45653a02 | ||
|
|
706d3cea3d | ||
|
|
2934c53c1d | ||
|
|
b60656245f | ||
|
|
d698b44438 | ||
|
|
1d0b5ed816 | ||
|
|
4c71a76c0a | ||
|
|
354c8c36a9 | ||
|
|
dbf2b3f562 | ||
|
|
9ed53e4e1c | ||
|
|
9f7244a77f | ||
|
|
6ec3dd410e | ||
|
|
bcf3b73dcf | ||
|
|
23e0541d9b | ||
|
|
c873ce3014 | ||
|
|
f3e78fd2fb | ||
|
|
4528bfa79c | ||
|
|
6498b872ea | ||
|
|
324c355318 | ||
|
|
184b6cb8a1 | ||
|
|
912407bef1 | ||
|
|
89f226eb38 | ||
|
|
f1031d5a0b | ||
|
|
3a5c4c194c | ||
|
|
d165e99f0b | ||
|
|
8b325041ee | ||
|
|
a46f28a63e | ||
|
|
f72de97943 | ||
|
|
f8d11f67a3 | ||
|
|
93cfb37540 | ||
|
|
9fe4628d02 | ||
|
|
1833d95896 | ||
|
|
42375bded5 | ||
|
|
a8e6b1d177 | ||
|
|
54a8070640 | ||
|
|
e1cefd0654 | ||
|
|
a0ef951586 | ||
|
|
bd2ba46a61 | ||
|
|
2df41a84c9 | ||
|
|
da1bab532a |
@@ -2,6 +2,36 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.64.0] - 2026-07-20
|
||||
|
||||
### Fixed
|
||||
|
||||
- Confidential OAuth clients can now revoke access tokens through the standard revocation endpoint when client secrets are stored as hashes. Invalid credentials fail closed, malformed or mixed authentication is rejected, backend failures remain retryable, and discovery metadata accurately advertises supported authentication methods.
|
||||
|
||||
No schema migrations.
|
||||
## [0.42.63.0] - 2026-07-20
|
||||
|
||||
**Schema commands now open the local brain you actually configured.**
|
||||
|
||||
If your PGLite brain lives at a custom path, commands such as `gbrain schema stats` previously ignored that path and could inspect the default brain instead. That made a healthy configured brain look empty or report the wrong schema counts. Schema commands now use the same complete database configuration as the rest of GBrain. PostgreSQL behavior is unchanged, and no migration is required.
|
||||
|
||||
### How to use it
|
||||
|
||||
Upgrade, then run the schema command normally:
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain schema stats --json
|
||||
```
|
||||
|
||||
The reported page and type counts now come from the `database_path` in `~/.gbrain/config.json` when the engine is PGLite.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Fixed
|
||||
- **Schema CLI commands preserve configured PGLite paths.** Engine construction and connection now receive the canonical complete engine configuration, including both `database_path` and `database_url` where applicable.
|
||||
- **CLI tests are isolated from ambient database URLs.** Schema subprocess tests explicitly clear inherited PostgreSQL URL variables, and a persistent-PGLite regression test proves `schema stats` reads the configured database rather than the default brain.
|
||||
|
||||
## [0.42.62.0] - 2026-07-17
|
||||
|
||||
**If your brain holds more than one source, everything now lands in the right one. Link extraction, timeline extraction, background cycles, and webhook captures used to quietly file some of their output under the default source; all of those paths now carry the correct source identity. Background agent jobs got tougher too: a failed database reconnect can no longer wedge the engine, and workers recover from dropped connections instead of crash-looping. If you run the admin dashboard behind a reverse proxy, the live activity panel finally connects. Long agent conversations cost less because repeated context is reused between turns on Anthropic calls. Local LiteLLM proxies work out of the box. Nested sources scan correctly again instead of reporting zero files. And the project's automated checks now include dependency vulnerability scanning, static code-security analysis, and signed provenance for release builds. Thirty merged changes in all, the largest batch to date, each one reviewed and verified against the live codebase before landing.**
|
||||
|
||||
@@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
|
||||
```
|
||||
|
||||
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
Vendored
+56
File diff suppressed because one or more lines are too long
Vendored
-56
File diff suppressed because one or more lines are too long
Vendored
+1
-1
@@ -7,7 +7,7 @@
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
|
||||
<script type="module" crossorigin src="/admin/assets/index-CoGEje3-.js"></script>
|
||||
<script type="module" crossorigin src="/admin/assets/index-BpDk4NI4.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/admin/assets/index-GxkWX7v3.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
+6
-2
@@ -5,13 +5,14 @@ import { AgentsPage } from './pages/Agents';
|
||||
import { RequestLogPage } from './pages/RequestLog';
|
||||
import { CalibrationPage } from './pages/Calibration';
|
||||
import { JobsWatchPage } from './pages/JobsWatch';
|
||||
import { SourcesPage } from './pages/Sources';
|
||||
import { api } from './api';
|
||||
|
||||
type Page = 'login' | 'dashboard' | 'agents' | 'log' | 'calibration' | 'jobs';
|
||||
type Page = 'login' | 'dashboard' | 'agents' | 'sources' | 'log' | 'calibration' | 'jobs';
|
||||
|
||||
function getPage(): Page {
|
||||
const hash = window.location.hash.replace('#', '') || 'dashboard';
|
||||
if (['login', 'dashboard', 'agents', 'log', 'calibration', 'jobs'].includes(hash)) return hash as Page;
|
||||
if (['login', 'dashboard', 'agents', 'sources', 'log', 'calibration', 'jobs'].includes(hash)) return hash as Page;
|
||||
return 'dashboard';
|
||||
}
|
||||
|
||||
@@ -54,6 +55,8 @@ export function App() {
|
||||
onClick={() => navigate('dashboard')}>Dashboard</a>
|
||||
<a className={`nav-item ${page === 'agents' ? 'active' : ''}`}
|
||||
onClick={() => navigate('agents')}>Agents</a>
|
||||
<a className={`nav-item ${page === 'sources' ? 'active' : ''}`}
|
||||
onClick={() => navigate('sources')}>Sources</a>
|
||||
<a className={`nav-item ${page === 'log' ? 'active' : ''}`}
|
||||
onClick={() => navigate('log')}>Request Log</a>
|
||||
<a className={`nav-item ${page === 'calibration' ? 'active' : ''}`}
|
||||
@@ -83,6 +86,7 @@ export function App() {
|
||||
<main className="main">
|
||||
{page === 'dashboard' && <DashboardPage />}
|
||||
{page === 'agents' && <AgentsPage />}
|
||||
{page === 'sources' && <SourcesPage />}
|
||||
{page === 'log' && <RequestLogPage />}
|
||||
{page === 'calibration' && <CalibrationPage />}
|
||||
{page === 'jobs' && <JobsWatchPage />}
|
||||
|
||||
@@ -52,4 +52,22 @@ export const api = {
|
||||
apiFetchText(`/admin/api/calibration/charts/${encodeURIComponent(type)}${holder ? `?holder=${encodeURIComponent(holder)}` : ''}`),
|
||||
// v0.41 D2 — live minion-jobs dashboard snapshot.
|
||||
jobsWatch: () => apiFetch('/admin/api/jobs/watch'),
|
||||
// v0.41.29 Sources tab + federated-read management
|
||||
sources: () => apiFetch('/admin/api/sources'),
|
||||
agentsFederatedRead: () => apiFetch('/admin/api/agents/federated-read'),
|
||||
grantRead: (clientId: string, sourceId: string) =>
|
||||
apiFetch(`/admin/api/agents/${encodeURIComponent(clientId)}/grant-read`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ source_id: sourceId }),
|
||||
}),
|
||||
revokeRead: (clientId: string, sourceId: string) =>
|
||||
apiFetch(`/admin/api/agents/${encodeURIComponent(clientId)}/revoke-read`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ source_id: sourceId }),
|
||||
}),
|
||||
setFederatedRead: (clientId: string, sourceIds: string[]) =>
|
||||
apiFetch(`/admin/api/agents/${encodeURIComponent(clientId)}/set-federated-read`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ source_ids: sourceIds }),
|
||||
}),
|
||||
};
|
||||
|
||||
@@ -381,8 +381,16 @@ function CredentialsModal({ credentials, onClose }: {
|
||||
);
|
||||
}
|
||||
|
||||
interface FederationState {
|
||||
source_id: string | null;
|
||||
federated_read: string[];
|
||||
}
|
||||
|
||||
function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: () => void; onRevoked: () => void }) {
|
||||
const [tab, setTab] = useState<'claude-code' | 'chatgpt' | 'claude-cowork' | 'perplexity' | 'cursor' | 'json'>('claude-code');
|
||||
const [federation, setFederation] = useState<FederationState | null>(null);
|
||||
const [allSources, setAllSources] = useState<string[]>([]);
|
||||
const [showFederation, setShowFederation] = useState(false);
|
||||
const copy = (text: string) => navigator.clipboard.writeText(text);
|
||||
const serverUrl = window.location.origin;
|
||||
|
||||
@@ -390,6 +398,30 @@ function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: ()
|
||||
const isOAuth = agent.auth_type === 'oauth';
|
||||
const agentName = agent.name || agent.client_name || 'unknown';
|
||||
|
||||
// Lazy-load federation state when the drawer opens for an OAuth client.
|
||||
// The /admin/api/agents endpoint doesn't carry source_id / federated_read,
|
||||
// so we fetch /admin/api/agents/federated-read separately and pair by id.
|
||||
useEffect(() => {
|
||||
if (!isOAuth || !cid) return;
|
||||
let cancelled = false;
|
||||
Promise.all([
|
||||
api.agentsFederatedRead().catch(() => ({ clients: [] })),
|
||||
api.sources().catch(() => ({ sources: [] })),
|
||||
]).then(([feds, srcs]: any) => {
|
||||
if (cancelled) return;
|
||||
const me = (feds.clients || []).find((c: any) => c.client_id === cid);
|
||||
setFederation(me ? { source_id: me.source_id, federated_read: me.federated_read || [] } : null);
|
||||
setAllSources((srcs.sources || []).map((s: any) => s.source_id));
|
||||
});
|
||||
return () => { cancelled = true; };
|
||||
}, [cid, isOAuth]);
|
||||
|
||||
const reloadFederation = async () => {
|
||||
const feds: any = await api.agentsFederatedRead().catch(() => ({ clients: [] }));
|
||||
const me = (feds.clients || []).find((c: any) => c.client_id === cid);
|
||||
setFederation(me ? { source_id: me.source_id, federated_read: me.federated_read || [] } : null);
|
||||
};
|
||||
|
||||
// For API keys, we can't show the actual token (it was shown once at creation).
|
||||
// For OAuth, we show the client_id and tell them to use their secret.
|
||||
|
||||
@@ -553,6 +585,34 @@ function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: ()
|
||||
<span>{agent.token_ttl ? (agent.token_ttl >= 31536000 ? 'No expiry' : agent.token_ttl >= 86400 ? `${Math.floor(agent.token_ttl / 86400)}d` : agent.token_ttl >= 3600 ? `${Math.floor(agent.token_ttl / 3600)}h` : `${agent.token_ttl}s`) : '1h (default)'}</span>
|
||||
</div>
|
||||
|
||||
{isOAuth && federation && (
|
||||
<>
|
||||
<div className="section-title" style={{ display: 'flex', alignItems: 'center', justifyContent: 'space-between' }}>
|
||||
<span>Federation</span>
|
||||
<button
|
||||
className="btn btn-secondary"
|
||||
style={{ padding: '4px 10px', fontSize: 12 }}
|
||||
onClick={() => setShowFederation(true)}
|
||||
>
|
||||
Manage reads
|
||||
</button>
|
||||
</div>
|
||||
<div style={{ display: 'grid', gridTemplateColumns: '120px 1fr', gap: '6px 12px', fontSize: 13 }}>
|
||||
<span style={{ color: 'var(--text-secondary)' }}>Write source</span>
|
||||
<span className="mono">{federation.source_id || '(none)'}</span>
|
||||
<span style={{ color: 'var(--text-secondary)' }}>Federated reads</span>
|
||||
<span style={{ fontSize: 12 }}>
|
||||
{federation.federated_read.length === 0
|
||||
? <span style={{ color: 'var(--text-muted)' }}>(empty — no federated reads)</span>
|
||||
: federation.federated_read.map((s) => (
|
||||
<span key={s} className="badge badge-read" style={{ marginRight: 4, marginBottom: 2 }}>{s}</span>
|
||||
))
|
||||
}
|
||||
</span>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
|
||||
{/*
|
||||
Config Export visible for both auth_type=oauth AND auth_type=api_key.
|
||||
Claude Code + Cursor + JSON tabs render real snippets regardless
|
||||
@@ -628,6 +688,142 @@ function AgentDrawer({ agent, onClose, onRevoked }: { agent: Agent; onClose: ()
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{showFederation && federation && (
|
||||
<FederationModal
|
||||
clientId={cid}
|
||||
clientName={agentName}
|
||||
allSources={allSources}
|
||||
currentReads={federation.federated_read}
|
||||
writeSource={federation.source_id}
|
||||
onClose={() => setShowFederation(false)}
|
||||
onSaved={async () => {
|
||||
await reloadFederation();
|
||||
setShowFederation(false);
|
||||
}}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* FederationModal — admin counterpart of `gbrain auth set-federated-read`.
|
||||
* Source checkbox list; "Save" submits the full new list via the
|
||||
* race-safe atomic SQL path in setFederatedReadCore. Per the CLI's
|
||||
* documented contract, this is wholesale-replace semantics — concurrent
|
||||
* grant/revoke from a CLI operator would be last-writer-wins against
|
||||
* a Save here.
|
||||
*/
|
||||
function FederationModal({
|
||||
clientId, clientName, allSources, currentReads, writeSource, onClose, onSaved,
|
||||
}: {
|
||||
clientId: string;
|
||||
clientName: string;
|
||||
allSources: string[];
|
||||
currentReads: string[];
|
||||
writeSource: string | null;
|
||||
onClose: () => void;
|
||||
onSaved: () => Promise<void> | void;
|
||||
}) {
|
||||
const [selected, setSelected] = useState<Set<string>>(new Set(currentReads));
|
||||
const [saving, setSaving] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
const toggle = (id: string) => {
|
||||
const next = new Set(selected);
|
||||
if (next.has(id)) next.delete(id); else next.add(id);
|
||||
setSelected(next);
|
||||
};
|
||||
|
||||
const handleSave = async () => {
|
||||
setSaving(true);
|
||||
setError(null);
|
||||
try {
|
||||
await api.setFederatedRead(clientId, Array.from(selected));
|
||||
await onSaved();
|
||||
} catch (e: any) {
|
||||
setError(e.message || 'save failed');
|
||||
setSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
// Union: all known sources + any current reads not in the source list
|
||||
// (e.g. orphan entries from before the source was deleted). The latter
|
||||
// surface as "(missing source)" so operators can revoke them.
|
||||
const allKnown = new Set([...allSources, ...currentReads]);
|
||||
const ordered = Array.from(allKnown).sort();
|
||||
|
||||
return (
|
||||
<div className="modal-overlay" onClick={onClose}>
|
||||
<div className="modal" onClick={(e) => e.stopPropagation()} style={{ maxWidth: 520 }}>
|
||||
<div className="modal-header">
|
||||
<div style={{ fontSize: 16, fontWeight: 600 }}>Manage federated reads</div>
|
||||
<div style={{ fontSize: 13, color: 'var(--text-secondary)', marginTop: 4 }}>
|
||||
<strong>{clientName}</strong> — pick which sources this client can read in addition to its
|
||||
{writeSource ? <> write source <code className="mono">{writeSource}</code></> : <> write source</>}.
|
||||
</div>
|
||||
</div>
|
||||
<div className="modal-body" style={{ maxHeight: '50vh', overflowY: 'auto' }}>
|
||||
{ordered.length === 0 && (
|
||||
<div style={{ color: 'var(--text-muted)', fontSize: 13 }}>
|
||||
No sources registered. Use <code>gbrain sources add <id> --path <dir></code> from the CLI first.
|
||||
</div>
|
||||
)}
|
||||
{ordered.map((id) => {
|
||||
const isOrphan = !allSources.includes(id);
|
||||
const isWriteSource = id === writeSource;
|
||||
return (
|
||||
<label
|
||||
key={id}
|
||||
style={{
|
||||
display: 'flex',
|
||||
alignItems: 'center',
|
||||
gap: 10,
|
||||
padding: '8px 10px',
|
||||
borderBottom: '1px solid var(--border)',
|
||||
cursor: 'pointer',
|
||||
fontSize: 13,
|
||||
}}
|
||||
>
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={selected.has(id)}
|
||||
onChange={() => toggle(id)}
|
||||
style={{ width: 16, height: 16, margin: 0, flexShrink: 0, cursor: 'pointer' }}
|
||||
/>
|
||||
<span className="mono" style={{ flex: 1, minWidth: 0, overflow: 'hidden', textOverflow: 'ellipsis', whiteSpace: 'nowrap' }}>{id}</span>
|
||||
{isWriteSource && <span className="badge badge-write" style={{ fontSize: 10, flexShrink: 0 }}>write source</span>}
|
||||
{isOrphan && <span className="badge badge-danger" style={{ fontSize: 10, flexShrink: 0 }}>missing source</span>}
|
||||
</label>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
{error && (
|
||||
<div style={{
|
||||
background: 'rgba(239,68,68,0.08)',
|
||||
border: '1px solid rgba(239,68,68,0.3)',
|
||||
color: '#ef4444',
|
||||
padding: '10px 12px',
|
||||
borderRadius: 6,
|
||||
margin: '12px 0',
|
||||
fontSize: 12,
|
||||
}}>
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
<div className="modal-footer">
|
||||
<button type="button" className="btn btn-secondary" onClick={onClose} disabled={saving}>Cancel</button>
|
||||
<button
|
||||
type="button"
|
||||
className="btn btn-primary"
|
||||
onClick={handleSave}
|
||||
disabled={saving}
|
||||
>
|
||||
{saving ? 'Saving…' : `Save (${selected.size} source${selected.size === 1 ? '' : 's'})`}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
import React, { useState, useEffect } from 'react';
|
||||
import { api } from '../api';
|
||||
|
||||
interface SourceRow {
|
||||
source_id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
sync_enabled: boolean;
|
||||
last_sync_at: string | null;
|
||||
staleness_hours: number | null;
|
||||
staleness_class: 'fresh' | 'stale' | 'severe' | 'unknown';
|
||||
last_commit: string | null;
|
||||
pages: number;
|
||||
chunks_total: number;
|
||||
chunks_unembedded: number;
|
||||
embedding_coverage_pct: number;
|
||||
}
|
||||
|
||||
interface FederatedClient {
|
||||
client_id: string;
|
||||
client_name: string;
|
||||
source_id: string | null;
|
||||
federated_read: string[];
|
||||
}
|
||||
|
||||
function timeAgo(iso: string | null): string {
|
||||
if (!iso) return 'never';
|
||||
const s = Math.floor((Date.now() - new Date(iso).getTime()) / 1000);
|
||||
if (s < 0) return 'in the future?';
|
||||
if (s < 60) return 'just now';
|
||||
if (s < 3600) return `${Math.floor(s / 60)}m ago`;
|
||||
if (s < 86400) return `${Math.floor(s / 3600)}h ago`;
|
||||
return `${Math.floor(s / 86400)}d ago`;
|
||||
}
|
||||
|
||||
function stalenessColor(cls: string): string {
|
||||
switch (cls) {
|
||||
case 'fresh': return '#4ade80';
|
||||
case 'stale': return '#fbbf24';
|
||||
case 'severe': return '#ef4444';
|
||||
default: return 'var(--text-muted)';
|
||||
}
|
||||
}
|
||||
|
||||
function coverageColor(pct: number): string {
|
||||
if (pct >= 99) return '#4ade80';
|
||||
if (pct >= 90) return '#fbbf24';
|
||||
return '#ef4444';
|
||||
}
|
||||
|
||||
export function SourcesPage() {
|
||||
const [sources, setSources] = useState<SourceRow[]>([]);
|
||||
const [clients, setClients] = useState<FederatedClient[]>([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
const load = async () => {
|
||||
setLoading(true);
|
||||
setError(null);
|
||||
try {
|
||||
const [srcReport, clientsResp] = await Promise.all([
|
||||
api.sources(),
|
||||
api.agentsFederatedRead(),
|
||||
]);
|
||||
setSources(srcReport.sources || []);
|
||||
setClients(clientsResp.clients || []);
|
||||
} catch (e: any) {
|
||||
setError(e.message || 'load failed');
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
};
|
||||
|
||||
useEffect(() => { load(); }, []);
|
||||
|
||||
// Reverse-lookup: for each source, which clients can read it?
|
||||
const readersBySource = (sourceId: string): string[] =>
|
||||
clients.filter((c) => c.federated_read.includes(sourceId)).map((c) => c.client_name);
|
||||
|
||||
// Reverse-lookup: which clients WRITE to this source (source_id == sourceId)?
|
||||
const writersBySource = (sourceId: string): string[] =>
|
||||
clients.filter((c) => c.source_id === sourceId).map((c) => c.client_name);
|
||||
|
||||
return (
|
||||
<div style={{ padding: 24, maxWidth: 1200 }}>
|
||||
<div style={{ display: 'flex', alignItems: 'center', justifyContent: 'space-between', marginBottom: 24 }}>
|
||||
<h1 style={{ fontSize: 24, margin: 0 }}>Sources</h1>
|
||||
<button
|
||||
onClick={load}
|
||||
style={{
|
||||
background: 'transparent',
|
||||
border: '1px solid var(--border)',
|
||||
color: 'var(--text-secondary)',
|
||||
padding: '6px 12px',
|
||||
borderRadius: 6,
|
||||
fontSize: 12,
|
||||
cursor: 'pointer',
|
||||
}}
|
||||
>
|
||||
Refresh
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{loading && <div style={{ color: 'var(--text-muted)' }}>Loading…</div>}
|
||||
{error && (
|
||||
<div style={{
|
||||
background: 'rgba(239,68,68,0.08)',
|
||||
border: '1px solid rgba(239,68,68,0.3)',
|
||||
color: '#ef4444',
|
||||
padding: 12,
|
||||
borderRadius: 6,
|
||||
marginBottom: 16,
|
||||
fontSize: 13,
|
||||
}}>
|
||||
Failed to load sources: {error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!loading && !error && sources.length === 0 && (
|
||||
<div style={{ color: 'var(--text-muted)', padding: 16 }}>
|
||||
No active sources with a local_path. Use{' '}
|
||||
<code style={{ background: 'var(--bg-elevated)', padding: '2px 6px', borderRadius: 4 }}>
|
||||
gbrain sources add <id> --path <dir>
|
||||
</code>{' '}
|
||||
to register one.
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!loading && !error && sources.length > 0 && (
|
||||
<div style={{ overflowX: 'auto' }}>
|
||||
<table style={{ width: '100%', borderCollapse: 'collapse', fontSize: 13 }}>
|
||||
<thead>
|
||||
<tr style={{ borderBottom: '1px solid var(--border)', textAlign: 'left', color: 'var(--text-muted)' }}>
|
||||
<th style={{ padding: '10px 12px' }}>ID</th>
|
||||
<th style={{ padding: '10px 12px', textAlign: 'right' }}>Pages</th>
|
||||
<th style={{ padding: '10px 12px', textAlign: 'right' }}>Chunks</th>
|
||||
<th style={{ padding: '10px 12px', textAlign: 'right' }}>Embed%</th>
|
||||
<th style={{ padding: '10px 12px' }}>Last Sync</th>
|
||||
<th style={{ padding: '10px 12px' }}>Writers</th>
|
||||
<th style={{ padding: '10px 12px' }}>Readers (federated)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{sources.map((s) => {
|
||||
const readers = readersBySource(s.source_id);
|
||||
const writers = writersBySource(s.source_id);
|
||||
return (
|
||||
<tr key={s.source_id} style={{ borderBottom: '1px solid var(--border)' }}>
|
||||
<td style={{ padding: '10px 12px', fontFamily: 'JetBrains Mono, monospace' }}>
|
||||
<div>{s.source_id}</div>
|
||||
{s.name !== s.source_id && (
|
||||
<div style={{ fontSize: 11, color: 'var(--text-muted)', fontFamily: 'inherit' }}>{s.name}</div>
|
||||
)}
|
||||
</td>
|
||||
<td style={{ padding: '10px 12px', textAlign: 'right', fontFamily: 'JetBrains Mono, monospace' }}>{s.pages.toLocaleString()}</td>
|
||||
<td style={{ padding: '10px 12px', textAlign: 'right', fontFamily: 'JetBrains Mono, monospace' }}>{s.chunks_total.toLocaleString()}</td>
|
||||
<td style={{ padding: '10px 12px', textAlign: 'right', color: coverageColor(s.embedding_coverage_pct), fontFamily: 'JetBrains Mono, monospace' }}>
|
||||
{s.embedding_coverage_pct.toFixed(0)}%
|
||||
</td>
|
||||
<td style={{ padding: '10px 12px', color: s.local_path == null ? 'var(--text-muted)' : stalenessColor(s.staleness_class) }}>
|
||||
{s.local_path == null ? 'push-only' : timeAgo(s.last_sync_at)}
|
||||
</td>
|
||||
<td style={{ padding: '10px 12px', fontSize: 12, color: 'var(--text-secondary)' }}>
|
||||
{writers.length === 0 ? <span style={{ color: 'var(--text-muted)' }}>none</span> : writers.join(', ')}
|
||||
</td>
|
||||
<td style={{ padding: '10px 12px', fontSize: 12, color: 'var(--text-secondary)' }}>
|
||||
{readers.length === 0 ? <span style={{ color: 'var(--text-muted)' }}>none</span> : readers.join(', ')}
|
||||
</td>
|
||||
</tr>
|
||||
);
|
||||
})}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div style={{
|
||||
marginTop: 24,
|
||||
padding: 12,
|
||||
background: 'var(--bg-elevated)',
|
||||
border: '1px solid var(--border)',
|
||||
borderRadius: 6,
|
||||
fontSize: 12,
|
||||
color: 'var(--text-muted)',
|
||||
lineHeight: 1.6,
|
||||
}}>
|
||||
<strong style={{ color: 'var(--text-secondary)' }}>Two scopes per OAuth client:</strong>{' '}
|
||||
<em>Writers</em> = clients with this source as their <code>source_id</code> (write authority).{' '}
|
||||
<em>Readers</em> = clients with this source in their <code>federated_read</code> list (read access via federation).
|
||||
Manage federation per-client from the <a href="#agents" style={{ color: '#60a5fa' }}>Agents</a> tab using the
|
||||
"Manage reads" action.
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
+6
-1
@@ -13,4 +13,9 @@ timeout = 60_000
|
||||
# fixtures still match the schema. v0.37's production default is ZE/1280;
|
||||
# tests that want the new default call configureGateway() explicitly in
|
||||
# their own beforeAll.
|
||||
preload = ["./test/helpers/legacy-embedding-preload.ts"]
|
||||
#
|
||||
# #2823: redirect GBRAIN_AUDIT_DIR to a per-run scratch dir BEFORE any test
|
||||
# runs, so audit-emitting code paths (content-sanity, shell-audit, etc.)
|
||||
# can't leak fixture events into the operator's real ~/.gbrain/audit/. See
|
||||
# test/helpers/audit-dir-preload.ts for the full rationale.
|
||||
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts"]
|
||||
|
||||
@@ -148,6 +148,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
Defense-in-depth layer for Postgres deployments that want the database itself
|
||||
to enforce source isolation, in addition to the mandatory app-layer filters
|
||||
(`sourceScopeOpts` — layer 1, always on).
|
||||
|
||||
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
|
||||
source-scoped read methods wrap their queries in a transaction that first runs
|
||||
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
|
||||
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
|
||||
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
|
||||
bound params). An RLS policy can then filter rows by
|
||||
`current_setting('app.scopes', true)`.
|
||||
|
||||
**Default off.** With the env var unset, reads call through on the shared pool
|
||||
exactly as before — no per-read transaction, no pool-slot hold (the search
|
||||
methods keep the transaction they always had for their `SET LOCAL
|
||||
statement_timeout`). Existing operators see zero behavior change.
|
||||
|
||||
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
|
||||
|
||||
```sql
|
||||
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY pages_scope_filter ON pages
|
||||
USING (current_setting('app.scopes', true) = '*'
|
||||
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
|
||||
|
||||
-- Required: connections that don't run through the scoped read helper
|
||||
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
|
||||
-- see zero rows once the policy exists:
|
||||
ALTER ROLE <runtime-role> SET app.scopes = '*';
|
||||
|
||||
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
|
||||
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
|
||||
```
|
||||
|
||||
Safe to enable in either order: the env var without a policy is a no-op
|
||||
setting; a policy without the env var is enforced only via the role default.
|
||||
|
||||
**Honest caveat:** only read paths routed through the scoped helper carry a
|
||||
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
|
||||
run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
|
||||
only.
|
||||
|
||||
`test/e2e/serve-http-oauth.test.ts` additionally pins confidential POST/Basic revocation, public-client SDK fallthrough, malformed/mixed authentication rejection, cross-client isolation, unknown-token opacity, metadata auth methods, no-store responses, strict post-revoke `401`, and retryable backend `503` semantics.
|
||||
|
||||
### Test command tiers
|
||||
|
||||
Seven test command tiers, each with a clear scope:
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -131,6 +131,16 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
history rewrite still hard-blocks even with `--skip-failed`. Run
|
||||
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
|
||||
|
||||
5. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
Interrupted `gbrain import <dir>` runs may leave
|
||||
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
|
||||
checkpoint `dir` is the absolute, resolved import target captured when
|
||||
import starts. It is not a cleanup instruction and it must not be
|
||||
re-derived from the process working directory. Checkpoints written by
|
||||
gbrain include `schema_version: 1`, `owner: "gbrain"`, and
|
||||
`kind: "import"` so downstream tools can validate the contract before
|
||||
deciding whether to resume.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. **Edit a file and search for the change.** Edit a brain markdown file,
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
# Multi-language full-text search
|
||||
|
||||
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
|
||||
The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE`
|
||||
environment variable. Default: `english`.
|
||||
|
||||
## How it works
|
||||
|
||||
Postgres text-search configurations control stemming and stop-word removal.
|
||||
`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on
|
||||
both sides of the search:
|
||||
|
||||
- **Query side** — `websearch_to_tsquery('<lang>', $query)` in both engines
|
||||
(Postgres and PGLite).
|
||||
- **Write side** — the `update_page_search_vector` and
|
||||
`update_chunk_search_vector` trigger functions that populate
|
||||
`pages.search_vector` and `content_chunks.search_vector`.
|
||||
|
||||
The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever
|
||||
interpolated into SQL (tsvector functions don't accept parameterized config
|
||||
names). Invalid values fall back to `english` with a warning.
|
||||
|
||||
## Built-in languages
|
||||
|
||||
Set the env var to any configuration your Postgres instance ships:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
export GBRAIN_FTS_LANGUAGE=spanish
|
||||
export GBRAIN_FTS_LANGUAGE=german
|
||||
```
|
||||
|
||||
List what's available:
|
||||
|
||||
```sql
|
||||
SELECT cfgname FROM pg_ts_config;
|
||||
```
|
||||
|
||||
PGLite (the embedded default engine) ships the same built-in snowball
|
||||
configurations as stock Postgres.
|
||||
|
||||
## First install vs. changing language later
|
||||
|
||||
On first install (or upgrade), the `configurable_fts_language` schema
|
||||
migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with
|
||||
that language. After the migration has run, changing the env var alone does
|
||||
NOT retokenize existing rows — the migration shows as applied and is skipped.
|
||||
Use the explicit command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview: language + row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command recreates both trigger functions under the new language and
|
||||
backfills every existing `pages` and `content_chunks` row in batches,
|
||||
streaming progress to stderr. It is idempotent: re-running with the same
|
||||
language produces identical vectors. `--json` prints a machine-readable
|
||||
result envelope but still requires `--yes` (or an interactive confirm).
|
||||
|
||||
## Recipe: accent-insensitive Portuguese (`pt_br`)
|
||||
|
||||
Brazilian Portuguese content often mixes accented and unaccented spellings
|
||||
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
|
||||
the `unaccent` extension, then stems with the portuguese snowball dictionary:
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS unaccent;
|
||||
|
||||
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
|
||||
|
||||
ALTER TEXT SEARCH CONFIGURATION pt_br
|
||||
ALTER MAPPING FOR hword, hword_part, word
|
||||
WITH unaccent, portuguese_stem;
|
||||
```
|
||||
|
||||
Then point GBrain at it:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br
|
||||
gbrain reindex-search-vector --yes
|
||||
```
|
||||
|
||||
Note: custom configurations require a real Postgres instance (e.g. the
|
||||
Supabase engine). The config must exist BEFORE the migration or the reindex
|
||||
command runs, or Postgres will reject the trigger recreation with
|
||||
`text search configuration "pt_br" does not exist`.
|
||||
|
||||
## Caveats
|
||||
|
||||
- One language per brain: the setting is global to the database, not
|
||||
per-source. Mixed-language brains should pick the dominant language (the
|
||||
vector-search arm is language-agnostic and covers the rest).
|
||||
- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that
|
||||
writes to the brain (CLI shells, MCP server, cron jobs) — a writer without
|
||||
the env var tokenizes new rows in `english` until the next reindex.
|
||||
@@ -114,8 +114,11 @@ Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
|
||||
Full subcommand reference:
|
||||
|
||||
```
|
||||
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated]
|
||||
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
|
||||
Register a source. id: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
|
||||
--path must be a git repo (or a subdirectory of one) — see
|
||||
"The git requirement for --path sources" below. --force
|
||||
skips that check to register before git-init exists.
|
||||
gbrain sources list [--json] List all sources with page counts + federation state.
|
||||
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
|
||||
Cascade-delete a source (pages, chunks, timeline).
|
||||
@@ -128,6 +131,47 @@ gbrain sources federate <id>
|
||||
gbrain sources unfederate <id>
|
||||
```
|
||||
|
||||
## The git requirement for --path sources
|
||||
|
||||
Every `--path` source must be a git repository (or live inside one — a
|
||||
subdirectory of a git repo works too) with at least one committed, tracked
|
||||
file under that path. `gbrain sources add` validates this at registration
|
||||
time and refuses a directory that doesn't qualify — no `.git` at all, a
|
||||
`git init` with no commit yet, or a commit made before `git add` — with an
|
||||
actionable error instead of silently registering a source that will fail
|
||||
(or worse, "succeed" while importing nothing) on its first `gbrain sync`.
|
||||
Fix it with:
|
||||
|
||||
```bash
|
||||
git -C <path> init
|
||||
git -C <path> add -A
|
||||
git -C <path> commit -m "initial import"
|
||||
gbrain sources add <id> --path <path>
|
||||
```
|
||||
|
||||
Two details that are easy to miss:
|
||||
|
||||
- **Files must actually be committed, not just present.** The sync walker
|
||||
reads files through git objects, so `git init` alone — even followed by an
|
||||
empty commit (`git commit --allow-empty`) — isn't enough. Registration
|
||||
checks for real tracked content (`git ls-tree HEAD` scoped to the path),
|
||||
not just a resolvable `HEAD`, so this footgun is caught immediately
|
||||
instead of surfacing later as a sync that imports nothing.
|
||||
- **`--force` registers the source anyway**, skipping the check. Use this if
|
||||
you're registering a path before an automated pipeline gets around to
|
||||
`git init`-ing it. GBrain never auto-`git init`s a `--path` source for
|
||||
you — it's your directory, not a gbrain-managed clone (same consent
|
||||
boundary as sync-time self-heal, which also never mutates a `--path`
|
||||
source without an explicit ask).
|
||||
|
||||
**If sync ever reports a problem with the sync anchor** (`last_commit`) —
|
||||
after a force-push, a history rewrite, or a from-scratch `git init` on a
|
||||
directory that was synced before — you do not need to reset anything by
|
||||
hand. `gbrain sync` detects an unreachable or non-ancestor anchor
|
||||
automatically and recovers: either a full reimport (anchor object missing)
|
||||
or a direct tree-to-tree diff against the orphaned bookmark (anchor present
|
||||
but rewritten), advancing the anchor to the new HEAD when it completes.
|
||||
|
||||
## Citation format for agents
|
||||
|
||||
When agents receive multi-source results they MUST cite pages in
|
||||
|
||||
@@ -1752,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
|
||||
```
|
||||
|
||||
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
@@ -2095,6 +2113,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
Defense-in-depth layer for Postgres deployments that want the database itself
|
||||
to enforce source isolation, in addition to the mandatory app-layer filters
|
||||
(`sourceScopeOpts` — layer 1, always on).
|
||||
|
||||
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
|
||||
source-scoped read methods wrap their queries in a transaction that first runs
|
||||
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
|
||||
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
|
||||
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
|
||||
bound params). An RLS policy can then filter rows by
|
||||
`current_setting('app.scopes', true)`.
|
||||
|
||||
**Default off.** With the env var unset, reads call through on the shared pool
|
||||
exactly as before — no per-read transaction, no pool-slot hold (the search
|
||||
methods keep the transaction they always had for their `SET LOCAL
|
||||
statement_timeout`). Existing operators see zero behavior change.
|
||||
|
||||
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
|
||||
|
||||
```sql
|
||||
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY pages_scope_filter ON pages
|
||||
USING (current_setting('app.scopes', true) = '*'
|
||||
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
|
||||
|
||||
-- Required: connections that don't run through the scoped read helper
|
||||
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
|
||||
-- see zero rows once the policy exists:
|
||||
ALTER ROLE <runtime-role> SET app.scopes = '*';
|
||||
|
||||
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
|
||||
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
|
||||
```
|
||||
|
||||
Safe to enable in either order: the env var without a policy is a no-op
|
||||
setting; a policy without the env var is enforced only via the role default.
|
||||
|
||||
**Honest caveat:** only read paths routed through the scoped helper carry a
|
||||
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
|
||||
run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
@@ -2767,6 +2830,16 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
history rewrite still hard-blocks even with `--skip-failed`. Run
|
||||
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
|
||||
|
||||
5. **Import checkpoints name the import target, not the caller's CWD.**
|
||||
Interrupted `gbrain import <dir>` runs may leave
|
||||
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
|
||||
checkpoint `dir` is the absolute, resolved import target captured when
|
||||
import starts. It is not a cleanup instruction and it must not be
|
||||
re-derived from the process working directory. Checkpoints written by
|
||||
gbrain include `schema_version: 1`, `owner: "gbrain"`, and
|
||||
`kind: "import"` so downstream tools can validate the contract before
|
||||
deciding whether to resume.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. **Edit a file and search for the change.** Edit a brain markdown file,
|
||||
|
||||
+1
-1
@@ -144,7 +144,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.62.0",
|
||||
"version": "0.42.64.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^1.19.13",
|
||||
"fast-uri": "^3.1.2",
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
// AUTO-GENERATED — do not edit by hand.
|
||||
// Run `bun run scripts/build-admin-embedded.ts` to regenerate.
|
||||
// Source: admin/dist/ at 2026-05-27.
|
||||
// Source: admin/dist/ at 2026-07-22.
|
||||
//
|
||||
// Bun resolves the file: imports to a path that works at runtime even
|
||||
// inside a compiled binary (`bun build --compile`). The manifest maps
|
||||
// the request path the express handler sees to (resolved-path, mime).
|
||||
|
||||
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
|
||||
import A_0_assets_index_CoGEje3__js from '../admin/dist/assets/index-CoGEje3-.js' with { type: 'file' };
|
||||
import A_0_assets_index_BpDk4NI4_js from '../admin/dist/assets/index-BpDk4NI4.js' with { type: 'file' };
|
||||
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
|
||||
import A_1_assets_index_GxkWX7v3_css from '../admin/dist/assets/index-GxkWX7v3.css' with { type: 'file' };
|
||||
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
|
||||
@@ -19,7 +19,7 @@ export interface AdminAsset {
|
||||
}
|
||||
|
||||
export const ADMIN_ASSETS: Record<string, AdminAsset> = {
|
||||
"/admin/assets/index-CoGEje3-.js": { path: A_0_assets_index_CoGEje3__js as unknown as string, mime: "application/javascript; charset=utf-8" },
|
||||
"/admin/assets/index-BpDk4NI4.js": { path: A_0_assets_index_BpDk4NI4_js as unknown as string, mime: "application/javascript; charset=utf-8" },
|
||||
"/admin/assets/index-GxkWX7v3.css": { path: A_1_assets_index_GxkWX7v3_css as unknown as string, mime: "text/css; charset=utf-8" },
|
||||
"/admin/index.html": { path: A_2_index_html as unknown as string, mime: "text/html; charset=utf-8" },
|
||||
};
|
||||
|
||||
+45
-2
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch']);
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -998,6 +998,13 @@ const THIN_CLIENT_REFUSED_COMMANDS = new Set([
|
||||
// - `code-def`/`code-refs`/`code-callers`/`code-callees` have NO MCP ops
|
||||
// in operations.ts:2630-2671; cannot be "fixed by routing" yet
|
||||
'pages', 'files', 'eval', 'code-def', 'code-refs', 'code-callers', 'code-callees',
|
||||
// scratch-DB audit: `config` get/set operate on the host brain's config
|
||||
// plane (DB rows / host file-plane). On a thin client they fabricated an
|
||||
// ephemeral local PGLite (full migration replay per call) and read/wrote
|
||||
// config nobody would ever see. NOTE: `jobs` is deliberately NOT here —
|
||||
// it gets a partial dispatch (list/get route over MCP engine-free, the
|
||||
// rest refuse) in the main dispatch before connectEngine().
|
||||
'config',
|
||||
]);
|
||||
|
||||
/**
|
||||
@@ -1035,6 +1042,9 @@ const THIN_CLIENT_REFUSE_HINTS: Record<string, string> = {
|
||||
'code-refs': '`code-refs` has no MCP op yet. Run on the host.',
|
||||
'code-callers': '`code-callers` has no MCP op yet. Run on the host.',
|
||||
'code-callees': '`code-callees` has no MCP op yet. Run on the host.',
|
||||
// scratch-DB audit additions
|
||||
config: "config reads/writes the host brain's config plane. Edit the host's .gbrain/config.json (file-plane keys) or run on the host with GBRAIN_HOME set.",
|
||||
jobs: '`jobs list` and `jobs get <id>` are thin-client routable; this subcommand runs against the host queue. Use the submit_job / list_jobs / get_job MCP tools from your agent, or run on the host with GBRAIN_HOME set.',
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -1593,6 +1603,27 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
// Thin-client `jobs` dispatch: `list` and `get` route over MCP (v0.32
|
||||
// routing branches in commands/jobs.ts) and never touch a local engine —
|
||||
// but falling through to connectEngine() below fabricates an empty
|
||||
// scratch PGLite in the thin-client GBRAIN_HOME and replays the entire
|
||||
// migration chain on every invocation before the remote call even runs.
|
||||
// Dispatch them engine-free here; every other jobs subcommand is
|
||||
// host-queue-bound, so refuse with a pinpoint hint instead of building
|
||||
// the scratch store.
|
||||
if (command === 'jobs') {
|
||||
const cfgJobs = loadConfig();
|
||||
if (isThinClient(cfgJobs)) {
|
||||
const jobsSub = args[0];
|
||||
if (jobsSub === 'list' || jobsSub === 'get') {
|
||||
const { runJobs } = await import('./commands/jobs.ts');
|
||||
await runJobs(null, args);
|
||||
return;
|
||||
}
|
||||
refuseThinClient('jobs', cfgJobs!.remote_mcp!.mcp_url);
|
||||
}
|
||||
}
|
||||
|
||||
// All remaining CLI-only commands need a DB connection
|
||||
const engine = await connectEngine();
|
||||
try {
|
||||
@@ -2001,6 +2032,15 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runReindexCodeCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reindex-search-vector': {
|
||||
// Explicit recreate of FTS trigger functions + batched backfill,
|
||||
// honoring GBRAIN_FTS_LANGUAGE. Use after changing the language
|
||||
// env var on a brain that already ran the configurable_fts_language
|
||||
// migration.
|
||||
const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts');
|
||||
await runReindexSearchVectorCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reindex-frontmatter': {
|
||||
// v0.29.1: recovery / explicit-rebuild path for pages.effective_date.
|
||||
// Mirror of reindex-code shape. Wraps the shared library function in
|
||||
@@ -2249,7 +2289,7 @@ IMPORT/EXPORT
|
||||
import <dir> [--no-embed] Import markdown directory
|
||||
sync [--repo <path>] [flags] Git-to-brain incremental sync
|
||||
sync --watch [--interval N] Continuous sync (loops until stopped)
|
||||
sync --install-cron Install persistent sync daemon
|
||||
See also: autopilot --install (continuous daemon).
|
||||
export [--dir ./out/] Export to markdown
|
||||
export --restore-only [--repo <p>] Restore missing supabase-only files
|
||||
[--type T] [--slug-prefix S] With optional filters
|
||||
@@ -2336,6 +2376,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II)
|
||||
query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0)
|
||||
reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0)
|
||||
reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0)
|
||||
reindex-search-vector [--dry-run] [--yes] [--json]
|
||||
Recreate FTS triggers + backfill under
|
||||
$GBRAIN_FTS_LANGUAGE (default 'english')
|
||||
sync --strategy code Sync code files into the brain
|
||||
|
||||
JOBS (Minions)
|
||||
|
||||
+586
-1
@@ -24,6 +24,8 @@ import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { sqlQueryForEngine, executeRawJsonb, type SqlQuery } from '../core/sql-query.ts';
|
||||
import { pgArray } from '../core/oauth-provider.ts';
|
||||
import { assertValidSourceId } from '../core/source-id.ts';
|
||||
|
||||
function hashToken(token: string): string {
|
||||
return createHash('sha256').update(token).digest('hex');
|
||||
@@ -165,6 +167,100 @@ async function list() {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* `gbrain auth list-clients [--json]` — read surface for OAuth 2.1 clients.
|
||||
*
|
||||
* The existing `gbrain auth list` shows LEGACY bearer tokens from
|
||||
* `access_tokens`; this is the parallel for v0.26+ OAuth clients. Separate
|
||||
* commands rather than merged output because the two models have different
|
||||
* field sets (legacy: lifecycle dates; OAuth: scopes + source_id +
|
||||
* federated_read).
|
||||
*
|
||||
* Human output is card-style (multi-line per client) instead of a fixed-
|
||||
* width table — federated_read can hold many ids per client and a wide
|
||||
* single-line layout truncates / wraps badly on terminals < 200 cols.
|
||||
* JSON output uses a `schema_version: 1` envelope; additive only.
|
||||
*/
|
||||
async function listClients(args: string[]) {
|
||||
const json = args.includes('--json');
|
||||
const includeDeleted = args.includes('--include-deleted');
|
||||
await withConfiguredSql(async (sql) => {
|
||||
// Codex finding #2 (medium): default-hide soft-deleted clients so admin
|
||||
// soft-deletes are honored by the CLI surface. Opt-in via flag.
|
||||
const rows = includeDeleted
|
||||
? await sql`
|
||||
SELECT client_id, client_name, scope, source_id, federated_read,
|
||||
grant_types, created_at, deleted_at
|
||||
FROM oauth_clients
|
||||
ORDER BY client_name
|
||||
`
|
||||
: await sql`
|
||||
SELECT client_id, client_name, scope, source_id, federated_read,
|
||||
grant_types, created_at, deleted_at
|
||||
FROM oauth_clients
|
||||
WHERE deleted_at IS NULL
|
||||
ORDER BY client_name
|
||||
`;
|
||||
if (json) {
|
||||
const clients = rows.map((r) => ({
|
||||
client_id: String(r.client_id),
|
||||
client_name: String(r.client_name),
|
||||
scope: r.scope == null ? null : String(r.scope),
|
||||
source_id: r.source_id == null ? null : String(r.source_id),
|
||||
federated_read: Array.isArray(r.federated_read)
|
||||
? (r.federated_read as string[]).map(String)
|
||||
: [],
|
||||
grant_types: Array.isArray(r.grant_types)
|
||||
? (r.grant_types as string[]).map(String)
|
||||
: [],
|
||||
created_at:
|
||||
r.created_at instanceof Date
|
||||
? r.created_at.toISOString()
|
||||
: r.created_at == null
|
||||
? null
|
||||
: String(r.created_at),
|
||||
deleted_at:
|
||||
r.deleted_at instanceof Date
|
||||
? r.deleted_at.toISOString()
|
||||
: r.deleted_at == null
|
||||
? null
|
||||
: String(r.deleted_at),
|
||||
}));
|
||||
process.stdout.write(JSON.stringify({ schema_version: 1, clients }, null, 2) + '\n');
|
||||
return;
|
||||
}
|
||||
if (rows.length === 0) {
|
||||
console.log(
|
||||
includeDeleted
|
||||
? 'No OAuth clients found (including deleted). Register one: gbrain auth register-client <name>'
|
||||
: 'No active OAuth clients found. Register one: gbrain auth register-client <name>'
|
||||
+ '\n(Use --include-deleted to also show soft-deleted clients.)',
|
||||
);
|
||||
return;
|
||||
}
|
||||
for (let i = 0; i < rows.length; i++) {
|
||||
const r = rows[i];
|
||||
const fed = Array.isArray(r.federated_read)
|
||||
? (r.federated_read as string[]).map(String)
|
||||
: [];
|
||||
const grants = Array.isArray(r.grant_types)
|
||||
? (r.grant_types as string[]).map(String)
|
||||
: [];
|
||||
const deletedAt = r.deleted_at;
|
||||
const status = deletedAt == null
|
||||
? ''
|
||||
: ` [SOFT-DELETED ${deletedAt instanceof Date ? deletedAt.toISOString() : String(deletedAt)}]`;
|
||||
console.log(`${sanitizeForTerminal(String(r.client_name))}${status}`);
|
||||
console.log(` client_id: ${sanitizeForTerminal(String(r.client_id))}`);
|
||||
console.log(` scope: ${r.scope == null ? '(none)' : sanitizeForTerminal(String(r.scope))}`);
|
||||
console.log(` grant types: ${grants.length ? sanitizeForTerminal(grants.join(', ')) : '(none)'}`);
|
||||
console.log(` write source: ${r.source_id == null ? '(none)' : sanitizeForTerminal(String(r.source_id))}`);
|
||||
console.log(` federated: ${fed.length ? sanitizeForTerminal(fed.join(', ')) : '(empty)'}`);
|
||||
if (i < rows.length - 1) console.log('');
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async function revoke(name: string) {
|
||||
if (!name) { console.error('Usage: auth revoke <name>'); process.exit(1); }
|
||||
await withConfiguredSql(async (sql) => {
|
||||
@@ -301,6 +397,475 @@ async function test(url: string, token: string) {
|
||||
console.log(`\n🧠 Your brain is live! (${elapsed}s)`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip ANSI escapes + C0/C1 control characters from a string before
|
||||
* printing it to the operator's terminal. Defense for the
|
||||
* codex-flagged terminal-control-injection class: a client_name or
|
||||
* source_id registered via DCR with `\x1b[2J` (clear-screen) or
|
||||
* `\x1b]0;TITLE\x07` (OSC title-change) would poison
|
||||
* `gbrain auth list-clients` output otherwise.
|
||||
*
|
||||
* Replaces unsafe bytes with their `\xNN` hex escape so the operator
|
||||
* sees that something weird is in the field, instead of silent
|
||||
* mutilation. Tab and newline are preserved as-is so legitimate
|
||||
* multi-line values render.
|
||||
*/
|
||||
export function sanitizeForTerminal(s: string): string {
|
||||
// ALL C0/C1 controls + DEL get escaped. Codex re-review caught that
|
||||
// preserving `\n` lets a DCR-registered client_name spoof additional
|
||||
// human-output lines in list-clients (a real attack — newline in the
|
||||
// name visually adds a fake row to the operator's terminal). Tab is
|
||||
// also escaped for the same reason — field-separator spoofing.
|
||||
// C0: 0x00-0x1F. DEL: 0x7F. C1: 0x80-0x9F.
|
||||
return s.replace(/[\x00-\x1f\x7f-\x9f]/g, (ch) =>
|
||||
`\\x${ch.charCodeAt(0).toString(16).padStart(2, '0')}`,
|
||||
);
|
||||
}
|
||||
|
||||
export interface ResolvedClient {
|
||||
client_id: string;
|
||||
client_name: string;
|
||||
source_id: string | null;
|
||||
federated_read: string[];
|
||||
deleted_at: Date | string | null;
|
||||
}
|
||||
|
||||
export type FederatedReadOutcome =
|
||||
| { kind: 'noop'; reason: 'already-granted' | 'not-present' | 'same-list'; client: ResolvedClient; current: string[] }
|
||||
| { kind: 'updated'; client: ResolvedClient; before: string[]; after: string[] };
|
||||
|
||||
/**
|
||||
* Resolve an OAuth client by client_id (exact) or client_name (unique).
|
||||
* Errors on no-match and on ambiguous client_name (>1 row). client_id
|
||||
* takes precedence — if a long hash is passed and matches, returns
|
||||
* immediately without ever querying by name.
|
||||
*
|
||||
* Legacy bearer tokens in `access_tokens` are NOT searched. Federated read
|
||||
* scope is an OAuth-client concept (oauth_clients.federated_read column);
|
||||
* legacy bearers have no source scope.
|
||||
*/
|
||||
/**
|
||||
* Resolve an OAuth client. Codex finding #2 (medium): default-hide
|
||||
* soft-deleted clients so admin-soft-deleted rows aren't mutated by the
|
||||
* CLI. The `includeDeleted` opt is reserved for future read-side surfaces;
|
||||
* grant/revoke/set ALWAYS filter active rows only.
|
||||
*/
|
||||
export async function resolveClient(
|
||||
sql: SqlQuery,
|
||||
nameOrId: string,
|
||||
opts: { includeDeleted?: boolean } = {},
|
||||
): Promise<ResolvedClient> {
|
||||
const allowDeleted = opts.includeDeleted === true;
|
||||
const byId = allowDeleted
|
||||
? await sql`
|
||||
SELECT client_id, client_name, source_id, federated_read, deleted_at
|
||||
FROM oauth_clients WHERE client_id = ${nameOrId} LIMIT 1
|
||||
`
|
||||
: await sql`
|
||||
SELECT client_id, client_name, source_id, federated_read, deleted_at
|
||||
FROM oauth_clients WHERE client_id = ${nameOrId} AND deleted_at IS NULL LIMIT 1
|
||||
`;
|
||||
if (byId.length === 1) return normalizeClientRow(byId[0]);
|
||||
const byName = allowDeleted
|
||||
? await sql`
|
||||
SELECT client_id, client_name, source_id, federated_read, deleted_at
|
||||
FROM oauth_clients WHERE client_name = ${nameOrId}
|
||||
`
|
||||
: await sql`
|
||||
SELECT client_id, client_name, source_id, federated_read, deleted_at
|
||||
FROM oauth_clients WHERE client_name = ${nameOrId} AND deleted_at IS NULL
|
||||
`;
|
||||
if (byName.length === 0) {
|
||||
throw new Error(
|
||||
`No active OAuth client found with name or id "${nameOrId}". ` +
|
||||
`Run \`gbrain auth register-client <name>\` to create one, ` +
|
||||
`or \`gbrain auth list-clients\` to see what exists. ` +
|
||||
`(Soft-deleted clients are hidden by default.)`,
|
||||
);
|
||||
}
|
||||
if (byName.length > 1) {
|
||||
const ids = byName.map((r) => ` ${String(r.client_id)}`).join('\n');
|
||||
throw new Error(
|
||||
`Multiple active OAuth clients named "${nameOrId}". Pass the full client_id instead:\n${ids}`,
|
||||
);
|
||||
}
|
||||
return normalizeClientRow(byName[0]);
|
||||
}
|
||||
|
||||
function normalizeClientRow(row: Record<string, unknown>): ResolvedClient {
|
||||
const fed = row.federated_read;
|
||||
return {
|
||||
client_id: String(row.client_id),
|
||||
client_name: String(row.client_name),
|
||||
source_id: row.source_id == null ? null : String(row.source_id),
|
||||
federated_read: Array.isArray(fed) ? (fed as string[]).map(String) : [],
|
||||
deleted_at: row.deleted_at == null
|
||||
? null
|
||||
: (row.deleted_at as Date | string),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the source_id shape AND DB existence. Codex finding #3 (medium):
|
||||
* a manually-INSERTed source row with weird chars (e.g. comma, quote)
|
||||
* would otherwise land in oauth_clients.federated_read as a never-deletable
|
||||
* malformed entry. Fail at the boundary before the existence query so
|
||||
* malformed input gets the validator's hint, not a "does not exist" hint
|
||||
* pointing at a non-creatable id.
|
||||
*/
|
||||
export async function assertSourceExists(sql: SqlQuery, sourceId: string): Promise<void> {
|
||||
assertValidSourceId(sourceId);
|
||||
const rows = await sql`SELECT id FROM sources WHERE id = ${sourceId} LIMIT 1`;
|
||||
if (rows.length === 0) {
|
||||
throw new Error(
|
||||
`Source "${sourceId}" does not exist. Run \`gbrain sources list\` to see registered sources, ` +
|
||||
`or \`gbrain sources add ${sourceId}\` to create it.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomic append: array_append + NOT-ANY guard so the row-lock fully
|
||||
* serializes concurrent grant/revoke against the same client. Codex
|
||||
* finding #1 (HIGH): the previous read-modify-write shape allowed a
|
||||
* concurrent revoke to be silently UNDONE by a racing grant.
|
||||
*
|
||||
* Returns the post-write federated_read array, or null when no rows
|
||||
* matched (already-granted, soft-deleted, or missing client). Callers
|
||||
* disambiguate via prior resolveClient + includes() check.
|
||||
*
|
||||
* `WHERE deleted_at IS NULL` is part of the atomic guard so a client
|
||||
* soft-deleted between resolveClient and the UPDATE can't be mutated.
|
||||
*/
|
||||
async function appendFederatedReadAtomic(
|
||||
sql: SqlQuery,
|
||||
clientId: string,
|
||||
sourceId: string,
|
||||
): Promise<string[] | null> {
|
||||
const rows = await sql`
|
||||
UPDATE oauth_clients
|
||||
SET federated_read = array_append(federated_read, ${sourceId})
|
||||
WHERE client_id = ${clientId}
|
||||
AND deleted_at IS NULL
|
||||
AND NOT (${sourceId} = ANY(federated_read))
|
||||
RETURNING federated_read
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
const fed = rows[0].federated_read;
|
||||
return Array.isArray(fed) ? (fed as string[]).map(String) : [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomic remove: array_remove + ANY guard. Same race-correctness story
|
||||
* as appendFederatedReadAtomic. Returns post-write array or null.
|
||||
*/
|
||||
async function removeFederatedReadAtomic(
|
||||
sql: SqlQuery,
|
||||
clientId: string,
|
||||
sourceId: string,
|
||||
): Promise<string[] | null> {
|
||||
const rows = await sql`
|
||||
UPDATE oauth_clients
|
||||
SET federated_read = array_remove(federated_read, ${sourceId})
|
||||
WHERE client_id = ${clientId}
|
||||
AND deleted_at IS NULL
|
||||
AND ${sourceId} = ANY(federated_read)
|
||||
RETURNING federated_read
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
const fed = rows[0].federated_read;
|
||||
return Array.isArray(fed) ? (fed as string[]).map(String) : [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Wholesale array overwrite for `set-federated-read`. Honors the
|
||||
* deleted_at filter. Last-writer-wins semantics under concurrent
|
||||
* `set` calls is acceptable — the user is asserting "this exact list"
|
||||
* intent; concurrent set+set just means whichever ran second wins.
|
||||
* Concurrent set+grant or set+revoke is also last-writer-wins, which
|
||||
* is the documented contract for `set`.
|
||||
*/
|
||||
async function replaceFederatedReadAtomic(
|
||||
sql: SqlQuery,
|
||||
clientId: string,
|
||||
next: string[],
|
||||
): Promise<string[] | null> {
|
||||
// TEXT[] binding via pgArray() string-literal escaping (see helper
|
||||
// for the security note). Our narrow SqlQuery surface
|
||||
// (src/core/sql-query.ts) doesn't bind JS arrays directly.
|
||||
const literal = pgArray(next);
|
||||
const rows = await sql`
|
||||
UPDATE oauth_clients
|
||||
SET federated_read = ${literal}
|
||||
WHERE client_id = ${clientId}
|
||||
AND deleted_at IS NULL
|
||||
RETURNING federated_read
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
const fed = rows[0].federated_read;
|
||||
return Array.isArray(fed) ? (fed as string[]).map(String) : [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure helper: dedupe a comma-separated source-id list while preserving
|
||||
* insertion order. Empty input → empty array. Exported so the CLI parser
|
||||
* and tests share one normalizer.
|
||||
*/
|
||||
export function parseSourceCsv(csv: string): string[] {
|
||||
const requested = csv.split(',').map((s) => s.trim()).filter(Boolean);
|
||||
const seen = new Set<string>();
|
||||
const out: string[] = [];
|
||||
for (const s of requested) {
|
||||
if (!seen.has(s)) {
|
||||
seen.add(s);
|
||||
out.push(s);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export interface FederatedReadOpts {
|
||||
/** When true, compute the outcome but skip the persisting UPDATE. */
|
||||
dryRun?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Core: append a source to the client's federated_read.
|
||||
*
|
||||
* Atomicity contract (Codex finding #1, HIGH):
|
||||
* The actual write goes through `appendFederatedReadAtomic` which
|
||||
* serializes at the row-lock so concurrent grant/revoke against the
|
||||
* same client cannot lose updates. The race vector that previously
|
||||
* silently restored revoked access is closed: under two operators
|
||||
* racing `revoke-read sensitive` + `grant-read harmless`, postgres
|
||||
* serializes the two UPDATEs and BOTH ops apply (sensitive removed,
|
||||
* harmless added), instead of one clobbering the other.
|
||||
*
|
||||
* The reported `before` is the snapshot at resolveClient time, which
|
||||
* may be stale relative to a concurrent racer. The `after` reflects
|
||||
* the post-UPDATE state from RETURNING (always fresh).
|
||||
*/
|
||||
export async function grantReadCore(
|
||||
sql: SqlQuery,
|
||||
nameOrId: string,
|
||||
sourceId: string,
|
||||
opts: FederatedReadOpts = {},
|
||||
): Promise<FederatedReadOutcome> {
|
||||
const client = await resolveClient(sql, nameOrId);
|
||||
await assertSourceExists(sql, sourceId);
|
||||
if (client.federated_read.includes(sourceId)) {
|
||||
return { kind: 'noop', reason: 'already-granted', client, current: client.federated_read };
|
||||
}
|
||||
if (opts.dryRun) {
|
||||
// Compute the would-be result without touching the row. Last-known
|
||||
// snapshot is best-effort under concurrent writes.
|
||||
const projected = [...client.federated_read, sourceId];
|
||||
return { kind: 'updated', client, before: client.federated_read, after: projected };
|
||||
}
|
||||
const after = await appendFederatedReadAtomic(sql, client.client_id, sourceId);
|
||||
if (after === null) {
|
||||
// Two equivalent failure modes: (a) racing grant-read already added
|
||||
// the source and the NOT-ANY guard suppressed our UPDATE, or
|
||||
// (b) the client was soft-deleted between resolveClient and UPDATE.
|
||||
// (a) is the more common path. Re-resolve to confirm + report.
|
||||
const reresolved = await resolveClient(sql, client.client_id, { includeDeleted: true });
|
||||
if (reresolved.deleted_at != null) {
|
||||
throw new Error(`Client "${client.client_name}" was soft-deleted before write could land.`);
|
||||
}
|
||||
return { kind: 'noop', reason: 'already-granted', client: reresolved, current: reresolved.federated_read };
|
||||
}
|
||||
return { kind: 'updated', client, before: client.federated_read, after };
|
||||
}
|
||||
|
||||
/**
|
||||
* Core: remove a source from the client's federated_read. Atomic via
|
||||
* array_remove + ANY-guard. Same race-correctness rationale as
|
||||
* grantReadCore — concurrent ops serialize at the row lock.
|
||||
*/
|
||||
export async function revokeReadCore(
|
||||
sql: SqlQuery,
|
||||
nameOrId: string,
|
||||
sourceId: string,
|
||||
opts: FederatedReadOpts = {},
|
||||
): Promise<FederatedReadOutcome> {
|
||||
const client = await resolveClient(sql, nameOrId);
|
||||
if (!client.federated_read.includes(sourceId)) {
|
||||
return { kind: 'noop', reason: 'not-present', client, current: client.federated_read };
|
||||
}
|
||||
if (opts.dryRun) {
|
||||
const projected = client.federated_read.filter((s) => s !== sourceId);
|
||||
return { kind: 'updated', client, before: client.federated_read, after: projected };
|
||||
}
|
||||
const after = await removeFederatedReadAtomic(sql, client.client_id, sourceId);
|
||||
if (after === null) {
|
||||
// Same disambiguation as grant: either a concurrent revoke already
|
||||
// removed the source (most common) or the client was soft-deleted.
|
||||
const reresolved = await resolveClient(sql, client.client_id, { includeDeleted: true });
|
||||
if (reresolved.deleted_at != null) {
|
||||
throw new Error(`Client "${client.client_name}" was soft-deleted before write could land.`);
|
||||
}
|
||||
return { kind: 'noop', reason: 'not-present', client: reresolved, current: reresolved.federated_read };
|
||||
}
|
||||
return { kind: 'updated', client, before: client.federated_read, after };
|
||||
}
|
||||
|
||||
/**
|
||||
* Core: replace the whole federated_read list. Idempotent on same list.
|
||||
*
|
||||
* Race semantics: wholesale-overwrite + deleted_at guard. Concurrent
|
||||
* set+set is last-writer-wins (documented contract for `set` — the
|
||||
* operator is asserting the exact list). Concurrent set+grant or
|
||||
* set+revoke is also last-writer-wins. If a strict-merge semantics is
|
||||
* needed, use grant-read / revoke-read individually.
|
||||
*/
|
||||
export async function setFederatedReadCore(
|
||||
sql: SqlQuery,
|
||||
nameOrId: string,
|
||||
sourceCsv: string,
|
||||
opts: FederatedReadOpts = {},
|
||||
): Promise<FederatedReadOutcome> {
|
||||
const next = parseSourceCsv(sourceCsv);
|
||||
const client = await resolveClient(sql, nameOrId);
|
||||
for (const s of next) {
|
||||
await assertSourceExists(sql, s);
|
||||
}
|
||||
const prev = client.federated_read;
|
||||
const same = prev.length === next.length && prev.every((v, i) => v === next[i]);
|
||||
if (same) {
|
||||
return { kind: 'noop', reason: 'same-list', client, current: prev };
|
||||
}
|
||||
if (opts.dryRun) {
|
||||
return { kind: 'updated', client, before: prev, after: next };
|
||||
}
|
||||
const after = await replaceFederatedReadAtomic(sql, client.client_id, next);
|
||||
if (after === null) {
|
||||
throw new Error(`Client "${client.client_name}" was soft-deleted before write could land.`);
|
||||
}
|
||||
return { kind: 'updated', client, before: prev, after };
|
||||
}
|
||||
|
||||
function printOutcome(
|
||||
verb: 'grant' | 'revoke' | 'set',
|
||||
sourceArg: string,
|
||||
outcome: FederatedReadOutcome,
|
||||
dryRun: boolean,
|
||||
): void {
|
||||
// Terminal-injection defense (Codex finding #5, low): a client_name
|
||||
// registered via DCR with ANSI escapes or control chars would
|
||||
// otherwise poison this output. Sanitize ALL strings that round-trip
|
||||
// from the DB before printing.
|
||||
const s = sanitizeForTerminal;
|
||||
const prefix = dryRun ? '[dry-run] ' : '';
|
||||
if (outcome.kind === 'noop') {
|
||||
const name = s(outcome.client.client_name);
|
||||
if (outcome.reason === 'already-granted') {
|
||||
console.log(`${prefix}No change: "${name}" already reads "${s(sourceArg)}".`);
|
||||
} else if (outcome.reason === 'not-present') {
|
||||
console.log(`${prefix}No change: "${name}" did not read "${s(sourceArg)}".`);
|
||||
} else {
|
||||
console.log(`${prefix}No change: "${name}" federated_read already matches.`);
|
||||
}
|
||||
console.log(` federated_read: ${outcome.current.map(s).join(', ') || '(empty)'}`);
|
||||
return;
|
||||
}
|
||||
const { client, before, after } = outcome;
|
||||
const name = s(client.client_name);
|
||||
const wouldOrDid = dryRun ? 'Would' : 'Did';
|
||||
if (verb === 'grant') {
|
||||
console.log(`${prefix}${wouldOrDid} grant: "${name}" can now read "${s(sourceArg)}".`);
|
||||
console.log(` federated_read: ${after.map(s).join(', ')}`);
|
||||
} else if (verb === 'revoke') {
|
||||
console.log(`${prefix}${wouldOrDid} revoke: "${name}" no longer reads "${s(sourceArg)}".`);
|
||||
console.log(` federated_read: ${after.map(s).join(', ') || '(empty — client has no federated reads)'}`);
|
||||
} else {
|
||||
console.log(`${prefix}${wouldOrDid} update "${name}" federated_read:`);
|
||||
console.log(` before: ${before.map(s).join(', ') || '(empty)'}`);
|
||||
console.log(` after: ${after.map(s).join(', ') || '(empty)'}`);
|
||||
}
|
||||
if (after.length === 0) {
|
||||
console.log(
|
||||
'Warning: client now reads no sources via federation. Queries through this ' +
|
||||
'client will only see content scoped explicitly via its write source.',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip `--dry-run` from a positional-arg list. Returns the filtered list
|
||||
* plus the flag value. Kept positional-tolerant — the existing
|
||||
* `auth grant-read alice source` shape MUST keep working, AND
|
||||
* `auth grant-read alice source --dry-run` AND `auth grant-read --dry-run alice source`.
|
||||
*/
|
||||
export function extractDryRun(args: string[]): { dryRun: boolean; rest: string[] } {
|
||||
let dryRun = false;
|
||||
const rest: string[] = [];
|
||||
for (const a of args) {
|
||||
if (a === '--dry-run') {
|
||||
dryRun = true;
|
||||
continue;
|
||||
}
|
||||
rest.push(a);
|
||||
}
|
||||
return { dryRun, rest };
|
||||
}
|
||||
|
||||
async function grantRead(args: string[]): Promise<void> {
|
||||
const { dryRun, rest } = extractDryRun(args);
|
||||
const [nameOrId, sourceId] = rest;
|
||||
if (!nameOrId || !sourceId) {
|
||||
console.error('Usage: gbrain auth grant-read <client-name-or-id> <source-id> [--dry-run]');
|
||||
process.exit(1);
|
||||
}
|
||||
try {
|
||||
await withConfiguredSql(async (sql) => {
|
||||
const outcome = await grantReadCore(sql, nameOrId, sourceId, { dryRun });
|
||||
printOutcome('grant', sourceId, outcome, dryRun);
|
||||
});
|
||||
} catch (e: any) {
|
||||
console.error('Error:', e.message);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
async function revokeRead(args: string[]): Promise<void> {
|
||||
const { dryRun, rest } = extractDryRun(args);
|
||||
const [nameOrId, sourceId] = rest;
|
||||
if (!nameOrId || !sourceId) {
|
||||
console.error('Usage: gbrain auth revoke-read <client-name-or-id> <source-id> [--dry-run]');
|
||||
process.exit(1);
|
||||
}
|
||||
try {
|
||||
await withConfiguredSql(async (sql) => {
|
||||
const outcome = await revokeReadCore(sql, nameOrId, sourceId, { dryRun });
|
||||
printOutcome('revoke', sourceId, outcome, dryRun);
|
||||
});
|
||||
} catch (e: any) {
|
||||
console.error('Error:', e.message);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
async function setFederatedRead(args: string[]): Promise<void> {
|
||||
const { dryRun, rest } = extractDryRun(args);
|
||||
const [nameOrId, sourceCsv] = rest;
|
||||
if (!nameOrId || sourceCsv === undefined) {
|
||||
console.error(
|
||||
'Usage: gbrain auth set-federated-read <client-name-or-id> <source-id1,source-id2,...> [--dry-run]',
|
||||
);
|
||||
console.error('Pass an empty string ("") to clear all federated reads.');
|
||||
process.exit(1);
|
||||
}
|
||||
try {
|
||||
await withConfiguredSql(async (sql) => {
|
||||
const outcome = await setFederatedReadCore(sql, nameOrId, sourceCsv, { dryRun });
|
||||
printOutcome('set', sourceCsv, outcome, dryRun);
|
||||
});
|
||||
} catch (e: any) {
|
||||
console.error('Error:', e.message);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
async function revokeClient(clientId: string) {
|
||||
if (!clientId) {
|
||||
console.error('Usage: auth revoke-client <client_id>');
|
||||
@@ -319,7 +884,7 @@ async function revokeClient(clientId: string) {
|
||||
console.error(`No client found with id "${clientId}"`);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(`OAuth client revoked: "${rows[0].client_name}" (${clientId})`);
|
||||
console.log(`OAuth client revoked: "${sanitizeForTerminal(String(rows[0].client_name))}" (${clientId})`);
|
||||
console.log('Tokens and authorization codes purged via cascade.');
|
||||
});
|
||||
} catch (e: any) {
|
||||
@@ -440,6 +1005,15 @@ export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
|
||||
if (!grantTypesSet && out.redirectUris.length > 0) {
|
||||
out.grantTypes = ['authorization_code', 'refresh_token'];
|
||||
}
|
||||
// Codex re-review (medium): validate source_id shape at the CLI boundary
|
||||
// so register-client can't seed malformed entries into source_id /
|
||||
// federated_read that subsequent grant/revoke/set commands can't manage.
|
||||
assertValidSourceId(out.sourceId);
|
||||
if (out.federatedRead) {
|
||||
for (const s of out.federatedRead) {
|
||||
assertValidSourceId(s);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
@@ -557,6 +1131,10 @@ export async function runAuth(args: string[]): Promise<void> {
|
||||
}
|
||||
case 'register-client': await registerClient(rest[0], rest.slice(1)); return;
|
||||
case 'revoke-client': await revokeClient(rest[0]); return;
|
||||
case 'list-clients': await listClients(rest); return;
|
||||
case 'grant-read': await grantRead(rest); return;
|
||||
case 'revoke-read': await revokeRead(rest); return;
|
||||
case 'set-federated-read': await setFederatedRead(rest); return;
|
||||
case 'test': {
|
||||
const tokenIdx = rest.indexOf('--token');
|
||||
const url = rest.find(a => !a.startsWith('--') && a !== rest[tokenIdx + 1]);
|
||||
@@ -594,6 +1172,13 @@ Usage:
|
||||
--bound-max-concurrent <n> Bound submit_agent concurrency (default: 1)
|
||||
--budget-usd-per-day <usd> Bound submit_agent daily spend cap
|
||||
gbrain auth revoke-client <client_id> Hard-delete an OAuth 2.1 client (cascades to tokens + codes)
|
||||
gbrain auth list-clients [--json] List OAuth 2.1 clients with scope + write source + federated_read.
|
||||
gbrain auth grant-read <name|client_id> <source-id> [--dry-run]
|
||||
Add a source to the client's federated_read list (idempotent).
|
||||
gbrain auth revoke-read <name|client_id> <source-id> [--dry-run]
|
||||
Remove a source from the client's federated_read list (idempotent).
|
||||
gbrain auth set-federated-read <name|client_id> "<id1,id2,...>" [--dry-run]
|
||||
Replace the client's whole federated_read list. Pass "" to clear.
|
||||
gbrain auth test <url> --token <token> Smoke-test a remote MCP server
|
||||
`);
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
* gbrain autopilot --status [--json]
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync, chmodSync } from 'fs';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { join } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
@@ -109,7 +109,21 @@ function logError(phase: string, e: unknown) {
|
||||
*/
|
||||
export function resolveGbrainCliPath(): string {
|
||||
try {
|
||||
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
// #2747: `env: process.env` is required under Bun. Bun's execSync
|
||||
// snapshots process.env at Bun's OWN startup, not at call time — a
|
||||
// runtime PATH mutation (dotenv/config loading, shell-profile sourcing
|
||||
// in a wrapper, etc.) happening between Bun boot and this call is
|
||||
// invisible to `which` without explicitly forwarding the current env.
|
||||
// This is why "which gbrain" succeeds when run standalone (fresh Bun
|
||||
// process, no prior mutation) but can fail from inside autopilot's own
|
||||
// process at this exact call site. Same fix already applied to
|
||||
// detectTini() in spawn-helpers.ts (see its comment) — this call site
|
||||
// was missed.
|
||||
const which = execSync('which gbrain', {
|
||||
encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
env: process.env,
|
||||
}).trim();
|
||||
if (which) return which;
|
||||
} catch { /* not on $PATH — fall through */ }
|
||||
|
||||
@@ -123,7 +137,14 @@ export function resolveGbrainCliPath(): string {
|
||||
return arg1;
|
||||
}
|
||||
|
||||
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
|
||||
// #2747: include what we actually saw so an operator (or a future bug
|
||||
// report) doesn't have to guess whether PATH/execPath/argv[1] looked
|
||||
// sane at the moment of failure.
|
||||
throw new Error(
|
||||
'Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH ' +
|
||||
'(e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly. ' +
|
||||
`Debug: PATH=${JSON.stringify(process.env.PATH ?? '')} execPath=${JSON.stringify(exec)} argv1=${JSON.stringify(arg1)}`,
|
||||
);
|
||||
}
|
||||
|
||||
export function shouldSpawnAutopilotWorker(args: string[]): boolean {
|
||||
@@ -1242,7 +1263,14 @@ function installLaunchd(wrapperPath: string, home: string, repoPath: string) {
|
||||
try {
|
||||
const agentsDir = join(home, 'Library', 'LaunchAgents');
|
||||
mkdirSync(agentsDir, { recursive: true });
|
||||
writeFileSync(plistPath(), plist);
|
||||
writeFileSync(plistPath(), plist, { mode: 0o644 });
|
||||
// launchd rejects group/world-writable agent plists: bootstrap/load fails
|
||||
// with the opaque "Bootstrap failed: 5: Input/output error" and the login
|
||||
// scan skips the file silently. writeFileSync's mode only applies on
|
||||
// create — a reinstall over an existing plist keeps the old bits (a 0666
|
||||
// plist written under an umask-0 parent stays 0666 forever) — so
|
||||
// normalize unconditionally.
|
||||
chmodSync(plistPath(), 0o644);
|
||||
execSync(`launchctl load "${plistPath()}"`, { stdio: 'pipe' });
|
||||
console.log('Installed launchd service: com.gbrain.autopilot');
|
||||
console.log(` Repo: ${repoPath}`);
|
||||
@@ -1332,7 +1360,11 @@ export function migrateSystemdUnitToRestartAlways(): { rewritten: boolean; reaso
|
||||
return { rewritten: false, reason: 'hand-edited' };
|
||||
}
|
||||
try {
|
||||
writeFileSync(unitPath, generateSystemdUnit(execMatch![1]));
|
||||
writeFileSync(unitPath, generateSystemdUnit(execMatch![1]), { mode: 0o644 });
|
||||
// This path always rewrites an EXISTING unit, so writeFileSync's mode
|
||||
// never applies — chmod is the only thing that normalizes a unit born
|
||||
// 0666 under a umask-0 parent (systemd warns on world-writable units).
|
||||
chmodSync(unitPath, 0o644);
|
||||
try {
|
||||
execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 10_000 });
|
||||
} catch {
|
||||
@@ -1349,7 +1381,10 @@ function installSystemd(wrapperPath: string, repoPath: string) {
|
||||
try {
|
||||
const unitPath = systemdUnitPath();
|
||||
mkdirSync(join(process.env.HOME || '', '.config', 'systemd', 'user'), { recursive: true });
|
||||
writeFileSync(unitPath, unit);
|
||||
writeFileSync(unitPath, unit, { mode: 0o644 });
|
||||
// Same umask-0 hardening as the launchd path (systemd warns on
|
||||
// world-writable units); mode only applies on create, so normalize.
|
||||
chmodSync(unitPath, 0o644);
|
||||
execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 10_000 });
|
||||
execSync('systemctl --user enable --now gbrain-autopilot.service', { stdio: 'pipe', timeout: 15_000 });
|
||||
console.log('Installed systemd user service: gbrain-autopilot.service');
|
||||
|
||||
+22
-15
@@ -3056,7 +3056,7 @@ export async function computeConversationFactsBacklogCheck(
|
||||
const typesRaw = await engine.getConfig(
|
||||
'cycle.conversation_facts_backfill.types',
|
||||
);
|
||||
let types = ['conversation', 'meeting', 'slack', 'email'];
|
||||
let types = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'];
|
||||
if (typesRaw) {
|
||||
try {
|
||||
const parsed = JSON.parse(typesRaw);
|
||||
@@ -4345,7 +4345,7 @@ export async function buildChecks(
|
||||
|
||||
// 2. Skill conformance (SKILL group — gated)
|
||||
if (scope === 'all' && skillsDir) {
|
||||
const conformanceResult = checkSkillConformance(skillsDir);
|
||||
const conformanceResult = skillConformanceCheck(skillsDir);
|
||||
checks.push(conformanceResult);
|
||||
}
|
||||
|
||||
@@ -4927,8 +4927,8 @@ export async function buildChecks(
|
||||
try {
|
||||
const { readConversationBodyForParsing } = await import('../core/conversation-parser/body.ts');
|
||||
const { parseConversation } = await import('../core/conversation-parser/parse.ts');
|
||||
const allowedTypes = ['conversation', 'meeting', 'slack', 'email'] as const;
|
||||
// PageFilters supports singular `type` only; iterate the 4 types
|
||||
const allowedTypes = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'] as const;
|
||||
// PageFilters supports singular `type` only; iterate the allowed types
|
||||
// and cap at ~50/each to land at ~200 total max.
|
||||
const sample: import('../core/types.ts').Page[] = [];
|
||||
for (const t of allowedTypes) {
|
||||
@@ -7178,9 +7178,17 @@ export async function buildChecks(
|
||||
let vanished = 0;
|
||||
const vanishedPaths: string[] = [];
|
||||
const fs = await import('node:fs');
|
||||
const nodePath = await import('node:path');
|
||||
// storage_path is repo-relative for sync-ingested assets. Resolving
|
||||
// against cwd made this check a false-positive WARN whenever doctor
|
||||
// ran outside the brain repo.
|
||||
const repoRoot = (await engine.getConfig('sync.repo_path')) ?? process.cwd();
|
||||
for (const r of rows) {
|
||||
const abs = nodePath.isAbsolute(r.storage_path)
|
||||
? r.storage_path
|
||||
: nodePath.join(repoRoot, r.storage_path);
|
||||
try {
|
||||
fs.statSync(r.storage_path);
|
||||
fs.statSync(abs);
|
||||
} catch {
|
||||
vanished++;
|
||||
if (vanishedPaths.length < 5) vanishedPaths.push(r.storage_path);
|
||||
@@ -7424,15 +7432,13 @@ function printAutoFixReport(report: AutoFixReport, dryRun: boolean, jsonOutput:
|
||||
|
||||
|
||||
/** Quick skill conformance check — frontmatter + required sections */
|
||||
function checkSkillConformance(skillsDir: string): Check {
|
||||
const manifestPath = join(skillsDir, 'manifest.json');
|
||||
if (!existsSync(manifestPath)) {
|
||||
return { name: 'skill_conformance', status: 'warn', message: 'manifest.json not found' };
|
||||
}
|
||||
|
||||
export function skillConformanceCheck(skillsDir: string): Check {
|
||||
try {
|
||||
const manifest = JSON.parse(readFileSync(manifestPath, 'utf-8'));
|
||||
const skills = manifest.skills || [];
|
||||
// Host workspaces are allowed to omit a gbrain-specific manifest. Keep
|
||||
// conformance aligned with resolver_health and skill_brain_first by using
|
||||
// the canonical fallback that derives entries from direct SKILL.md files.
|
||||
const manifest = loadOrDeriveManifest(skillsDir);
|
||||
const skills = manifest.skills;
|
||||
let passing = 0;
|
||||
const failing: string[] = [];
|
||||
|
||||
@@ -7452,7 +7458,8 @@ function checkSkillConformance(skillsDir: string): Check {
|
||||
}
|
||||
|
||||
if (failing.length === 0) {
|
||||
return { name: 'skill_conformance', status: 'ok', message: `${passing}/${skills.length} skills pass` };
|
||||
const derivedNote = manifest.derived ? ' (derived from SKILL.md files)' : '';
|
||||
return { name: 'skill_conformance', status: 'ok', message: `${passing}/${skills.length} skills pass${derivedNote}` };
|
||||
}
|
||||
return {
|
||||
name: 'skill_conformance',
|
||||
@@ -7460,7 +7467,7 @@ function checkSkillConformance(skillsDir: string): Check {
|
||||
message: `${passing}/${skills.length} pass. Failing: ${failing.join(', ')}`,
|
||||
};
|
||||
} catch {
|
||||
return { name: 'skill_conformance', status: 'warn', message: 'Could not parse manifest.json' };
|
||||
return { name: 'skill_conformance', status: 'warn', message: 'Could not load or derive skills manifest' };
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -76,6 +76,18 @@ interface DreamArgs {
|
||||
drain: boolean;
|
||||
/** Drain wallclock budget in seconds. Default 300 (5 min). */
|
||||
windowSeconds: number;
|
||||
/**
|
||||
* issue #2860 — `--once`. One-shot bypass of the named `--phase`'s own
|
||||
* `dream.<phase>.enabled` / `cycle.<phase>.enabled` config gate, for this
|
||||
* invocation only. Never reads or writes config — unlike the old
|
||||
* "toggle enabled true, run, toggle back to false" workaround, a crash
|
||||
* mid-run can't leave any global state stuck. Requires an explicit
|
||||
* `--phase <name>`; bare `--once` is a usage error (there'd be no single
|
||||
* phase to target). Applies only to phases with a config `.enabled` gate
|
||||
* (patterns, synthesize, conversation_facts_backfill, enrich_thin,
|
||||
* skillopt) — a no-op for phases that always run when named directly.
|
||||
*/
|
||||
once: boolean;
|
||||
}
|
||||
|
||||
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}$/;
|
||||
@@ -105,6 +117,14 @@ function collectFlagValues(args: string[], flag: string): string[] | null {
|
||||
|
||||
function parseArgs(args: string[]): DreamArgs {
|
||||
const phaseIdx = args.indexOf('--phase');
|
||||
// issue #2860 (Codex P3): captured BEFORE --input/--drain get a chance to
|
||||
// implicitly default `phase` below, so --once's validation can require
|
||||
// the user actually TYPED --phase, not merely that some phase ended up
|
||||
// resolved. Without this, `--input <f> --once` and `--drain --once`
|
||||
// slip past the "explicit --phase required" contract (the derived
|
||||
// `phase` value is already non-null by the time that check runs) and
|
||||
// --once becomes silently ineffective for both.
|
||||
const phaseWasExplicit = phaseIdx !== -1;
|
||||
const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null;
|
||||
let phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
|
||||
? (rawPhase as CyclePhase)
|
||||
@@ -214,6 +234,35 @@ function parseArgs(args: string[]): DreamArgs {
|
||||
}
|
||||
}
|
||||
|
||||
// issue #2860: --once requires an EXPLICIT single --phase target (typed
|
||||
// by the user, not merely implied by --input/--drain — see
|
||||
// `phaseWasExplicit` above). Bare `--once` (full/default cycle) has no
|
||||
// single phase to bypass the gate for, and force-enabling EVERY
|
||||
// currently-disabled phase at once would be exactly the kind of
|
||||
// surprise-spend risk the flag exists to prevent. An implicit phase
|
||||
// (from --input or --drain) is rejected too: --drain returns before
|
||||
// onceForPhase is ever read, and --input already bypasses the
|
||||
// synthesize gate on its own, so --once would silently do nothing in
|
||||
// either case — reject loudly instead of pretending it worked (Codex
|
||||
// review finding).
|
||||
//
|
||||
// Codex review finding: `--help` must short-circuit BEFORE this exits(2),
|
||||
// matching the "IRON RULE" pinned by test/dream.test.ts's
|
||||
// "--help --source whatever prints help and exits 0" case — `gbrain
|
||||
// dream --help --once` (no --phase) must show help, not a usage error.
|
||||
const once = args.includes('--once');
|
||||
const wantsHelp = args.includes('--help') || args.includes('-h');
|
||||
if (once && !phaseWasExplicit && !wantsHelp) {
|
||||
console.error(
|
||||
'--once requires an explicit --phase <name> (bypasses that one ' +
|
||||
'phase\'s dream.<phase>.enabled / cycle.<phase>.enabled gate for ' +
|
||||
'this run only; never touches config). A phase implied by --input ' +
|
||||
'or --drain does not count — --once would silently do nothing for ' +
|
||||
'those. Usage: gbrain dream --phase <name> --once',
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
return {
|
||||
json: args.includes('--json'),
|
||||
dryRun: args.includes('--dry-run'),
|
||||
@@ -229,6 +278,7 @@ function parseArgs(args: string[]): DreamArgs {
|
||||
source,
|
||||
drain,
|
||||
windowSeconds,
|
||||
once,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -310,6 +360,17 @@ Options:
|
||||
"--dry-run" does NOT mean "zero LLM calls."
|
||||
--json Emit the CycleReport as JSON (agent-readable)
|
||||
--phase <name> Run a single phase: ${ALL_PHASES.join(' | ')}
|
||||
--once With --phase <name>: run that phase once even if its
|
||||
own dream.<phase>.enabled / cycle.<phase>.enabled
|
||||
config gate is false. Never reads or writes config —
|
||||
unlike toggling the flag on/off around the run, a
|
||||
crash mid-invocation can't leave it stuck. Applies to
|
||||
patterns, synthesize, conversation_facts_backfill,
|
||||
enrich_thin, skillopt; no-op on phases with no such
|
||||
gate. Requires an EXPLICIT --phase <name> — a phase
|
||||
implied by --input or --drain does not count (bare
|
||||
--once, or --once with --input/--drain and no
|
||||
explicit --phase, is a usage error).
|
||||
--pull git pull the brain repo before syncing (default: no pull)
|
||||
--dir <path> Brain directory (default: configured brain). On a
|
||||
postgres/remote brain with no local checkout, the
|
||||
@@ -353,6 +414,7 @@ Examples:
|
||||
gbrain dream
|
||||
gbrain dream --dry-run --json
|
||||
gbrain dream --phase lint
|
||||
gbrain dream --phase patterns --once # run once, ignore dream.patterns.enabled=false
|
||||
gbrain dream --phase synthesize --input ~/transcripts/2026-04-25.txt
|
||||
gbrain dream --phase synthesize --from 2026-04-01 --to 2026-04-25
|
||||
0 2 * * * gbrain dream --json # nightly via cron
|
||||
@@ -594,6 +656,9 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom
|
||||
synthFrom: opts.from ?? undefined,
|
||||
synthTo: opts.to ?? undefined,
|
||||
synthBypassDreamGuard: opts.bypassDreamGuard,
|
||||
// issue #2860: opts.phase is guaranteed non-null here when opts.once is
|
||||
// set (parseArgs enforces --once requires --phase).
|
||||
onceForPhase: opts.once ? opts.phase! : undefined,
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
|
||||
@@ -33,7 +33,7 @@ import type { BrainEngine } from '../core/engine.ts';
|
||||
import type { EnrichCandidate, PageType } from '../core/types.ts';
|
||||
import { operations } from '../core/operations.ts';
|
||||
import type { OperationContext } from '../core/operations.ts';
|
||||
import { isAvailable, chat, getChatModel, withBudgetTracker } from '../core/ai/gateway.ts';
|
||||
import { configureGatewayIfUninitialized, isAvailable, chat, getChatModel, withBudgetTracker } from '../core/ai/gateway.ts';
|
||||
import { BudgetTracker, BudgetExhausted } from '../core/budget/budget-tracker.ts';
|
||||
import { hybridSearch } from '../core/search/hybrid.ts';
|
||||
import { serializeMarkdown } from '../core/markdown.ts';
|
||||
@@ -807,7 +807,9 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Chat gateway required for non-dry-run.
|
||||
// Chat gateway is required for non-dry-run. Recover a cold singleton before
|
||||
// reporting an availability error (#2590).
|
||||
if (!parsed.dryRun && !isAvailable('chat')) configureGatewayIfUninitialized();
|
||||
if (!parsed.dryRun && !isAvailable('chat')) {
|
||||
console.error('Chat gateway unavailable. Configure a chat model (e.g. `gbrain config set chat_model anthropic:claude-haiku-4-5`), or pass --dry-run to preview candidates.');
|
||||
process.exit(1);
|
||||
|
||||
@@ -71,7 +71,7 @@ import {
|
||||
extractFactsFromTurn,
|
||||
isFactsExtractionEnabled,
|
||||
} from '../core/facts/extract.ts';
|
||||
import { isAvailable, withBudgetTracker } from '../core/ai/gateway.ts';
|
||||
import { configureGatewayIfUninitialized, isAvailable, withBudgetTracker } from '../core/ai/gateway.ts';
|
||||
import { BudgetTracker, BudgetExhausted } from '../core/budget/budget-tracker.ts';
|
||||
import { listSources } from '../core/sources-ops.ts';
|
||||
import {
|
||||
@@ -81,7 +81,6 @@ import {
|
||||
} from '../core/op-checkpoint.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions, maybeBackground } from '../core/cli-options.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { createHash } from 'crypto';
|
||||
// v0.41.15.0 (T5): worker-pool primitive + per-source-clamp wrapper +
|
||||
// per-page advisory lock + delete-orphans-first replay safety. See plan
|
||||
@@ -141,7 +140,14 @@ export const DEFAULT_MAX_COST_USD = 5.0;
|
||||
* `--types` flag is an explicit per-run override; cycle config is
|
||||
* the single source of truth.
|
||||
*/
|
||||
export const ALLOWED_TYPES = ['conversation', 'meeting', 'slack', 'email'] as const;
|
||||
export const ALLOWED_TYPES = [
|
||||
'conversation',
|
||||
'meeting',
|
||||
'slack',
|
||||
'email',
|
||||
'imessage',
|
||||
'imessage-daily',
|
||||
] as const;
|
||||
export type AllowedType = (typeof ALLOWED_TYPES)[number];
|
||||
|
||||
/**
|
||||
@@ -757,6 +763,12 @@ async function processPage(
|
||||
source_markdown_slug: page.slug,
|
||||
source: PER_SEGMENT_SOURCE_PREFIX,
|
||||
source_session: sessionId,
|
||||
// Preserve the conversation's valid time instead of defaulting every
|
||||
// extracted fact to extraction time. Epoch-anchored parses have no
|
||||
// trustworthy date, so they retain the existing now() fallback.
|
||||
...(seg.startIso && !seg.startIso.startsWith('1970-')
|
||||
? { valid_from: new Date(seg.startIso) }
|
||||
: {}),
|
||||
context:
|
||||
fact.context ?? `from ${page.slug} segment ${seg.startIso}..${seg.endIso}`,
|
||||
}));
|
||||
@@ -1069,7 +1081,8 @@ export async function runExtractConversationFactsCore(
|
||||
}
|
||||
// Fall through to receipt+rollup write so the partial run is
|
||||
// still observable in extract_health doctor + extracts/ pages.
|
||||
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
|
||||
// ...but not under --dry-run: a preview must not persist cache state.
|
||||
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
|
||||
// Return partial result — caller (CLI / Minion) decides how to
|
||||
// surface. NOT a thrown failure.
|
||||
return result;
|
||||
@@ -1081,7 +1094,9 @@ export async function runExtractConversationFactsCore(
|
||||
// (queryable + citable per D-EXTRACT-17/19) AND UPSERTs the per-day
|
||||
// rollup row (best-effort cache per F-OUT-19). Both are best-effort —
|
||||
// failures stderr-warn but never fail the parent operation.
|
||||
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
|
||||
// --dry-run must not persist cache/knowledge state: skip the rollup UPSERT +
|
||||
// receipt-page write so a preview leaves no extract cache row behind.
|
||||
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -1351,7 +1366,9 @@ export async function runExtractConversationFacts(
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Chat gateway is required for non-dry-run.
|
||||
// Chat gateway is required for non-dry-run. Recover a cold singleton before
|
||||
// reporting an availability error (#2590).
|
||||
if (!parsed.dryRun && !isAvailable('chat')) configureGatewayIfUninitialized();
|
||||
if (!parsed.dryRun && !isAvailable('chat')) {
|
||||
console.error('Chat gateway unavailable. Configure an Anthropic or compatible chat model, or pass --dry-run to preview segmentation.');
|
||||
process.exit(1);
|
||||
|
||||
@@ -188,7 +188,7 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string
|
||||
// Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't
|
||||
// call isSyncable at all — so it descended into `node_modules/`, emitted
|
||||
// markdown files from there, AND ignored the canonical exclusion list
|
||||
// (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor
|
||||
// (`.raw/`, README.md, etc.). Now: pruneDir skips entire vendor
|
||||
// subtrees before recursion (saving IO), and isSyncable filters the emit
|
||||
// set against the canonical markdown-strategy rules.
|
||||
const files: { path: string; relPath: string }[] = [];
|
||||
|
||||
+79
-9
@@ -11,6 +11,7 @@ import {
|
||||
isCodeFilePath,
|
||||
isMarkdownFilePath,
|
||||
isImageFilePath as isImageFilePathFromSync,
|
||||
matchesAnyGlob,
|
||||
pruneDir,
|
||||
SYNC_SKIP_FILES,
|
||||
type SyncStrategy,
|
||||
@@ -20,6 +21,7 @@ import {
|
||||
loadCheckpoint,
|
||||
saveCheckpoint,
|
||||
clearCheckpoint,
|
||||
resolveImportTargetDir,
|
||||
resumeFilter,
|
||||
} from '../core/import-checkpoint.ts';
|
||||
|
||||
@@ -46,7 +48,25 @@ export interface RunImportResult {
|
||||
export async function runImport(
|
||||
engine: BrainEngine,
|
||||
args: string[],
|
||||
opts: { commit?: string; strategy?: SyncStrategy; sourceId?: string; managedBookmark?: boolean } = {},
|
||||
opts: {
|
||||
commit?: string;
|
||||
strategy?: SyncStrategy;
|
||||
sourceId?: string;
|
||||
managedBookmark?: boolean;
|
||||
/**
|
||||
* #753/#774: glob patterns to exclude from the import (same semantics as
|
||||
* `isSyncable`'s `exclude` — matched against the dir-relative path).
|
||||
* Threaded by performFullSync for `gbrain sync --exclude`.
|
||||
*/
|
||||
exclude?: string[];
|
||||
/**
|
||||
* #753/#774 monorepo subdir-source support: when set, slugs and
|
||||
* `source_path` are computed relative to this root (the git repo root)
|
||||
* instead of `dir` (the sync scope), so `wiki/page1.md` lands as slug
|
||||
* `wiki/page1` consistently across full and incremental sync.
|
||||
*/
|
||||
slugRoot?: string;
|
||||
} = {},
|
||||
): Promise<RunImportResult> {
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
const fresh = args.includes('--fresh');
|
||||
@@ -168,7 +188,19 @@ export async function runImport(
|
||||
console.error('Usage: gbrain import <dir> [--no-embed] [--workers N] [--fresh] [--source-id <id>] [--json]');
|
||||
process.exit(1);
|
||||
}
|
||||
const dir: string = dirArg; // narrowed; survives closure capture
|
||||
// #1728: capture the import target ONCE as an absolute real path. Every
|
||||
// downstream consumer of `dir` (collection, checkpoint load/save, resume
|
||||
// filtering) sees the same canonical identity — never the caller's `.`/
|
||||
// relative spelling, which would make the persisted checkpoint `dir`
|
||||
// resolve against whatever CWD a later process happens to run from.
|
||||
let dir: string;
|
||||
try {
|
||||
dir = resolveImportTargetDir(dirArg);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(`Import target is not readable: ${dirArg} (${msg})`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// v0.31.2: collect under the right strategy. Pre-fix this called
|
||||
// collectMarkdownFiles unconditionally — code-strategy first sync
|
||||
@@ -177,13 +209,30 @@ export async function runImport(
|
||||
const strategy: SyncStrategy = opts.strategy ?? 'markdown';
|
||||
const _walkT0 = Date.now();
|
||||
console.error(`[gbrain phase] import.collect_files start dir=${dir} strategy=${strategy}`);
|
||||
const allFiles = collectSyncableFiles(dir, { strategy });
|
||||
let allFiles = collectSyncableFiles(dir, { strategy });
|
||||
console.error(
|
||||
`[gbrain phase] import.collect_files done ${Date.now() - _walkT0}ms files=${allFiles.length}`,
|
||||
);
|
||||
const fileTypeLabel = strategy === 'code' ? 'code'
|
||||
: strategy === 'auto' ? 'syncable' : 'markdown';
|
||||
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
|
||||
// #753/#774: apply --exclude glob patterns (threaded by performFullSync).
|
||||
if (opts.exclude && opts.exclude.length > 0) {
|
||||
const beforeExclude = allFiles.length;
|
||||
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(dir, abs), opts.exclude));
|
||||
console.log(
|
||||
`Found ${allFiles.length} ${fileTypeLabel} files ` +
|
||||
`(${beforeExclude - allFiles.length} excluded by --exclude patterns)`,
|
||||
);
|
||||
// NAV-4: everything excluded is almost always a mistyped pattern — warn.
|
||||
if (beforeExclude > 0 && allFiles.length === 0) {
|
||||
console.warn(
|
||||
`[gbrain sync] No files matched after applying ${opts.exclude.length} --exclude pattern(s). ` +
|
||||
`Check your --exclude flags. Patterns: ${JSON.stringify(opts.exclude)}`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
|
||||
}
|
||||
|
||||
// Sort newest-first so date-prefixed brain paths get embedded before older ones.
|
||||
// See src/core/sort-newest-first.ts for the policy.
|
||||
@@ -229,6 +278,11 @@ export async function runImport(
|
||||
|
||||
async function processFile(eng: BrainEngine, filePath: string) {
|
||||
const relativePath = relative(dir, filePath);
|
||||
// #753/#774: slug + source_path base. When performFullSync syncs a
|
||||
// monorepo subdir, slugRoot is the git root so slugs stay git-root-
|
||||
// relative (matching the incremental path's git-diff paths). The
|
||||
// checkpoint (`completed`) stays dir-relative — resumeFilter's contract.
|
||||
const importRelPath = opts.slugRoot ? relative(opts.slugRoot, filePath) : relativePath;
|
||||
// v0.31.2 (D5): per-file slow-path log. Fires only when a single
|
||||
// file takes >5s. The user's hang surfaces as one file taking
|
||||
// forever — without this, the agent can't see which file.
|
||||
@@ -239,8 +293,8 @@ export async function runImport(
|
||||
// up images when GBRAIN_EMBEDDING_MULTIMODAL=true so this branch is
|
||||
// unreachable when the gate is off; defense-in-depth check anyway.
|
||||
const result = isImageFilePath(relativePath) && process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true'
|
||||
? await importImageFile(eng, filePath, relativePath, { noEmbed, sourceId })
|
||||
: await importFile(eng, filePath, relativePath, { noEmbed, sourceId, activePack: importActivePack });
|
||||
? await importImageFile(eng, filePath, importRelPath, { noEmbed, sourceId })
|
||||
: await importFile(eng, filePath, importRelPath, { noEmbed, sourceId, activePack: importActivePack });
|
||||
const _fileMs = Date.now() - _fileT0;
|
||||
if (_fileMs > 5000) {
|
||||
console.error(`[gbrain phase] import.process_file slow ${_fileMs}ms ${relativePath}`);
|
||||
@@ -256,7 +310,9 @@ export async function runImport(
|
||||
if (result.error && result.error !== 'unchanged') {
|
||||
console.error(` Skipped ${relativePath}: ${result.error}`);
|
||||
// Bug 9 — non-"unchanged" skips carry a real error reason.
|
||||
failures.push({ path: relativePath, error: result.error });
|
||||
// #774: ledger paths use the slug base so an incremental sync's
|
||||
// success at the same (git-root-relative) path clears the row.
|
||||
failures.push({ path: importRelPath, error: result.error });
|
||||
} else {
|
||||
// 'unchanged' or no-error skip: content_hash matched a prior
|
||||
// successful import, so this file IS done for checkpoint purposes.
|
||||
@@ -274,7 +330,7 @@ export async function runImport(
|
||||
}
|
||||
errors++;
|
||||
skipped++;
|
||||
failures.push({ path: relativePath, error: msg });
|
||||
failures.push({ path: importRelPath, error: msg });
|
||||
}
|
||||
processed++;
|
||||
tickProgress();
|
||||
@@ -288,6 +344,9 @@ export async function runImport(
|
||||
catch { /* non-fatal */ }
|
||||
}
|
||||
saveCheckpoint(checkpointPath, {
|
||||
schema_version: 1,
|
||||
owner: 'gbrain',
|
||||
kind: 'import',
|
||||
dir,
|
||||
completedPaths: Array.from(completed),
|
||||
timestamp: new Date().toISOString(),
|
||||
@@ -510,10 +569,21 @@ function isCollectibleForWalker(
|
||||
strategy: SyncStrategy,
|
||||
multimodalOn: boolean,
|
||||
): boolean {
|
||||
// #2607: apply the SAME segment-level prune gate as incremental sync's
|
||||
// `classifySync` (core/sync.ts). The FS walk below prunes at descent time,
|
||||
// but the git fast path enumerates via `git ls-files` and historically
|
||||
// filtered only by extension — so `sync --full` imported (and resurrected
|
||||
// previously-deleted) pages under dot-dirs / vendored trees that incremental
|
||||
// sync excludes. Full and incremental must agree on the exclusion set.
|
||||
// (In the FS-walk route `path` is a basename, so this is the same dot-file
|
||||
// check pruneDir already applied there — no behavior change on that route.)
|
||||
const segments = path.split('/');
|
||||
if (segments.some((seg) => !pruneDir(seg))) return false;
|
||||
|
||||
// Metafiles are directory scaffolding (READMEs / index / log / schema /
|
||||
// resolver), not typed brain pages — same exclusion `sync`'s `isSyncable`
|
||||
// applies. Guards both the FS-walk and the git-fast-path collection routes.
|
||||
const basename = path.split('/').pop() || '';
|
||||
const basename = segments[segments.length - 1] || '';
|
||||
if ((SYNC_SKIP_FILES as readonly string[]).includes(basename)) return false;
|
||||
|
||||
switch (strategy) {
|
||||
|
||||
+78
-16
@@ -7,7 +7,7 @@ import type { BrainEngine } from '../core/engine.ts';
|
||||
import { MinionQueue } from '../core/minions/queue.ts';
|
||||
import { MinionWorker } from '../core/minions/worker.ts';
|
||||
import { WORKER_EXIT_RSS_WATCHDOG } from '../core/minions/worker-exit-codes.ts';
|
||||
import type { MinionJob, MinionJobStatus } from '../core/minions/types.ts';
|
||||
import type { MinionHandler, MinionJob, MinionJobStatus } from '../core/minions/types.ts';
|
||||
import type { PaceKeyOverrides } from '../core/pace-mode.ts';
|
||||
import { loadConfig, isThinClient } from '../core/config.ts';
|
||||
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
|
||||
@@ -22,6 +22,49 @@ function hasFlag(args: string[], flag: string): boolean {
|
||||
return args.includes(flag);
|
||||
}
|
||||
|
||||
/**
|
||||
* Long-lived workers outlive operator config changes. Re-stamp the AI gateway
|
||||
* from DB-backed model config immediately before queued jobs enter gateway-backed
|
||||
* paths, so a stale process-level default cannot route new work to the wrong
|
||||
* provider.
|
||||
*/
|
||||
async function refreshGatewayForJob(engine: BrainEngine): Promise<void> {
|
||||
const { reconfigureGatewayWithEngine } = await import('../core/ai/gateway.ts');
|
||||
await reconfigureGatewayWithEngine(engine);
|
||||
}
|
||||
|
||||
const GATEWAY_REFRESH_JOB_NAMES = new Set([
|
||||
'embed',
|
||||
'extract-conversation-facts',
|
||||
'enrich',
|
||||
'contextual_reindex_per_chunk',
|
||||
'autopilot-cycle',
|
||||
'synthesize',
|
||||
'patterns',
|
||||
'consolidate',
|
||||
'extract_facts',
|
||||
'extract-atoms-drain',
|
||||
'embed-backfill',
|
||||
'extract-takes-from-pages',
|
||||
'embed-catch-up',
|
||||
]);
|
||||
|
||||
function registerBuiltinJob(
|
||||
worker: MinionWorker,
|
||||
engine: BrainEngine,
|
||||
name: string,
|
||||
handler: MinionHandler,
|
||||
): void {
|
||||
if (!GATEWAY_REFRESH_JOB_NAMES.has(name)) {
|
||||
worker.register(name, handler);
|
||||
return;
|
||||
}
|
||||
worker.register(name, async (job) => {
|
||||
await refreshGatewayForJob(engine);
|
||||
return await handler(job);
|
||||
});
|
||||
}
|
||||
|
||||
/** Parse `--max-waiting N` from CLI args. Returns undefined if absent.
|
||||
* Throws on malformed input (caller should surface the error and exit).
|
||||
* Clamps to [1, 100] to match the queue-layer clamp in MinionQueue.add.
|
||||
@@ -132,9 +175,23 @@ function formatJobDetail(job: MinionJob): string {
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export async function runJobs(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
export async function runJobs(engineOrNull: BrainEngine | null, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
|
||||
// Thin-client dispatch (cli.ts) passes engine=null for the subcommands
|
||||
// with remote MCP routing (`list`, `get`) so no scratch local engine is
|
||||
// ever built. Any other subcommand arriving with a null engine is a
|
||||
// routing bug upstream of this function — refuse instead of crashing
|
||||
// inside MinionQueue.
|
||||
if (!engineOrNull && sub !== 'list' && sub !== 'get') {
|
||||
console.error(`\`gbrain jobs ${sub ?? ''}\` needs a local engine and cannot run on a thin client.`);
|
||||
process.exit(1);
|
||||
}
|
||||
// Null only ever reaches the MCP-routed `list`/`get` branches, which
|
||||
// never touch the engine — narrowed once here so the host-only cases
|
||||
// below typecheck unchanged.
|
||||
const engine = engineOrNull as BrainEngine;
|
||||
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
console.log(`gbrain jobs — Minions job queue
|
||||
|
||||
@@ -217,6 +274,8 @@ HANDLER TYPES (built in)
|
||||
return;
|
||||
}
|
||||
|
||||
// The constructor just stores the reference; on the null (thin-client
|
||||
// list/get) paths no queue method is ever reached.
|
||||
const queue = new MinionQueue(engine);
|
||||
|
||||
switch (sub) {
|
||||
@@ -1423,7 +1482,7 @@ export async function registerBuiltinHandlers(
|
||||
return { ...result, embed_job_id: embedJobId, embed_skip_reason: embedSkipReason };
|
||||
});
|
||||
|
||||
worker.register('embed', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'embed', async (job) => {
|
||||
const { runEmbedCore } = await import('./embed.ts');
|
||||
// Primary Minion progress channel is job.updateProgress (DB-backed,
|
||||
// readable via `gbrain jobs get <id>`). Stderr from the worker daemon
|
||||
@@ -1470,7 +1529,7 @@ export async function registerBuiltinHandlers(
|
||||
// BudgetTracker inside its own process. BudgetExhausted is caught at
|
||||
// the core level and returned as `result.budget_exhausted: true` (NOT
|
||||
// a job failure) so the user can resume with a higher cap.
|
||||
worker.register('extract-conversation-facts', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'extract-conversation-facts', async (job) => {
|
||||
const { runExtractConversationFactsCore } = await import('./extract-conversation-facts.ts');
|
||||
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
|
||||
if (!sourceId) {
|
||||
@@ -1481,7 +1540,7 @@ export async function registerBuiltinHandlers(
|
||||
}
|
||||
const types = Array.isArray(job.data.types)
|
||||
? (job.data.types as string[]).filter((t) =>
|
||||
['conversation', 'meeting', 'slack', 'email'].includes(t),
|
||||
['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'].includes(t),
|
||||
)
|
||||
: undefined;
|
||||
const result = await runExtractConversationFactsCore(engine, {
|
||||
@@ -1529,7 +1588,7 @@ export async function registerBuiltinHandlers(
|
||||
// at the core level and returned as result.budget_exhausted (NOT a failure).
|
||||
// Strict per-source: the CLI fans out one job per source when --source is
|
||||
// omitted, so a job ALWAYS carries data.sourceId.
|
||||
worker.register('enrich', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'enrich', async (job) => {
|
||||
const { runEnrichCore } = await import('./enrich.ts');
|
||||
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
|
||||
if (!sourceId) {
|
||||
@@ -1669,13 +1728,13 @@ export async function registerBuiltinHandlers(
|
||||
const { makeContextualReindexHandler } = await import(
|
||||
'../core/minions/handlers/contextual-reindex-per-chunk.ts'
|
||||
);
|
||||
worker.register('contextual_reindex_per_chunk', makeContextualReindexHandler({ engine }));
|
||||
registerBuiltinJob(worker, engine, 'contextual_reindex_per_chunk', makeContextualReindexHandler({ engine }));
|
||||
}
|
||||
|
||||
// derivation); the handler returns { partial, status, report } so
|
||||
// `gbrain jobs get <id>` shows the full structured report. Does NOT
|
||||
// throw on partial: a flaky phase must not block every future cycle.
|
||||
worker.register('autopilot-cycle', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'autopilot-cycle', async (job) => {
|
||||
const { runCycle } = await import('../core/cycle.ts');
|
||||
// v0.41.30 (T2): fall back to null (NOT cwd '.') when no repo is configured.
|
||||
// The queued cycle is the same primitive `gbrain dream` uses; a checkout-less
|
||||
@@ -1780,6 +1839,7 @@ export async function registerBuiltinHandlers(
|
||||
brainDir: effectiveBrainDir,
|
||||
pull,
|
||||
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
|
||||
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
|
||||
...(sourceId ? { sourceId } : {}),
|
||||
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
|
||||
yieldBetweenPhases: async () => {
|
||||
@@ -1817,6 +1877,7 @@ export async function registerBuiltinHandlers(
|
||||
brainDir: repoPath,
|
||||
pull: false, // brain-wide DB/maintenance work never git-pulls
|
||||
signal: job.signal,
|
||||
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
|
||||
phases,
|
||||
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
|
||||
});
|
||||
@@ -1962,17 +2023,18 @@ export async function registerBuiltinHandlers(
|
||||
brainDir: repoPath,
|
||||
phases: [phase as any],
|
||||
signal: job.signal,
|
||||
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
|
||||
});
|
||||
return { phase, status: report.status, report };
|
||||
};
|
||||
|
||||
// PROTECTED — internally spawn subagent children
|
||||
worker.register('synthesize', makePhaseHandler('synthesize'));
|
||||
worker.register('patterns', makePhaseHandler('patterns'));
|
||||
worker.register('consolidate', makePhaseHandler('consolidate'));
|
||||
registerBuiltinJob(worker, engine, 'synthesize', makePhaseHandler('synthesize'));
|
||||
registerBuiltinJob(worker, engine, 'patterns', makePhaseHandler('patterns'));
|
||||
registerBuiltinJob(worker, engine, 'consolidate', makePhaseHandler('consolidate'));
|
||||
|
||||
// Open — DB writes only, no LLM spend
|
||||
worker.register('extract_facts', makePhaseHandler('extract_facts'));
|
||||
registerBuiltinJob(worker, engine, 'extract_facts', makePhaseHandler('extract_facts'));
|
||||
worker.register('resolve_symbol_edges', makePhaseHandler('resolve_symbol_edges'));
|
||||
worker.register('recompute_emotional_weight', makePhaseHandler('recompute_emotional_weight'));
|
||||
|
||||
@@ -1982,7 +2044,7 @@ export async function registerBuiltinHandlers(
|
||||
// window / defer behavior. On LockUnavailableError (the routine cycle holds
|
||||
// the per-source lock) the job completes `{ deferred: true }` and retries
|
||||
// next tick instead of failing — cooperative interleave (CODEX accepted).
|
||||
worker.register('extract-atoms-drain', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'extract-atoms-drain', async (job) => {
|
||||
const { runExtractAtomsDrainForSource } = await import('../core/cycle/extract-atoms-drain.ts');
|
||||
const { LockUnavailableError } = await import('../core/db-lock.ts');
|
||||
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
|
||||
@@ -2010,7 +2072,7 @@ export async function registerBuiltinHandlers(
|
||||
// Cost-bounded via D6 ($10/job BudgetTracker) + D19 (source-level cooldown
|
||||
// + 24h rolling cap, gated at submit time). NOT in PROTECTED_JOB_NAMES —
|
||||
// embedding-only spend, no API-by-the-minute risk like subagent.
|
||||
worker.register('embed-backfill', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'embed-backfill', async (job) => {
|
||||
const { makeEmbedBackfillHandler } = await import('../core/minions/handlers/embed-backfill.ts');
|
||||
return await makeEmbedBackfillHandler(engine)(job);
|
||||
});
|
||||
@@ -2031,7 +2093,7 @@ export async function registerBuiltinHandlers(
|
||||
// (LLM-bearing). Two-gate consent enforced at the handler boundary:
|
||||
// refuses to run unless takes.bootstrap_enabled config is true, even
|
||||
// when allowProtectedSubmit was set at queue.add time.
|
||||
worker.register('extract-takes-from-pages', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'extract-takes-from-pages', async (job) => {
|
||||
const { extractTakesFromPages } = await import('../core/extract-takes-from-pages.ts');
|
||||
const data = (job.data ?? {}) as { sourceId?: string; maxPages?: number };
|
||||
const bootstrapCfg = await engine.getConfig('takes.bootstrap_enabled');
|
||||
@@ -2058,7 +2120,7 @@ export async function registerBuiltinHandlers(
|
||||
// remediation pipeline. Wraps runEmbedCore with stale + catchUp + the
|
||||
// priority/batchSize the recommendation supplies. NOT in
|
||||
// PROTECTED_JOB_NAMES (embedding spend only).
|
||||
worker.register('embed-catch-up', async (job) => {
|
||||
registerBuiltinJob(worker, engine, 'embed-catch-up', async (job) => {
|
||||
const { runEmbedCore } = await import('./embed.ts');
|
||||
const data = (job.data ?? {}) as {
|
||||
sourceId?: string;
|
||||
|
||||
+43
-14
@@ -7,6 +7,7 @@
|
||||
|
||||
import { listRecipes, getRecipe } from '../core/ai/recipes/index.ts';
|
||||
import { configureGateway, embedOne, isAvailable as gwIsAvailable, chat as gwChat } from '../core/ai/gateway.ts';
|
||||
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
|
||||
import { probeOllama, probeLMStudio } from '../core/ai/probes.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { AIConfigError, AITransientError } from '../core/ai/errors.ts';
|
||||
@@ -33,16 +34,19 @@ interface ProviderOption {
|
||||
|
||||
function configureFromEnv(): void {
|
||||
const config = loadConfig();
|
||||
configureGateway({
|
||||
embedding_model: config?.embedding_model,
|
||||
embedding_dimensions: config?.embedding_dimensions,
|
||||
expansion_model: config?.expansion_model,
|
||||
chat_model: config?.chat_model,
|
||||
chat_fallback_chain: config?.chat_fallback_chain,
|
||||
base_urls: config?.provider_base_urls,
|
||||
provider_chat_options: config?.provider_chat_options,
|
||||
env: { ...process.env },
|
||||
});
|
||||
// Route through buildGatewayConfig — the single ownership seam that folds
|
||||
// file-plane API keys (openrouter_api_key, zeroentropy_api_key, ...) into
|
||||
// the gateway env — instead of hand-assembling AIGatewayConfig field by
|
||||
// field. Hand-building it here let this diagnostic report a provider as
|
||||
// missing env even when ~/.gbrain/config.json had it and the real gateway
|
||||
// path resolved it fine (#2728). Pre-init (no file-plane config yet) falls
|
||||
// back to a bare env passthrough so the command still works before
|
||||
// `gbrain init`.
|
||||
if (config) {
|
||||
configureGateway(buildGatewayConfig(config));
|
||||
return;
|
||||
}
|
||||
configureGateway({ env: { ...process.env } });
|
||||
}
|
||||
|
||||
export function envReady(recipe: Recipe, env: NodeJS.ProcessEnv = process.env): boolean {
|
||||
@@ -137,7 +141,12 @@ EXAMPLES
|
||||
}
|
||||
|
||||
function runList(_args: string[]): void {
|
||||
console.log(formatRecipeTable(listRecipes()));
|
||||
// Same env the gateway actually sees (file-plane keys folded in), not bare
|
||||
// process.env — keeps this table's STATUS column honest with what
|
||||
// `providers test` (and the real init/gateway path) would report.
|
||||
const cfg = loadConfig();
|
||||
const env = cfg ? buildGatewayConfig(cfg).env : process.env;
|
||||
console.log(formatRecipeTable(listRecipes(), env));
|
||||
}
|
||||
|
||||
async function runTest(args: string[]): Promise<void> {
|
||||
@@ -164,8 +173,18 @@ async function runTest(args: string[]): Promise<void> {
|
||||
// the divergence at the top of the test so the recovery experience
|
||||
// doesn't repeat the bug-reporter's "providers test ✓ but import still
|
||||
// broken" trap.
|
||||
//
|
||||
// #2863: `cfg` is lifted out of the try block (not just used for the
|
||||
// warning) so the configureGateway calls below can reuse it. Before this
|
||||
// fix, the --model override only forwarded embedding_model/chat_model +
|
||||
// env, dropping config.provider_base_urls entirely — a probe against a
|
||||
// custom endpoint (e.g. a regional DashScope base URL) would silently
|
||||
// fall back to the recipe's hardcoded default endpoint and fail with a
|
||||
// misleading "Incorrect API key" error even though the key was valid for
|
||||
// the configured endpoint.
|
||||
let cfg: ReturnType<typeof loadConfig> | null = null;
|
||||
try {
|
||||
const cfg = loadConfig();
|
||||
cfg = loadConfig();
|
||||
const configuredModel = tpArg === 'embedding' ? cfg?.embedding_model : cfg?.chat_model;
|
||||
if (!configuredModel) {
|
||||
console.error(
|
||||
@@ -181,17 +200,27 @@ async function runTest(args: string[]): Promise<void> {
|
||||
}
|
||||
} catch { /* loadConfig throws when no brain configured — first-time install path; the no-config branch above handles it. */ }
|
||||
|
||||
// Reuse the SAME resolver the production path uses (buildGatewayConfig —
|
||||
// also used by cli.ts#connectEngine and init-embed-check.ts) so the probe
|
||||
// sees the identical base_urls / provider_chat_options / folded API keys
|
||||
// that a real `gbrain import`/`gbrain query` call would. Only the
|
||||
// touchpoint's model (+ embedding dims) is overridden on top, so an
|
||||
// isolated `--model` probe still targets exactly the requested model —
|
||||
// it just resolves that model's endpoint the way the brain actually
|
||||
// would. Falls back to bare env when no brain is configured yet (cfg is
|
||||
// null on first-time install, matching the old behavior for that case).
|
||||
const baseGatewayConfig = cfg ? buildGatewayConfig(cfg) : { env: { ...process.env } };
|
||||
if (tpArg === 'embedding') {
|
||||
const dims = recipe?.touchpoints.embedding?.default_dims ?? 1536;
|
||||
configureGateway({
|
||||
...baseGatewayConfig,
|
||||
embedding_model: modelArg,
|
||||
embedding_dimensions: dims,
|
||||
env: { ...process.env },
|
||||
});
|
||||
} else {
|
||||
configureGateway({
|
||||
...baseGatewayConfig,
|
||||
chat_model: modelArg,
|
||||
env: { ...process.env },
|
||||
});
|
||||
}
|
||||
void modelId; // intentionally unused but preserved for readability
|
||||
|
||||
@@ -0,0 +1,282 @@
|
||||
/**
|
||||
* `gbrain reindex-search-vector` — recreate FTS trigger functions and
|
||||
* backfill existing rows under the language configured via
|
||||
* GBRAIN_FTS_LANGUAGE.
|
||||
*
|
||||
* Why this command exists: schema migration v123 (configurable_fts_language)
|
||||
* stamps the trigger functions with the configured language at first apply.
|
||||
* After that, changing the env var has no effect on the write side because
|
||||
* v123 already shows as "applied" — the migrations runner will skip it.
|
||||
* This command is the documented escape hatch: it re-runs the same
|
||||
* recreate-and-backfill logic v123 uses, gated on an explicit user
|
||||
* action so the operation is intentional and visible (writes touch
|
||||
* every row in pages and content_chunks).
|
||||
*
|
||||
* Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces
|
||||
* the same trigger function bodies and the same tokenized vectors.
|
||||
*
|
||||
* Flags:
|
||||
* --dry-run Show what would happen, exit 0 without touching DB.
|
||||
* --yes Skip interactive [y/N]. Required for non-TTY (including --json).
|
||||
* --json Machine-readable result envelope. Does NOT imply --yes.
|
||||
*
|
||||
* Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE)
|
||||
* so a large brain never holds one giant row lock, and streams progress
|
||||
* through the shared reporter (stderr; stdout stays clean for --json).
|
||||
*
|
||||
* Cost: trigger recreate is sub-millisecond. Backfill is one tsvector
|
||||
* rebuild per page + per chunk. On a 20K-page brain with 80K chunks,
|
||||
* expect ~5-15s depending on Postgres CPU and content size.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { getFtsLanguage } from '../core/fts-language.ts';
|
||||
import { createInterface } from 'readline';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export interface ReindexSearchVectorOpts {
|
||||
dryRun?: boolean;
|
||||
yes?: boolean;
|
||||
json?: boolean;
|
||||
}
|
||||
|
||||
export interface ReindexSearchVectorResult {
|
||||
status: 'ok' | 'dry_run' | 'cancelled';
|
||||
language: string;
|
||||
pagesUpdated: number;
|
||||
chunksUpdated: number;
|
||||
triggersRecreated: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
interface CountRow {
|
||||
pages: number;
|
||||
chunks: number;
|
||||
}
|
||||
|
||||
/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */
|
||||
export const BACKFILL_BATCH_SIZE = 5000;
|
||||
|
||||
/**
|
||||
* Keyset-batched UPDATE: applies `setClause` to `table` rows where
|
||||
* search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking
|
||||
* the shared progress reporter after each batch. Terminates when a batch
|
||||
* returns fewer rows than the batch size (or none).
|
||||
*/
|
||||
async function batchedBackfill(
|
||||
engine: BrainEngine,
|
||||
table: 'pages' | 'content_chunks',
|
||||
setClause: string,
|
||||
tick: (n: number) => void
|
||||
): Promise<void> {
|
||||
let cursor = 0;
|
||||
for (;;) {
|
||||
const rows = await engine.executeRaw<{ id: number }>(`
|
||||
UPDATE ${table} SET ${setClause}
|
||||
WHERE id IN (
|
||||
SELECT id FROM ${table}
|
||||
WHERE search_vector IS NOT NULL AND id > ${cursor}
|
||||
ORDER BY id
|
||||
LIMIT ${BACKFILL_BATCH_SIZE}
|
||||
)
|
||||
RETURNING id
|
||||
`);
|
||||
if (rows.length === 0) break;
|
||||
tick(rows.length);
|
||||
cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor);
|
||||
if (rows.length < BACKFILL_BATCH_SIZE) break;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Programmatic entrypoint — takes a typed opts object. Used by tests and
|
||||
* future internal callers. The CLI wrapper is `runReindexSearchVectorCli`
|
||||
* defined at the bottom of this file.
|
||||
*/
|
||||
export async function runReindexSearchVector(
|
||||
engine: BrainEngine,
|
||||
opts: ReindexSearchVectorOpts
|
||||
): Promise<ReindexSearchVectorResult> {
|
||||
const lang = getFtsLanguage();
|
||||
const startedAt = Date.now();
|
||||
|
||||
// Inventory: how many rows will the backfill touch?
|
||||
const counts = await engine.executeRaw<CountRow>(
|
||||
`SELECT
|
||||
(SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages,
|
||||
(SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks`
|
||||
);
|
||||
const pagesCount = counts[0]?.pages ?? 0;
|
||||
const chunksCount = counts[0]?.chunks ?? 0;
|
||||
|
||||
if (opts.dryRun) {
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'dry_run',
|
||||
language: lang,
|
||||
pagesUpdated: pagesCount,
|
||||
chunksUpdated: chunksCount,
|
||||
triggersRecreated: 0,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`);
|
||||
console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`);
|
||||
console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Confirm unless --yes. --json does NOT bypass the gate — a machine
|
||||
// caller must pass --yes explicitly (mirrors reindex-code, #1784).
|
||||
if (!opts.yes) {
|
||||
if (!process.stdin.isTTY) {
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify({
|
||||
error: {
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'reindex_requires_yes',
|
||||
message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to preview.',
|
||||
},
|
||||
language: lang,
|
||||
pages: pagesCount,
|
||||
chunks: chunksCount,
|
||||
}));
|
||||
} else {
|
||||
console.error('Refusing to run without --yes in non-TTY environment.');
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
||||
const answer = await new Promise<string>(resolve => {
|
||||
rl.question(
|
||||
`Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `,
|
||||
resolve
|
||||
);
|
||||
});
|
||||
rl.close();
|
||||
|
||||
if (!/^y(es)?$/i.test(answer.trim())) {
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'cancelled',
|
||||
language: lang,
|
||||
pagesUpdated: 0,
|
||||
chunksUpdated: 0,
|
||||
triggersRecreated: 0,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log('Cancelled.');
|
||||
}
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
// Recreate trigger functions. The strings are intentionally identical to
|
||||
// the v124 migration body — keeping them in lockstep is the contract.
|
||||
// `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening:
|
||||
// CREATE OR REPLACE resets proconfig, so omitting it here would strip the
|
||||
// hardening from every brain that runs this command.
|
||||
//
|
||||
// #2704: compiled_truth (the unbounded whole-page body) is deliberately
|
||||
// NOT indexed here — it overflows Postgres's 1MB tsvector cap on large
|
||||
// pages, and content_chunks.search_vector (populated separately, chunk-
|
||||
// grain, well under the cap) is what searchKeyword() actually queries.
|
||||
// See migrate.ts's v124 for the full rationale; keep this copy in sync.
|
||||
const recreatePagesFn = `
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
const recreateChunksFn = `
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
await engine.executeRaw(recreatePagesFn);
|
||||
await engine.executeRaw(recreateChunksFn);
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// Backfill: UPDATE-to-self forces the pages trigger to re-fire
|
||||
// (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a
|
||||
// direct vector compute since the column itself is what we want.
|
||||
progress.start('reindex_search_vector.pages', pagesCount);
|
||||
await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n));
|
||||
progress.finish();
|
||||
|
||||
progress.start('reindex_search_vector.chunks', chunksCount);
|
||||
await batchedBackfill(
|
||||
engine,
|
||||
'content_chunks',
|
||||
`search_vector =
|
||||
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`,
|
||||
n => progress.tick(n)
|
||||
);
|
||||
progress.finish();
|
||||
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'ok',
|
||||
language: lang,
|
||||
pagesUpdated: pagesCount,
|
||||
chunksUpdated: chunksCount,
|
||||
triggersRecreated: 2,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`✅ Recreated 2 trigger functions with language='${lang}'`);
|
||||
console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector.
|
||||
* Matches the style of `reindex-code`: --dry-run, --yes/-y, --json.
|
||||
*
|
||||
* Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes.
|
||||
*/
|
||||
export async function runReindexSearchVectorCli(
|
||||
engine: BrainEngine,
|
||||
args: string[]
|
||||
): Promise<void> {
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const yes = args.includes('--yes') || args.includes('-y');
|
||||
const json = args.includes('--json');
|
||||
|
||||
await runReindexSearchVector(engine, { dryRun, yes, json });
|
||||
}
|
||||
+18
-12
@@ -105,13 +105,19 @@ function printHelp(): void {
|
||||
async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>, args: string[]): Promise<void> {
|
||||
const { json, timeoutMs } = parseFlags(args);
|
||||
|
||||
let submitted: { id: number; name: string; state: string };
|
||||
// submit_job / get_job return the MinionJob row verbatim — the lifecycle
|
||||
// field is `status` (src/core/minions/types.ts), not `state`. Reading
|
||||
// `state` here made every poll see `undefined`, so the terminal check
|
||||
// never matched and ping always exhausted its timeout (exit 1) even when
|
||||
// the cycle completed. The ping's own JSON *output* keys (`state`,
|
||||
// `last_state`) are kept as-is for consumers.
|
||||
let submitted: { id: number; name: string; status: string };
|
||||
try {
|
||||
const res = await callRemoteTool(config, 'submit_job', {
|
||||
name: 'autopilot-cycle',
|
||||
data: { phases: ['sync', 'extract', 'embed'] },
|
||||
});
|
||||
submitted = unpackToolResult<{ id: number; name: string; state: string }>(res);
|
||||
submitted = unpackToolResult<{ id: number; name: string; status: string }>(res);
|
||||
} catch (e) {
|
||||
return failPing(e, json);
|
||||
}
|
||||
@@ -122,43 +128,43 @@ async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>,
|
||||
|
||||
const startMs = Date.now();
|
||||
let attempt = 0;
|
||||
let lastState = submitted.state;
|
||||
let lastState = submitted.status;
|
||||
while (Date.now() - startMs < timeoutMs) {
|
||||
const elapsed = Date.now() - startMs;
|
||||
const intervalMs = elapsed < 30_000 ? 1_000 : elapsed < 5 * 60_000 + 30_000 ? 5_000 : 10_000;
|
||||
await sleep(intervalMs);
|
||||
attempt++;
|
||||
|
||||
let job: { id: number; state: string; failed_reason?: string };
|
||||
let job: { id: number; status: string; failed_reason?: string };
|
||||
try {
|
||||
const res = await callRemoteTool(config, 'get_job', { id: submitted.id });
|
||||
job = unpackToolResult<{ id: number; state: string; failed_reason?: string }>(res);
|
||||
job = unpackToolResult<{ id: number; status: string; failed_reason?: string }>(res);
|
||||
} catch (e) {
|
||||
// Network blip mid-poll: log and keep going. Surface only if persistent.
|
||||
if (!json) console.error(` poll #${attempt} failed (${e instanceof Error ? e.message : String(e)}); continuing...`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (job.state !== lastState) {
|
||||
lastState = job.state;
|
||||
if (!json) console.error(` job #${submitted.id} → ${job.state}`);
|
||||
if (job.status !== lastState) {
|
||||
lastState = job.status;
|
||||
if (!json) console.error(` job #${submitted.id} → ${job.status}`);
|
||||
}
|
||||
|
||||
const terminal = ['completed', 'failed', 'dead', 'cancelled'];
|
||||
if (terminal.includes(job.state)) {
|
||||
const ok = job.state === 'completed';
|
||||
if (terminal.includes(job.status)) {
|
||||
const ok = job.status === 'completed';
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
status: ok ? 'success' : 'error',
|
||||
job_id: submitted.id,
|
||||
state: job.state,
|
||||
state: job.status,
|
||||
...(job.failed_reason ? { failed_reason: job.failed_reason } : {}),
|
||||
elapsed_ms: Date.now() - startMs,
|
||||
}));
|
||||
} else {
|
||||
console.log(ok
|
||||
? `\nautopilot-cycle complete (${Math.round((Date.now() - startMs) / 1000)}s).`
|
||||
: `\nautopilot-cycle ended ${job.state}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
|
||||
: `\nautopilot-cycle ended ${job.status}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
|
||||
}
|
||||
process.exit(ok ? 0 : 1);
|
||||
}
|
||||
|
||||
@@ -48,7 +48,7 @@ import {
|
||||
} from '../core/schema-pack/index.ts';
|
||||
import type { SchemaPackManifest, PackPrimitive } from '../core/schema-pack/manifest-v1.ts';
|
||||
import { PACK_PRIMITIVES } from '../core/schema-pack/manifest-v1.ts';
|
||||
import { gbrainPath, loadConfig, configPath } from '../core/config.ts';
|
||||
import { gbrainPath, loadConfig, configPath, toEngineConfig } from '../core/config.ts';
|
||||
|
||||
export async function runSchema(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
@@ -434,16 +434,12 @@ function parseFlags(args: string[]): ParsedFlags {
|
||||
|
||||
async function withConnectedEngine<T>(fn: (engine: import('../core/engine.ts').BrainEngine) => Promise<T>): Promise<T> {
|
||||
const { createEngine } = await import('../core/engine-factory.ts');
|
||||
const cfg = loadConfig() ?? {};
|
||||
const engineKind = (cfg as { engine?: string }).engine === 'postgres' ? 'postgres' : 'pglite';
|
||||
const cfg = loadConfig() ?? { engine: 'pglite' as const };
|
||||
// PR #1321 (closed) defensive fix retained: build the EngineConfig once and
|
||||
// pass it to BOTH createEngine and engine.connect. The factory captures
|
||||
// config at construction; explicit re-pass at connect() is defense in depth
|
||||
// against future engine implementations that read URL from connect-time.
|
||||
const connectConfig: import('../core/types.ts').EngineConfig = {
|
||||
engine: engineKind,
|
||||
database_url: (cfg as { database_url?: string }).database_url,
|
||||
};
|
||||
const connectConfig = toEngineConfig(cfg);
|
||||
const engine = await createEngine(connectConfig);
|
||||
await engine.connect(connectConfig);
|
||||
try {
|
||||
|
||||
@@ -22,6 +22,7 @@ import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/
|
||||
import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
|
||||
import { mcpAuthRouter } from '@modelcontextprotocol/sdk/server/auth/router.js';
|
||||
import { requireBearerAuth } from '@modelcontextprotocol/sdk/server/auth/middleware/bearerAuth.js';
|
||||
import { OAuthTokenRevocationRequestSchema } from '@modelcontextprotocol/sdk/shared/auth.js';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { operations, OperationError } from '../core/operations.ts';
|
||||
import type { OperationContext, AuthInfo } from '../core/operations.ts';
|
||||
@@ -37,6 +38,7 @@ import { VERSION } from '../version.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { sqlQueryForEngine, executeRawJsonb } from '../core/sql-query.ts';
|
||||
import { MinionQueue } from '../core/minions/queue.ts';
|
||||
import { isRetryableError } from '../core/retry-matcher.ts';
|
||||
import {
|
||||
computeContentHash,
|
||||
validateIngestionEvent,
|
||||
@@ -363,6 +365,42 @@ export interface AgentClientSpend {
|
||||
inflight_count: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* `/admin/api/sources` source list — the input rows for buildSyncStatusReport.
|
||||
*
|
||||
* Queries the JSONB config column directly (listSources doesn't carry it,
|
||||
* but buildSyncStatusReport needs syncEnabled / strategy fields).
|
||||
*
|
||||
* Deliberately does NOT filter on local_path: in a push-only deployment
|
||||
* (content arrives via MCP put_page / capture / ingest, not `gbrain sync`
|
||||
* of a server checkout) every source has a null local_path — filtering on
|
||||
* it would empty both the Sources tab AND the federation source-picker.
|
||||
* buildSyncStatusReport does no disk I/O, so null-local_path sources
|
||||
* report fine (pages/chunks from SQL, staleness 'unknown' / never-synced).
|
||||
*/
|
||||
export async function queryAdminSources(engine: BrainEngine): Promise<
|
||||
Array<{ id: string; name: string; local_path: string | null; config: Record<string, unknown> }>
|
||||
> {
|
||||
const rows = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown> | string | null;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources
|
||||
WHERE archived IS NOT TRUE
|
||||
ORDER BY id`,
|
||||
);
|
||||
return rows.map((r) => ({
|
||||
id: r.id,
|
||||
name: r.name,
|
||||
local_path: r.local_path,
|
||||
config: typeof r.config === 'string'
|
||||
? (JSON.parse(r.config) as Record<string, unknown>)
|
||||
: (r.config ?? {}),
|
||||
}));
|
||||
}
|
||||
|
||||
export async function queryAgentClientSpend(engine: BrainEngine): Promise<AgentClientSpend[]> {
|
||||
const sql = sqlQueryForEngine(engine);
|
||||
const rows = await sql`
|
||||
@@ -745,6 +783,93 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
}
|
||||
});
|
||||
|
||||
// The SDK's /revoke handler compares the presented secret with
|
||||
// client.client_secret as plaintext. GBrain stores only a SHA-256 hash, so
|
||||
// confidential clients need the same hash-aware validation used above for
|
||||
// authorization_code and refresh_token exchanges. Public clients present no
|
||||
// secret and continue through to the SDK's PKCE-compatible handler.
|
||||
app.post('/revoke', ccRateLimiter, express.urlencoded({ extended: false }), async (req, res, next) => {
|
||||
res.setHeader('Cache-Control', 'no-store');
|
||||
|
||||
const rawClientId: unknown = req.body?.client_id;
|
||||
const rawBodySecret: unknown = req.body?.client_secret;
|
||||
const authHeader = (req.headers.authorization ?? '').toString();
|
||||
|
||||
// RFC 6749 §2.3: one client-authentication method per request. Reject
|
||||
// duplicates/arrays from express.urlencoded rather than letting them reach
|
||||
// hashToken() as non-strings and become a misleading invalid_client error.
|
||||
const hasBasicAuth = /^Basic\b/i.test(authHeader);
|
||||
if (
|
||||
(rawClientId !== undefined && typeof rawClientId !== 'string') ||
|
||||
(rawBodySecret !== undefined && typeof rawBodySecret !== 'string') ||
|
||||
(hasBasicAuth && (rawClientId !== undefined || rawBodySecret !== undefined))
|
||||
) {
|
||||
res.status(400).json({ error: 'invalid_request', error_description: 'Malformed or mixed client authentication' });
|
||||
return;
|
||||
}
|
||||
|
||||
let clientId = typeof rawClientId === 'string' ? rawClientId : undefined;
|
||||
let presentedSecret = typeof rawBodySecret === 'string' && rawBodySecret.length > 0
|
||||
? rawBodySecret
|
||||
: undefined;
|
||||
if (hasBasicAuth) {
|
||||
try {
|
||||
const match = authHeader.match(/^Basic\s+([^\s]+)$/i);
|
||||
if (!match) throw new Error('Malformed Basic authentication');
|
||||
const decoded = Buffer.from(match[1], 'base64').toString('utf8');
|
||||
const idx = decoded.indexOf(':');
|
||||
if (idx < 1) throw new Error('Malformed Basic authentication');
|
||||
clientId = decodeURIComponent(decoded.slice(0, idx).replace(/\+/g, ' '));
|
||||
presentedSecret = decodeURIComponent(decoded.slice(idx + 1).replace(/\+/g, ' '));
|
||||
if (!presentedSecret) throw new Error('Malformed Basic authentication');
|
||||
} catch {
|
||||
res.setHeader('WWW-Authenticate', 'Basic realm="gbrain"');
|
||||
res.status(401).json({ error: 'invalid_client', error_description: 'Invalid client' });
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (!clientId || !presentedSecret) return next();
|
||||
|
||||
const parsedRequest = OAuthTokenRevocationRequestSchema.safeParse(req.body);
|
||||
if (!parsedRequest.success || parsedRequest.data.token.length === 0) {
|
||||
res.status(400).json({ error: 'invalid_request', error_description: 'Valid token required' });
|
||||
return;
|
||||
}
|
||||
|
||||
let client;
|
||||
try {
|
||||
client = await oauthProvider.verifyConfidentialClientSecret(clientId, presentedSecret);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : '';
|
||||
if (msg === 'Invalid client' || msg === 'Client has been revoked') {
|
||||
if (hasBasicAuth) res.setHeader('WWW-Authenticate', 'Basic realm="gbrain"');
|
||||
res.status(401).json({ error: 'invalid_client', error_description: 'Invalid client' });
|
||||
return;
|
||||
}
|
||||
console.error('[serve-http] revoke client verification failed:', msg || 'Unknown error');
|
||||
const retryable = isRetryableError(e);
|
||||
res.status(retryable ? 503 : 500).json({
|
||||
error: retryable ? 'temporarily_unavailable' : 'server_error',
|
||||
error_description: retryable ? 'Token revocation temporarily unavailable' : 'Token revocation failed',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
await oauthProvider.revokeToken(client, parsedRequest.data);
|
||||
// RFC 7009 §2.2: successful revocation, including an unknown token, is 200.
|
||||
res.status(200).end();
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : 'Unknown error';
|
||||
console.error('[serve-http] token revocation failed:', msg);
|
||||
const retryable = isRetryableError(e);
|
||||
res.status(retryable ? 503 : 500).json({
|
||||
error: retryable ? 'temporarily_unavailable' : 'server_error',
|
||||
error_description: retryable ? 'Token revocation temporarily unavailable' : 'Token revocation failed',
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// MCP SDK Auth Router (OAuth endpoints)
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -796,6 +921,16 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
if (body?.grant_types_supported && !body.grant_types_supported.includes('client_credentials')) {
|
||||
body.grant_types_supported.push('client_credentials');
|
||||
}
|
||||
if (body?.token_endpoint_auth_methods_supported) {
|
||||
for (const method of ['client_secret_basic', 'none']) {
|
||||
if (!body.token_endpoint_auth_methods_supported.includes(method)) {
|
||||
body.token_endpoint_auth_methods_supported.push(method);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (body?.revocation_endpoint_auth_methods_supported && !body.revocation_endpoint_auth_methods_supported.includes('client_secret_basic')) {
|
||||
body.revocation_endpoint_auth_methods_supported.push('client_secret_basic');
|
||||
}
|
||||
return origJson(body);
|
||||
};
|
||||
}
|
||||
@@ -1412,6 +1547,111 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Sources tab — read-only view of registered sources with sync + embed
|
||||
// coverage stats. Drives the admin SPA's `Sources` page.
|
||||
//
|
||||
// Returns the same shape `gbrain sources status --json` prints, so the
|
||||
// SPA stays in lockstep with the CLI surface.
|
||||
// ---------------------------------------------------------------------------
|
||||
app.get('/admin/api/sources', requireAdmin, async (_req: Request, res: Response) => {
|
||||
try {
|
||||
const { buildSyncStatusReport } = await import('./sync.ts');
|
||||
const report = await buildSyncStatusReport(engine, await queryAdminSources(engine));
|
||||
res.json(report);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
res.status(503).json({ error: 'service_unavailable', detail: msg });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Federated-read management (admin-side counterparts of the CLI commands
|
||||
// `gbrain auth grant-read / revoke-read / set-federated-read`). All three
|
||||
// route through the same *Core helpers as the CLI so race-safety,
|
||||
// soft-delete filter, and source-id shape validation apply uniformly.
|
||||
//
|
||||
// The admin SPA's `Agents` page renders "Manage reads" actions per
|
||||
// client backed by these endpoints.
|
||||
// ---------------------------------------------------------------------------
|
||||
app.get('/admin/api/agents/federated-read', requireAdmin, async (_req: Request, res: Response) => {
|
||||
try {
|
||||
const rows = await sql`
|
||||
SELECT client_id, client_name, source_id, federated_read
|
||||
FROM oauth_clients
|
||||
WHERE deleted_at IS NULL
|
||||
ORDER BY client_name
|
||||
`;
|
||||
const clients = rows.map((r) => ({
|
||||
client_id: String(r.client_id),
|
||||
client_name: String(r.client_name),
|
||||
source_id: r.source_id == null ? null : String(r.source_id),
|
||||
federated_read: Array.isArray(r.federated_read)
|
||||
? (r.federated_read as string[]).map(String)
|
||||
: [],
|
||||
}));
|
||||
res.json({ clients });
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
res.status(503).json({ error: 'service_unavailable', detail: msg });
|
||||
}
|
||||
});
|
||||
|
||||
app.post('/admin/api/agents/:clientId/grant-read', requireAdmin, express.json(), async (req: Request, res: Response) => {
|
||||
const clientId = String(req.params.clientId ?? '');
|
||||
const sourceId = String(req.body?.source_id ?? '').trim();
|
||||
if (!clientId || !sourceId) {
|
||||
res.status(400).json({ error: 'invalid_request', detail: 'clientId path param + source_id body required' });
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const { grantReadCore } = await import('./auth.ts');
|
||||
const outcome = await grantReadCore(sql, clientId, sourceId);
|
||||
res.json({ outcome });
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
res.status(400).json({ error: 'mutation_failed', detail: msg });
|
||||
}
|
||||
});
|
||||
|
||||
app.post('/admin/api/agents/:clientId/revoke-read', requireAdmin, express.json(), async (req: Request, res: Response) => {
|
||||
const clientId = String(req.params.clientId ?? '');
|
||||
const sourceId = String(req.body?.source_id ?? '').trim();
|
||||
if (!clientId || !sourceId) {
|
||||
res.status(400).json({ error: 'invalid_request', detail: 'clientId path param + source_id body required' });
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const { revokeReadCore } = await import('./auth.ts');
|
||||
const outcome = await revokeReadCore(sql, clientId, sourceId);
|
||||
res.json({ outcome });
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
res.status(400).json({ error: 'mutation_failed', detail: msg });
|
||||
}
|
||||
});
|
||||
|
||||
app.post('/admin/api/agents/:clientId/set-federated-read', requireAdmin, express.json(), async (req: Request, res: Response) => {
|
||||
const clientId = String(req.params.clientId ?? '');
|
||||
const rawIds = req.body?.source_ids;
|
||||
if (!clientId || !Array.isArray(rawIds)) {
|
||||
res.status(400).json({ error: 'invalid_request', detail: 'clientId path param + source_ids[] body required' });
|
||||
return;
|
||||
}
|
||||
// Encode the array as CSV so the same setFederatedReadCore signature
|
||||
// (string CSV input) the CLI uses applies here. Empty array → empty
|
||||
// string → clears the list.
|
||||
const csv = rawIds.map((s) => String(s).trim()).filter(Boolean).join(',');
|
||||
try {
|
||||
const { setFederatedReadCore } = await import('./auth.ts');
|
||||
const outcome = await setFederatedReadCore(sql, clientId, csv);
|
||||
res.json({ outcome });
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
res.status(400).json({ error: 'mutation_failed', detail: msg });
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// SSE live activity feed
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
+18
-5
@@ -7,7 +7,9 @@
|
||||
* full story.
|
||||
*
|
||||
* Subcommands:
|
||||
* gbrain sources add <id> --path <path> [--name <display>] [--federated|--no-federated]
|
||||
* gbrain sources add <id> --path <path> [--name <display>] [--federated|--no-federated] [--force]
|
||||
* --path must be a git-initialized repo (files committed,
|
||||
* not just present) — #2707. --force skips the check.
|
||||
* gbrain sources list [--json]
|
||||
* gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
|
||||
* gbrain sources rename <id> <new-name>
|
||||
@@ -120,7 +122,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
if (!id) {
|
||||
console.error(
|
||||
'Usage: gbrain sources add <id> [--path <path> | --url <https-url>] ' +
|
||||
'[--name <display>] [--federated|--no-federated] [--clone-dir <path>]',
|
||||
'[--name <display>] [--federated|--no-federated] [--clone-dir <path>] [--force]',
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
@@ -132,6 +134,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
let cloneDir: string | undefined;
|
||||
let patFile: string | undefined;
|
||||
let noHarden = false;
|
||||
let force = false;
|
||||
|
||||
for (let i = 1; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
@@ -143,6 +146,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
if (a === '--clone-dir') { cloneDir = args[++i]; continue; }
|
||||
if (a === '--pat-file') { patFile = args[++i]; continue; }
|
||||
if (a === '--no-harden') { noHarden = true; continue; }
|
||||
if (a === '--force') { force = true; continue; }
|
||||
console.error(`Unknown flag: ${a}`);
|
||||
process.exit(2);
|
||||
}
|
||||
@@ -162,6 +166,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
remoteUrl,
|
||||
federated,
|
||||
cloneDir,
|
||||
force,
|
||||
});
|
||||
|
||||
// Topology A discovery: if the just-added source carries a brain-resident
|
||||
@@ -1178,7 +1183,14 @@ async function runAudit(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
// frontmatter.type and estimates per-page segment count from body
|
||||
// bytes. Estimated per-segment Sonnet cost is a rough heuristic
|
||||
// (~2000 in + 500 out tokens at $3/MTok in + $15/MTok out ≈ $0.013).
|
||||
const FACTS_BACKFILL_ALLOWED = ['conversation', 'meeting', 'slack', 'email'];
|
||||
const FACTS_BACKFILL_ALLOWED = [
|
||||
'conversation',
|
||||
'meeting',
|
||||
'slack',
|
||||
'email',
|
||||
'imessage',
|
||||
'imessage-daily',
|
||||
];
|
||||
const FACTS_BACKFILL_CHARS_PER_SEGMENT = 6500; // matches SEGMENT_TEXT_CHAR_LIMIT
|
||||
const FACTS_BACKFILL_USD_PER_SEGMENT = 0.013;
|
||||
let factsBackfillPages = 0;
|
||||
@@ -1368,8 +1380,9 @@ function printHelp(): void {
|
||||
console.log(`gbrain sources — manage multi-source brain configuration (v0.26.5)
|
||||
|
||||
Subcommands:
|
||||
add <id> --path <p> [--name <n>] [--federated|--no-federated]
|
||||
Register a new source.
|
||||
add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
|
||||
Register a new source. --path must be a git repo
|
||||
with committed files; --force skips that check.
|
||||
list [--json] List registered sources with page counts.
|
||||
remove <id> [--confirm-destructive] [--dry-run]
|
||||
Permanently delete a source and all its data.
|
||||
|
||||
+611
-58
@@ -1,4 +1,4 @@
|
||||
import { existsSync, readFileSync, writeFileSync, statSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync, statSync, realpathSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
@@ -9,6 +9,7 @@ import { createInterface } from 'readline';
|
||||
import {
|
||||
isSyncable,
|
||||
unsyncableReason,
|
||||
matchesAnyGlob,
|
||||
resolveSlugForPath,
|
||||
unacknowledgedSyncFailures,
|
||||
acknowledgeFailures,
|
||||
@@ -742,6 +743,27 @@ export interface SyncOpts {
|
||||
sourceId?: string;
|
||||
/** Multi-repo: sync strategy override (markdown, code, auto). */
|
||||
strategy?: 'markdown' | 'code' | 'auto';
|
||||
/**
|
||||
* #753/#774 — sync only files under this subdirectory of the git repo.
|
||||
* Git operations (pull, diff, rev-parse) still run against the repo root
|
||||
* (discovered via `git rev-parse --show-toplevel`); file walking, imports,
|
||||
* deletes and renames are scoped to the subpath. Slugs are git-root-relative
|
||||
* (`wiki/page1.md` → slug `wiki/page1`) so full and incremental syncs of
|
||||
* the same scope agree. Enables N logical sources in one git repo.
|
||||
*
|
||||
* SECURITY (NAV-1/NAV-2): the resolved subpath must realpath-resolve inside
|
||||
* the git root — `../escape` and symlinked subdirs pointing outside the repo
|
||||
* are rejected before any git op runs.
|
||||
*/
|
||||
srcSubpath?: string;
|
||||
/**
|
||||
* #753/#774 — glob patterns for files to exclude from sync (repeatable
|
||||
* `--exclude` on the CLI). Matched against the scope-relative path in both
|
||||
* the full-sync and incremental paths. Excluded files are never imported;
|
||||
* exclusion does NOT delete previously-imported pages (conservative,
|
||||
* matching the #1433 metafile posture).
|
||||
*/
|
||||
exclude?: string[];
|
||||
/**
|
||||
* Number of parallel workers for the import phase. When > 1, each worker
|
||||
* gets its own small Postgres connection pool and files are dispatched via
|
||||
@@ -897,14 +919,210 @@ export function buildAutoEmbedArgs(slugs: string[], sourceId?: string): string[]
|
||||
* 100 MiB is generous but still bounded — a 100K-file diff with long
|
||||
* paths tops out around 10–20 MiB in practice.
|
||||
*/
|
||||
function git(repoPath: string, args: string[], configs: string[] = []): string {
|
||||
function git(repoPath: string, args: string[], configs: string[] = [], timeoutMs = 30000): string {
|
||||
return execFileSync('git', buildGitInvocation(repoPath, args, configs), {
|
||||
encoding: 'utf-8',
|
||||
timeout: 30000,
|
||||
timeout: timeoutMs,
|
||||
maxBuffer: 100 * 1024 * 1024,
|
||||
}).trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* #753/#774: walk up from inputPath to the nearest git repo root via
|
||||
* `git -C <path> rev-parse --show-toplevel`. Handles worktrees and submodules
|
||||
* natively (git itself resolves them). Throws a user-friendly error when no
|
||||
* git repo is found.
|
||||
*/
|
||||
export function discoverGitRoot(inputPath: string): string {
|
||||
try {
|
||||
return git(inputPath, ['rev-parse', '--show-toplevel']);
|
||||
} catch {
|
||||
throw new Error(
|
||||
`Not inside a git repository: ${inputPath}. GBrain sync requires a git-initialized repo (or a subdirectory of one).`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #2964: snapshot the CURRENT on-disk state of a gbrain-owned brain dir as
|
||||
* a baseline commit — used both right after a self-healing `git init` (no
|
||||
* `.git` at all) and to recover a repo left with `.git` but zero commits
|
||||
* (an interrupted prior self-heal, or a `git init` from some other source
|
||||
* that never got a first commit). Respects `.gitignore` (written first) so
|
||||
* future incremental syncs diff against what's actually here rather than
|
||||
* an empty tree — an empty initial commit would make every existing file
|
||||
* look "added" again on the next sync, even though the full-sync pass that
|
||||
* follows already imported them from disk directly.
|
||||
*
|
||||
* `--no-gpg-sign` + explicit `-c user.name/user.email`: this runs from a
|
||||
* headless nightly cron/launchd invocation, which has no reason to have
|
||||
* git signing/identity configured, and must not block on an unavailable
|
||||
* signing agent or pinentry prompt.
|
||||
*
|
||||
* db_only exclusion is recomputed directly and passed to `git add` as
|
||||
* negative pathspecs, rather than relying solely on `manageGitignore`
|
||||
* having written `.gitignore` successfully: that helper is deliberately
|
||||
* best-effort (a broken gbrain.yml parse, or an unwritable .gitignore,
|
||||
* only warns and returns — the right default for its OTHER callers, where
|
||||
* .gitignore management is a side effect that must never kill the sync
|
||||
* job). For a commit we are about to create ourselves, "fail open" there
|
||||
* would mean silently committing db_only content into git history. Fail
|
||||
* closed instead: db_only exclusion doesn't depend on the .gitignore
|
||||
* write having succeeded. `loadStorageConfig` throwing (unreadable
|
||||
* gbrain.yml, or a semantic overlap) propagates — better to leave this
|
||||
* self-heal wedged with a clear error than commit unknown content.
|
||||
*/
|
||||
function createSyncBaselineCommit(repoPath: string): void {
|
||||
// #2964: db_only exclusion is computed directly from loadStorageConfig
|
||||
// and passed to `git add` as pathspecs — deliberately NOT via
|
||||
// manageGitignore/.gitignore, for two independent reasons:
|
||||
//
|
||||
// 1. Ordering (Codex review round 6, P1): `collectSyncableFiles` — the
|
||||
// file enumeration `performFullSync` runs right after this function
|
||||
// returns — honors `.gitignore` via `git ls-files --exclude-standard`.
|
||||
// Writing db_only entries into `.gitignore` BEFORE that first import
|
||||
// would silently exclude those pages from the database entirely.
|
||||
// That's the exact bug class `runSync`'s existing "manage .gitignore
|
||||
// ONLY on successful sync" ordering (this file, `manageGitignoreAtGitRoot`
|
||||
// callers below — itself a prior Codex P1 fix) exists to prevent. Leave
|
||||
// `.gitignore` untouched here; the existing post-sync flow writes it
|
||||
// once this sync completes, same as it does for every other sync.
|
||||
// 2. Fail-closed (rounds 5-6): `manageGitignore`'s "warn and return" on a
|
||||
// broken gbrain.yml/unwritable .gitignore is the right default for its
|
||||
// OTHER callers (a side effect that must never kill the sync job), but
|
||||
// wrong for a commit we are creating ourselves — silently committing
|
||||
// db_only content into git history.
|
||||
const storageConfig = loadStorageConfig(repoPath);
|
||||
const dbOnlyDirs = storageConfig?.db_only ?? [];
|
||||
// Sniff-test fail-closed (round 6, P2): `loadStorageConfig` warns-and-
|
||||
// returns an EMPTY config for syntactically-valid-but-unsupported YAML
|
||||
// (e.g. flow-style `db_only: [dir/]` — the narrow custom parser only
|
||||
// handles block-style lists), which would silently resolve zero
|
||||
// exclusions from a file that clearly intended some. If gbrain.yml
|
||||
// exists and mentions db_only (or its deprecated pre-v0.22.11 alias
|
||||
// `supabase_only` — same keep-out-of-git semantics, still a supported
|
||||
// backward-compat key per storage-config.ts) but nothing resolved from
|
||||
// it, refuse rather than guess "genuinely empty" vs "syntax ignored".
|
||||
//
|
||||
// Known false-positive (round 8 review): a genuinely, intentionally
|
||||
// empty `db_only: []` mentioning the word also refuses, and can't be
|
||||
// told apart from the unsupported-syntax case — `loadStorageConfig`
|
||||
// returns the IDENTICAL `{db_tracked:[],db_only:[]}` for both (verified
|
||||
// directly: flow-style `[dir/]` and literal `[]` both collapse to that
|
||||
// same shape). Distinguishing them would mean teaching this function
|
||||
// about the parser's internal line-recognition rules, which belongs in
|
||||
// storage-config.ts, not here. Accepted trade-off: the false-positive
|
||||
// cost is low and self-resolving (the brain stays wedged with a clear,
|
||||
// actionable error until the user drops the pointless empty stanza or
|
||||
// fixes their syntax; retried on every subsequent sync); the
|
||||
// false-negative this guards against — silently committing db_only
|
||||
// content into permanent git history — is high-cost and hard to undo.
|
||||
if (dbOnlyDirs.length === 0) {
|
||||
const yamlPath = join(repoPath, 'gbrain.yml');
|
||||
const yamlContent = existsSync(yamlPath) ? readFileSync(yamlPath, 'utf-8') : '';
|
||||
// A YAML KEY line (`db_only:` / `supabase_only:`, ignoring leading
|
||||
// whitespace and `#` comments), not a bare substring search — round 9,
|
||||
// P2: a comment or unrelated prose value that happens to mention the
|
||||
// word (e.g. `# db_only handling TBD`) must not trip this guard on an
|
||||
// otherwise-genuinely-config-free gbrain.yml.
|
||||
const mentionsUnresolvedKey = yamlContent.split('\n').some((line) => {
|
||||
const trimmed = line.trim();
|
||||
return !trimmed.startsWith('#') && /^(db_only|supabase_only)\s*:/.test(trimmed);
|
||||
});
|
||||
if (mentionsUnresolvedKey) {
|
||||
throw new Error(
|
||||
`${yamlPath} mentions db_only but no directories resolved from it — refusing to ` +
|
||||
`auto-commit (cannot tell "genuinely empty" from "unsupported syntax silently ignored"). ` +
|
||||
`Fix gbrain.yml's storage.db_only syntax, or git-init this directory manually.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// #2964 (round 9, P1): every db_only dir is ALWAYS pathspec-excluded,
|
||||
// unconditionally — never pre-filtered against what an existing
|
||||
// `.gitignore` claims to already cover. An earlier version checked
|
||||
// `git check-ignore -q dir` first and skipped the pathspec when it
|
||||
// already reported "ignored" (to dodge the advisory error below), but
|
||||
// `check-ignore` on a directory can say "ignored" even when a
|
||||
// pre-existing `.gitignore` re-includes a child via negation (e.g.
|
||||
// `private-cache/*` + `!private-cache/index.md`) — the filter would
|
||||
// then skip excluding it via pathspec, and `git add -A` would stage
|
||||
// that re-included child despite the whole directory being declared
|
||||
// db_only. Our OWN pathspec exclusion is unconditional and doesn't
|
||||
// consult `.gitignore` at all, so it can't be defeated by ANY
|
||||
// .gitignore content, negated or not. `:(exclude,literal)dir` (not the
|
||||
// `:!dir` shorthand) so a db_only dir name that itself starts with a
|
||||
// pathspec magic character like `:` is excluded literally rather than
|
||||
// reinterpreted (round 9, P2).
|
||||
const excludePathspecs = dbOnlyDirs.map((dir) => `:(exclude,literal)${dir}`);
|
||||
// Clear the index before staging (round 6, P1): the unborn-HEAD
|
||||
// recovery site can reach this function with a repo whose index
|
||||
// already has entries staged from some OTHER prior operation (a manual
|
||||
// `git add`, an interrupted workflow) before gbrain ever touched it.
|
||||
// `add -A` only adds/updates — it does not drop an already-staged path
|
||||
// that our exclusion pathspecs above now want excluded. `read-tree
|
||||
// --empty` resets the index without touching the working tree; a
|
||||
// no-op on a freshly-`git init`-ed repo, whose index is already empty.
|
||||
git(repoPath, ['read-tree', '--empty']);
|
||||
try {
|
||||
// #2964: 10 minutes, not the shared git() helper's 30s default — this
|
||||
// full-tree `git add -A` walks a legacy brain that may hold years of
|
||||
// accumulated content. A 30s timeout would abort staging after `git
|
||||
// init` already created `.git`, leaving an unborn repo that every
|
||||
// subsequent sync would retry (and time out identically) forever;
|
||||
// the unborn-HEAD recovery path exists for OTHER causes of that
|
||||
// state, not to be this one's normal first outcome.
|
||||
git(repoPath, ['add', '-A', '--', '.', ...excludePathspecs], [], 600_000);
|
||||
} catch (err) {
|
||||
// Now that exclusion is always applied (never pre-filtered), an
|
||||
// explicit pathspec exclusion for a path a pre-existing `.gitignore`
|
||||
// ALSO happens to cover trips git's advice.addIgnoredFile: nonzero
|
||||
// exit + "paths ignored by one of your .gitignore files, use -f",
|
||||
// even though the add otherwise fully succeeded (verified directly:
|
||||
// `git status --short` right after this exact error shows every
|
||||
// non-excluded path staged correctly). Recognize and swallow ONLY
|
||||
// this exact advisory; anything else (timeout, permission denied,
|
||||
// real corruption) rethrows.
|
||||
const stderr = err && typeof err === 'object' && 'stderr' in err ? String((err as { stderr: unknown }).stderr) : '';
|
||||
if (!stderr.includes('ignored by one of your .gitignore files')) throw err;
|
||||
}
|
||||
git(
|
||||
repoPath,
|
||||
// --no-verify only skips pre-commit/commit-msg — prepare-commit-msg
|
||||
// and (worse, since it runs AFTER the commit object already exists,
|
||||
// synchronously inside this same git invocation) post-commit are
|
||||
// NOT covered by it. An operator's global core.hooksPath or
|
||||
// init.templateDir can wire either, expecting project tooling,
|
||||
// prompting interactively, or hanging — none of which a headless
|
||||
// self-heal commit can satisfy, and a hanging post-commit hook would
|
||||
// burn the 600s budget above without even being the slow step.
|
||||
// `-c core.hooksPath=/dev/null` (in configs, below) makes git look
|
||||
// for hook scripts inside a location that can't contain any,
|
||||
// disabling the entire hooks path for this one invocation — the
|
||||
// complete form of what --no-verify only partially covers, kept for
|
||||
// explicitness on the two hooks it does name.
|
||||
[
|
||||
'commit', '--quiet', '--allow-empty', '--no-gpg-sign', '--no-verify',
|
||||
'-m', 'gbrain: initial commit (auto-init by sync)',
|
||||
],
|
||||
['user.name=gbrain', 'user.email=gbrain@localhost', 'core.hooksPath=/dev/null'],
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* #774 NAV-1 TOCTOU: true only if filePath realpath-resolves inside gitRoot.
|
||||
* Guards symlink escape at the per-file level (a committed symlink whose
|
||||
* target lives outside the repo), not just at scope entry.
|
||||
*/
|
||||
function isPathSafe(filePath: string, gitRoot: string): boolean {
|
||||
try {
|
||||
const real = realpathSync(filePath);
|
||||
const rootReal = realpathSync(gitRoot);
|
||||
return real === rootReal || real.startsWith(rootReal + '/');
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function hasOriginRemote(repoPath: string): boolean {
|
||||
try {
|
||||
execFileSync('git', buildGitInvocation(repoPath, ['remote', 'get-url', 'origin']), {
|
||||
@@ -956,6 +1174,65 @@ async function readSyncAnchor(
|
||||
return await engine.getConfig(`sync.${which}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* #2964: is `repoPath` gbrain's own default-brain anchor, as opposed to a
|
||||
* path some caller merely happened to pass through unchanged?
|
||||
*
|
||||
* `!opts.sourceId` alone is NOT sufficient — and neither is rejecting
|
||||
* `opts.sourceId` outright: migration `sources_table_additive` (v20)
|
||||
* seeds a `'default'` source row whose `local_path` is copied FROM
|
||||
* `config.sync.repo_path` on every brain that has ever run it (i.e.
|
||||
* effectively all of them by now), and `writeSyncAnchor` keeps that row's
|
||||
* `local_path` current on every sync thereafter. So on a real installed
|
||||
* brain, `resolveSourceForDir` (dream cycle) and the CLI's bare `gbrain
|
||||
* sync` both resolve `sourceId: 'default'`, NOT `undefined` — rejecting
|
||||
* all non-empty `sourceId` (an earlier, insufficiently-reviewed version
|
||||
* of this check) made self-heal never fire on that real path either,
|
||||
* masked in tests only because a freshly-`initSchema()`'d test brain's
|
||||
* `'default'` row has a null `local_path` (Codex review round 5).
|
||||
*
|
||||
* The actual boundary: `'default'` is gbrain's own bootstrap identity,
|
||||
* not something a caller names — a DIFFERENT, non-default `sourceId` is
|
||||
* what an explicit `sources add <id> --path <dir>` registration (a
|
||||
* user's own external directory) looks like, and that's what must keep
|
||||
* failing loudly. So: permit `sourceId` when it's exactly `undefined` or
|
||||
* `'default'`, reject any other id, and for BOTH permitted cases prove
|
||||
* ownership by VALUE — reread the live anchor for that same identity
|
||||
* (`sources.default.local_path` when sourceId='default', else
|
||||
* `config.sync.repo_path`) and require the resolved `repoPath` to
|
||||
* REALPATH-equal it (not raw string equality: `dream`'s `resolveBrainDir`
|
||||
* normalizes via `path.resolve`, so a trailing slash or `..` in the
|
||||
* stored anchor must not defeat the match — Codex review round 5, P2).
|
||||
* An arbitrary caller-supplied path (e.g. an admin-scope
|
||||
* `submit_job({name:'sync', data:{repoPath}})`) only passes this check
|
||||
* if it already equals gbrain's own anchor by realpath identity — at
|
||||
* which point self-healing it is exactly the legitimate case, not an
|
||||
* escalation.
|
||||
*
|
||||
* `opts.srcSubpath` disqualifies unconditionally: a subpath-scoped sync
|
||||
* only wants THAT subdirectory captured, but the self-heal baseline
|
||||
* commit runs `git add -A` at the git root (there's no file list yet to
|
||||
* scope it to — collection happens after this point) — see the P2 review
|
||||
* finding on `createSyncBaselineCommit`'s callers.
|
||||
*/
|
||||
async function isAnchorOwnedSyncPath(
|
||||
engine: BrainEngine,
|
||||
opts: SyncOpts,
|
||||
repoPath: string,
|
||||
): Promise<boolean> {
|
||||
if (opts.srcSubpath) return false;
|
||||
if (opts.sourceId && opts.sourceId !== 'default') return false;
|
||||
const anchor = await readSyncAnchor(engine, opts.sourceId, 'repo_path');
|
||||
if (anchor === null) return false;
|
||||
try {
|
||||
return realpathSync(anchor) === realpathSync(repoPath);
|
||||
} catch {
|
||||
// Anchor or repoPath doesn't realpath-resolve (dangling/nonexistent) —
|
||||
// can't prove identity, so don't self-heal.
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function writeSyncAnchor(
|
||||
engine: BrainEngine,
|
||||
sourceId: string | undefined,
|
||||
@@ -1567,17 +1844,72 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
}
|
||||
|
||||
// Validate git repo
|
||||
if (!existsSync(join(repoPath, '.git'))) {
|
||||
throw new Error(`Not a git repository: ${repoPath}. GBrain sync requires a git-initialized repo.`);
|
||||
// #753/#774: discover the git root instead of requiring `.git` at repoPath
|
||||
// directly. Supports subdir-of-git-repo sources (monorepo pattern): either
|
||||
// an explicit `--src-subpath` under a git-root repoPath, or a repoPath that
|
||||
// IS a subdirectory (auto-discovery). Two axes fall out:
|
||||
// - gitContextRoot: ALL git operations (pull, rev-parse, diff, cat-file)
|
||||
// - syncScopeRoot: file walking, imports, deletes, renames
|
||||
// In the common case (repoPath == git root, no subpath) they are identical.
|
||||
serr(`[gbrain phase] sync.discover_git_root`);
|
||||
// #2964: a legacy `sync.repo_path`-anchored default brain can reach here
|
||||
// having never been `git init`-ed — e.g. a brain-pages dir that predates
|
||||
// git-backed sync, or one rsync'd from another machine without its
|
||||
// `.git`. gbrain owns that directory outright, so self-heal by
|
||||
// initializing it in place instead of failing the sync phase every
|
||||
// single run. Mirrors the recloneIfMissing self-recovery above for
|
||||
// owned remote clones. Ownership is proven by VALUE (resolved repoPath
|
||||
// equals gbrain's persisted anchor) via `isAnchorOwnedSyncPath`, not by
|
||||
// the mere absence of `opts.sourceId`/`opts.repoPath` — see that
|
||||
// function's docstring. `!opts.dryRun`: a preview must never write.
|
||||
let gitContextRoot: string;
|
||||
try {
|
||||
gitContextRoot = realpathSync(discoverGitRoot(repoPath));
|
||||
} catch (err) {
|
||||
if (
|
||||
opts.dryRun ||
|
||||
opts.signal?.aborted ||
|
||||
!existsSync(repoPath) ||
|
||||
!(await isAnchorOwnedSyncPath(engine, opts, repoPath))
|
||||
) {
|
||||
throw err;
|
||||
}
|
||||
serr(`[gbrain] auto-recovery: git-initializing brain dir ${repoPath} (no git repo found).`);
|
||||
git(repoPath, ['init', '--quiet']);
|
||||
createSyncBaselineCommit(repoPath);
|
||||
gitContextRoot = realpathSync(discoverGitRoot(repoPath));
|
||||
}
|
||||
const rawScopeRoot = opts.srcSubpath ? join(repoPath, opts.srcSubpath) : repoPath;
|
||||
if (!existsSync(rawScopeRoot)) {
|
||||
throw new Error(`Sync scope does not exist: ${rawScopeRoot}`);
|
||||
}
|
||||
const syncScopeRoot = realpathSync(rawScopeRoot);
|
||||
// NAV-1/NAV-2 scope-entry guard: the realpath-resolved scope must live
|
||||
// inside the realpath-resolved git root. Catches `--src-subpath ../escape`
|
||||
// AND a symlinked subdir pointing outside the repo, before any git op runs.
|
||||
if (syncScopeRoot !== gitContextRoot && !syncScopeRoot.startsWith(gitContextRoot + '/')) {
|
||||
throw new Error(
|
||||
`Sync scope ${syncScopeRoot} resolves outside git repo ${gitContextRoot}. ` +
|
||||
`Refusing to sync: possible path traversal via --src-subpath.`,
|
||||
);
|
||||
}
|
||||
// Relative path from git root to sync scope ('' when scope == root).
|
||||
const syncScopeRelPath = syncScopeRoot === gitContextRoot ? '' : relative(gitContextRoot, syncScopeRoot);
|
||||
const scoped = syncScopeRelPath !== '';
|
||||
// Anchor written back to sync state (sources.local_path / sync.repo_path):
|
||||
// the SCOPE path, so a follow-up bare `gbrain sync` auto-discovers the same
|
||||
// scope. Unchanged (the caller's repoPath spelling) when no --src-subpath.
|
||||
const anchorPath = opts.srcSubpath ? rawScopeRoot : repoPath;
|
||||
const fullSyncRoots = { gitContextRoot, syncScopeRoot, anchorPath };
|
||||
|
||||
serr(`[gbrain phase] sync.detect_head`);
|
||||
// Detect detached HEAD up front so the working-tree fallback fires for both
|
||||
// the default sync and `--no-pull` callers. Only the actual git pull is
|
||||
// gated on opts.noPull.
|
||||
const detachedHead = isDetachedHead(repoPath);
|
||||
const detachedHead = isDetachedHead(gitContextRoot);
|
||||
if (detachedHead && !opts.noPull) {
|
||||
// Print the caller's repoPath spelling (not the realpathed git root) —
|
||||
// it's what the operator recognizes, and tests pin it.
|
||||
serr(`Detached HEAD on ${repoPath}; skipping git pull. Syncing from local working tree.`);
|
||||
}
|
||||
|
||||
@@ -1587,7 +1919,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// hardening that cloneRepo applies. Route through pullRepo from
|
||||
// git-remote.ts so the flag set is consistent across initial clone and
|
||||
// ongoing pulls — single source of truth for the defensive flags.
|
||||
const originRemotePresent = !opts.noPull && !detachedHead ? hasOriginRemote(repoPath) : false;
|
||||
const originRemotePresent = !opts.noPull && !detachedHead ? hasOriginRemote(gitContextRoot) : false;
|
||||
if (!opts.noPull && !detachedHead && !originRemotePresent) {
|
||||
serr(`No origin remote on ${repoPath}; skipping git pull. Syncing from local working tree.`);
|
||||
}
|
||||
@@ -1626,8 +1958,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// We pass a safe default (the operator's full --timeout if set, else
|
||||
// pullRepo's own 300s default). The catch below distinguishes
|
||||
// timeout (ETIMEDOUT / SIGTERM on err.cause) from ordinary pull
|
||||
// failure.
|
||||
pullRepo(repoPath);
|
||||
// failure. Pull applies to the whole git repo (gitContextRoot), not
|
||||
// just the sync scope — git has no per-subdir pull.
|
||||
pullRepo(gitContextRoot);
|
||||
serr(`[gbrain phase] sync.git_pull done ${Date.now() - _t0}ms`);
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -1668,11 +2001,56 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// Get current HEAD
|
||||
let headCommit: string;
|
||||
try {
|
||||
headCommit = git(repoPath, ['rev-parse', 'HEAD']);
|
||||
headCommit = git(gitContextRoot, ['rev-parse', 'HEAD']);
|
||||
} catch {
|
||||
throw new Error(`No commits in repo ${repoPath}. Make at least one commit before syncing.`);
|
||||
// #2964: unborn-HEAD recovery. `.git` exists (discoverGitRoot succeeded
|
||||
// above) but there are zero commits — e.g. a prior self-heal `git init`
|
||||
// ran but the process died before the baseline commit landed, leaving
|
||||
// this brain permanently wedged on "No commits in repo" every night
|
||||
// thereafter. Finish the same baseline-commit self-heal the
|
||||
// discoverGitRoot catch above would have done, gated the same way
|
||||
// (ownership proven by value, never on a dry-run preview) PLUS a scope
|
||||
// check: `discoverGitRoot` walks UP from `repoPath`, so it can resolve
|
||||
// to an ANCESTOR repo, not `repoPath` itself (most plausible for a
|
||||
// `--src-subpath` sync, but `isAnchorOwnedSyncPath` already refuses
|
||||
// that case — kept here too as defense in depth against any other path
|
||||
// where gitContextRoot could diverge from repoPath). Committing at an
|
||||
// ancestor (`git add -A` at gitContextRoot) would capture sibling
|
||||
// files well outside the sync scope — refuse instead of guessing.
|
||||
if (
|
||||
opts.dryRun ||
|
||||
opts.signal?.aborted ||
|
||||
gitContextRoot !== realpathSync(repoPath) ||
|
||||
!(await isAnchorOwnedSyncPath(engine, opts, repoPath))
|
||||
) {
|
||||
throw new Error(`No commits in repo ${repoPath}. Make at least one commit before syncing.`);
|
||||
}
|
||||
serr(`[gbrain] auto-recovery: repo has no commits yet, creating baseline commit ${gitContextRoot}.`);
|
||||
createSyncBaselineCommit(gitContextRoot);
|
||||
headCommit = git(gitContextRoot, ['rev-parse', 'HEAD']);
|
||||
}
|
||||
|
||||
// #2964: self-heal deliberately does NOT special-case db_only/.gitignore
|
||||
// interaction beyond the COMMIT itself (createSyncBaselineCommit's
|
||||
// pathspec exclusion, which stands on its own regardless of what
|
||||
// .gitignore says). db_only content is documented as DB-sourced ("bulk
|
||||
// machine-generated content... written to disk as a local cache", see
|
||||
// docs/storage-tiering.md) — it reaches the database via ingest-specific
|
||||
// paths, never via gbrain sync's git-diff-based file collection, and
|
||||
// `.gitignore` management there is entirely about keeping db_only out of
|
||||
// git history, not about what sync imports. An earlier version of this
|
||||
// fix (Codex review rounds 6-7) tried to also guarantee db_only markdown
|
||||
// gets imported on this first sync and that .gitignore gets written
|
||||
// post-success even when called outside runSync — solving a problem
|
||||
// that, per the docs above, isn't actually in scope for what sync is
|
||||
// for. Reverted in round 8 review discussion in favor of this simpler
|
||||
// design: after self-heal, the import + any subsequent .gitignore
|
||||
// management behave EXACTLY the same as for any other brain, self-healed
|
||||
// or not (runSync's existing post-success manageGitignoreAtGitRoot call
|
||||
// covers the CLI path identically either way; the dream cycle not
|
||||
// calling it is a separate, pre-existing characteristic of the dream
|
||||
// cycle in general, not something this fix introduces or worsens).
|
||||
|
||||
// #1970: bookmark reachability. The ONLY thing that should force a full
|
||||
// reconcile is a truly-absent object; a present-but-non-ancestor bookmark
|
||||
// (history rewrite: force-push, master→main consolidation, squash) is still
|
||||
@@ -1690,7 +2068,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (lastCommit) {
|
||||
let objectPresent = true;
|
||||
try {
|
||||
git(repoPath, ['cat-file', '-t', lastCommit]);
|
||||
git(gitContextRoot, ['cat-file', '-t', lastCommit]);
|
||||
} catch {
|
||||
objectPresent = false;
|
||||
}
|
||||
@@ -1699,7 +2077,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// back to the authoritative full reconcile (which now also purges stale
|
||||
// pages for deleted files; see performFullSync's delete-reconcile pass).
|
||||
serr(`Sync anchor ${lastCommit.slice(0, 8)} object missing (gc'd after history rewrite). Running full reimport.`);
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
|
||||
// Observability only — NOT control flow. A non-ancestor bookmark is still
|
||||
@@ -1707,7 +2085,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// failure mode (#1970) is visible in the logs.
|
||||
let isAncestor = true;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', lastCommit, headCommit]);
|
||||
git(gitContextRoot, ['merge-base', '--is-ancestor', lastCommit, headCommit]);
|
||||
} catch {
|
||||
isAncestor = false;
|
||||
}
|
||||
@@ -1722,7 +2100,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
|
||||
// First sync
|
||||
if (!lastCommit) {
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
|
||||
// v0.42.x (#1794): resumable incremental sync — resolve the PINNED target.
|
||||
@@ -1744,7 +2122,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (storedTarget) {
|
||||
let pinReachable = false;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', storedTarget, headCommit]);
|
||||
git(gitContextRoot, ['merge-base', '--is-ancestor', storedTarget, headCommit]);
|
||||
pinReachable = true;
|
||||
} catch {
|
||||
pinReachable = false;
|
||||
@@ -1778,7 +2156,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
const currentVersion = String(CHUNKER_VERSION);
|
||||
const versionMismatch = storedVersion !== null && storedVersion !== currentVersion;
|
||||
const versionNeverSet = storedVersion === null && opts.sourceId !== undefined;
|
||||
const detachedWorkingTreeManifest = detachedHead ? buildDetachedWorkingTreeManifest(repoPath) : null;
|
||||
const detachedWorkingTreeManifest = detachedHead ? buildDetachedWorkingTreeManifest(gitContextRoot) : null;
|
||||
const hasDetachedWorkingTreeChanges = detachedWorkingTreeManifest !== null &&
|
||||
(detachedWorkingTreeManifest.added.length > 0 ||
|
||||
detachedWorkingTreeManifest.modified.length > 0 ||
|
||||
@@ -1814,7 +2192,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
`[sync] chunker_version gate: stored=${storedVersion ?? 'unset'}, current=${currentVersion}. ` +
|
||||
`Forcing full re-chunk pass (git HEAD unchanged but pipeline version advanced).`,
|
||||
);
|
||||
const result = await performFullSync(engine, repoPath, headCommit, opts);
|
||||
const result = await performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
await writeChunkerVersion(engine, opts.sourceId, currentVersion);
|
||||
return result;
|
||||
}
|
||||
@@ -1835,7 +2213,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// buffer, and a gc'd anchor object can't be diffed at all. On either
|
||||
// `unavailable`, fall back to the authoritative full reconcile instead of
|
||||
// throwing — a slow correct reconcile beats a hard error or a silent walk.
|
||||
const delta = computeSyncDelta(repoPath, lastCommit, pin, {
|
||||
const delta = computeSyncDelta(gitContextRoot, lastCommit, pin, {
|
||||
detachedManifest: detachedWorkingTreeManifest,
|
||||
});
|
||||
if (delta.status === 'unavailable') {
|
||||
@@ -1843,30 +2221,60 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
`[sync] delta ${lastCommit.slice(0, 8)}..${pin.slice(0, 8)} unavailable ` +
|
||||
`(${delta.reason}) — falling back to full reconcile.`,
|
||||
);
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
return performFullSync(engine, fullSyncRoots, headCommit, opts);
|
||||
}
|
||||
const manifest = delta.manifest;
|
||||
|
||||
// Filter to syncable files (strategy-aware)
|
||||
// #753/#774 scope filter: git-diff paths are git-root-relative; when a
|
||||
// subpath scope is active, only paths under it participate. Back-compat:
|
||||
// syncScopeRelPath is '' when scope == root, so inScope is always true and
|
||||
// the filters below reduce to the pre-#774 behavior exactly.
|
||||
const inScope = (p: string): boolean =>
|
||||
!scoped || p === syncScopeRelPath || p.startsWith(syncScopeRelPath + '/');
|
||||
// --exclude patterns match the SCOPE-relative path (what the user of a
|
||||
// scoped source thinks in), same form runImport matches on full sync.
|
||||
const scopeRel = (p: string): string =>
|
||||
scoped && p.startsWith(syncScopeRelPath + '/') ? p.slice(syncScopeRelPath.length + 1) : p;
|
||||
const excluded = (p: string): boolean =>
|
||||
opts.exclude !== undefined && opts.exclude.length > 0 && matchesAnyGlob(scopeRel(p), opts.exclude);
|
||||
|
||||
// Filter to syncable files (strategy-aware + scope-aware + exclude-aware)
|
||||
const syncOpts = opts.strategy ? { strategy: opts.strategy } : undefined;
|
||||
// #1970 (F-C): a rename whose DESTINATION is unsyncable drops out of BOTH
|
||||
// `renamed` (only `r.to` is kept below) AND `deleted` (git emits it as `R`,
|
||||
// not `D`), leaving the OLD page stale. Fold the source side into the delete
|
||||
// set. isSyncable(r.from) excludes metafiles automatically, so a rename of a
|
||||
// metafile is left untouched (matching the #1433 metafile-skip invariant).
|
||||
// #774: a rename whose destination LEFT the scope is the same class — the
|
||||
// old page's backing file is gone from this source's slice of the repo.
|
||||
const renamedToUnsyncable = manifest.renamed
|
||||
.filter(r => isSyncable(r.from, syncOpts) && !isSyncable(r.to, syncOpts))
|
||||
.filter(r => inScope(r.from) && isSyncable(r.from, syncOpts) &&
|
||||
!(inScope(r.to) && isSyncable(r.to, syncOpts)))
|
||||
.map(r => r.from);
|
||||
const filtered: SyncManifest = {
|
||||
added: manifest.added.filter(p => isSyncable(p, syncOpts)),
|
||||
modified: manifest.modified.filter(p => isSyncable(p, syncOpts)),
|
||||
added: manifest.added.filter(p => inScope(p) && !excluded(p) && isSyncable(p, syncOpts)),
|
||||
modified: manifest.modified.filter(p => inScope(p) && !excluded(p) && isSyncable(p, syncOpts)),
|
||||
deleted: unique([
|
||||
...manifest.deleted.filter(p => isSyncable(p, syncOpts)),
|
||||
...manifest.deleted.filter(p => inScope(p) && isSyncable(p, syncOpts)),
|
||||
...renamedToUnsyncable,
|
||||
]),
|
||||
renamed: manifest.renamed.filter(r => isSyncable(r.to, syncOpts)),
|
||||
renamed: manifest.renamed.filter(r => inScope(r.to) && !excluded(r.to) && isSyncable(r.to, syncOpts)),
|
||||
};
|
||||
|
||||
// NAV-4: warn when --exclude filtered out every candidate change — almost
|
||||
// always a mistyped pattern, and otherwise indistinguishable from
|
||||
// "up to date" in the output.
|
||||
if (opts.exclude && opts.exclude.length > 0) {
|
||||
const excludeCandidates = [...manifest.added, ...manifest.modified]
|
||||
.filter(p => inScope(p) && isSyncable(p, syncOpts));
|
||||
if (excludeCandidates.length > 0 && excludeCandidates.every(excluded)) {
|
||||
console.warn(
|
||||
`[gbrain sync] No files matched after applying ${opts.exclude.length} --exclude pattern(s). ` +
|
||||
`Check your --exclude flags. Patterns: ${JSON.stringify(opts.exclude)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Delete pages that became un-syncable (modified but filtered out).
|
||||
// v0.20.0 Cathedral II SP-5: resolveSlugForPath picks the right slug shape
|
||||
// (markdown vs code) based on the chunker's classifier, so a Rust file that
|
||||
@@ -1890,14 +2298,20 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// delete the page. That's the same pre-fix behavior — removing the
|
||||
// page requires `gbrain pages purge-deleted` or a direct MCP delete.
|
||||
// Filed as v0.42+ follow-up for a `gbrain pages remove <slug>` surface.
|
||||
const unsyncableModified = manifest.modified.filter(p => !isSyncable(p, syncOpts));
|
||||
const unsyncableModified = manifest.modified.filter(p => inScope(p) && !isSyncable(p, syncOpts));
|
||||
// v0.18.0+ multi-source: scope getPage + deletePage to opts.sourceId so
|
||||
// unsyncable cleanup in source A doesn't accidentally sweep same-slug
|
||||
// pages in sources B/C/D.
|
||||
const pageOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined;
|
||||
for (const path of unsyncableModified) {
|
||||
// v0.41.13 #1433: never delete on metafile classification.
|
||||
if (unsyncableReason(path, syncOpts) === 'metafile') continue;
|
||||
// #2404 hardening: same for 'pruned-dir' — a page under a pruned
|
||||
// directory can only exist via a deliberate put_page (sync never
|
||||
// imports those paths), so "the file was modified" is not evidence
|
||||
// the page is stale. Deleting here silently destroyed put-created
|
||||
// pages every time their materialized file landed in a commit.
|
||||
const reason = unsyncableReason(path, syncOpts);
|
||||
if (reason === 'metafile' || reason === 'pruned-dir') continue;
|
||||
const slug = await resolveSlugByPathOrSourcePath(engine, path, opts.sourceId);
|
||||
try {
|
||||
const existing = await engine.getPage(slug, pageOpts);
|
||||
@@ -1938,7 +2352,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// (#1794): advance to the PINNED target, and clear any checkpoint (a resume
|
||||
// whose remaining range turned out to have no syncable changes still
|
||||
// completes cleanly here).
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(gitContextRoot, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
@@ -2325,8 +2739,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// throw here crashes the whole sync mid-run and freezes the checkpoint,
|
||||
// defeating --skip-failed. A `skipped` result carrying an error is also
|
||||
// captured so the failure is recorded rather than silently dropped.
|
||||
const filePath = join(repoPath, to);
|
||||
if (existsSync(filePath)) {
|
||||
// Paths from git diff are relative to gitContextRoot; join from there.
|
||||
// NAV-1 TOCTOU: refuse a destination that realpath-resolves outside the
|
||||
// repo (committed symlink pointing out).
|
||||
const filePath = join(gitContextRoot, to);
|
||||
if (existsSync(filePath) && isPathSafe(filePath, gitContextRoot)) {
|
||||
try {
|
||||
const result = await importFile(engine, filePath, to, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack });
|
||||
if (result.status === 'imported') chunksCreated += result.chunks;
|
||||
@@ -2411,8 +2828,8 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
progress.start('sync.imports', importsToDo.length);
|
||||
|
||||
// Core import logic shared by serial and parallel paths.
|
||||
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
|
||||
const syncRepoPath = repoPath!;
|
||||
// Paths from git diff are relative to gitContextRoot; join from there.
|
||||
const syncRepoPath = gitContextRoot;
|
||||
// paced-backfill (T3 / C9 / CX4): ONE shared pacer across all worker
|
||||
// engines. This is the multi-pool permit case — each parallel worker owns a
|
||||
// separate PostgresEngine, so a single worker count can't bound TOTAL
|
||||
@@ -2500,6 +2917,16 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
// #774 NAV-1 TOCTOU: re-validate the file's realpath at import time so a
|
||||
// committed symlink pointing outside the repo (or one swapped in after
|
||||
// the scope-entry check) is never read. Recorded as a failure —
|
||||
// fail-closed: the bookmark won't advance past a symlink escape.
|
||||
if (!isPathSafe(filePath, gitContextRoot)) {
|
||||
failedFiles.push({ path, error: 'path resolves outside git repo (symlink escape)' });
|
||||
progressAt.last = Date.now();
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
// v0.41.37.0 #1569: per-file BEGIN heartbeat, emitted BEFORE importFile so a
|
||||
// hang names the stalling file (the progress.tick below only fires AFTER
|
||||
// importFile returns — useless when one file wedges). Off by default
|
||||
@@ -2703,11 +3130,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// - pin NOT an ancestor of HEAD (history REWRITE / reset / force-push) →
|
||||
// the tree we imported against is gone. Block; do not advance.
|
||||
try {
|
||||
const currentHead = git(repoPath, ['rev-parse', 'HEAD']);
|
||||
const currentHead = git(gitContextRoot, ['rev-parse', 'HEAD']);
|
||||
if (currentHead !== pin) {
|
||||
let pinStillReachable = false;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', pin, currentHead]);
|
||||
git(gitContextRoot, ['merge-base', '--is-ancestor', pin, currentHead]);
|
||||
pinStillReachable = true;
|
||||
} catch {
|
||||
pinStillReachable = false;
|
||||
@@ -2748,9 +3175,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// "fresh". The checkpoint rows clear here — CONVERGENCE CONTRACT: sync
|
||||
// convergence == IMPORT convergence; downstream extract/facts/embed is
|
||||
// decoupled (its own resumable stale sweeps).
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(gitContextRoot, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
await clearOpCheckpoint(engine, ckpt.target);
|
||||
@@ -2799,7 +3226,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// checkpoint is INTENTIONALLY left in place — the banked completed set lets
|
||||
// the next run skip the drained files and re-attempt only the failures.
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
// v0.42.x (#1794): surface banked progress so a blocked run doesn't read as
|
||||
// total loss (last_commit is unchanged by design; the checkpoint is banked).
|
||||
serr(
|
||||
@@ -2870,8 +3297,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (!opts.noExtract && totalChanges <= 100 && pagesAffected.length > 0) {
|
||||
try {
|
||||
const { extractLinksForSlugs, extractTimelineForSlugs, stampExtracted } = await import('./extract.ts');
|
||||
const linksCreated = await extractLinksForSlugs(engine, repoPath, pagesAffected, extractOpts);
|
||||
const timelineCreated = await extractTimelineForSlugs(engine, repoPath, pagesAffected, extractOpts);
|
||||
// #774: pages' source_path is git-root-relative, so extract resolves
|
||||
// files from gitContextRoot (== repoPath realpath when unscoped).
|
||||
const linksCreated = await extractLinksForSlugs(engine, gitContextRoot, pagesAffected, extractOpts);
|
||||
const timelineCreated = await extractTimelineForSlugs(engine, gitContextRoot, pagesAffected, extractOpts);
|
||||
if (linksCreated > 0 || timelineCreated > 0) {
|
||||
slog(` Extracted: ${linksCreated} links, ${timelineCreated} timeline entries`);
|
||||
}
|
||||
@@ -2976,11 +3405,21 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
|
||||
async function performFullSync(
|
||||
engine: BrainEngine,
|
||||
repoPath: string,
|
||||
// #753/#774: the three roots resolved once at the top of performSyncInner.
|
||||
// gitContextRoot — git repo root (git ops, slug base for scoped syncs)
|
||||
// syncScopeRoot — where files are walked/imported (== gitContextRoot
|
||||
// when no subpath scope is active)
|
||||
// anchorPath — what gets written back to sync.repo_path/local_path
|
||||
roots: { gitContextRoot: string; syncScopeRoot: string; anchorPath: string },
|
||||
headCommit: string,
|
||||
opts: SyncOpts,
|
||||
): Promise<SyncResult> {
|
||||
// Dry-run: walk the repo, count syncable files, return without writing.
|
||||
const { gitContextRoot, syncScopeRoot, anchorPath } = roots;
|
||||
// Scoped sync → slugs/source_path are git-root-relative (matches the
|
||||
// incremental path's git-diff paths). Unscoped → undefined (dir-relative,
|
||||
// the pre-#774 behavior, byte-for-byte).
|
||||
const slugRoot = syncScopeRoot !== gitContextRoot ? gitContextRoot : undefined;
|
||||
// Dry-run: walk the scope, count syncable files, return without writing.
|
||||
// Fixes the silent-write-on-dry-run bug where performFullSync called
|
||||
// runImport unconditionally regardless of opts.dryRun.
|
||||
//
|
||||
@@ -2990,11 +3429,14 @@ async function performFullSync(
|
||||
// code --dry-run` always reported zero files even when ~1500 code
|
||||
// files were waiting.
|
||||
if (opts.dryRun) {
|
||||
const allFiles = collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' });
|
||||
let allFiles = collectSyncableFiles(syncScopeRoot, { strategy: opts.strategy ?? 'markdown' });
|
||||
if (opts.exclude && opts.exclude.length > 0) {
|
||||
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(syncScopeRoot, abs), opts.exclude));
|
||||
}
|
||||
slog(
|
||||
`Full-sync dry run (strategy=${opts.strategy ?? 'markdown'}): ` +
|
||||
`${allFiles.length} file(s) would be imported ` +
|
||||
`from ${repoPath} @ ${headCommit.slice(0, 8)}.`,
|
||||
`from ${syncScopeRoot} @ ${headCommit.slice(0, 8)}.`,
|
||||
);
|
||||
return {
|
||||
status: 'dry_run',
|
||||
@@ -3017,21 +3459,24 @@ async function performFullSync(
|
||||
// sync and the jobs handler.
|
||||
const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER;
|
||||
const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency);
|
||||
slog(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
|
||||
slog(`Running full import of ${syncScopeRoot}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs = [repoPath];
|
||||
const importArgs = [syncScopeRoot];
|
||||
if (opts.noEmbed) importArgs.push('--no-embed');
|
||||
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
|
||||
// v0.31.2: thread strategy through so code-strategy first sync
|
||||
// actually enumerates code files (closes bug 1).
|
||||
// v0.30.x: thread sourceId so performFullSync routes pages to the named
|
||||
// source (incremental path already does this).
|
||||
// #753/#774: thread exclude (--exclude CLI) + slugRoot (monorepo subdir).
|
||||
const _fullImportT0 = Date.now();
|
||||
serr(`[gbrain phase] sync.fullsync.import start strategy=${opts.strategy ?? 'markdown'}`);
|
||||
const result = await runImport(engine, importArgs, {
|
||||
commit: headCommit,
|
||||
strategy: opts.strategy,
|
||||
sourceId: opts.sourceId,
|
||||
exclude: opts.exclude,
|
||||
slugRoot,
|
||||
// issue #1939: performFullSync owns the failure ledger + bookmark via the
|
||||
// shared gate below; don't let runImport double-record or write its own.
|
||||
managedBookmark: true,
|
||||
@@ -3055,9 +3500,9 @@ async function performFullSync(
|
||||
const advanceFull = async (): Promise<void> => {
|
||||
// Persist sync state so the next sync is incremental. Routed through
|
||||
// writeSyncAnchor so --source pins the right sources row.
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(repoPath));
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(gitContextRoot));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
};
|
||||
|
||||
@@ -3084,7 +3529,7 @@ async function performFullSync(
|
||||
);
|
||||
}
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', anchorPath);
|
||||
return {
|
||||
status: 'blocked_by_failures',
|
||||
fromCommit: null,
|
||||
@@ -3140,16 +3585,24 @@ async function performFullSync(
|
||||
// backslash paths while a stored source_path can hold git-derived forward
|
||||
// slashes; without normalization every file-backed page mismatches, looks
|
||||
// stale, and the reconcile wipes the whole source.
|
||||
const currentFiles = collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' })
|
||||
.map(abs => relative(repoPath, abs));
|
||||
// #774: scoped syncs store git-root-relative source_paths (slugRoot), so
|
||||
// relativize the walk to the same base — otherwise every page mismatches
|
||||
// and the mass-delete valve trips on a perfectly healthy scoped source.
|
||||
const currentFiles = collectSyncableFiles(syncScopeRoot, { strategy: opts.strategy ?? 'markdown' })
|
||||
.map(abs => relative(slugRoot ?? syncScopeRoot, abs));
|
||||
const rows = await engine.executeRaw<{ slug: string; source_path: string | null }>(
|
||||
`SELECT slug, source_path FROM pages WHERE source_id = $1 AND source_path IS NOT NULL AND deleted_at IS NULL`,
|
||||
[sid],
|
||||
);
|
||||
// #774: a scoped full sync is authoritative ONLY for its scope — pages
|
||||
// whose source_path lives outside the subpath (e.g. from an earlier
|
||||
// root-level sync of this source) are out of this walk's sight and must
|
||||
// not be treated as stale.
|
||||
const scopePrefix = slugRoot ? relative(gitContextRoot, syncScopeRoot) + '/' : '';
|
||||
const plan = planReconcileDeletes(
|
||||
rows,
|
||||
currentFiles,
|
||||
p => isSyncable(p, reconcileSyncOpts),
|
||||
p => (scopePrefix === '' || p.startsWith(scopePrefix)) && isSyncable(p, reconcileSyncOpts),
|
||||
);
|
||||
if (plan.staleSlugs.length > 0 && plan.massDelete && !massReconcileAllowed()) {
|
||||
// #2828 mass-delete safety valve: a reconcile that would sweep more than
|
||||
@@ -3169,9 +3622,45 @@ async function performFullSync(
|
||||
`GBRAIN_ALLOW_MASS_RECONCILE=1 to restore the old behavior.`,
|
||||
);
|
||||
} else if (plan.staleSlugs.length > 0) {
|
||||
// #2426: a stale page whose source_path was NEVER committed to git is
|
||||
// DB-only write-through (the file was written into the clone but never
|
||||
// committed/pushed, then lost — e.g. a fresh clone). "Absent from git"
|
||||
// is the SYMPTOM of that bug, not evidence the content is disposable.
|
||||
// Keep those pages and re-export their markdown to the working tree so
|
||||
// they're file-backed again; only pages whose file once existed in git
|
||||
// history (i.e. was genuinely deleted) are reconcile-deleted.
|
||||
const everCommitted = listEverCommittedPaths(gitContextRoot);
|
||||
const pathBySlug = new Map(rows.map(r => [r.slug, r.source_path]));
|
||||
let deletableSlugs = plan.staleSlugs;
|
||||
const dbOnlySlugs: string[] = [];
|
||||
if (everCommitted) {
|
||||
deletableSlugs = [];
|
||||
for (const slug of plan.staleSlugs) {
|
||||
const sp = pathBySlug.get(slug);
|
||||
if (sp && !everCommitted.has(sp.replace(/\\/g, '/'))) dbOnlySlugs.push(slug);
|
||||
else deletableSlugs.push(slug);
|
||||
}
|
||||
}
|
||||
if (dbOnlySlugs.length > 0) {
|
||||
let reExported = 0;
|
||||
try {
|
||||
const { writePageThrough } = await import('../core/write-through.ts');
|
||||
for (const slug of dbOnlySlugs) {
|
||||
const r = await writePageThrough(engine, slug, { sourceId: sid });
|
||||
if (r.written) reExported++;
|
||||
}
|
||||
} catch { /* best-effort — pages are preserved either way */ }
|
||||
serr(
|
||||
`\n Kept ${dbOnlySlugs.length} page(s) whose markdown was never committed to git ` +
|
||||
`(DB-only write-through — not deleting).` +
|
||||
(reExported > 0 ? ` Re-exported ${reExported} of them to the working tree.` : '') +
|
||||
`\n Commit + push them (e.g. scripts/brain-commit-push.sh, or 'gbrain sources harden') ` +
|
||||
`so the next sync sees them as file-backed.`,
|
||||
);
|
||||
}
|
||||
const deleteScopedOpts = { sourceId: sid };
|
||||
for (let i = 0; i < plan.staleSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = plan.staleSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
for (let i = 0; i < deletableSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = deletableSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
try {
|
||||
const deleted = await engine.deletePages(batch, deleteScopedOpts);
|
||||
reconciledDeletes += deleted.length;
|
||||
@@ -3290,6 +3779,34 @@ export function planReconcileDeletes(
|
||||
return { staleSlugs, reconcilableCount: reconcilable.length, massDelete };
|
||||
}
|
||||
|
||||
/**
|
||||
* #2426: every repo-relative path that ever appeared as an ADD in git history
|
||||
* (rename detection off, so a `git mv` destination still counts as an add).
|
||||
* Used by the full-sync reconcile to distinguish "file was committed and later
|
||||
* deleted" (genuine delete → reconcile) from "file was NEVER committed"
|
||||
* (DB-only write-through → preserve). Returns null when `repoPath` isn't a git
|
||||
* work tree or git is unavailable — callers keep the plain-directory behavior.
|
||||
* Forward-slash-normalized to match `normalizeReconcilePath` membership tests.
|
||||
*/
|
||||
export function listEverCommittedPaths(repoPath: string): Set<string> | null {
|
||||
let stdout: string;
|
||||
try {
|
||||
stdout = execFileSync(
|
||||
'git',
|
||||
['-C', repoPath, '-c', 'core.quotepath=off', 'log', '--all', '--no-renames',
|
||||
'--diff-filter=A', '--format=', '--name-only'],
|
||||
{ encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] },
|
||||
);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const set = new Set<string>();
|
||||
for (const line of stdout.split('\n')) {
|
||||
if (line) set.add(line.replace(/\\/g, '/'));
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
/**
|
||||
* #2828 escape hatch: `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the pre-valve
|
||||
* behavior for the rare intentional bulk removal. Env-only (an incident-time
|
||||
@@ -3400,6 +3917,19 @@ export function composeAbortSignals(
|
||||
return AbortSignal.any(live);
|
||||
}
|
||||
|
||||
/**
|
||||
* #753/#774: `.gitignore` must be managed at the git ROOT — when a source's
|
||||
* local_path (or --repo) points at a monorepo subdirectory, writing ignore
|
||||
* entries into the subdir would create a stray `.gitignore` git doesn't
|
||||
* consult for the repo-level db_only rules. Best-effort: falls back to the
|
||||
* given path when git discovery fails (manageGitignore no-ops on non-repos).
|
||||
*/
|
||||
function manageGitignoreAtGitRoot(path: string, engineKind?: 'pglite' | 'postgres'): void {
|
||||
let root = path;
|
||||
try { root = discoverGitRoot(path); } catch { /* best-effort */ }
|
||||
manageGitignore(root, engineKind);
|
||||
}
|
||||
|
||||
export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
// v0.40 Federated Sync v2: `gbrain sync trigger` subcommand
|
||||
// Routes to runSyncTrigger which queues a 'sync' minion job with
|
||||
@@ -3432,6 +3962,13 @@ Options:
|
||||
--repo <path> Path to the brain repo. Defaults to the path
|
||||
saved by 'gbrain init'.
|
||||
--full Force a full re-sync (rare; usually incremental).
|
||||
--src-subpath <dir> Sync only this subdirectory of the git repo (monorepo
|
||||
pattern: N logical sources in one repo). Git pull/diff
|
||||
run at the repo root; imports are scoped to the subdir
|
||||
and slugs stay root-relative (wiki/page1). Passing the
|
||||
subdirectory directly as --repo also works.
|
||||
--exclude <glob> Exclude files matching the glob from sync (repeatable;
|
||||
matched against the scope-relative path).
|
||||
--dry-run Show what would be synced without writing.
|
||||
--skip-failed Acknowledge previously-recorded sync failures so
|
||||
the bookmark can advance past unparseable files.
|
||||
@@ -3591,6 +4128,20 @@ See also:
|
||||
process.exit(1);
|
||||
}
|
||||
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
|
||||
// #753/#774: monorepo subdir-source flags. --exclude is repeatable.
|
||||
const srcSubpath = args.find((a, i) => args[i - 1] === '--src-subpath') || undefined;
|
||||
const excludePatterns: string[] = [];
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--exclude' && i + 1 < args.length) excludePatterns.push(args[i + 1]);
|
||||
}
|
||||
if (syncAll && (srcSubpath || excludePatterns.length > 0)) {
|
||||
console.error(
|
||||
`--src-subpath/--exclude scope a single sync invocation; they cannot be combined with --all. ` +
|
||||
`For --all runs, register the subdirectory as the source's local_path instead ` +
|
||||
`(gbrain sources add <id> --path <repo>/<subdir>).`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
|
||||
const parallelStr = args.find((a, i) => args[i - 1] === '--parallel');
|
||||
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
|
||||
@@ -3850,7 +4401,7 @@ See also:
|
||||
result.status !== 'blocked_by_failures' &&
|
||||
result.status !== 'partial'
|
||||
) {
|
||||
manageGitignore(src.local_path!, engine.kind);
|
||||
manageGitignoreAtGitRoot(src.local_path!, engine.kind);
|
||||
}
|
||||
// D18: auto-enqueue embed-backfill per source (unless opted out).
|
||||
// v0.41.13.0 (T7 / D-V3-5): partial excluded — the next clean sync
|
||||
@@ -4038,6 +4589,8 @@ See also:
|
||||
const opts: SyncOpts = {
|
||||
repoPath, dryRun, full, noPull, noEmbed, noExtract, skipFailed, retryFailed, noSchemaPack, sourceId,
|
||||
strategy: strategyArg, concurrency,
|
||||
srcSubpath,
|
||||
exclude: excludePatterns.length > 0 ? excludePatterns : undefined,
|
||||
signal: composeAbortSignals(singleSourceInterrupt.signal, singleSourceController?.signal),
|
||||
};
|
||||
|
||||
@@ -4121,7 +4674,7 @@ See also:
|
||||
) {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
manageGitignoreAtGitRoot(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
// v0.42.42.0 (#2139, Step 4b): the inline gate auto-deferred this run's
|
||||
@@ -4170,7 +4723,7 @@ See also:
|
||||
) {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
manageGitignoreAtGitRoot(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
|
||||
+12
-6
@@ -101,12 +101,18 @@ async function getPageId(engine: BrainEngine, slug: string, sourceId?: string):
|
||||
return rows[0].id;
|
||||
}
|
||||
|
||||
async function resolveTakesSourceId(engine: BrainEngine): Promise<string | undefined> {
|
||||
try {
|
||||
return await resolveSourceId(engine, null);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
// Fail-closed (#2698 residual, TODOS.md): `resolveSourceId` only ever
|
||||
// throws when a source WAS explicitly in play — an invalid or
|
||||
// unregistered `GBRAIN_SOURCE`, a `.gbrain-source` dotfile pointing at a
|
||||
// source that doesn't exist, or a genuine DB error — never for "nothing
|
||||
// configured" (that path resolves cleanly to the seeded `'default'`
|
||||
// source, tier 6 of resolveSourceId). Swallowing those errors here used
|
||||
// to fall back to the unscoped slug-only page lookup, silently
|
||||
// reintroducing the pre-#2698 cross-source write bug whenever resolution
|
||||
// merely errored instead of resolving cleanly. Let it propagate so the
|
||||
// write is blocked instead of silently unscoped.
|
||||
async function resolveTakesSourceId(engine: BrainEngine): Promise<string> {
|
||||
return resolveSourceId(engine, null);
|
||||
}
|
||||
|
||||
function readBodyOrEmpty(path: string): string {
|
||||
|
||||
@@ -90,6 +90,38 @@ export function isValidOpenAITextEmbedding3Dim(modelId: string, dims: number): b
|
||||
return Number.isInteger(dims) && dims >= 1 && dims <= max;
|
||||
}
|
||||
|
||||
// NVIDIA NIM hosted embedding models use asymmetric input_type values. Most
|
||||
// emit fixed natural dimensions, but llama-nemotron-embed-1b-v2 accepts
|
||||
// Matryoshka-style dimension overrides (e.g. matching an existing 1280d
|
||||
// brain column without re-embedding through another provider).
|
||||
const NVIDIA_EMBEDDING_DIMS: Record<string, number> = {
|
||||
'nvidia/nv-embedqa-e5-v5': 1024,
|
||||
'nvidia/llama-nemotron-embed-1b-v2': 2048,
|
||||
'nvidia/nv-embed-v1': 4096,
|
||||
'nvidia/nv-embedcode-7b-v1': 4096,
|
||||
};
|
||||
|
||||
const NVIDIA_EMBEDDING_DIM_OPTIONS: Record<string, number[]> = {
|
||||
'nvidia/llama-nemotron-embed-1b-v2': [1024, 1280, 1536, 2048],
|
||||
};
|
||||
|
||||
export function isNvidiaEmbeddingModel(modelId: string): boolean {
|
||||
return modelId in NVIDIA_EMBEDDING_DIMS;
|
||||
}
|
||||
|
||||
export function nvidiaEmbeddingDim(modelId: string): number | undefined {
|
||||
return NVIDIA_EMBEDDING_DIMS[modelId];
|
||||
}
|
||||
|
||||
export function nvidiaEmbeddingDimOptions(modelId: string): number[] | undefined {
|
||||
return NVIDIA_EMBEDDING_DIM_OPTIONS[modelId];
|
||||
}
|
||||
|
||||
export function supportsNvidiaEmbeddingDimension(modelId: string, dims: number): boolean {
|
||||
const options = nvidiaEmbeddingDimOptions(modelId);
|
||||
return !!options && options.includes(dims);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the providerOptions blob for embedMany() that pins output dimensions.
|
||||
*
|
||||
@@ -194,6 +226,17 @@ export function dimsProviderOptions(
|
||||
},
|
||||
};
|
||||
}
|
||||
// NVIDIA NIM hosted embeddings are OpenAI-compatible but require
|
||||
// asymmetric input_type. Use passage for indexing/document-side vectors
|
||||
// and query for search-side vectors. Only llama-nemotron-embed-1b-v2
|
||||
// supports a dimensions override; fixed-dim models reject it.
|
||||
if (isNvidiaEmbeddingModel(modelId)) {
|
||||
const opts: Record<string, any> = {
|
||||
input_type: inputType === 'query' ? 'query' : 'passage',
|
||||
};
|
||||
if (supportsNvidiaEmbeddingDimension(modelId, dims)) opts.dimensions = dims;
|
||||
return { openaiCompatible: opts };
|
||||
}
|
||||
// OpenAI text-embedding-3 family on the openai-compatible adapter
|
||||
// (Azure OpenAI hosts these via its OpenAI-compatible /embeddings
|
||||
// endpoint). The provider defaults to the model's native size (3072
|
||||
|
||||
+157
-1
@@ -23,6 +23,7 @@
|
||||
|
||||
import { embed as aiEmbed, embedMany, generateObject, generateText, jsonSchema } from 'ai';
|
||||
import { AsyncLocalStorage } from 'node:async_hooks';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { listRecipes } from './recipes/index.ts';
|
||||
import { createOpenAI } from '@ai-sdk/openai';
|
||||
import { createGoogleGenerativeAI } from '@ai-sdk/google';
|
||||
@@ -52,6 +53,8 @@ import { dimsProviderOptions } from './dims.ts';
|
||||
import { hasAnthropicKey } from './anthropic-key.ts';
|
||||
import { AIConfigError, AITransientError, normalizeAIError } from './errors.ts';
|
||||
import { runGuardrails, hasGuardrails, type GuardrailHook } from '../guardrails.ts';
|
||||
import { loadConfig } from '../config.ts';
|
||||
import { buildGatewayConfig } from './build-gateway-config.ts';
|
||||
|
||||
// ---- Gateway-wide AI-HTTP timeout (v0.42.20.0, #1762/#1775) ----
|
||||
//
|
||||
@@ -116,6 +119,18 @@ const DEFAULT_RERANKER_MODEL = 'zeroentropyai:zerank-2';
|
||||
let _config: AIGatewayConfig | null = null;
|
||||
const _modelCache = new Map<string, any>();
|
||||
|
||||
/**
|
||||
* Recover the process-global gateway for foreground command entrypoints that
|
||||
* were reached without cli.ts's normal engine-connect initialization (#2590).
|
||||
* Existing configured gateways, including their DB-resolved model overrides,
|
||||
* are deliberately left unchanged.
|
||||
*/
|
||||
export function configureGatewayIfUninitialized(): void {
|
||||
if (_config) return;
|
||||
const config = loadConfig();
|
||||
if (config) configureGateway(buildGatewayConfig(config));
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.31.12 recipe-models merge: per-gateway-instance set of model ids the
|
||||
* user opted into via config. Keyed by provider id (`anthropic`, `openai`,
|
||||
@@ -506,6 +521,20 @@ export async function reconfigureGatewayWithEngine(engine: BrainEngine): Promise
|
||||
const expansionFull = newExpansion.includes(':') ? newExpansion : prefixWithProviderFrom(cfg.expansion_model ?? DEFAULT_EXPANSION_MODEL, newExpansion);
|
||||
const chatFull = newChat.includes(':') ? newChat : prefixWithProviderFrom(cfg.chat_model ?? DEFAULT_CHAT_MODEL, newChat);
|
||||
|
||||
// ALSO resolve the four tier models and register them as extended models.
|
||||
// assertTouchpoint's contract (model-resolver.ts) says config-chosen models —
|
||||
// `models.default` and `models.tier.*` included — bypass the native recipe
|
||||
// allowlist, but pre-fix only chat/expansion/embedding/reranker were
|
||||
// registered. A model reachable ONLY through a tier (e.g. `models.tier.deep`
|
||||
// set to an Opus newer than the recipe list) failed `probeChatModel` at call
|
||||
// time and silently degraded think/auto_think to the gather-only stub.
|
||||
// Resolving per-tier also honors `models.default` (it sits above tiers in
|
||||
// the resolveModel chain).
|
||||
const tierModels: string[] = [];
|
||||
for (const tier of ['utility', 'reasoning', 'deep', 'subagent'] as const) {
|
||||
tierModels.push(await resolveModel(engine, { tier, fallback: TIER_DEFAULTS[tier] }));
|
||||
}
|
||||
|
||||
_config = { ...cfg, expansion_model: expansionFull, chat_model: chatFull };
|
||||
_modelCache.clear();
|
||||
_shrinkState.clear();
|
||||
@@ -517,6 +546,7 @@ export async function reconfigureGatewayWithEngine(engine: BrainEngine): Promise
|
||||
_config.chat_model,
|
||||
_config.reranker_model,
|
||||
...(_config.chat_fallback_chain ?? []),
|
||||
...tierModels,
|
||||
]) {
|
||||
if (m) registerExtendedModel(m);
|
||||
}
|
||||
@@ -1044,6 +1074,30 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
|
||||
* float[] (not base64), so the Layer 2 cap compares against the JSON
|
||||
* payload size of each embedding rather than a base64 string length.
|
||||
*/
|
||||
/**
|
||||
* NVIDIA NIM compatibility shim. NVIDIA uses the OpenAI embeddings wire
|
||||
* shape but requires asymmetric input_type values: query for retrieval and
|
||||
* passage for indexed documents. The generic gateway store carries
|
||||
* query/document across the AI SDK boundary; map document to passage here.
|
||||
*/
|
||||
const nvidiaCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
let baseInit: RequestInit = init ?? {};
|
||||
if (baseInit.body && typeof baseInit.body === 'string') {
|
||||
try {
|
||||
const parsed = JSON.parse(baseInit.body);
|
||||
if (parsed && typeof parsed === 'object' && parsed.input_type === undefined) {
|
||||
parsed.input_type = __embedInputTypeStore.getStore() === 'query' ? 'query' : 'passage';
|
||||
const headers = new Headers(baseInit.headers ?? {});
|
||||
headers.delete('content-length');
|
||||
baseInit = { ...baseInit, body: JSON.stringify(parsed), headers };
|
||||
}
|
||||
} catch {
|
||||
// Preserve the provider response when the SDK body is unexpectedly non-JSON.
|
||||
}
|
||||
}
|
||||
return fetch(input as any, baseInit);
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
// OUTBOUND: normalize URL, rewrite path /embeddings → /models/embed, then
|
||||
// rewrite body. fetch accepts RequestInfo (string | Request) | URL; we
|
||||
@@ -1304,6 +1358,8 @@ function instantiateEmbedding(recipe: Recipe, modelId: string, cfg: AIGatewayCon
|
||||
? voyageCompatFetch
|
||||
: recipe.id === 'zeroentropyai'
|
||||
? zeroEntropyCompatFetch
|
||||
: recipe.id === 'nvidia'
|
||||
? nvidiaCompatFetch
|
||||
: openAICompatAsymmetricFetch);
|
||||
const client = createOpenAICompatible({
|
||||
name: recipe.id,
|
||||
@@ -2849,6 +2905,30 @@ async function classifyGatewayGuardrail(input: {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive OpenAI's `prompt_cache_key` (the AI SDK's `providerOptions.openai.
|
||||
* promptCacheKey`). It's a ROUTING hint, not a cache breakpoint: OpenAI caches
|
||||
* prefixes automatically, and a stable key makes requests sharing a prefix
|
||||
* land on the same engine, raising the hit rate (OpenAI cites 60%→87%).
|
||||
*
|
||||
* Hash the system prompt + sorted tool names — that's the stable prefix
|
||||
* gbrain's repeated loops (enrich, page-summary, skillopt, subagent) actually
|
||||
* share. Returns undefined when there's no system prompt (nothing stable to
|
||||
* key on), so one-off requests don't get pinned to a single engine. An
|
||||
* explicit key can still be set per provider/model via
|
||||
* `provider_chat_options` config, which overrides the derived key.
|
||||
*
|
||||
* @internal exported for tests; not part of the public gateway API.
|
||||
*/
|
||||
export function openAIPromptCacheKey(args: {
|
||||
system?: string;
|
||||
toolNames?: string[];
|
||||
}): string | undefined {
|
||||
if (!args.system) return undefined;
|
||||
const basis = `${args.system} ${(args.toolNames ?? []).slice().sort().join(',')}`;
|
||||
return `gbrain:${createHash('sha256').update(basis).digest('hex').slice(0, 32)}`;
|
||||
}
|
||||
|
||||
export function toAISDKTools(tools: ChatToolDef[] | undefined): Record<string, any> | undefined {
|
||||
if (!tools || tools.length === 0) return undefined;
|
||||
return tools.reduce((acc, t) => {
|
||||
@@ -2957,10 +3037,68 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
|
||||
const providerOptions: Record<string, any> = {};
|
||||
if (useCache) {
|
||||
// Call-level `providerOptions.anthropic.cacheControl` is NOT a no-op:
|
||||
// @ai-sdk/anthropic 3.0.47+ passes it through as a top-level
|
||||
// `cache_control` field on the Anthropic request body, which the
|
||||
// Messages API resolves as its documented "auto-cache the last
|
||||
// cacheable block in the request" shorthand (see Anthropic's
|
||||
// prompt-caching docs — "top-level auto-caching ... is the simplest
|
||||
// option when you don't need fine-grained placement"). Keep it: it's
|
||||
// what gives a growing multi-turn conversation (toolLoop()) a rolling
|
||||
// cache breakpoint on each turn's tail for free, without us having to
|
||||
// hand-roll the marker-walking logic subagent.ts's raw-SDK path uses.
|
||||
//
|
||||
// But "last cacheable block" is the wrong block for gbrain#2490's
|
||||
// actual callers (page-summary, skillopt, enrich): those are
|
||||
// single-turn calls with a STABLE system prompt and a DIFFERENT user
|
||||
// message every time, so the auto-marker lands on the ever-varying
|
||||
// tail — every call WRITES a fresh cache entry and never READS a prior
|
||||
// one (cache_read_input_tokens stays 0 forever). Caching the stable
|
||||
// prefix needs an EXPLICIT breakpoint on the system block itself,
|
||||
// which is applied below via a `SystemModelMessage` (round-trips its
|
||||
// own `providerOptions`) instead of a bare string.
|
||||
providerOptions.anthropic = { cacheControl: { type: 'ephemeral' } };
|
||||
}
|
||||
// OpenAI prompt_cache_key (native-openai only): a stable per-prefix routing
|
||||
// hint that keeps requests sharing a system prompt + tool set on the same
|
||||
// inference engine, lifting OpenAI's automatic prefix-cache hit rate. The
|
||||
// openai-compatible path (litellm/azure/groq/...) ignores
|
||||
// providerOptions.openai, so it gets nothing. Applied BEFORE the configured
|
||||
// provider options so `provider_chat_options.openai.promptCacheKey` from
|
||||
// config still overrides the derived key.
|
||||
if (recipe.implementation === 'native-openai') {
|
||||
const promptCacheKey = openAIPromptCacheKey({
|
||||
system: opts.system,
|
||||
toolNames: (opts.tools ?? []).map(t => t.name),
|
||||
});
|
||||
if (promptCacheKey) providerOptions.openai = { promptCacheKey };
|
||||
}
|
||||
applyConfiguredChatProviderOptions(providerOptions, cfg, recipe.id, modelId);
|
||||
|
||||
// Derive ONE canonical cache-control value AFTER config merging and reuse
|
||||
// it for every breakpoint (system block, last tool def, call-level). If
|
||||
// `provider_chat_options.anthropic.cacheControl` overrides the TTL (e.g.
|
||||
// `{ type: 'ephemeral', ttl: '1h' }`), that override lands in
|
||||
// `providerOptions.anthropic.cacheControl` via the deep-merge above —
|
||||
// reusing it here (instead of hardcoding `{ type: 'ephemeral' }` per
|
||||
// breakpoint) keeps every marker in the request on the same TTL.
|
||||
const cacheControlValue: { type: 'ephemeral'; ttl?: '5m' | '1h' } | undefined = useCache
|
||||
? (providerOptions.anthropic?.cacheControl ?? { type: 'ephemeral' })
|
||||
: undefined;
|
||||
|
||||
// Anthropic-only secondary breakpoint: mark the LAST tool def too (mirrors
|
||||
// subagent.ts's raw-SDK path — Anthropic caches everything up to and
|
||||
// including the last `cache_control` block it sees in the request, so
|
||||
// marking the last tool extends the cached prefix through the whole tool
|
||||
// list). `tool.providerOptions.anthropic.cacheControl` is the shape
|
||||
// @ai-sdk/anthropic 3.x reads for tool-def breakpoints.
|
||||
if (cacheControlValue && opts.tools && opts.tools.length > 0 && tools) {
|
||||
const lastTool = tools[opts.tools[opts.tools.length - 1]!.name];
|
||||
if (lastTool) {
|
||||
lastTool.providerOptions = { anthropic: { cacheControl: cacheControlValue } };
|
||||
}
|
||||
}
|
||||
|
||||
let _budgetRecorded = false;
|
||||
const _recordBudget = (modelLabel: string, inputTokens: number, outputTokens: number): void => {
|
||||
if (!tracker || _budgetRecorded) return;
|
||||
@@ -2977,10 +3115,28 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
}
|
||||
};
|
||||
|
||||
// The actual Anthropic system-prompt cache breakpoint. A bare string
|
||||
// `system` produces `{ role: 'system', content }` with no `providerOptions`
|
||||
// field (ai@6's convertToLanguageModelPrompt), so @ai-sdk/anthropic's
|
||||
// getCacheControl(providerOptions) on that block always resolves to
|
||||
// nothing. Passing a `SystemModelMessage` object instead — the shape `ai`
|
||||
// documents specifically for "additional provider options (e.g. for
|
||||
// caching)" — round-trips `providerOptions` onto that block. Byte-identical
|
||||
// to the old bare-string form when useCache is false. Reuses
|
||||
// `cacheControlValue` (the config-merged value) so this breakpoint's TTL
|
||||
// always matches the last-tool and call-level breakpoints.
|
||||
const systemParam = cacheControlValue && opts.system
|
||||
? {
|
||||
role: 'system' as const,
|
||||
content: opts.system,
|
||||
providerOptions: { anthropic: { cacheControl: cacheControlValue } },
|
||||
}
|
||||
: opts.system;
|
||||
|
||||
try {
|
||||
const result = await _generateTextTransport({
|
||||
model,
|
||||
system: opts.system,
|
||||
system: systemParam,
|
||||
messages: toModelMessages(repairToolPairing(opts.messages)) as any,
|
||||
tools: opts.tools && opts.tools.length > 0 ? tools : undefined,
|
||||
maxOutputTokens: opts.maxTokens ?? defaultMaxOutputTokens(modelStr),
|
||||
|
||||
@@ -17,13 +17,16 @@ export const anthropic: Recipe = {
|
||||
touchpoints: {
|
||||
// No embedding model available.
|
||||
expansion: {
|
||||
models: ['claude-haiku-4-5-20251001', 'claude-sonnet-4-6'],
|
||||
models: ['claude-haiku-4-5-20251001', 'claude-sonnet-5', 'claude-sonnet-4-6'],
|
||||
cost_per_1m_tokens_usd: 0.25,
|
||||
price_last_verified: '2026-05-10',
|
||||
},
|
||||
chat: {
|
||||
models: [
|
||||
'claude-fable-5',
|
||||
'claude-opus-4-8',
|
||||
'claude-opus-4-7',
|
||||
'claude-sonnet-5',
|
||||
'claude-sonnet-4-6',
|
||||
'claude-haiku-4-5-20251001',
|
||||
],
|
||||
|
||||
@@ -23,6 +23,9 @@ import { zhipu } from './zhipu.ts';
|
||||
import { azureOpenAI } from './azure-openai.ts';
|
||||
import { zeroentropyai } from './zeroentropyai.ts';
|
||||
import { llamaServerReranker } from './llama-server-reranker.ts';
|
||||
import { moonshot } from './moonshot.ts';
|
||||
import { mistral } from './mistral.ts';
|
||||
import { nvidia } from './nvidia.ts';
|
||||
|
||||
const ALL: Recipe[] = [
|
||||
openai,
|
||||
@@ -42,6 +45,9 @@ const ALL: Recipe[] = [
|
||||
zhipu,
|
||||
azureOpenAI,
|
||||
zeroentropyai,
|
||||
moonshot,
|
||||
mistral,
|
||||
nvidia,
|
||||
];
|
||||
|
||||
/** Map from `provider:id` key to recipe. */
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Mistral AI exposes an OpenAI-compatible API at https://api.mistral.ai/v1
|
||||
* (/embeddings + /chat/completions). EU-hosted — the reason this recipe
|
||||
* exists: a brain that must stay inside EU jurisdiction can run embed +
|
||||
* expansion + chat on a single provider without a US hop.
|
||||
*
|
||||
* Verified against the live API on 2026-07-19 (model catalog, embedding
|
||||
* dimensions, dimension-parameter rejection, and the batch ceiling — see
|
||||
* the notes on each field below).
|
||||
*
|
||||
* DIMENSIONS — mistral-embed is FIXED 1024 and accepts NO dimension
|
||||
* parameter at all. Both spellings are rejected upstream:
|
||||
* {"dimensions": 512} -> 400 extra_forbidden (not in the API schema)
|
||||
* {"output_dimension": 512} -> 400 "This model does not support output_dimension"
|
||||
* The generic `openai-compatible` branch of dims.ts:dimsProviderOptions()
|
||||
* already falls through to `return undefined` for these model ids, so no
|
||||
* dimension field is emitted. Do NOT add mistral-embed to any of the
|
||||
* flexible-dim allowlists there — it would 400 every embed call. Same
|
||||
* contract as voyage-4-nano, for the same reason.
|
||||
*
|
||||
* codestral-embed / codestral-embed-2505 are deliberately NOT listed: they
|
||||
* return 1536 dims, and a touchpoint carries a single `default_dims`.
|
||||
* Mixing them under a 1024 declaration is the mixed-dim footgun
|
||||
* embedding-dim-check.ts exists to catch. They are code-retrieval models
|
||||
* anyway; a prose brain wants mistral-embed.
|
||||
*/
|
||||
export const mistral: Recipe = {
|
||||
id: 'mistral',
|
||||
name: 'Mistral AI',
|
||||
tier: 'openai-compat',
|
||||
implementation: 'openai-compatible',
|
||||
base_url_default: 'https://api.mistral.ai/v1',
|
||||
auth_env: {
|
||||
required: ['MISTRAL_API_KEY'],
|
||||
setup_url: 'https://console.mistral.ai/api-keys',
|
||||
},
|
||||
touchpoints: {
|
||||
embedding: {
|
||||
models: ['mistral-embed', 'mistral-embed-2312'],
|
||||
default_dims: 1024,
|
||||
// Mistral's published list price. Advisory only — canonical embedding
|
||||
// spend accounting lives in src/core/embedding-pricing.ts.
|
||||
cost_per_1m_tokens_usd: 0.1,
|
||||
price_last_verified: '2026-07-19',
|
||||
// Measured ceiling, not a doc guess: the /embeddings endpoint accepts a
|
||||
// 65,286-token batch and rejects 66,960 with
|
||||
// 400 code 3210 "Too many tokens overall, split into more batches."
|
||||
// -> the real cap is 65,536 (64K) tokens per request.
|
||||
max_batch_tokens: 65_536,
|
||||
// chars_per_token is a DIVISOR in splitByTokenBudget()
|
||||
// (estTokens = text.length / charsPerToken), so a LOWER value is the
|
||||
// conservative direction. The module default of 4 is an English-prose
|
||||
// assumption; German prose measured 3.58 here, and code/JSON/CJK runs
|
||||
// denser still. 2 keeps the estimate above the real token count for
|
||||
// every content shape we see.
|
||||
chars_per_token: 2,
|
||||
// With safety_factor 0.5 the pre-split budget is 32,768 estimated
|
||||
// tokens = 65,536 chars. Worst realistic density (~1.5 chars/token)
|
||||
// puts that at ~43.7K real tokens — still clear of the 64K ceiling.
|
||||
safety_factor: 0.5,
|
||||
},
|
||||
expansion: {
|
||||
models: ['ministral-3b-latest', 'mistral-small-latest'],
|
||||
price_last_verified: '2026-07-19',
|
||||
},
|
||||
chat: {
|
||||
models: [
|
||||
'mistral-small-latest', 'mistral-medium-latest', 'mistral-large-latest',
|
||||
'ministral-3b-latest', 'ministral-8b-latest', 'magistral-small-latest',
|
||||
],
|
||||
supports_tools: true,
|
||||
// Same call as the Moonshot recipe: ordinary tool calls are fine, but
|
||||
// gbrain's subagent loop stays Anthropic-pinned for stable tool_use_id
|
||||
// behavior across crashes/replays.
|
||||
supports_subagent_loop: false,
|
||||
supports_prompt_cache: false,
|
||||
max_context_tokens: 262144,
|
||||
price_last_verified: '2026-07-19',
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://console.mistral.ai/api-keys, then `export MISTRAL_API_KEY=...` and use `mistral:mistral-embed` (1024 dims) for embeddings.',
|
||||
};
|
||||
@@ -0,0 +1,42 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Moonshot AI / Kimi Open Platform. Kimi exposes an OpenAI-compatible
|
||||
* /v1/chat/completions API at https://api.moonshot.ai/v1.
|
||||
*
|
||||
* Verified against Kimi API docs and live /v1/models on 2026-06-23.
|
||||
* The recipe is local-production glue until upstream GBrain carries a native
|
||||
* Moonshot recipe; keep it registered in the local patch registry.
|
||||
*/
|
||||
export const moonshot: Recipe = {
|
||||
id: 'moonshot',
|
||||
name: 'Moonshot AI / Kimi',
|
||||
tier: 'openai-compat',
|
||||
implementation: 'openai-compatible',
|
||||
base_url_default: 'https://api.moonshot.ai/v1',
|
||||
auth_env: {
|
||||
required: ['MOONSHOT_API_KEY'],
|
||||
setup_url: 'https://platform.kimi.ai/console/api-keys',
|
||||
},
|
||||
touchpoints: {
|
||||
expansion: {
|
||||
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
|
||||
// Kimi pricing varies by current promotional/account terms; do not use
|
||||
// this advisory field for budget enforcement. Canonical budget pricing
|
||||
// belongs in src/core/model-pricing.ts when verified for the account.
|
||||
price_last_verified: '2026-06-23',
|
||||
},
|
||||
chat: {
|
||||
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
|
||||
supports_tools: true,
|
||||
// Kimi tool calling is enough for ordinary chat/tool calls. GBrain's
|
||||
// subagent loop remains Anthropic-pinned because upstream requires stable
|
||||
// Anthropic-style tool_use_id behavior across crashes/replays.
|
||||
supports_subagent_loop: false,
|
||||
supports_prompt_cache: false,
|
||||
max_context_tokens: 256000,
|
||||
price_last_verified: '2026-06-23',
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://platform.kimi.ai/console/api-keys, then `export MOONSHOT_API_KEY=...` and use `moonshot:kimi-k2.7-code`.',
|
||||
};
|
||||
@@ -0,0 +1,71 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* NVIDIA NIM / API Catalog exposes OpenAI-compatible /v1/chat/completions
|
||||
* and /v1/embeddings APIs.
|
||||
*
|
||||
* Retrieval models use asymmetric encoding. The gateway maps gbrain's
|
||||
* document/query distinction to NVIDIA's wire values:
|
||||
* document -> input_type: passage
|
||||
* query -> input_type: query
|
||||
*
|
||||
* The model ids below intentionally keep NVIDIA's full catalog ids because
|
||||
* the hosted endpoint expects values like `nvidia/nv-embedqa-e5-v5` in the
|
||||
* request body. Short aliases are provided for CLI ergonomics.
|
||||
*/
|
||||
export const nvidia: Recipe = {
|
||||
id: 'nvidia',
|
||||
name: 'NVIDIA NIM',
|
||||
tier: 'openai-compat',
|
||||
implementation: 'openai-compatible',
|
||||
base_url_default: 'https://integrate.api.nvidia.com/v1',
|
||||
auth_env: {
|
||||
required: ['NVIDIA_API_KEY'],
|
||||
setup_url: 'https://build.nvidia.com',
|
||||
},
|
||||
aliases: {
|
||||
'nv-embedqa-e5-v5': 'nvidia/nv-embedqa-e5-v5',
|
||||
'llama-nemotron-embed-1b-v2': 'nvidia/llama-nemotron-embed-1b-v2',
|
||||
'nemotron-3-super': 'nvidia/nemotron-3-super-120b-a12b',
|
||||
'nemotron-3-super-120b-a12b': 'nvidia/nemotron-3-super-120b-a12b',
|
||||
'nv-embed-v1': 'nvidia/nv-embed-v1',
|
||||
'nv-embedcode-7b-v1': 'nvidia/nv-embedcode-7b-v1',
|
||||
},
|
||||
// No resolveAuth override: NVIDIA is plain `Authorization: Bearer <key>`,
|
||||
// which defaultResolveAuth derives from auth_env.required. IRON RULE
|
||||
// (test/ai/recipes-existing-regression.test.ts): only Azure overrides
|
||||
// resolveAuth.
|
||||
touchpoints: {
|
||||
chat: {
|
||||
models: [
|
||||
'nvidia/nemotron-3-super-120b-a12b',
|
||||
],
|
||||
supports_tools: false,
|
||||
supports_subagent_loop: false,
|
||||
// Do not treat Nemotron as a Minions subagent driver until tool-calling
|
||||
// and replay stability are proven through a separate adapter test.
|
||||
max_context_tokens: 128000,
|
||||
price_last_verified: '2026-05-24',
|
||||
},
|
||||
embedding: {
|
||||
models: [
|
||||
'nvidia/nv-embedqa-e5-v5',
|
||||
'nvidia/llama-nemotron-embed-1b-v2',
|
||||
'nvidia/nv-embed-v1',
|
||||
'nvidia/nv-embedcode-7b-v1',
|
||||
],
|
||||
// Default to the lightest tested hosted model. Larger NVIDIA models are
|
||||
// supported via explicit embedding_dimensions (2048 or 4096).
|
||||
default_dims: 1024,
|
||||
dims_options: [1024, 2048, 4096],
|
||||
// Conservative split; hosted NVIDIA embedding endpoints require
|
||||
// input_type and may reject large payloads before tokenizing.
|
||||
max_batch_tokens: 8192,
|
||||
chars_per_token: 4,
|
||||
safety_factor: 0.75,
|
||||
cost_per_1m_tokens_usd: undefined,
|
||||
price_last_verified: '2026-05-24',
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://build.nvidia.com, then `export NVIDIA_API_KEY=...`.',
|
||||
};
|
||||
@@ -100,7 +100,15 @@ function gbrainHome(): string {
|
||||
* core→commands import). which gbrain → process.execPath → argv[1] → "gbrain". */
|
||||
function resolveGbrainCliPath(): string {
|
||||
try {
|
||||
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
// #2747: `env: process.env` required under Bun — see the sibling copy
|
||||
// of this function in commands/autopilot.ts for the full explanation
|
||||
// (Bun snapshots process.env at its own startup; execSync without an
|
||||
// explicit env is blind to any PATH mutation since then).
|
||||
const which = execSync('which gbrain', {
|
||||
encoding: 'utf-8',
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
env: process.env,
|
||||
}).trim();
|
||||
if (which) return which;
|
||||
} catch { /* not on PATH */ }
|
||||
const exec = process.execPath ?? '';
|
||||
@@ -183,15 +191,17 @@ if [ "\${1:-}" = "--push-only" ]; then
|
||||
fi
|
||||
|
||||
_msg="\${1:?usage: brain-commit-push.sh <message> <path> [paths...]}"; shift || true
|
||||
# Pull first so the local tree is current before we stage.
|
||||
git fetch origin >/dev/null 2>&1 || true
|
||||
git pull --rebase origin "$_branch" || { git rebase --abort >/dev/null 2>&1 || true; echo "rebase conflict: manual attention needed" >&2; exit 3; }
|
||||
|
||||
# EXPLICIT paths only — never a blind 'git add -A' (would risk committing
|
||||
# secrets, temp files, or unrelated edits).
|
||||
if [ "$#" -eq 0 ]; then
|
||||
echo "refusing blind 'git add -A' — pass explicit path(s) to commit" >&2; exit 2
|
||||
fi
|
||||
# COMMIT BEFORE PULL (#2426): the old order (fetch + pull --rebase, THEN stage)
|
||||
# aborted on any dirty tree — 'cannot pull with rebase: You have unstaged
|
||||
# changes' — so the helper could never commit a MODIFIED page (exactly the
|
||||
# write-through case). Stage + commit first; brain_push below already handles
|
||||
# a remote that advanced (push -> rejected -> pull --rebase -> push).
|
||||
git add -- "$@"
|
||||
if git diff --cached --quiet; then echo "nothing to commit"; exit 0; fi
|
||||
git commit -m "$_msg"
|
||||
@@ -337,6 +347,47 @@ function uninstallLocalHook(repoPath: string): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the gbrain durability post-commit hook is installed — i.e. the
|
||||
* user opted this repo into push-durability via `gbrain sources harden`.
|
||||
* Cheap (one git-config read + one file read); used as the gate for
|
||||
* write-through auto-commit (#2426).
|
||||
*/
|
||||
export function isDurabilityHardened(repoPath: string): boolean {
|
||||
try {
|
||||
const { dir } = resolveHooksDir(repoPath);
|
||||
const hookPath = join(dir, 'post-commit');
|
||||
return existsSync(hookPath) && readFileSync(hookPath, 'utf-8').includes(HOOK_BANNER);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #2426: best-effort commit of a single write-through artifact so DB writes
|
||||
* reach git (the post-commit hook then background-pushes). Pre-fix,
|
||||
* write-through `.md` accumulated uncommitted forever: it never reached the
|
||||
* remote, froze `last_sync_at` (HEAD never moved), and a later `sync --full`
|
||||
* delete-reconcile treated the never-committed pages as disposable.
|
||||
*
|
||||
* Path-limited (`git commit -- <path>`) so unrelated staged/dirty edits are
|
||||
* never swept into the commit. Never throws; returns false on any failure
|
||||
* (index.lock contention, nothing changed, detached states) — the DB row and
|
||||
* the on-disk file remain the durable sinks either way.
|
||||
*/
|
||||
export function commitWriteThroughFile(repoPath: string, absPath: string, slug: string): boolean {
|
||||
try {
|
||||
const rel = relative(repoPath, absPath);
|
||||
if (!rel || rel.startsWith('..') || isAbsolute(rel)) return false;
|
||||
const gitOpts = { stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV } } as const;
|
||||
execFileSync('git', ['-C', repoPath, 'add', '--', rel], gitOpts);
|
||||
execFileSync('git', ['-C', repoPath, 'commit', '-m', `gbrain: write-through ${slug}`, '--', rel], gitOpts);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Committed helper ────────────────────────────────────────────────────────
|
||||
|
||||
function installHelper(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } {
|
||||
|
||||
+33
-26
@@ -409,6 +409,11 @@ export interface ScanOpts {
|
||||
visitDir?: (dirPath: string) => void;
|
||||
}
|
||||
|
||||
/** Timeout-arm winner for the COUNT-vs-deadline race in scanBrainSources.
|
||||
* A unique object so it can never collide with a legitimate COUNT result
|
||||
* (number | null). Module-private. */
|
||||
const DEADLINE_SENTINEL: unique symbol = Symbol('gbrain.scan.deadline');
|
||||
|
||||
export async function scanBrainSources(
|
||||
engine: BrainEngine,
|
||||
opts: ScanOpts = {},
|
||||
@@ -480,41 +485,43 @@ export async function scanBrainSources(
|
||||
// pool can make this await hang past the budget. Without the race, we'd
|
||||
// wait indefinitely AND defeat the wall-clock guarantee.
|
||||
let dbPageCount: number | null = null;
|
||||
// Set when the deadline race's timeout arm wins: the verdict that the
|
||||
// budget is spent, independent of any later Date.now() reading. Timer
|
||||
// callbacks on loaded runners can fire measurably EARLY relative to the
|
||||
// wall clock (a +1ms pad was drifted past in practice — see the flake
|
||||
// lineage in test/brain-writer-partial-scan.test.ts and issue #2946), so
|
||||
// the hung-COUNT path must not re-derive "did the deadline fire?" from
|
||||
// the clock the timer just raced against.
|
||||
let deadlineHit = false;
|
||||
if (opts.dbPageCountForSource) {
|
||||
try {
|
||||
if (opts.deadline) {
|
||||
const remainingMs = opts.deadline - Date.now();
|
||||
if (remainingMs <= 0) {
|
||||
dbPageCount = null;
|
||||
deadlineHit = true;
|
||||
} else {
|
||||
// Race COUNT against the deadline so a hung query can't eat the budget.
|
||||
//
|
||||
// Boundary overshoot (+1ms): the post-await deadline check at line
|
||||
// ~512 uses `Date.now() >= deadline`. setTimeout fires AT OR AFTER
|
||||
// the requested delay, so in theory the check always passes. In
|
||||
// practice on heavily-loaded CI runners (8 parallel shards × 4
|
||||
// concurrent test files = ~32 concurrent bun processes) we saw
|
||||
// intermittent failures where the timer callback resolved
|
||||
// microseconds BEFORE the wall-clock boundary, leaving Date.now()
|
||||
// a tick below deadline and the skip-check evaluating false. The
|
||||
// src-a scan then ran on a populated dir before src-b's
|
||||
// between-source check caught up — causing
|
||||
// `firstSource.status === 'skipped'` to receive 'scanned'.
|
||||
//
|
||||
// Adding 1ms guarantees the timer fires past the deadline by at
|
||||
// least one millisecond regardless of runner timer drift. Cost is
|
||||
// 1ms additional wall-clock latency on hung COUNT queries, which
|
||||
// is operationally negligible. Flake repro:
|
||||
// https://github.com/garrytan/gbrain/actions/runs/77611667786
|
||||
dbPageCount = await Promise.race([
|
||||
// Race COUNT against the deadline so a hung query can't eat the
|
||||
// budget. The timeout arm resolves a private sentinel — NOT null —
|
||||
// so a deadline win is distinguishable from a COUNT that resolved
|
||||
// null (failed/absent count keeps its existing semantics).
|
||||
const raced = await Promise.race([
|
||||
opts.dbPageCountForSource(src.id),
|
||||
new Promise<null>(resolve => setTimeout(() => resolve(null), remainingMs + 1)),
|
||||
new Promise<typeof DEADLINE_SENTINEL>(resolve =>
|
||||
setTimeout(() => resolve(DEADLINE_SENTINEL), remainingMs)),
|
||||
]);
|
||||
if (raced === DEADLINE_SENTINEL) {
|
||||
dbPageCount = null;
|
||||
deadlineHit = true;
|
||||
} else {
|
||||
dbPageCount = raced;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
dbPageCount = await opts.dbPageCountForSource(src.id);
|
||||
}
|
||||
} catch {
|
||||
// A throwing COUNT is a failed count, not a deadline verdict.
|
||||
dbPageCount = null;
|
||||
}
|
||||
}
|
||||
@@ -524,11 +531,11 @@ export async function scanBrainSources(
|
||||
// status='partial' with files_scanned=0, which is misleading ("partial
|
||||
// scan" when actually nothing was scanned). Mark this source + remainder
|
||||
// as 'skipped' so the doctor message is honest.
|
||||
// `>=` matches the between-source check above (line 445). The Promise.race
|
||||
// setTimeout resolves null at exactly `remainingMs` from now, so post-await
|
||||
// Date.now() often equals deadline within integer-ms precision — strict `>`
|
||||
// missed those landings on CI and let the next scanOneSource run anyway.
|
||||
if (opts.signal?.aborted || (opts.deadline && Date.now() >= opts.deadline)) {
|
||||
// `deadlineHit` is the authoritative verdict for the hung-COUNT path (the
|
||||
// sentinel above); the wall-clock re-check (`>=`, matching the
|
||||
// between-source check at line ~445) still covers a COUNT that RESOLVED
|
||||
// slowly enough to eat the budget without the timer winning.
|
||||
if (opts.signal?.aborted || deadlineHit || (opts.deadline && Date.now() >= opts.deadline)) {
|
||||
if (abortedAtSource === null) {
|
||||
abortedAtSource = src.id;
|
||||
}
|
||||
|
||||
@@ -26,7 +26,17 @@ export interface ChronicleJudgeInput {
|
||||
effectiveDate: string | null; // depth page effective_date (deterministic when)
|
||||
attendees: string[]; // deterministic who from frontmatter
|
||||
}
|
||||
export interface ChronicleJudgeResult { events: ChronicleEventProposal[] }
|
||||
export interface ChronicleJudgeResult {
|
||||
events: ChronicleEventProposal[];
|
||||
/**
|
||||
* #2606 — distinct judge-failure signal so an unusable response is never
|
||||
* recorded as a legitimate `no_events`:
|
||||
* - 'truncated': the model hit the output-token cap (stopReason 'length');
|
||||
* the JSON array was cut mid-stream and must not be parsed as complete.
|
||||
* - 'parse_failed': the model returned text but no valid JSON array.
|
||||
*/
|
||||
failure?: 'truncated' | 'parse_failed';
|
||||
}
|
||||
export type ChronicleJudge = (input: ChronicleJudgeInput) => Promise<ChronicleJudgeResult>;
|
||||
|
||||
export interface ChronicleExtractResult {
|
||||
@@ -126,6 +136,12 @@ export async function runChronicleExtract(
|
||||
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: 'judge_error' };
|
||||
}
|
||||
|
||||
// #2606: a truncated or unparseable judge response is a FAILURE, not an
|
||||
// empty page. Record it as a distinct skipped reason so operators (and
|
||||
// retries) can tell it apart from a genuine no_events.
|
||||
if (result?.failure) {
|
||||
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: `judge_${result.failure}` };
|
||||
}
|
||||
const proposals = Array.isArray(result?.events) ? result.events : [];
|
||||
if (proposals.length === 0) return { slug: opts.slug, status: 'no_events', events_written: 0 };
|
||||
// PARSE BARRIER — reject the WHOLE batch on any malformed proposal; no partial writes.
|
||||
@@ -167,11 +183,25 @@ const JUDGE_SYSTEM = `You segment a meeting/transcript page into discrete timeli
|
||||
Return ONLY a JSON array. Each element: {"when": ISO datetime or YYYY-MM-DD, "who": [entity slugs/names], "what": one-clause summary, "where": optional string, "kind": one of meeting|call|meal|solo|travel|work|commitment|decision|intro|conflict|milestone|event}.
|
||||
Prefer the page's known date for "when" when the text gives no explicit time. Use the provided attendee slugs for "who" when the text does not name participants. No prose, no markdown — just the JSON array.`;
|
||||
|
||||
/**
|
||||
* #2606: default output-token cap for the judge. Raised from the original
|
||||
* 1500 (which event-dense pages overflowed, silently truncating the JSON
|
||||
* array). Override via `chronicle.judge_max_tokens`.
|
||||
*/
|
||||
const DEFAULT_JUDGE_MAX_TOKENS = 4000;
|
||||
|
||||
function defaultJudge(engine: BrainEngine): ChronicleJudge {
|
||||
return async (input) => {
|
||||
const { isAvailable, chat } = await import('../ai/gateway.ts');
|
||||
if (!isAvailable('chat')) return { events: [] };
|
||||
const body = (input.body || '').slice(0, 12_000);
|
||||
// #2606: configurable cap so event-dense pages have headroom.
|
||||
let maxTokens = DEFAULT_JUDGE_MAX_TOKENS;
|
||||
const capRaw = await engine.getConfig('chronicle.judge_max_tokens').catch(() => null);
|
||||
if (capRaw) {
|
||||
const n = parseInt(capRaw, 10);
|
||||
if (Number.isFinite(n) && n > 0) maxTokens = n;
|
||||
}
|
||||
let text: string;
|
||||
try {
|
||||
const res = await chat({
|
||||
@@ -183,32 +213,44 @@ function defaultJudge(engine: BrainEngine): ChronicleJudge {
|
||||
`${input.title}\n\n${body}\n</page>\n\n` +
|
||||
`Known attendees: ${input.attendees.slice(0, 10).join(', ') || '(none)'}.\nExtract the events.`,
|
||||
}],
|
||||
maxTokens: 1500,
|
||||
maxTokens,
|
||||
});
|
||||
if (res.stopReason === 'refusal' || res.stopReason === 'content_filter') return { events: [] };
|
||||
// #2606: output hit the token cap — the JSON array is cut mid-stream.
|
||||
// Do NOT feed it to the parser as if complete; surface the truncation.
|
||||
if (res.stopReason === 'length') return { events: [], failure: 'truncated' };
|
||||
text = res.text;
|
||||
} catch (err) {
|
||||
if ((err as Error)?.name === 'AbortError') throw err;
|
||||
return { events: [] };
|
||||
}
|
||||
const parsed = parseJudgeJson(text);
|
||||
// #2606: non-empty model text with no parseable JSON array is a parse
|
||||
// failure, distinct from the model legitimately answering `[]`.
|
||||
if (parsed === null) return { events: [], failure: 'parse_failed' };
|
||||
return { events: parsed };
|
||||
};
|
||||
}
|
||||
|
||||
/** Tolerant JSON-array extraction from a model response (mirrors facts parser). */
|
||||
export function parseJudgeJson(text: string): ChronicleEventProposal[] {
|
||||
if (!text) return [];
|
||||
/**
|
||||
* Tolerant JSON-array extraction from a model response (mirrors facts parser).
|
||||
*
|
||||
* #2606: returns `null` on parse FAILURE (empty text, no `[...]` found,
|
||||
* JSON.parse throw, non-array result) so callers can distinguish "the model
|
||||
* said no events" (a legitimate `[]`) from "the response was unusable".
|
||||
*/
|
||||
export function parseJudgeJson(text: string): ChronicleEventProposal[] | null {
|
||||
if (!text) return null;
|
||||
let s = text.trim();
|
||||
const fence = s.match(/```(?:json)?\s*([\s\S]*?)```/i);
|
||||
if (fence) s = fence[1].trim();
|
||||
const start = s.indexOf('[');
|
||||
const end = s.lastIndexOf(']');
|
||||
if (start === -1 || end === -1 || end < start) return [];
|
||||
if (start === -1 || end === -1 || end < start) return null;
|
||||
try {
|
||||
const arr = JSON.parse(s.slice(start, end + 1));
|
||||
return Array.isArray(arr) ? arr : [];
|
||||
return Array.isArray(arr) ? arr : null;
|
||||
} catch {
|
||||
return [];
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -114,6 +114,26 @@ function resolveFlushGraceMs(): number {
|
||||
/** Default per-sink drain budget (matches drainAllBackgroundWorkForCliExit). */
|
||||
const DEFAULT_DRAIN_TIMEOUT_MS = 2_000;
|
||||
|
||||
/**
|
||||
* Resolve the per-sink drain budget: `GBRAIN_DRAIN_TIMEOUT_MS` env override
|
||||
* (slow-provider escape hatch, same env-only pattern as
|
||||
* GBRAIN_TEARDOWN_DEADLINE_MS) over the 2000ms default. An explicit
|
||||
* `drainTimeoutMs` from a call site still wins — the env replaces only the
|
||||
* DEFAULT. The 2s default assumes a sub-second cloud chat provider; a
|
||||
* self-hosted model (e.g. ollama at 10-20s per completion) can never finish a
|
||||
* fire-and-forget facts:absorb extraction inside it, so every one-shot CLI
|
||||
* exit — sync timers especially — aborts the in-flight chat and the
|
||||
* extraction never lands, retrying (and re-aborting) on each subsequent sync
|
||||
* of the same page. Raising the budget via env lets those installs drain
|
||||
* instead of abort; computeTeardownDeadlineMs already scales the backstop
|
||||
* from the resolved value, so the deadline widens with it.
|
||||
*/
|
||||
export function resolveDrainTimeoutMs(): number {
|
||||
const env = Number(process.env.GBRAIN_DRAIN_TIMEOUT_MS);
|
||||
if (Number.isFinite(env) && env > 0) return env;
|
||||
return DEFAULT_DRAIN_TIMEOUT_MS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Backstop deadline for drain + disconnect COMBINED, computed from the bounds
|
||||
* it guards so it fires only when a component violated its own bound (#2084
|
||||
@@ -262,7 +282,10 @@ export function flushThenExit(code: number, opts: FlushThenExitOpts = {}): void
|
||||
export interface FinishCliTeardownOpts {
|
||||
/** Engine to disconnect. A disconnect throw is warned + swallowed (D3). */
|
||||
engine: { disconnect(): Promise<void> };
|
||||
/** Per-sink drain budget. Default 2000 (the registry default). */
|
||||
/**
|
||||
* Per-sink drain budget. Default: `GBRAIN_DRAIN_TIMEOUT_MS` env override,
|
||||
* else 2000 (the registry default).
|
||||
*/
|
||||
drainTimeoutMs?: number;
|
||||
/** Test seam — wins over the env override and the computed formula. */
|
||||
deadlineMs?: number;
|
||||
@@ -284,7 +307,7 @@ export interface FinishCliTeardownOpts {
|
||||
* exit in here, and it means a component violated its own bound.
|
||||
*/
|
||||
export async function finishCliTeardown(opts: FinishCliTeardownOpts): Promise<void> {
|
||||
const drainTimeoutMs = opts.drainTimeoutMs ?? DEFAULT_DRAIN_TIMEOUT_MS;
|
||||
const drainTimeoutMs = opts.drainTimeoutMs ?? resolveDrainTimeoutMs();
|
||||
const warn = opts.warn ?? ((m: string) => console.warn(m));
|
||||
const drain = opts.drain ?? drainAllBackgroundWorkForCliExit;
|
||||
const deadlineMs =
|
||||
|
||||
@@ -271,6 +271,8 @@ export interface GBrainConfig {
|
||||
verdict_model?: string;
|
||||
max_prompt_tokens?: number;
|
||||
max_chunks_per_transcript?: number;
|
||||
subagent_timeout_ms?: number;
|
||||
subagent_wait_timeout_ms?: number;
|
||||
};
|
||||
patterns?: {
|
||||
lookback_days?: number;
|
||||
@@ -710,6 +712,12 @@ export async function loadConfigWithEngine(
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isFinite(n) && n > 0 ? n : undefined;
|
||||
}
|
||||
async function dbNum(key: string): Promise<number | undefined> {
|
||||
const v = await dbStr(key);
|
||||
if (v === undefined) return undefined;
|
||||
const n = Number(v);
|
||||
return Number.isNaN(n) ? undefined : n;
|
||||
}
|
||||
const dbWarnBytes = await dbInt('content_sanity.bytes_warn');
|
||||
const dbBlockBytes = await dbInt('content_sanity.bytes_block');
|
||||
const dbJunkEnabled = await dbBool('content_sanity.junk_patterns_enabled');
|
||||
@@ -759,6 +767,8 @@ export async function loadConfigWithEngine(
|
||||
const dbVerdictModel = await dbStr('dream.synthesize.verdict_model');
|
||||
const dbMaxPromptTokens = await dbInt('dream.synthesize.max_prompt_tokens');
|
||||
const dbMaxChunksPerTranscript = await dbInt('dream.synthesize.max_chunks_per_transcript');
|
||||
const dbSubagentTimeoutMs = await dbNum('dream.synthesize.subagent_timeout_ms');
|
||||
const dbSubagentWaitTimeoutMs = await dbNum('dream.synthesize.subagent_wait_timeout_ms');
|
||||
const dbLookbackDays = await dbInt('dream.patterns.lookback_days');
|
||||
const dbMinEvidence = await dbInt('dream.patterns.min_evidence');
|
||||
|
||||
@@ -783,6 +793,12 @@ export async function loadConfigWithEngine(
|
||||
if (mergedSynth.max_chunks_per_transcript === undefined && dbMaxChunksPerTranscript !== undefined) {
|
||||
mergedSynth.max_chunks_per_transcript = dbMaxChunksPerTranscript;
|
||||
}
|
||||
if (mergedSynth.subagent_timeout_ms === undefined && dbSubagentTimeoutMs !== undefined) {
|
||||
mergedSynth.subagent_timeout_ms = dbSubagentTimeoutMs;
|
||||
}
|
||||
if (mergedSynth.subagent_wait_timeout_ms === undefined && dbSubagentWaitTimeoutMs !== undefined) {
|
||||
mergedSynth.subagent_wait_timeout_ms = dbSubagentWaitTimeoutMs;
|
||||
}
|
||||
if (mergedPatterns.lookback_days === undefined && dbLookbackDays !== undefined) {
|
||||
mergedPatterns.lookback_days = dbLookbackDays;
|
||||
}
|
||||
@@ -854,6 +870,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// subagent handler's error message tells users to `config set` this, so it
|
||||
// must be a known key or `config set` rejects it without --force.
|
||||
'agent.use_gateway_loop',
|
||||
// #2778: per-turn output-token cap for the subagent loop (default 8192).
|
||||
'agent.max_output_tokens',
|
||||
// DB-plane (v0.32.3 search modes + related)
|
||||
'search.mode',
|
||||
'search.cache.enabled',
|
||||
@@ -888,6 +906,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'models.chat',
|
||||
'models.eval.longmemeval',
|
||||
'facts.extraction_model',
|
||||
// #2113: output-token cap for the per-turn facts extractor (default 4000).
|
||||
'facts.extraction_max_tokens',
|
||||
// Dream cycle config
|
||||
'dream.synthesize.session_corpus_dir',
|
||||
'dream.synthesize.meeting_transcripts_dir',
|
||||
@@ -895,8 +915,16 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'dream.synthesize.verdict_model',
|
||||
'dream.synthesize.max_prompt_tokens',
|
||||
'dream.synthesize.max_chunks_per_transcript',
|
||||
// #2415: top-level namespace for synthesize/patterns output (default 'wiki').
|
||||
'dream.synthesize.output_root',
|
||||
'dream.synthesize.subagent_timeout_ms',
|
||||
'dream.synthesize.subagent_wait_timeout_ms',
|
||||
'dream.patterns.lookback_days',
|
||||
'dream.patterns.min_evidence',
|
||||
// #2782-family: patterns-phase subagent timeouts (mirror of the
|
||||
// dream.synthesize.* pair from #1594).
|
||||
'dream.patterns.subagent_timeout_ms',
|
||||
'dream.patterns.subagent_wait_timeout_ms',
|
||||
// Emotional weight (v0.29)
|
||||
'emotional_weight.high_tags',
|
||||
'emotional_weight.user_holder',
|
||||
@@ -945,6 +973,10 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// operator had to discover --force by reading source. Same class as the
|
||||
// spend-controls registration above.
|
||||
'auto_chronicle',
|
||||
// #2606: chronicle judge output-token cap (default 4000). Event-dense
|
||||
// pages overflowed the old hardcoded 1500 and were misrecorded as
|
||||
// no_events; the cap is now configurable and truncation is surfaced.
|
||||
'chronicle.judge_max_tokens',
|
||||
// Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate
|
||||
// consent reads this key, and enabling it is the documented path to
|
||||
// `gbrain takes extract --from-pages` — same unregistered-key class.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
/**
|
||||
* v0.41.16.0 — Built-in conversation parser pattern registry.
|
||||
*
|
||||
* Fourteen hand-vetted patterns covering the chat-export formats this
|
||||
* Fifteen hand-vetted patterns covering the chat-export formats this
|
||||
* codebase is most likely to encounter. Each pattern's regex was
|
||||
* derived from a public format reference (source_doc field) so future
|
||||
* maintainers can verify against the wild shape.
|
||||
@@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string {
|
||||
return stripped || raw.trim();
|
||||
}
|
||||
|
||||
/** The 14 hand-vetted built-in patterns. */
|
||||
/** The 15 hand-vetted built-in patterns. */
|
||||
export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
||||
// -------------------------------------------------------------------
|
||||
// INLINE-DATE patterns (date in every line; less ambiguous; tried first).
|
||||
@@ -178,6 +178,41 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
|
||||
'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)',
|
||||
},
|
||||
|
||||
{
|
||||
// iMessage sync's time-only 12-hour shape. AM/PM is required so this
|
||||
// cannot shadow bold-paren-time's 24-hour form or imessage-slack's
|
||||
// full-date form.
|
||||
id: 'bold-paren-time-12h',
|
||||
origin: 'builtin',
|
||||
regex: /^\*\*(.+?)\*\*\s*\((\d{1,2}):(\d{2})\s*(AM|PM|am|pm)\)\s*:\s*(.*)$/,
|
||||
captures: {
|
||||
speaker_group: 1,
|
||||
hour_group: 2,
|
||||
minute_group: 3,
|
||||
ampm_group: 4,
|
||||
text_group: 5,
|
||||
},
|
||||
date_source: 'frontmatter',
|
||||
time_format: '12h_ampm',
|
||||
timezone_policy: 'utc_assumed_with_warn',
|
||||
multi_line: false,
|
||||
quick_reject: /^\*\*/,
|
||||
test_positive: [
|
||||
'**Me** (9:04 AM): sounds good, see you then',
|
||||
'**+155****0135** (9:39 AM): Will do',
|
||||
'**Alice Example** (12:00 PM): noon message',
|
||||
'**Bob Example** (5:38 pm): lowercase ampm',
|
||||
],
|
||||
test_negative: [
|
||||
'**Alice** (00:00): 24h shape',
|
||||
'**Alice Example** (2024-03-15 9:00 AM): full-date iMessage shape',
|
||||
'**[18:37] G T:** telegram bracket',
|
||||
'Alice (9:00 AM): missing the bold',
|
||||
],
|
||||
source_doc:
|
||||
'Time-only 12h AM/PM iMessage export shape: `**Speaker** (H:MM AM): text`',
|
||||
},
|
||||
|
||||
{
|
||||
// Fathom/phone-call raw transcripts in this workspace use a plain
|
||||
// `Speaker A: ...` / `Speaker B: ...` shape with no per-line time.
|
||||
|
||||
@@ -321,11 +321,22 @@ export function applyPattern(
|
||||
if (!body) return [];
|
||||
const out: MatchedMessage[] = [];
|
||||
const lines = body.split(/\r?\n/);
|
||||
// Some multi-day conversation exports use markdown date headings instead
|
||||
// of repeating a date on every message. Keep the caller's context immutable
|
||||
// while advancing a local date anchor as those headings are encountered.
|
||||
const runningCtx: DateContext = { ...dateCtx };
|
||||
const dateHeaderRe = /^#{1,4}\s+(\d{4}-\d{2}-\d{2})\s*$/;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const rawLine = lines[i];
|
||||
const line = rawLine.trim();
|
||||
if (!line) continue;
|
||||
|
||||
const dateHeader = dateHeaderRe.exec(line);
|
||||
if (dateHeader) {
|
||||
runningCtx.fallbackDate = dateHeader[1];
|
||||
continue;
|
||||
}
|
||||
|
||||
// Quick-reject fast path.
|
||||
if (entry.quick_reject && !entry.quick_reject.test(line)) {
|
||||
// Continuation handling for orphan lines.
|
||||
@@ -339,7 +350,7 @@ export function applyPattern(
|
||||
|
||||
const m = entry.regex.exec(line);
|
||||
if (m) {
|
||||
const iso = buildIso(m, entry, dateCtx);
|
||||
const iso = buildIso(m, entry, runningCtx);
|
||||
if (iso === null) continue; // reconstruction failed; skip line
|
||||
const rawSpeaker = m[entry.captures.speaker_group] ?? '';
|
||||
const speaker = cleanSpeaker(rawSpeaker, entry.speaker_clean);
|
||||
@@ -380,7 +391,7 @@ function getNonBlankLines(body: string, headCap?: number): string[] {
|
||||
* window) and `scorePatternFull` (whole body) delegate here so the
|
||||
* quick_reject + regex loop lives in one place. Reused by
|
||||
* `parseConversation`'s fallback path which pre-splits ONCE and
|
||||
* passes the array to all 12 candidates (saves 11 redundant body
|
||||
* passes the array to all 15 candidates (saves 14 redundant body
|
||||
* splits per fallback pass).
|
||||
*/
|
||||
function scoreFromLines(
|
||||
|
||||
+48
-2
@@ -479,6 +479,37 @@ export interface CycleOpts {
|
||||
* Validated via `assertValidSourceId` in `cycleLockIdFor` (defense-in-depth).
|
||||
*/
|
||||
sourceId?: string;
|
||||
/**
|
||||
* issue #2860 — one-shot per-invocation bypass of a phase's own
|
||||
* `dream.<phase>.enabled` / `cycle.<phase>.enabled` config gate. Wired
|
||||
* from `gbrain dream --phase <name> --once`.
|
||||
*
|
||||
* Deliberately typed as the SINGLE named `CyclePhase`, not a boolean —
|
||||
* each gated phase's dispatch block below only honors the override when
|
||||
* `onceForPhase` matches ITS OWN phase name, so the bypass can never leak
|
||||
* to a different phase even if a caller passes a wider `phases` array
|
||||
* than the CLI does (the CLI always restricts to `phases: [phase]`).
|
||||
*
|
||||
* Never reads or writes config — the phase still evaluates its config
|
||||
* gate every call; this only overrides the boolean OUTCOME for that one
|
||||
* call. Applies to: patterns, synthesize, conversation_facts_backfill,
|
||||
* enrich_thin, skillopt (the phases that gate on a `.enabled` config
|
||||
* key read inside the phase's own module). Does NOT apply to
|
||||
* extract_atoms / synthesize_concepts — those are pack-gated via
|
||||
* `packDeclaresPhase`, a different mechanism with its own existing
|
||||
* one-shot escape hatch (`--drain` for extract_atoms).
|
||||
*/
|
||||
onceForPhase?: CyclePhase;
|
||||
/**
|
||||
* Absolute wall-clock deadline (epoch ms) of the enclosing minion job,
|
||||
* from `MinionJobContext.deadlineAtMs` (the claim-time `timeout_at`
|
||||
* stamp). Phases that spawn bounded sub-work (patterns' subagent) clamp
|
||||
* their own timeouts to the REMAINING time so one phase's fixed
|
||||
* worst-case can't blow past the job budget and dead-letter the whole
|
||||
* cycle mid-phase (#2781). Unset for direct callers (`gbrain dream`) —
|
||||
* phases then use their configured timeouts unchanged.
|
||||
*/
|
||||
deadlineAtMs?: number | null;
|
||||
}
|
||||
|
||||
// ─── Lock primitives ───────────────────────────────────────────────
|
||||
@@ -1682,6 +1713,10 @@ export async function runCycle(
|
||||
from: opts.synthFrom,
|
||||
to: opts.synthTo,
|
||||
bypassDreamGuard: opts.synthBypassDreamGuard,
|
||||
// #1586: scope synthesized writes to the cycle's resolved source
|
||||
// (explicit --source wins, else derived from the checkout dir).
|
||||
sourceId: cycleSourceId,
|
||||
once: opts.onceForPhase === 'synthesize',
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
@@ -1885,6 +1920,8 @@ export async function runCycle(
|
||||
brainDir,
|
||||
dryRun,
|
||||
yieldDuringPhase: opts.yieldDuringPhase,
|
||||
once: opts.onceForPhase === 'patterns',
|
||||
deadlineAtMs: opts.deadlineAtMs ?? null,
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
@@ -2100,7 +2137,11 @@ export async function runCycle(
|
||||
progress.start('cycle.conversation_facts_backfill');
|
||||
const { runPhaseConversationFactsBackfill } = await import('./cycle/conversation-facts-backfill.ts');
|
||||
const { result, duration_ms } = await timePhase(() =>
|
||||
runPhaseConversationFactsBackfill(engine, { dryRun, signal: opts.signal }),
|
||||
runPhaseConversationFactsBackfill(engine, {
|
||||
dryRun,
|
||||
signal: opts.signal,
|
||||
once: opts.onceForPhase === 'conversation_facts_backfill',
|
||||
}),
|
||||
);
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
@@ -2128,7 +2169,11 @@ export async function runCycle(
|
||||
progress.start('cycle.enrich_thin');
|
||||
const { runPhaseEnrichThin } = await import('./cycle/enrich-thin.ts');
|
||||
const { result, duration_ms } = await timePhase(() =>
|
||||
runPhaseEnrichThin(engine, { dryRun, signal: opts.signal }),
|
||||
runPhaseEnrichThin(engine, {
|
||||
dryRun,
|
||||
signal: opts.signal,
|
||||
once: opts.onceForPhase === 'enrich_thin',
|
||||
}),
|
||||
);
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
@@ -2159,6 +2204,7 @@ export async function runCycle(
|
||||
runPhaseSkillopt({
|
||||
engine,
|
||||
dryRun,
|
||||
once: opts.onceForPhase === 'skillopt',
|
||||
...(opts.signal ? { signal: opts.signal } : {}),
|
||||
}),
|
||||
);
|
||||
|
||||
@@ -57,6 +57,14 @@ import {
|
||||
export interface ConversationFactsBackfillPhaseOpts {
|
||||
dryRun?: boolean;
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* issue #2860 — `gbrain dream --phase conversation_facts_backfill --once`.
|
||||
* Bypasses the `cycle.conversation_facts_backfill.enabled` gate for THIS
|
||||
* call only; never reads or writes config. Per-source + brain-wide cost/
|
||||
* walltime caps still apply — the override lifts the on/off switch, not
|
||||
* the spend guards.
|
||||
*/
|
||||
once?: boolean;
|
||||
}
|
||||
|
||||
/** Phase return shape (matches PhaseResult contract from cycle.ts). */
|
||||
@@ -155,17 +163,23 @@ export async function runPhaseConversationFactsBackfill(
|
||||
const cfg = await loadCfg(engine);
|
||||
|
||||
if (!cfg.enabled) {
|
||||
return {
|
||||
phase: 'conversation_facts_backfill',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'cycle.conversation_facts_backfill.enabled=false (default OFF)',
|
||||
details: {
|
||||
reason: 'disabled',
|
||||
enable_hint:
|
||||
'gbrain config set cycle.conversation_facts_backfill.enabled true',
|
||||
},
|
||||
};
|
||||
if (!opts.once) {
|
||||
return {
|
||||
phase: 'conversation_facts_backfill',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'cycle.conversation_facts_backfill.enabled=false (default OFF)',
|
||||
details: {
|
||||
reason: 'disabled',
|
||||
enable_hint:
|
||||
'gbrain config set cycle.conversation_facts_backfill.enabled true',
|
||||
},
|
||||
};
|
||||
}
|
||||
process.stderr.write(
|
||||
'[dream] --once: cycle.conversation_facts_backfill.enabled is false but ' +
|
||||
'--phase conversation_facts_backfill --once forces this run (config untouched)\n',
|
||||
);
|
||||
}
|
||||
|
||||
const startedAt = Date.now();
|
||||
|
||||
@@ -45,6 +45,12 @@ import {
|
||||
export interface EnrichThinPhaseOpts {
|
||||
dryRun?: boolean;
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* issue #2860 — `gbrain dream --phase enrich_thin --once`. Bypasses the
|
||||
* `cycle.enrich_thin.enabled` gate for THIS call only; never reads or
|
||||
* writes config. Per-source + brain-wide cost/walltime caps still apply.
|
||||
*/
|
||||
once?: boolean;
|
||||
}
|
||||
|
||||
export interface EnrichThinPhaseResult {
|
||||
@@ -139,16 +145,22 @@ export async function runPhaseEnrichThin(
|
||||
const cfg = await loadCfg(engine);
|
||||
|
||||
if (!cfg.enabled) {
|
||||
return {
|
||||
phase: 'enrich_thin',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'cycle.enrich_thin.enabled=false (default OFF)',
|
||||
details: {
|
||||
reason: 'disabled',
|
||||
enable_hint: 'gbrain config set cycle.enrich_thin.enabled true',
|
||||
},
|
||||
};
|
||||
if (!opts.once) {
|
||||
return {
|
||||
phase: 'enrich_thin',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'cycle.enrich_thin.enabled=false (default OFF)',
|
||||
details: {
|
||||
reason: 'disabled',
|
||||
enable_hint: 'gbrain config set cycle.enrich_thin.enabled true',
|
||||
},
|
||||
};
|
||||
}
|
||||
process.stderr.write(
|
||||
'[dream] --once: cycle.enrich_thin.enabled is false but ' +
|
||||
'--phase enrich_thin --once forces this run (config untouched)\n',
|
||||
);
|
||||
}
|
||||
|
||||
const startedAt = Date.now();
|
||||
|
||||
+125
-30
@@ -11,15 +11,17 @@
|
||||
* 1. Reads the markdown body (DB-side fetch via engine.getPage).
|
||||
* 2. Parses the `## Facts` fence with parseFactsFence.
|
||||
* 3. Maps ParsedFact → FenceExtractedFact via extractFactsFromFenceText.
|
||||
* 4. Wipes the page's DB index via deleteFactsForPage.
|
||||
* 5. Re-inserts via engine.insertFacts batch.
|
||||
* 4. De-dupes rows by canonical (claim, source) content key.
|
||||
* 5. Reconciles the page-scoped DB index: no-op when already in sync,
|
||||
* insert only missing keys when possible, or wipe/reinsert when stale
|
||||
* DB rows need cleanup (#1781 — the unconditional wipe-and-reinsert
|
||||
* made every cycle non-idempotent, re-appending duplicate rows).
|
||||
*
|
||||
* After the phase, the DB index for every affected page byte-matches
|
||||
* the fence (modulo embeddings + runtime-derived fields). Pages with
|
||||
* no fence go through delete-then-empty-insert — DB rows for that
|
||||
* page coordinate are wiped; legacy NULL-source_markdown_slug rows
|
||||
* survive because deleteFactsForPage targets source_markdown_slug =
|
||||
* slug only.
|
||||
* After the phase, the DB index for every affected page matches the
|
||||
* fence's canonical (claim, source) row set (modulo embeddings +
|
||||
* runtime-derived fields). Pages with no fence wipe DB rows for that
|
||||
* page coordinate only; legacy NULL-source_markdown_slug rows survive
|
||||
* because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
*
|
||||
* Empty-fence guard (Codex R2-#7): the phase refuses to do its
|
||||
* destructive reconciliation pass when legacy rows (row_num IS NULL,
|
||||
@@ -35,7 +37,11 @@ import type { BrainEngine } from '../engine.ts';
|
||||
import { writeReceipt } from '../extract/receipt-writer.ts';
|
||||
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
|
||||
import { parseFactsFence } from '../facts-fence.ts';
|
||||
import { extractFactsFromFenceText } from '../facts/extract-from-fence.ts';
|
||||
import {
|
||||
extractFactsFromFenceText,
|
||||
FENCE_SOURCE_DEFAULT,
|
||||
type FenceExtractedFact,
|
||||
} from '../facts/extract-from-fence.ts';
|
||||
import {
|
||||
runPhantomRedirectPass,
|
||||
emptyPhantomPassResult,
|
||||
@@ -44,6 +50,51 @@ import {
|
||||
import { embed, isAvailable } from '../ai/gateway.ts';
|
||||
import { isAborted } from '../abort-check.ts';
|
||||
|
||||
interface ExistingPageFact {
|
||||
fact: string;
|
||||
source: string | null;
|
||||
row_num: number | string | null;
|
||||
}
|
||||
|
||||
function factContentKey(fact: string, source: string | null | undefined): string {
|
||||
return `${fact}\u0000${source ?? FENCE_SOURCE_DEFAULT}`;
|
||||
}
|
||||
|
||||
function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFact[] {
|
||||
const seen = new Set<string>();
|
||||
const deduped: FenceExtractedFact[] = [];
|
||||
for (const fact of facts) {
|
||||
const key = factContentKey(fact.fact, fact.source);
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
deduped.push(fact);
|
||||
}
|
||||
return deduped;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fence-owned DB rows for one page coordinate. Excludes `cli:`-origin
|
||||
* conversation facts (#1928) — they are not fence-owned, so they must
|
||||
* neither count as "stale" (which would force a wipe every cycle) nor
|
||||
* be compared against the fence's row set. Mirrors the
|
||||
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
|
||||
*/
|
||||
async function listExistingFactsForPage(
|
||||
engine: BrainEngine,
|
||||
slug: string,
|
||||
sourceId: string,
|
||||
): Promise<ExistingPageFact[]> {
|
||||
return engine.executeRaw<ExistingPageFact>(
|
||||
`SELECT fact, source, row_num
|
||||
FROM facts
|
||||
WHERE source_id = $1
|
||||
AND source_markdown_slug = $2
|
||||
AND COALESCE(source, '') NOT LIKE 'cli:%'
|
||||
ORDER BY row_num ASC, id ASC`,
|
||||
[sourceId, slug],
|
||||
);
|
||||
}
|
||||
|
||||
export interface ExtractFactsOpts {
|
||||
/** Subset of slugs to reconcile. undefined = walk every page in the brain. */
|
||||
slugs?: string[];
|
||||
@@ -220,28 +271,70 @@ export async function runExtractFacts(
|
||||
|
||||
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
|
||||
|
||||
if (opts.dryRun) continue;
|
||||
|
||||
// Wipe-and-reinsert per page. The delete targets source_markdown_slug =
|
||||
// slug only, so NULL-source_markdown_slug legacy rows survive (the
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts (conversation
|
||||
// facts from extract-conversation-facts) are NOT fence-owned — the page
|
||||
// carries no `## Facts` fence to recreate them — so they MUST survive this
|
||||
// reconcile. Exclude them from the wipe.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
|
||||
if (parsed.facts.length === 0) continue;
|
||||
|
||||
// v0.35.4 (D-ENG-1) — thread page.effective_date as the fallback
|
||||
// valid_from. Without this, fence rows without explicit `validFrom:`
|
||||
// land with `valid_from = now()` (import timestamp) and every
|
||||
// trajectory query against the page returns import dates instead of
|
||||
// claim dates.
|
||||
const pageEffectiveDate = page.effective_date ? new Date(page.effective_date) : null;
|
||||
const extracted = extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate });
|
||||
const extracted = dedupeFactsByContentKey(
|
||||
extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate }),
|
||||
);
|
||||
|
||||
if (opts.dryRun) continue;
|
||||
|
||||
// #1781 — reconcile instead of unconditional wipe-and-reinsert. Compare
|
||||
// the fence's canonical (claim, source) row set against the page's
|
||||
// fence-owned DB rows: no-op when already in sync, insert only missing
|
||||
// keys when possible, wipe/reinsert only when stale rows need cleanup.
|
||||
const existing = await listExistingFactsForPage(engine, slug, sourceId);
|
||||
const existingKeys = new Set(existing.map(f => factContentKey(f.fact, f.source)));
|
||||
const desiredByKey = new Map(extracted.map(f => [factContentKey(f.fact, f.source), f]));
|
||||
|
||||
if (extracted.length === 0) {
|
||||
if (existing.length > 0) {
|
||||
// The delete targets source_markdown_slug = slug only, so
|
||||
// NULL-source_markdown_slug legacy rows survive (the
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
|
||||
// (conversation facts from extract-conversation-facts) are NOT
|
||||
// fence-owned — the page carries no `## Facts` fence to recreate
|
||||
// them — so they MUST survive this reconcile.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const hasStaleExisting = existing.some(f => !desiredByKey.has(factContentKey(f.fact, f.source)));
|
||||
const hasDuplicateExisting = existing.length !== existingKeys.size;
|
||||
const hasRowNumDrift = existing.some(f => {
|
||||
const desired = desiredByKey.get(factContentKey(f.fact, f.source));
|
||||
return desired !== undefined && Number(f.row_num) !== desired.row_num;
|
||||
});
|
||||
|
||||
if (
|
||||
existing.length === extracted.length &&
|
||||
!hasStaleExisting &&
|
||||
!hasDuplicateExisting &&
|
||||
!hasRowNumDrift
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let toInsert = extracted.filter(f => !existingKeys.has(factContentKey(f.fact, f.source)));
|
||||
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
|
||||
// Fall back to the legacy page-level reconcile when old DB rows must
|
||||
// be removed. Same delete scoping as above: legacy
|
||||
// NULL-source_markdown_slug rows and `cli:`-origin conversation
|
||||
// facts (#1928) survive.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
toInsert = extracted;
|
||||
}
|
||||
|
||||
// v0.35.4 (D-CDX-3) — batch-embed before insert. Without this,
|
||||
// cycle-inserted facts land with `embedding = NULL`, which breaks
|
||||
@@ -250,17 +343,17 @@ export async function runExtractFacts(
|
||||
// unavailable (no API key configured), facts still insert with
|
||||
// NULL embeddings — drift_score gracefully returns null and
|
||||
// clustering falls back to recency.
|
||||
if (isAvailable('embedding') && extracted.length > 0) {
|
||||
if (isAvailable('embedding') && toInsert.length > 0) {
|
||||
try {
|
||||
const texts = extracted.map(e => e.fact);
|
||||
const texts = toInsert.map(e => e.fact);
|
||||
// #1972: forward the abort signal so a cancelled cycle's in-flight
|
||||
// batch embed (a network call) is itself abortable, not just the loop.
|
||||
const embeddings = await embed(texts, { abortSignal: opts.signal });
|
||||
// Defensive: embed should return one vector per input; if the
|
||||
// gateway returns a partial array (provider partial-batch retry
|
||||
// returning fewer than requested), only fill what we have.
|
||||
for (let i = 0; i < extracted.length && i < embeddings.length; i++) {
|
||||
extracted[i].embedding = embeddings[i];
|
||||
for (let i = 0; i < toInsert.length && i < embeddings.length; i++) {
|
||||
toInsert[i].embedding = embeddings[i];
|
||||
}
|
||||
} catch (err) {
|
||||
// Embedding failure is non-fatal — facts still get inserted, just
|
||||
@@ -271,7 +364,9 @@ export async function runExtractFacts(
|
||||
}
|
||||
}
|
||||
|
||||
const inserted = await engine.insertFacts(extracted, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB
|
||||
if (toInsert.length === 0) continue;
|
||||
|
||||
const inserted = await engine.insertFacts(toInsert, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB
|
||||
result.factsInserted += inserted.inserted;
|
||||
}
|
||||
|
||||
|
||||
+182
-41
@@ -19,7 +19,7 @@
|
||||
*/
|
||||
|
||||
import { join, dirname } from 'node:path';
|
||||
import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
|
||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||||
import { MinionQueue } from '../minions/queue.ts';
|
||||
@@ -27,11 +27,73 @@ import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.
|
||||
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
import type { Page, PageType } from '../types.ts';
|
||||
// #2415: allow-list + output-root resolution shared with the synthesize
|
||||
// phase — both phases must agree on the configured namespace.
|
||||
import { loadAllowedSlugPrefixes, loadOutputRoot } from './synthesize.ts';
|
||||
import { probeChatModel } from '../ai/gateway.ts';
|
||||
import { normalizeModelId } from '../model-id.ts';
|
||||
|
||||
export interface PatternsPhaseOpts {
|
||||
brainDir: string;
|
||||
dryRun: boolean;
|
||||
yieldDuringPhase?: () => Promise<void>;
|
||||
/**
|
||||
* issue #2860 — `gbrain dream --phase patterns --once`. Bypasses the
|
||||
* `dream.patterns.enabled` gate for THIS call only; never reads or
|
||||
* writes config.
|
||||
*/
|
||||
once?: boolean;
|
||||
/**
|
||||
* Absolute deadline (epoch ms) of the enclosing minion job, or null for
|
||||
* direct callers (`gbrain dream`). When set, the subagent's job timeout
|
||||
* and the wait timeout are clamped so the phase finishes (or times out)
|
||||
* BEFORE the parent job's budget expires — a fixed 30/35-min default
|
||||
* inside an interval-derived cycle budget dead-letters the whole cycle
|
||||
* mid-phase and starves every tail phase (#2781).
|
||||
*/
|
||||
deadlineAtMs?: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop-margin reserved under the parent deadline when clamping subagent
|
||||
* budgets. NOT a promise that tail phases complete — the cycle is allowed
|
||||
* to go partial and resume next tick. This only guarantees the phase's
|
||||
* wait returns and the handler unwinds cleanly before the worker's abort
|
||||
* fires: wait poll interval (5s) + worker force-evict grace (30s) + lock
|
||||
* and DB cleanup headroom.
|
||||
*/
|
||||
export const CYCLE_DEADLINE_RESERVE_MS = 60 * 1000;
|
||||
|
||||
/**
|
||||
* Smallest remaining budget worth submitting a subagent for. Below this,
|
||||
* the LLM call is near-certain to be killed mid-flight — wasted spend and
|
||||
* a guaranteed-timeout child — so the phase skips honestly instead
|
||||
* (`insufficient_cycle_budget`) and the next cycle retries with a fresh
|
||||
* budget.
|
||||
*/
|
||||
export const MIN_PATTERNS_SUBAGENT_BUDGET_MS = 2 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* Clamp the configured subagent budgets to the remaining parent-job time.
|
||||
* Both timeouts derive from the SAME absolute child deadline
|
||||
* (`deadlineAtMs - reserve`) so the child job's kill switch and our wait
|
||||
* agree. Returns null when the remaining budget is below the minimum —
|
||||
* caller should skip the phase without submitting.
|
||||
*/
|
||||
export function clampSubagentBudgets(
|
||||
config: { subagentTimeoutMs: number; subagentWaitTimeoutMs: number },
|
||||
deadlineAtMs: number | null | undefined,
|
||||
nowMs: number,
|
||||
): { timeoutMs: number; waitTimeoutMs: number } | null {
|
||||
if (deadlineAtMs == null) {
|
||||
return { timeoutMs: config.subagentTimeoutMs, waitTimeoutMs: config.subagentWaitTimeoutMs };
|
||||
}
|
||||
const childBudgetMs = deadlineAtMs - CYCLE_DEADLINE_RESERVE_MS - nowMs;
|
||||
if (childBudgetMs < MIN_PATTERNS_SUBAGENT_BUDGET_MS) return null;
|
||||
return {
|
||||
timeoutMs: Math.min(config.subagentTimeoutMs, childBudgetMs),
|
||||
waitTimeoutMs: Math.min(config.subagentWaitTimeoutMs, childBudgetMs),
|
||||
};
|
||||
}
|
||||
|
||||
export async function runPhasePatterns(
|
||||
@@ -43,11 +105,17 @@ export async function runPhasePatterns(
|
||||
const config = await loadPatternsConfig(engine);
|
||||
|
||||
if (!config.enabled) {
|
||||
return skipped('disabled', 'dream.patterns.enabled is false');
|
||||
if (!opts.once) {
|
||||
return skipped('disabled', 'dream.patterns.enabled is false');
|
||||
}
|
||||
process.stderr.write(
|
||||
'[dream] --once: dream.patterns.enabled is false but ' +
|
||||
'--phase patterns --once forces this run (config untouched)\n',
|
||||
);
|
||||
}
|
||||
|
||||
// Gather reflections within lookback window.
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays);
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays, config.outputRoot);
|
||||
if (reflections.length < config.minEvidence) {
|
||||
return skipped(
|
||||
'insufficient_evidence',
|
||||
@@ -63,27 +131,51 @@ export async function runPhasePatterns(
|
||||
});
|
||||
}
|
||||
|
||||
// Submit one subagent for pattern detection.
|
||||
if (!process.env.ANTHROPIC_API_KEY) {
|
||||
return skipped('no_api_key', 'ANTHROPIC_API_KEY unset; pattern detection skipped');
|
||||
// Submit one subagent for pattern detection. The subagent dispatches via
|
||||
// the gateway model-tier resolver, so gate on "is the resolved model's
|
||||
// provider reachable" rather than ANTHROPIC_API_KEY specifically — a
|
||||
// hardcoded env gate misclassified non-Anthropic stacks (litellm,
|
||||
// deepseek, openrouter, ...) as "no upstream" even though the subagent
|
||||
// routes them through the gateway (agent.use_gateway_loop), and it missed
|
||||
// Anthropic keys set via `gbrain config set anthropic_api_key`. Same
|
||||
// probe semantics as think/index.ts + synthesize's makeJudgeClient:
|
||||
// unknown provider/model or Anthropic-without-key skips cheaply; other
|
||||
// providers' auth is checked lazily at dispatch and surfaces in the job
|
||||
// outcome. (Takeover of PR #2279's intent by @brettdavies.)
|
||||
const probe = probeChatModel(normalizeModelId(config.model));
|
||||
if (!probe.ok) {
|
||||
return skipped('no_provider', `pattern detection skipped: ${probe.detail}`);
|
||||
}
|
||||
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
}
|
||||
|
||||
// #2781: budget the subagent from the REMAINING parent-job time, not
|
||||
// the fixed config default. Checked after the cheap gates (disabled /
|
||||
// insufficient_evidence / no_provider) so a skip for budget reasons
|
||||
// only fires when the phase would otherwise have submitted.
|
||||
const budgets = clampSubagentBudgets(config, opts.deadlineAtMs, Date.now());
|
||||
if (budgets === null) {
|
||||
return skipped(
|
||||
'insufficient_cycle_budget',
|
||||
`remaining cycle budget under ${Math.round(MIN_PATTERNS_SUBAGENT_BUDGET_MS / 1000)}s ` +
|
||||
`(reserve ${Math.round(CYCLE_DEADLINE_RESERVE_MS / 1000)}s); next cycle retries with a fresh budget`,
|
||||
);
|
||||
}
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
const data: SubagentHandlerData = {
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence),
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
};
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
max_stalled: 3,
|
||||
timeout_ms: 30 * 60 * 1000,
|
||||
timeout_ms: budgets.timeoutMs,
|
||||
};
|
||||
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
|
||||
allowProtectedSubmit: true,
|
||||
@@ -92,13 +184,23 @@ export async function runPhasePatterns(
|
||||
let outcome: string;
|
||||
try {
|
||||
const final = await waitForCompletion(queue, job.id, {
|
||||
timeoutMs: 35 * 60 * 1000,
|
||||
timeoutMs: budgets.waitTimeoutMs,
|
||||
pollMs: 5 * 1000,
|
||||
});
|
||||
outcome = final.status;
|
||||
} catch (e) {
|
||||
if (e instanceof TimeoutError) outcome = 'timeout';
|
||||
else throw e;
|
||||
if (e instanceof TimeoutError) {
|
||||
outcome = 'timeout';
|
||||
// The child's own timeout_ms clock starts at ITS claim, not at
|
||||
// submit — a child that sat queued behind other work can outlive
|
||||
// the parent deadline this wait was clamped to. Cancel it so the
|
||||
// subagent can't keep spending/writing after the phase gave up
|
||||
// (waiting child → cancelled immediately; active child → lock
|
||||
// stripped, worker abort fires on next renew tick).
|
||||
try { await queue.cancelJob(job.id); } catch { /* best-effort */ }
|
||||
} else {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
if (opts.yieldDuringPhase) {
|
||||
@@ -113,13 +215,47 @@ export async function runPhasePatterns(
|
||||
// Reverse-write to fs.
|
||||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
|
||||
|
||||
return ok(`${writtenRefs.length} pattern page(s) written/updated (${outcome})`, {
|
||||
const details = {
|
||||
reflections_considered: reflections.length,
|
||||
patterns_written: writtenRefs.length,
|
||||
reverse_write_count: reverseWriteCount,
|
||||
child_outcome: outcome,
|
||||
job_id: job.id,
|
||||
});
|
||||
};
|
||||
|
||||
// #2782: the phase status must reflect the child outcome. Pre-fix this
|
||||
// returned status:ok even when the subagent timed out (e.g. no
|
||||
// subagent-capable worker slot free for the whole wait window) and zero
|
||||
// pattern pages were written — a silent no-op for days.
|
||||
if (outcome !== 'complete') {
|
||||
if (writtenRefs.length === 0) {
|
||||
return {
|
||||
phase: 'patterns',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: `pattern-detection subagent job ${job.id} ended '${outcome}'; nothing was written`,
|
||||
details,
|
||||
error: makeError(
|
||||
outcome === 'timeout' ? 'Timeout' : 'InternalError',
|
||||
`PATTERNS_CHILD_${outcome.toUpperCase()}`,
|
||||
`subagent job ${job.id} outcome '${outcome}' with zero pattern pages written`,
|
||||
outcome === 'timeout'
|
||||
? 'A timeout with zero writes usually means no subagent-capable worker claimed the job. Check `gbrain jobs list` and worker capacity.'
|
||||
: undefined,
|
||||
),
|
||||
};
|
||||
}
|
||||
// Partial: the child died/timed out but some pages landed first.
|
||||
return {
|
||||
phase: 'patterns',
|
||||
status: 'warn',
|
||||
duration_ms: 0,
|
||||
summary: `${writtenRefs.length} pattern page(s) written but subagent job ${job.id} ended '${outcome}'`,
|
||||
details,
|
||||
};
|
||||
}
|
||||
|
||||
return ok(`${writtenRefs.length} pattern page(s) written/updated (${outcome})`, details);
|
||||
} catch (e) {
|
||||
return failed(makeError('InternalError', 'PATTERNS_PHASE_FAIL',
|
||||
e instanceof Error ? (e.message || 'patterns phase threw') : String(e)));
|
||||
@@ -135,6 +271,22 @@ interface PatternsConfig {
|
||||
lookbackDays: number;
|
||||
minEvidence: number;
|
||||
model: string;
|
||||
/** #2415: shared output namespace (dream.synthesize.output_root, default 'wiki'). */
|
||||
outputRoot: string;
|
||||
/** #1594-family: subagent job timeout, config `dream.patterns.subagent_timeout_ms`. */
|
||||
subagentTimeoutMs: number;
|
||||
/** #1594-family: waitForCompletion timeout, config `dream.patterns.subagent_wait_timeout_ms`. */
|
||||
subagentWaitTimeoutMs: number;
|
||||
}
|
||||
|
||||
const DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS = 30 * 60 * 1000;
|
||||
const DEFAULT_PATTERNS_SUBAGENT_WAIT_TIMEOUT_MS = 35 * 60 * 1000;
|
||||
|
||||
async function getNumberConfig(engine: BrainEngine, key: string, fallback: number): Promise<number> {
|
||||
const raw = await engine.getConfig(key);
|
||||
if (raw === undefined || raw === null) return fallback;
|
||||
const value = Number(raw);
|
||||
return Number.isNaN(value) ? fallback : value;
|
||||
}
|
||||
|
||||
async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig> {
|
||||
@@ -155,6 +307,13 @@ async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig>
|
||||
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
|
||||
minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3,
|
||||
model,
|
||||
outputRoot: await loadOutputRoot(engine),
|
||||
subagentTimeoutMs: await getNumberConfig(
|
||||
engine, 'dream.patterns.subagent_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS,
|
||||
),
|
||||
subagentWaitTimeoutMs: await getNumberConfig(
|
||||
engine, 'dream.patterns.subagent_wait_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_WAIT_TIMEOUT_MS,
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -169,16 +328,19 @@ interface ReflectionRef {
|
||||
async function gatherReflections(
|
||||
engine: BrainEngine,
|
||||
lookbackDays: number,
|
||||
outputRoot = 'wiki',
|
||||
): Promise<ReflectionRef[]> {
|
||||
const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString();
|
||||
// #2415: reflections live under the configured output root (bound as a
|
||||
// parameter; outputRoot is slug-grammar-validated by loadOutputRoot).
|
||||
const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>(
|
||||
`SELECT slug, title, compiled_truth
|
||||
FROM pages
|
||||
WHERE slug LIKE 'wiki/personal/reflections/%'
|
||||
WHERE slug LIKE $2
|
||||
AND updated_at >= $1::timestamptz
|
||||
ORDER BY updated_at DESC
|
||||
LIMIT 100`,
|
||||
[since],
|
||||
[since, `${outputRoot}/personal/reflections/%`],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
slug: r.slug,
|
||||
@@ -189,7 +351,7 @@ async function gatherReflections(
|
||||
|
||||
// ── Prompt ────────────────────────────────────────────────────────────
|
||||
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string {
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number, outputRoot = 'wiki'): string {
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const corpus = reflections
|
||||
.map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`)
|
||||
@@ -199,15 +361,15 @@ function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number):
|
||||
|
||||
OUTPUT POLICY
|
||||
- Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections.
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks).
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[${outputRoot}/personal/reflections/...]] wikilinks).
|
||||
- Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one.
|
||||
- Pattern slug format: \`wiki/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- Pattern slug format: \`${outputRoot}/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics.
|
||||
|
||||
DO NOT WRITE
|
||||
- A "patterns from today" digest (that's the dream-cycle-summaries page; not your job).
|
||||
- Patterns with <${minEvidence} reflections cited.
|
||||
- Anything outside wiki/personal/patterns/.
|
||||
- Anything outside ${outputRoot}/personal/patterns/.
|
||||
|
||||
CONTEXT
|
||||
- Today: ${today}
|
||||
@@ -298,27 +460,6 @@ function renderPageToMarkdown(page: Page, tags: string[]): string {
|
||||
);
|
||||
}
|
||||
|
||||
// ── Allow-list (shared with synthesize.ts) ───────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
const candidates = [
|
||||
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
|
||||
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
|
||||
];
|
||||
for (const path of candidates) {
|
||||
if (!existsSync(path)) continue;
|
||||
try {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
// ── Status helpers ───────────────────────────────────────────────────
|
||||
|
||||
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
|
||||
import { randomUUID, createHash } from 'node:crypto';
|
||||
import { BaseCyclePhase, type ScopedReadOpts, type BasePhaseOpts } from './base-phase.ts';
|
||||
import { chat as gatewayChat } from '../ai/gateway.ts';
|
||||
import { chat as gatewayChat, getChatModel } from '../ai/gateway.ts';
|
||||
import { writeReceipt } from '../extract/receipt-writer.ts';
|
||||
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
|
||||
import { GBrainError } from '../types.ts';
|
||||
@@ -330,6 +330,8 @@ class ProposeTakesPhase extends BaseCyclePhase {
|
||||
opts.reporter.start('propose_takes.pages' as never, pages.length);
|
||||
}
|
||||
|
||||
const modelId = opts.model ?? getChatModel();
|
||||
|
||||
for (const page of pages) {
|
||||
result.pages_scanned += 1;
|
||||
this.tick(opts);
|
||||
@@ -359,7 +361,7 @@ class ProposeTakesPhase extends BaseCyclePhase {
|
||||
|
||||
// Budget pre-check before the LLM call. Estimate: ~1500 input tokens + 500 output.
|
||||
const budget = this.checkBudget({
|
||||
modelId: opts.model ?? 'claude-sonnet-4-6',
|
||||
modelId,
|
||||
estimatedInputTokens: 1500,
|
||||
maxOutputTokens: 500,
|
||||
});
|
||||
@@ -408,7 +410,7 @@ class ProposeTakesPhase extends BaseCyclePhase {
|
||||
p.weight,
|
||||
p.domain ?? null,
|
||||
JSON.stringify(existingTakes),
|
||||
opts.model ?? 'claude-sonnet-4-6',
|
||||
modelId,
|
||||
],
|
||||
);
|
||||
result.proposals_inserted += 1;
|
||||
|
||||
@@ -23,7 +23,13 @@ import type { PhaseResult } from '../cycle.ts';
|
||||
import type { ProgressReporter } from '../progress.ts';
|
||||
import { writeReceipt } from '../extract/receipt-writer.ts';
|
||||
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
|
||||
import { chat as gatewayChat } from '../ai/gateway.ts';
|
||||
import { chat as gatewayChat, isAvailable } from '../ai/gateway.ts';
|
||||
// #2163: concept pages route through importFromContent (the same
|
||||
// parse→chunk→embed pipeline put_page uses) instead of a bare engine.putPage,
|
||||
// so they land in the retrieval surface (content_chunks + embeddings) where
|
||||
// source-boost's 1.3× 'concepts/' weighting can actually reach them.
|
||||
import { importFromContent } from '../import-file.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
|
||||
const DEFAULT_BUDGET_USD = 1.5;
|
||||
const TIER_T1_MIN = 10;
|
||||
@@ -216,19 +222,23 @@ export async function runPhaseSynthesizeConcepts(
|
||||
|
||||
if (!opts.dryRun) {
|
||||
const title = group.conceptSlug.split('/').pop() ?? group.conceptSlug;
|
||||
await engine.putPage(`concepts/${title}`, {
|
||||
title: title.replace(/-/g, ' '),
|
||||
type: 'concept',
|
||||
compiled_truth: narrative,
|
||||
frontmatter: {
|
||||
type: 'concept',
|
||||
// #2163: serialize to markdown and import via the canonical pipeline so
|
||||
// the page is chunked (+ embedded when a provider is configured) —
|
||||
// mirrors put_page's isAvailable('embedding') → noEmbed gate.
|
||||
const md = serializeMarkdown(
|
||||
{
|
||||
tier: group.tier,
|
||||
mention_count: group.atomTitles.length,
|
||||
composite_score: group.atomTitles.length,
|
||||
synthesized_at: new Date().toISOString(),
|
||||
synthesized_by: 'synthesize_concepts-v0.41',
|
||||
},
|
||||
timeline: '',
|
||||
narrative,
|
||||
'',
|
||||
{ type: 'concept', title: title.replace(/-/g, ' '), tags: [] },
|
||||
);
|
||||
await importFromContent(engine, `concepts/${title}`, md, {
|
||||
noEmbed: !isAvailable('embedding'),
|
||||
});
|
||||
}
|
||||
conceptsWritten++;
|
||||
|
||||
+164
-25
@@ -75,6 +75,8 @@ const MIN_PROMPT_TOKENS = 100_000;
|
||||
const DEFAULT_MAX_CHUNKS = 24;
|
||||
/** Conservative default budget when model is unknown (200K × HEADROOM_RATIO). */
|
||||
const UNKNOWN_MODEL_BUDGET_TOKENS = 180_000;
|
||||
const DEFAULT_SUBAGENT_TIMEOUT_MS = 30 * 60 * 1000;
|
||||
const DEFAULT_SUBAGENT_WAIT_TIMEOUT_MS = 35 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* Compute per-chunk character budget for the resolved model + config override.
|
||||
@@ -242,6 +244,21 @@ export interface SynthesizePhaseOpts {
|
||||
* the synthesize loop. Caller must opt in explicitly.
|
||||
*/
|
||||
bypassDreamGuard?: boolean;
|
||||
/**
|
||||
* #1586: the cycle's resolved brain source (cycleSourceId from cycle.ts —
|
||||
* explicit --source wins, else derived from the checkout dir). Threaded to
|
||||
* every subagent child as `source_id` so put_page writes land in this
|
||||
* source, and stamped onto collected refs so reverse-writes read the
|
||||
* correct (source_id, slug) row. Unset → legacy 'default'.
|
||||
*/
|
||||
sourceId?: string;
|
||||
/**
|
||||
* issue #2860 — `gbrain dream --phase synthesize --once`. Bypasses the
|
||||
* `dream.synthesize.enabled` gate for THIS call only (does NOT bypass
|
||||
* the `session_corpus_dir` not-configured check — there's nothing to
|
||||
* run without a corpus). Never reads or writes config.
|
||||
*/
|
||||
once?: boolean;
|
||||
}
|
||||
|
||||
export async function runPhaseSynthesize(
|
||||
@@ -275,8 +292,14 @@ export async function runPhaseSynthesize(
|
||||
'dream.synthesize.session_corpus_dir is unset');
|
||||
}
|
||||
if (!opts.inputFile && !config.enabled) {
|
||||
return skipped('not_configured',
|
||||
'dream.synthesize.enabled is explicitly false');
|
||||
if (!opts.once) {
|
||||
return skipped('not_configured',
|
||||
'dream.synthesize.enabled is explicitly false');
|
||||
}
|
||||
process.stderr.write(
|
||||
'[dream] --once: dream.synthesize.enabled is false but ' +
|
||||
'--phase synthesize --once forces this run (config untouched)\n',
|
||||
);
|
||||
}
|
||||
|
||||
// Cooldown check (skipped for explicit --input / --date / --from / --to runs).
|
||||
@@ -397,7 +420,7 @@ export async function runPhaseSynthesize(
|
||||
|
||||
// Fan-out: submit one subagent per worth-processing transcript (or one
|
||||
// per chunk for transcripts that exceed the model's per-prompt budget).
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
@@ -460,10 +483,13 @@ export async function runPhaseSynthesize(
|
||||
: config.model;
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const childData: SubagentHandlerData = {
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock),
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock, config.outputRoot),
|
||||
model: subagentModel,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
// #1586: scope every child tool call to the cycle's resolved source
|
||||
// so put_page writes land there instead of the hardcoded 'default'.
|
||||
...(opts.sourceId ? { source_id: opts.sourceId } : {}),
|
||||
};
|
||||
// Idempotency key parity:
|
||||
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
|
||||
@@ -478,7 +504,7 @@ export async function runPhaseSynthesize(
|
||||
max_stalled: 3,
|
||||
on_child_fail: 'continue',
|
||||
idempotency_key,
|
||||
timeout_ms: 30 * 60 * 1000, // 30 min per chunk
|
||||
timeout_ms: config.subagentTimeoutMs,
|
||||
};
|
||||
const child = await queue.add(
|
||||
'subagent',
|
||||
@@ -499,7 +525,7 @@ export async function runPhaseSynthesize(
|
||||
for (const jobId of childIds) {
|
||||
try {
|
||||
const job = await waitForCompletion(queue, jobId, {
|
||||
timeoutMs: 35 * 60 * 1000,
|
||||
timeoutMs: config.subagentWaitTimeoutMs,
|
||||
pollMs: 5 * 1000,
|
||||
});
|
||||
childOutcomes.push({ jobId, status: job.status });
|
||||
@@ -522,20 +548,29 @@ export async function runPhaseSynthesize(
|
||||
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
|
||||
// even if Sonnet drops the chunk suffix.
|
||||
// v0.32.8: refs carry source_id so reverseWriteRefs picks the correct
|
||||
// (source, slug) row (currently always 'default' from subagent put_page).
|
||||
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
|
||||
// (source, slug) row. #1586: refs are stamped with the cycle's resolved
|
||||
// source (children write there via SubagentHandlerData.source_id).
|
||||
const cycleSourceId = opts.sourceId ?? 'default';
|
||||
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId);
|
||||
|
||||
const summaryDate = opts.date ?? today();
|
||||
|
||||
// #2569: persist the dream-output identity marker into the DB frontmatter
|
||||
// of every child-written page BEFORE reverse-rendering, so generated pages
|
||||
// are queryable (`frontmatter->>'dream_generated'`) and a later put_page
|
||||
// write-through (which re-renders from the DB row) can't erase the stamp.
|
||||
await stampDreamProvenance(engine, writtenRefs, summaryDate);
|
||||
|
||||
// Dual-write: reverse-render each DB row → markdown file.
|
||||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
|
||||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs, cycleSourceId);
|
||||
|
||||
// Summary index page (deterministic; orchestrator-written via direct
|
||||
// engine.putPage so no allow-list path needed).
|
||||
const summaryDate = opts.date ?? today();
|
||||
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
|
||||
// Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs.
|
||||
const writtenSlugs = writtenRefs.map(r => r.slug);
|
||||
if (SUMMARY_SLUG_RE.test(summarySlug)) {
|
||||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
|
||||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes, cycleSourceId);
|
||||
}
|
||||
|
||||
// Write completion timestamp ON SUCCESS only.
|
||||
@@ -593,6 +628,27 @@ interface SynthConfig {
|
||||
* `dream.synthesize.max_chunks_per_transcript`.
|
||||
*/
|
||||
maxChunksPerTranscript: number;
|
||||
/**
|
||||
* #2415: top-level namespace for synthesized output (reflections, originals,
|
||||
* patterns). Config key `dream.synthesize.output_root`; default 'wiki' —
|
||||
* zero behavior change unless set. No trailing slash. Must satisfy the slug
|
||||
* grammar; invalid values fall back to 'wiki' with a stderr warning.
|
||||
*/
|
||||
outputRoot: string;
|
||||
subagentTimeoutMs: number;
|
||||
subagentWaitTimeoutMs: number;
|
||||
}
|
||||
|
||||
/** #2415: shared output-root resolution (synthesize + patterns phases). */
|
||||
export async function loadOutputRoot(engine: BrainEngine): Promise<string> {
|
||||
const raw = await engine.getConfig('dream.synthesize.output_root');
|
||||
if (!raw) return 'wiki';
|
||||
const trimmed = raw.trim().replace(/^\/+|\/+$/g, '');
|
||||
if (SUMMARY_SLUG_RE.test(trimmed)) return trimmed;
|
||||
process.stderr.write(
|
||||
`[dream] dream.synthesize.output_root "${raw}" is not a valid slug prefix; falling back to "wiki".\n`,
|
||||
);
|
||||
return 'wiki';
|
||||
}
|
||||
|
||||
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
@@ -621,6 +677,16 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
|
||||
const maxPromptTokensStr = await engine.getConfig('dream.synthesize.max_prompt_tokens');
|
||||
const maxChunksStr = await engine.getConfig('dream.synthesize.max_chunks_per_transcript');
|
||||
const subagentTimeoutMs = await getNumberConfig(
|
||||
engine,
|
||||
'dream.synthesize.subagent_timeout_ms',
|
||||
DEFAULT_SUBAGENT_TIMEOUT_MS,
|
||||
);
|
||||
const subagentWaitTimeoutMs = await getNumberConfig(
|
||||
engine,
|
||||
'dream.synthesize.subagent_wait_timeout_ms',
|
||||
DEFAULT_SUBAGENT_WAIT_TIMEOUT_MS,
|
||||
);
|
||||
|
||||
let excludePatterns: string[] = ['medical', 'therapy'];
|
||||
if (excludeStr) {
|
||||
@@ -658,9 +724,23 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
|
||||
maxPromptTokens,
|
||||
maxChunksPerTranscript,
|
||||
outputRoot: await loadOutputRoot(engine),
|
||||
subagentTimeoutMs,
|
||||
subagentWaitTimeoutMs,
|
||||
};
|
||||
}
|
||||
|
||||
async function getNumberConfig(
|
||||
engine: BrainEngine,
|
||||
key: string,
|
||||
fallback: number,
|
||||
): Promise<number> {
|
||||
const raw = await engine.getConfig(key);
|
||||
if (raw === undefined || raw === null) return fallback;
|
||||
const value = Number(raw);
|
||||
return Number.isNaN(value) ? fallback : value;
|
||||
}
|
||||
|
||||
async function checkCooldown(
|
||||
engine: BrainEngine,
|
||||
hours: number,
|
||||
@@ -677,7 +757,13 @@ async function checkCooldown(
|
||||
|
||||
// ── Allow-list source of truth ───────────────────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
/**
|
||||
* #2415: `outputRoot` remaps the canonical `wiki/`-rooted globs to the
|
||||
* configured namespace (e.g. `notes/personal/reflections/*`). Default 'wiki'
|
||||
* returns the globs verbatim. Shared by the patterns phase (imported there —
|
||||
* the two phases must enforce the same allow-list).
|
||||
*/
|
||||
export async function loadAllowedSlugPrefixes(outputRoot = 'wiki'): Promise<string[]> {
|
||||
// Search a few known locations relative to the binary / repo. The first
|
||||
// hit wins; if none found, return [].
|
||||
const candidates = [
|
||||
@@ -691,7 +777,10 @@ async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
if (outputRoot === 'wiki') return globs as string[];
|
||||
return (globs as string[]).map(g =>
|
||||
g.startsWith('wiki/') ? `${outputRoot}/${g.slice('wiki/'.length)}` : g,
|
||||
);
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
@@ -939,6 +1028,7 @@ function buildSynthesisPrompt(
|
||||
chunkIdx: number,
|
||||
chunkTotal: number,
|
||||
priorContradictionsBlock = '',
|
||||
outputRoot = 'wiki',
|
||||
): string {
|
||||
const dateHint = t.inferredDate ?? today();
|
||||
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
|
||||
@@ -964,13 +1054,14 @@ OUTPUT POLICY (ALL of these are required)
|
||||
2. Cross-reference compulsively: every new page MUST contain at least one wikilink (e.g., \`[ref](people/jane-doe)\` or \`[[people/jane-doe]]\`) to existing brain content. Use the search tool to find existing pages first.
|
||||
3. Do NOT write to any path outside the allow-list shown in the put_page schema.
|
||||
4. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated segments. NO underscores, NO file extensions.
|
||||
5. Self-contained opening: begin every new page's body with a 2-3 sentence summary that a reader unfamiliar with this transcript could understand on its own, before any quotes or detail. Do not assume the reader has the source conversation for context.
|
||||
|
||||
TASKS
|
||||
A. Reflections (self-knowledge, pattern recognition, emotional processing):
|
||||
slug: \`wiki/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||||
slug: \`${outputRoot}/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||||
|
||||
B. Originals (new ideas, frames, theses, mental models):
|
||||
slug: \`wiki/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||||
slug: \`${outputRoot}/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||||
|
||||
C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries — your job is the reflection/original synthesis, NOT modifying existing person pages).
|
||||
|
||||
@@ -1011,6 +1102,7 @@ async function collectChildPutPageSlugs(
|
||||
engine: BrainEngine,
|
||||
childIds: number[],
|
||||
chunkInfo: Map<number, { idx: number; hash6: string }>,
|
||||
sourceId = 'default',
|
||||
): Promise<Array<{ slug: string; source_id: string }>> {
|
||||
if (childIds.length === 0) return [];
|
||||
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
|
||||
@@ -1020,10 +1112,10 @@ async function collectChildPutPageSlugs(
|
||||
//
|
||||
// v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent
|
||||
// put_page tool schema doesn't expose source_id (subagents are scoped to
|
||||
// a single source); default to 'default' for the current dream-cycle
|
||||
// product behavior. Threading the source_id through reverseWriteRefs
|
||||
// guarantees getPage targets the correct (source, slug) row instead of
|
||||
// the first DB match.
|
||||
// a single source). #1586: the orchestrator scopes each child to the
|
||||
// cycle's resolved source via SubagentHandlerData.source_id, and stamps
|
||||
// the SAME source here so reverseWriteRefs / provenance reads target the
|
||||
// correct (source_id, slug) row. Unset → legacy 'default'.
|
||||
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
|
||||
`SELECT job_id,
|
||||
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
|
||||
@@ -1039,7 +1131,7 @@ async function collectChildPutPageSlugs(
|
||||
const ci = chunkInfo.get(r.job_id);
|
||||
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
|
||||
}
|
||||
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' }));
|
||||
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId }));
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1068,12 +1160,52 @@ async function hasLegacySingleChunkCompletion(
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
// ── Dream-provenance DB stamp (#2569) ────────────────────────────────
|
||||
|
||||
/**
|
||||
* Persist the dream-output identity marker (`dream_generated: true` +
|
||||
* `dream_cycle_date`) into the `pages.frontmatter` JSONB row for every page
|
||||
* a synthesize child wrote. Render-time `frontmatterOverrides` alone only
|
||||
* reach the markdown FILE — the DB row stayed unstamped, so DB consumers
|
||||
* couldn't enumerate generated pages and a later put_page write-through
|
||||
* (which re-renders from the DB row) silently erased the marker.
|
||||
*
|
||||
* Plain UPDATE through executeRawJsonb (raw object bound to $3::jsonb —
|
||||
* never JSON.stringify into a ::jsonb cast; engine-parity safe, no new
|
||||
* engine method). Best-effort per row: a stamp failure never kills the
|
||||
* phase (the render-time override still covers the file).
|
||||
*/
|
||||
async function stampDreamProvenance(
|
||||
engine: BrainEngine,
|
||||
refs: Array<{ slug: string; source_id: string }>,
|
||||
cycleDate: string,
|
||||
): Promise<void> {
|
||||
if (refs.length === 0) return;
|
||||
const { executeRawJsonb } = await import('../sql-query.ts');
|
||||
for (const { slug, source_id } of refs) {
|
||||
try {
|
||||
await executeRawJsonb(
|
||||
engine,
|
||||
`UPDATE pages
|
||||
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb
|
||||
WHERE slug = $1 AND source_id = $2`,
|
||||
[slug, source_id],
|
||||
[{ dream_generated: true, dream_cycle_date: cycleDate }],
|
||||
);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[dream] provenance stamp ${slug}@${source_id} failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Reverse-write DB rows → markdown files ───────────────────────────
|
||||
|
||||
async function reverseWriteRefs(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
refs: Array<{ slug: string; source_id: string }>,
|
||||
nativeSourceId = 'default',
|
||||
): Promise<number> {
|
||||
let count = 0;
|
||||
for (const { slug, source_id } of refs) {
|
||||
@@ -1084,10 +1216,11 @@ async function reverseWriteRefs(
|
||||
const tags = await engine.getTags(slug, { sourceId: source_id });
|
||||
try {
|
||||
const md = renderPageToMarkdown(page, tags);
|
||||
// v0.32.8 F6: non-default sources land at brainDir/.sources/<id>/<slug>.md
|
||||
// so same-slug-different-source pages don't collide. Default-source
|
||||
// pages stay at brainDir/<slug>.md so single-source brains see no change.
|
||||
const filePath = source_id === 'default'
|
||||
// v0.32.8 F6: foreign-source pages land at brainDir/.sources/<id>/<slug>.md
|
||||
// so same-slug-different-source pages don't collide. Pages belonging to
|
||||
// the cycle's own source (#1586: brainDir IS that source's checkout —
|
||||
// legacy 'default' when unscoped) stay at brainDir/<slug>.md.
|
||||
const filePath = source_id === nativeSourceId
|
||||
? join(brainDir, `${slug}.md`)
|
||||
: join(brainDir, '.sources', source_id, `${slug}.md`);
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
@@ -1134,6 +1267,7 @@ async function writeSummaryPage(
|
||||
summaryDate: string,
|
||||
writtenSlugs: string[],
|
||||
childOutcomes: Array<{ jobId: number; status: string }>,
|
||||
sourceId = 'default',
|
||||
): Promise<void> {
|
||||
const completed = childOutcomes.filter(c => c.status === 'completed').length;
|
||||
const failed = childOutcomes.length - completed;
|
||||
@@ -1171,13 +1305,15 @@ async function writeSummaryPage(
|
||||
// unnecessarily; we go straight to the engine.
|
||||
const { parseMarkdown } = await import('../markdown.ts');
|
||||
const parsed = parseMarkdown(fullMarkdown);
|
||||
// #1586: summary lands in the cycle's resolved source too — otherwise the
|
||||
// children live in the named source while the index drifts to 'default'.
|
||||
await engine.putPage(summarySlug, {
|
||||
type: parsed.type,
|
||||
title: parsed.title,
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline,
|
||||
frontmatter: parsed.frontmatter,
|
||||
});
|
||||
}, { sourceId });
|
||||
|
||||
// Also write to disk (orchestrator dual-write).
|
||||
try {
|
||||
@@ -1242,4 +1378,7 @@ function makeError(cls: string, code: string, message: string, hint?: string): P
|
||||
// double-encoded jsonb regression). Not part of the runtime contract.
|
||||
export const __testing = {
|
||||
collectChildPutPageSlugs,
|
||||
buildSynthesisPrompt,
|
||||
stampDreamProvenance,
|
||||
reverseWriteRefs,
|
||||
};
|
||||
|
||||
@@ -206,6 +206,22 @@ export async function embedStaleForSource(
|
||||
chunk_source: c.chunk_source,
|
||||
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
// Carry through per-chunk metadata. upsertChunks writes these as
|
||||
// EXCLUDED.<col> (not COALESCE), so omitting them here resets image
|
||||
// rows to modality='text' (breaking the image search arm's
|
||||
// modality='image' filter) and wipes code-chunk symbol metadata on
|
||||
// every embed-stale pass. embedding_image is deliberately NOT
|
||||
// carried: the upsert COALESCEs it, and getChunks returns the
|
||||
// pgvector as a string which upsertChunks would mis-serialize.
|
||||
modality: c.modality ?? undefined,
|
||||
language: c.language ?? undefined,
|
||||
symbol_name: c.symbol_name ?? undefined,
|
||||
symbol_type: c.symbol_type ?? undefined,
|
||||
start_line: c.start_line ?? undefined,
|
||||
end_line: c.end_line ?? undefined,
|
||||
parent_symbol_path: c.parent_symbol_path ?? undefined,
|
||||
doc_comment: c.doc_comment ?? undefined,
|
||||
symbol_name_qualified: c.symbol_name_qualified ?? undefined,
|
||||
}));
|
||||
await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId }));
|
||||
// v0.41.31: stamp provenance only when EVERY chunk was stale (fully
|
||||
|
||||
@@ -29,6 +29,9 @@ import {
|
||||
isOpenAITextEmbedding3Model,
|
||||
isValidOpenAITextEmbedding3Dim,
|
||||
maxOpenAITextEmbedding3Dim,
|
||||
nvidiaEmbeddingDim,
|
||||
nvidiaEmbeddingDimOptions,
|
||||
supportsNvidiaEmbeddingDimension,
|
||||
} from './ai/dims.ts';
|
||||
|
||||
/**
|
||||
@@ -366,7 +369,9 @@ function validateDimAgainstTouchpoint(
|
||||
dimsOptions: number[] | undefined,
|
||||
requestedDims: number | undefined,
|
||||
): ResolveSchemaDimResult {
|
||||
const dim = requestedDims ?? defaultDims;
|
||||
const nvidiaNaturalDims = recipe.id === 'nvidia' ? nvidiaEmbeddingDim(modelId) : undefined;
|
||||
const effectiveDefaultDims = nvidiaNaturalDims ?? defaultDims;
|
||||
const dim = requestedDims ?? effectiveDefaultDims;
|
||||
|
||||
if (!Number.isInteger(dim) || dim <= 0) {
|
||||
return {
|
||||
@@ -396,7 +401,7 @@ function validateDimAgainstTouchpoint(
|
||||
dim,
|
||||
model: `${recipe.id}:${modelId}`,
|
||||
provider: recipe.id,
|
||||
recipeDefault: defaultDims,
|
||||
recipeDefault: effectiveDefaultDims,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -411,6 +416,22 @@ function isCustomDimValidForProvider(
|
||||
requestedDims: number,
|
||||
dimsOptions: number[] | undefined,
|
||||
): CustomDimCheck {
|
||||
// NVIDIA models are mixed: some fixed-dim, one Matryoshka-style. Handle
|
||||
// them before generic recipe dims_options so llama-nemotron can use 1280d.
|
||||
if (recipe.id === 'nvidia') {
|
||||
const naturalDims = nvidiaEmbeddingDim(modelId);
|
||||
if (naturalDims !== undefined && requestedDims === naturalDims) return { valid: true, error: '' };
|
||||
if (supportsNvidiaEmbeddingDimension(modelId, requestedDims)) return { valid: true, error: '' };
|
||||
const options = nvidiaEmbeddingDimOptions(modelId);
|
||||
return {
|
||||
valid: false,
|
||||
error:
|
||||
`NVIDIA model "${modelId}" does not support dimensions ${requestedDims}. ` +
|
||||
`Natural dimensions: ${naturalDims ?? 'unknown'}. ` +
|
||||
(options ? `Supported overrides: ${options.join(', ')}.` : 'No dimension overrides are supported for this NVIDIA model.'),
|
||||
};
|
||||
}
|
||||
|
||||
// Tier 1: recipe-declared dims_options.
|
||||
if (dimsOptions && dimsOptions.length > 0) {
|
||||
if (dimsOptions.includes(requestedDims)) return { valid: true, error: '' };
|
||||
|
||||
@@ -37,6 +37,9 @@ export const EMBEDDING_PRICING: Record<string, EmbeddingPricing> = {
|
||||
'voyage:voyage-4-large': { pricePerMTok: 0.18 },
|
||||
// ZeroEntropy (https://zeroentropy.dev/pricing — zembed-1)
|
||||
'zeroentropyai:zembed-1': { pricePerMTok: 0.05 },
|
||||
// Mistral (https://mistral.ai/pricing/api/, verified 2026-07-19)
|
||||
'mistral:mistral-embed': { pricePerMTok: 0.10 },
|
||||
'mistral:mistral-embed-2312': { pricePerMTok: 0.10 },
|
||||
};
|
||||
|
||||
export type PriceLookupResult =
|
||||
|
||||
@@ -936,6 +936,27 @@ export interface BrainEngine {
|
||||
|
||||
// Search
|
||||
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
/**
|
||||
* fix/title-retrieval-arm (D1): page-grain title candidate arm.
|
||||
*
|
||||
* content_chunks.search_vector never includes the page TITLE (it is
|
||||
* doc_comment + symbol_name_qualified + chunk_text), so a page whose
|
||||
* title tokens are absent from its body is unreachable by searchKeyword.
|
||||
* This arm queries the PAGE-GRAIN DOCUMENT vector pages.search_vector —
|
||||
* NOT titles alone: per trg_pages_search_vector it is title (weight 'A')
|
||||
* + compiled_truth ('B') + timeline text ('C'). Ranked by ts_rank_cd,
|
||||
* the 'A'-weighted title dominates, but body/timeline matches also
|
||||
* produce (lower-ranked) candidates. Returns page-grain hits joined to
|
||||
* ONE representative chunk per page (compiled_truth preferred, else
|
||||
* lowest chunk_index) so rows are shaped like searchKeyword's output and
|
||||
* can enter RRF fusion in hybridSearch.
|
||||
*
|
||||
* Deliberately NO query-length gating — unlike the alias hop (≤6-token
|
||||
* guard) and the title-phrase re-rank boost, this arm must GENERATE
|
||||
* candidates for long exact-title queries, which is exactly where
|
||||
* chunk-grain AND FTS is weakest.
|
||||
*/
|
||||
searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
/**
|
||||
* Hydrate embeddings for chunks already known by id. v0.36 (D9):
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { TakeBatchInput, TakeKind } from './engine.ts';
|
||||
import { chat, isAvailable } from './ai/gateway.ts';
|
||||
import { chat, getChatModel, isAvailable } from './ai/gateway.ts';
|
||||
|
||||
export const ALLOWED_PAGE_TYPES = [
|
||||
'concept', 'atom', 'lore', 'briefing', 'writing', 'originals',
|
||||
@@ -190,7 +190,11 @@ export async function extractTakesFromPages(
|
||||
let response: { text: string };
|
||||
try {
|
||||
response = await chat({
|
||||
model: opts.model ?? 'anthropic:claude-haiku-4-5',
|
||||
// #2997 — default to the configured chat model (file-plane gateway
|
||||
// config, same idiom as enrich.ts) instead of hardcoded cloud Haiku.
|
||||
// On OAuth/local-only installs the hardcoded model made every takes
|
||||
// extraction die with llm_unavailable despite a working chat_model.
|
||||
model: opts.model || getChatModel(),
|
||||
system: CLASSIFIER_SYSTEM,
|
||||
messages: [
|
||||
{
|
||||
|
||||
+51
-12
@@ -67,6 +67,23 @@ export async function getFactsExtractionModel(engine?: BrainEngine): Promise<str
|
||||
return normalizeModelId(resolved);
|
||||
}
|
||||
|
||||
/**
|
||||
* #2113: output-token cap for the extractor call. The pre-fix hardcoded 1500
|
||||
* silently truncated output on mandatory-reasoning models (thinking tokens
|
||||
* count toward the cap), so the JSON never parsed and extraction returned
|
||||
* zero facts with no signal. Configurable via
|
||||
* `gbrain config set facts.extraction_max_tokens <n>`; default 4000.
|
||||
*/
|
||||
export const DEFAULT_EXTRACTION_MAX_TOKENS = 4000;
|
||||
|
||||
export async function getFactsExtractionMaxTokens(engine?: BrainEngine): Promise<number> {
|
||||
if (!engine) return DEFAULT_EXTRACTION_MAX_TOKENS;
|
||||
const raw = await engine.getConfig('facts.extraction_max_tokens').catch(() => null);
|
||||
if (raw == null || raw.trim() === '') return DEFAULT_EXTRACTION_MAX_TOKENS;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n > 0 ? Math.floor(n) : DEFAULT_EXTRACTION_MAX_TOKENS;
|
||||
}
|
||||
|
||||
export const ALL_EXTRACT_KINDS: readonly FactKind[] = [
|
||||
'event', 'preference', 'commitment', 'belief', 'fact',
|
||||
] as const;
|
||||
@@ -164,24 +181,46 @@ export async function extractFactsFromTurn(input: ExtractInput): Promise<Extract
|
||||
|
||||
const cap = Math.max(1, Math.min(input.maxFactsPerTurn ?? 10, 25));
|
||||
const defaultModel = await getFactsExtractionModel(input.engine);
|
||||
const maxTokens = await getFactsExtractionMaxTokens(input.engine);
|
||||
const model = input.model ?? defaultModel;
|
||||
const userContent = `<turn>\n${cleaned}\n</turn>\n\nExtract up to ${cap} facts.${
|
||||
input.entityHints && input.entityHints.length
|
||||
? ` Known entity slugs the user already mentioned: ${input.entityHints.slice(0, 5).join(', ')}.`
|
||||
: ''
|
||||
}`;
|
||||
let result: ChatResult;
|
||||
try {
|
||||
result = await chat({
|
||||
model: input.model ?? defaultModel,
|
||||
model,
|
||||
system: EXTRACTOR_SYSTEM,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: `<turn>\n${cleaned}\n</turn>\n\nExtract up to ${cap} facts.${
|
||||
input.entityHints && input.entityHints.length
|
||||
? ` Known entity slugs the user already mentioned: ${input.entityHints.slice(0, 5).join(', ')}.`
|
||||
: ''
|
||||
}`,
|
||||
},
|
||||
],
|
||||
maxTokens: 1500,
|
||||
messages: [{ role: 'user', content: userContent }],
|
||||
maxTokens,
|
||||
abortSignal: input.abortSignal,
|
||||
});
|
||||
// #2113: never checked pre-fix — a truncated response (stopReason
|
||||
// 'length', e.g. reasoning tokens eating the cap on mandatory-reasoning
|
||||
// models) produced unparseable JSON and silently extracted zero facts.
|
||||
// Retry ONCE at double the cap, then surface the truncation loudly.
|
||||
if (result.stopReason === 'length') {
|
||||
process.stderr.write(
|
||||
`[facts-extract] WARN: extractor output truncated at maxTokens=${maxTokens} ` +
|
||||
`(model=${model}); retrying once at ${maxTokens * 2}\n`,
|
||||
);
|
||||
result = await chat({
|
||||
model,
|
||||
system: EXTRACTOR_SYSTEM,
|
||||
messages: [{ role: 'user', content: userContent }],
|
||||
maxTokens: maxTokens * 2,
|
||||
abortSignal: input.abortSignal,
|
||||
});
|
||||
if (result.stopReason === 'length') {
|
||||
process.stderr.write(
|
||||
`[facts-extract] WARN: extractor output STILL truncated at maxTokens=${maxTokens * 2} ` +
|
||||
`(model=${model}); facts for this turn are likely lost. ` +
|
||||
`Raise the cap: gbrain config set facts.extraction_max_tokens <n>\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
// Re-throw aborts; absorb other errors as "no extraction" — caller's
|
||||
// `put_page` backstop will still record the page itself.
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
/**
|
||||
* Full-text search language configuration.
|
||||
*
|
||||
* Postgres tsvector/tsquery require a text search configuration name (e.g.
|
||||
* 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded
|
||||
* 'english' across engines and trigger functions, which broke search
|
||||
* quality for non-English brains (no stemming, no stop-word removal).
|
||||
*
|
||||
* This helper centralizes the choice. Default stays 'english' for backward
|
||||
* compatibility — only users who set GBRAIN_FTS_LANGUAGE see different
|
||||
* behavior.
|
||||
*
|
||||
* Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent +
|
||||
* portuguese stemmer) are supported as long as the configuration exists
|
||||
* in the target Postgres instance. See docs/guides/multi-language-fts.md
|
||||
* for setup instructions.
|
||||
*
|
||||
* Validation: only allow lowercase letters, digits, and underscores. This
|
||||
* prevents SQL injection when the value is interpolated into queries
|
||||
* (Postgres tsvector functions don't accept parameterized config names —
|
||||
* they must be literals or identifiers).
|
||||
*/
|
||||
|
||||
const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/;
|
||||
const DEFAULT_LANGUAGE = 'english';
|
||||
|
||||
let cachedLanguage: string | null = null;
|
||||
|
||||
/**
|
||||
* Returns the configured Postgres text search configuration name.
|
||||
*
|
||||
* Resolution order:
|
||||
* 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid)
|
||||
* 2. 'english' (default — preserves existing behavior)
|
||||
*
|
||||
* The return value is safe to interpolate directly into SQL because it
|
||||
* passes the VALID_CONFIG_NAME guard. If validation fails, falls back to
|
||||
* the default and emits a one-time warning.
|
||||
*
|
||||
* Cached on first call; reset with `resetFtsLanguageCache()` (test only).
|
||||
*/
|
||||
export function getFtsLanguage(): string {
|
||||
if (cachedLanguage !== null) return cachedLanguage;
|
||||
|
||||
const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim();
|
||||
if (!raw) {
|
||||
cachedLanguage = DEFAULT_LANGUAGE;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
if (!VALID_CONFIG_NAME.test(raw)) {
|
||||
console.warn(
|
||||
`[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` +
|
||||
`Falling back to '${DEFAULT_LANGUAGE}'.`
|
||||
);
|
||||
cachedLanguage = DEFAULT_LANGUAGE;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
cachedLanguage = raw;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resets the cached language. Tests only — don't use in production code.
|
||||
*/
|
||||
export function resetFtsLanguageCache(): void {
|
||||
cachedLanguage = null;
|
||||
}
|
||||
@@ -303,6 +303,83 @@ export function validateRepoState(
|
||||
return 'healthy';
|
||||
}
|
||||
|
||||
/**
|
||||
* True if `path` is itself a git repo OR a subdirectory of one, per
|
||||
* `git rev-parse --show-toplevel`. Mirrors the walk-up discovery
|
||||
* `sync.ts:discoverGitRoot` performs at sync time (#753/#774 — subdir-of-git
|
||||
* sources are valid), so a directory that passes this check is guaranteed
|
||||
* not to hit sync's "Not inside a git repository" error later. Used by
|
||||
* `addSource` (#2707) to validate `--path` at registration time instead of
|
||||
* deferring the failure to the first sync.
|
||||
*/
|
||||
export function isInsideGitRepo(path: string): boolean {
|
||||
try {
|
||||
execFileSync('git', ['-C', path, 'rev-parse', '--show-toplevel'], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
timeout: 10_000,
|
||||
env: { ...process.env, ...GIT_ENV },
|
||||
});
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The empty-tree object ID for `path`'s repo, derived (not hardcoded) so
|
||||
* this works for both the default SHA-1 object format and the opt-in
|
||||
* `--object-format=sha256` one (git 2.29+) — each has its own empty-tree
|
||||
* OID. `git hash-object -t tree --stdin < /dev/null` computes the hash of
|
||||
* a zero-entry tree using whatever hash algorithm `path`'s repo is
|
||||
* configured for, without needing to know which one that is. #2707 codex
|
||||
* round 4 (P2): an earlier version hardcoded the well-known SHA-1 constant
|
||||
* (`4b825dc6...`), which silently mismatched — and so let an empty
|
||||
* SHA-256 repo through — on a SHA-256 repo's real (different) empty-tree
|
||||
* OID.
|
||||
*/
|
||||
function emptyTreeOid(path: string): string {
|
||||
return execFileSync('git', ['-C', path, 'hash-object', '-t', 'tree', '--stdin'], {
|
||||
input: '',
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
timeout: 10_000,
|
||||
env: { ...process.env, ...GIT_ENV },
|
||||
}).toString().trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* True if `path`'s HEAD tree has at least one tracked entry scoped to
|
||||
* `path` itself. `-C path` + the `HEAD:./` revision syntax resolves the
|
||||
* tree object for `path` specifically (not the whole repo root), so this
|
||||
* is correct for both a repo's toplevel AND a subdirectory-of-a-repo
|
||||
* source — then a single OID comparison against that repo's empty-tree
|
||||
* object (see `emptyTreeOid`) tells us whether that tree is empty. #2707
|
||||
* codex round 3 (P2): unlike listing (`ls-tree`), this is O(1) output — no
|
||||
* `maxBuffer` exposure on a repo with a very large number of entries.
|
||||
*
|
||||
* Subsumes "no commits at all" (`HEAD:./` on an unborn repo fails to
|
||||
* resolve — there's no HEAD) AND "has a HEAD commit but it's empty"
|
||||
* (#2707 codex round 2): `git commit --allow-empty` followed by creating
|
||||
* untracked files resolves `HEAD:./` successfully (to the empty-tree OID)
|
||||
* but that tree has zero entries — a directory that would pass a bare
|
||||
* `rev-parse HEAD` check yet still can't sync (or worse, "succeeds"
|
||||
* importing nothing and then never notices the untracked files change —
|
||||
* the silent-staleness class #2707 exists to prevent). A directory
|
||||
* that's `git init`ed but never committed, or where this specific path
|
||||
* was never `git add`ed, fails this check either way.
|
||||
*/
|
||||
export function hasTrackedContent(path: string): boolean {
|
||||
try {
|
||||
const out = execFileSync('git', ['-C', path, 'rev-parse', '--verify', 'HEAD:./'], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
timeout: 10_000,
|
||||
env: { ...process.env, ...GIT_ENV },
|
||||
});
|
||||
return out.toString().trim() !== emptyTreeOid(path);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Durability helpers (v0.42.44) ───────────────────────────────────────────
|
||||
// Used by the brain-repo durability feature (`gbrain sources harden/pull`) and
|
||||
// the DB-free pull cron. These are the auth-capable, rebase-aware counterparts
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { existsSync, readFileSync, writeFileSync, renameSync, unlinkSync } from 'fs';
|
||||
import { relative, isAbsolute } from 'path';
|
||||
import { existsSync, readFileSync, writeFileSync, renameSync, unlinkSync, realpathSync } from 'fs';
|
||||
import { relative, isAbsolute, resolve } from 'path';
|
||||
|
||||
/**
|
||||
* Path-based import checkpoint.
|
||||
@@ -25,6 +25,12 @@ import { relative, isAbsolute } from 'path';
|
||||
* enter the set.
|
||||
*/
|
||||
export interface ImportCheckpoint {
|
||||
/** Checkpoint payload schema. v1 is path-based with explicit producer metadata. */
|
||||
schema_version: 1;
|
||||
/** Producer marker for downstream consumers that validate before acting. */
|
||||
owner: 'gbrain';
|
||||
/** Checkpoint kind. Prevents unrelated checkpoint files from being treated as import state. */
|
||||
kind: 'import';
|
||||
/** Absolute brain directory the checkpoint was created against. Mismatch on resume → discard. */
|
||||
dir: string;
|
||||
/**
|
||||
@@ -37,6 +43,21 @@ export interface ImportCheckpoint {
|
||||
}
|
||||
|
||||
const OLD_FORMAT_LOG = 'Older checkpoint format detected — re-walking (cheap via content_hash)';
|
||||
export const IMPORT_CHECKPOINT_SCHEMA_VERSION = 1;
|
||||
export const IMPORT_CHECKPOINT_OWNER = 'gbrain';
|
||||
export const IMPORT_CHECKPOINT_KIND = 'import';
|
||||
|
||||
/**
|
||||
* Capture the import target once at run start. `resolve()` removes caller
|
||||
* spelling such as `.` or `../staging`; `realpathSync()` collapses symlinks
|
||||
* and proves the target exists. The returned value is the only directory
|
||||
* identity import checkpoints should ever persist (#1728 — a raw `.` here
|
||||
* made the checkpoint `dir` resolve to whatever CWD the NEXT consumer ran
|
||||
* from, which downstream tooling treated as an owned staging directory).
|
||||
*/
|
||||
export function resolveImportTargetDir(dir: string): string {
|
||||
return realpathSync(resolve(dir));
|
||||
}
|
||||
|
||||
/**
|
||||
* Load a checkpoint and verify it's compatible with the current run.
|
||||
@@ -72,11 +93,21 @@ export function loadCheckpoint(path: string, currentDir: string): ImportCheckpoi
|
||||
}
|
||||
|
||||
if (typeof obj.dir !== 'string') return null;
|
||||
if (!isAbsolute(obj.dir)) return null;
|
||||
if (obj.dir !== currentDir) return null;
|
||||
// Self-describing metadata (#1728): absent fields are tolerated (legacy
|
||||
// path-based checkpoints predate them), but present-and-wrong means the
|
||||
// file was written by something else — don't resume from it.
|
||||
if (obj.schema_version !== undefined && obj.schema_version !== IMPORT_CHECKPOINT_SCHEMA_VERSION) return null;
|
||||
if (obj.owner !== undefined && obj.owner !== IMPORT_CHECKPOINT_OWNER) return null;
|
||||
if (obj.kind !== undefined && obj.kind !== IMPORT_CHECKPOINT_KIND) return null;
|
||||
if (typeof obj.timestamp !== 'string') return null;
|
||||
if (!obj.completedPaths.every((p): p is string => typeof p === 'string')) return null;
|
||||
|
||||
return {
|
||||
schema_version: IMPORT_CHECKPOINT_SCHEMA_VERSION,
|
||||
owner: IMPORT_CHECKPOINT_OWNER,
|
||||
kind: IMPORT_CHECKPOINT_KIND,
|
||||
dir: obj.dir,
|
||||
completedPaths: obj.completedPaths,
|
||||
timestamp: obj.timestamp,
|
||||
@@ -98,6 +129,9 @@ export function saveCheckpoint(path: string, cp: ImportCheckpoint): void {
|
||||
// Sort for stable serialization — keeps diffs across snapshots minimal
|
||||
// and tests deterministic.
|
||||
const payload: ImportCheckpoint = {
|
||||
schema_version: IMPORT_CHECKPOINT_SCHEMA_VERSION,
|
||||
owner: IMPORT_CHECKPOINT_OWNER,
|
||||
kind: IMPORT_CHECKPOINT_KIND,
|
||||
dir: cp.dir,
|
||||
completedPaths: [...cp.completedPaths].sort(),
|
||||
timestamp: cp.timestamp,
|
||||
|
||||
+167
-1
@@ -1,5 +1,6 @@
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
|
||||
/**
|
||||
* Schema migrations — run automatically on initSchema().
|
||||
@@ -134,7 +135,10 @@ export const MIGRATIONS: Migration[] = [
|
||||
}
|
||||
}
|
||||
}
|
||||
if (renamed > 0) console.log(` Renamed ${renamed} slugs`);
|
||||
// Migration progress goes to stderr — stdout must stay clean for
|
||||
// callers parsing JSON (e.g. `gbrain doctor --json | jq`); migrations
|
||||
// can run lazily inside ANY command's first DB connect.
|
||||
if (renamed > 0) process.stderr.write(` Renamed ${renamed} slugs\n`);
|
||||
},
|
||||
},
|
||||
{
|
||||
@@ -5505,6 +5509,168 @@ export const MIGRATIONS: Migration[] = [
|
||||
WHERE dimension IS NOT NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 123,
|
||||
name: 'configurable_fts_language',
|
||||
// Recreate the two search_vector trigger functions using the language
|
||||
// configured via GBRAIN_FTS_LANGUAGE (default 'english'). Idempotent:
|
||||
// CREATE OR REPLACE swaps the function body atomically; no trigger
|
||||
// recreation needed since the trigger references the function by name.
|
||||
//
|
||||
// Why a handler instead of a static SQL string: Postgres tsvector
|
||||
// functions don't accept parameterized config names — the language
|
||||
// must be a literal in the SQL. getFtsLanguage() validates the value
|
||||
// (lowercase letters/digits/underscores only) before interpolation.
|
||||
//
|
||||
// Function bodies mirror schema.sql / pglite-schema.ts exactly —
|
||||
// INCLUDING the `SET search_path = pg_catalog, public` hardening from
|
||||
// v120/#1647 (CREATE OR REPLACE resets proconfig, so omitting it here
|
||||
// would silently strip the hardening on every upgraded brain). Only
|
||||
// the text-search config name is parameterized. Keep all copies in
|
||||
// sync when the trigger logic changes.
|
||||
//
|
||||
// Backfill: after recreating the functions, re-tokenize existing rows
|
||||
// under the new language. Skipped when the configured language is
|
||||
// 'english' (trigger output identical — re-tokenizing is wasted I/O).
|
||||
// To change language after this migration has run, use
|
||||
// `gbrain reindex-search-vector`.
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
const lang = getFtsLanguage();
|
||||
|
||||
const recreatePagesFn = `
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
const recreateChunksFn = `
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
await engine.executeRaw(recreatePagesFn);
|
||||
await engine.executeRaw(recreateChunksFn);
|
||||
|
||||
if (lang === 'english') {
|
||||
// stderr, NOT stdout: migrations run lazily inside any command's
|
||||
// first DB connect — a console.log here polluted `doctor --json`
|
||||
// stdout and broke jq consumers (heavy-tests fm_wallclock).
|
||||
process.stderr.write(` v123: trigger functions recreated with language='english' (default — no backfill needed)\n`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Backfill existing rows under the new tokenizer. UPDATE-to-same-value
|
||||
// re-fires the pages trigger; chunks are rewritten directly with the
|
||||
// same expression as the trigger.
|
||||
await engine.executeRaw(`
|
||||
UPDATE pages SET id = id
|
||||
WHERE search_vector IS NOT NULL;
|
||||
`);
|
||||
|
||||
await engine.executeRaw(`
|
||||
UPDATE content_chunks
|
||||
SET search_vector =
|
||||
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')
|
||||
WHERE search_vector IS NOT NULL;
|
||||
`);
|
||||
|
||||
process.stderr.write(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows\n`);
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 124,
|
||||
name: 'page_search_vector_drop_compiled_truth',
|
||||
// #2704: a single markdown page whose compiled_truth exceeds Postgres's
|
||||
// hard 1,048,575-byte tsvector cap made update_page_search_vector()
|
||||
// throw "string is too long for tsvector" INSIDE the pages UPSERT
|
||||
// transaction — not a per-file ledger entry, a transaction abort. The
|
||||
// whole source's sync checkpoint stayed pinned (Sync BLOCKED) until the
|
||||
// oversized file was fixed or manually skipped, even though every
|
||||
// OTHER file in the run imported fine.
|
||||
//
|
||||
// Fix: drop compiled_truth (the unbounded whole-page body) from this
|
||||
// trigger. It was already redundant — content_chunks.search_vector
|
||||
// (Cathedral II Layer 3, v0.20.0) is the ACTUAL keyword-search source:
|
||||
// searchKeyword() in postgres-engine.ts/pglite-engine.ts ranks and
|
||||
// queries `cc.search_vector` exclusively; `pages.search_vector` is
|
||||
// written by this trigger but never read by any query in this
|
||||
// codebase (verified: no `pages.search_vector`/bare `search_vector`
|
||||
// appears on either side of a WHERE/ts_rank anywhere outside this
|
||||
// trigger's own definition and the reindex/backfill machinery that
|
||||
// maintains it). And chunking already bounds each chunk_text well
|
||||
// under the tsvector limit (chunkText() targets embedding-sized
|
||||
// pieces, several orders of magnitude smaller than 1MB) — the overflow
|
||||
// was specific to the whole-page grain this trigger no longer builds.
|
||||
//
|
||||
// title + timeline (both naturally small — a compiled_truth-sized
|
||||
// title or timeline field would be its own bug) stay, so
|
||||
// pages.search_vector keeps carrying SOME signal rather than going
|
||||
// fully inert; a future PR can drop the column outright once its
|
||||
// last non-search consumer (if any turns up) is confirmed gone.
|
||||
//
|
||||
// No backfill: existing rows keep whatever search_vector they already
|
||||
// computed until their next UPDATE (harmless — nothing reads this
|
||||
// column, so staleness has zero behavioral effect). The brains that
|
||||
// actually hit this bug never successfully wrote a value for the
|
||||
// oversized page in the first place, so there's nothing stale to fix
|
||||
// for them specifically — the NEXT sync of that exact file is what
|
||||
// proves the fix, not a backfill of already-working rows.
|
||||
//
|
||||
// Function body mirrors reindex-search-vector.ts's recreatePagesFn
|
||||
// (documented contract there: keep both in lockstep) and the fresh-
|
||||
// install baselines in pglite-schema.ts / schema-embedded.ts — all
|
||||
// four updated in the same commit as this migration.
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
const lang = getFtsLanguage();
|
||||
await engine.executeRaw(`
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`);
|
||||
process.stderr.write(` v124: update_page_search_vector() no longer indexes compiled_truth (was overflowing tsvector on large pages, #2704)
|
||||
`);
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -58,8 +58,29 @@ import { randomUUIDv7 } from 'bun';
|
||||
|
||||
const DEFAULT_MODEL = 'claude-sonnet-4-6';
|
||||
const DEFAULT_MAX_TURNS = 20;
|
||||
const DEFAULT_MAX_OUTPUT_TOKENS = 8192;
|
||||
const DEFAULT_RATE_KEY = 'anthropic:messages';
|
||||
|
||||
/**
|
||||
* Resolve the per-turn output-token cap (#2778). Per-job data wins, then the
|
||||
* `agent.max_output_tokens` config row, then the 8192 default (was a
|
||||
* hardcoded 4096 that made pages >~12KB unwritable via put_page). Invalid
|
||||
* values (NaN / zero / negative) fall through to the next tier.
|
||||
*/
|
||||
export function resolveMaxOutputTokens(
|
||||
perJob: number | undefined,
|
||||
configRaw: string | null | undefined,
|
||||
): number {
|
||||
if (typeof perJob === 'number' && Number.isFinite(perJob) && perJob > 0) {
|
||||
return Math.floor(perJob);
|
||||
}
|
||||
if (typeof configRaw === 'string' && configRaw.trim() !== '') {
|
||||
const n = Number(configRaw);
|
||||
if (Number.isFinite(n) && n > 0) return Math.floor(n);
|
||||
}
|
||||
return DEFAULT_MAX_OUTPUT_TOKENS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the rate-lease cap from the env var.
|
||||
*
|
||||
@@ -212,6 +233,11 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
fallback: TIER_DEFAULTS.subagent,
|
||||
});
|
||||
const maxTurns = data.max_turns ?? DEFAULT_MAX_TURNS;
|
||||
// #2778: per-turn output cap — data.max_tokens → config → 8192 default.
|
||||
const maxOutputTokens = resolveMaxOutputTokens(
|
||||
data.max_tokens,
|
||||
await engine.getConfig('agent.max_output_tokens').catch(() => null),
|
||||
);
|
||||
// v0.41 Approach C: systemPrompt is now built AFTER toolDefs (a few
|
||||
// lines below) so the renderer can splice a tool-usage preamble
|
||||
// listing each available tool's usage_hint. The renderer is
|
||||
@@ -246,6 +272,8 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
config,
|
||||
brainId: data.brain_id,
|
||||
allowedSlugPrefixes: data.allowed_slug_prefixes,
|
||||
// #1586: cycle-resolved source scope for tool-call OperationContexts.
|
||||
sourceId: data.source_id,
|
||||
});
|
||||
const toolDefs = data.allowed_tools && data.allowed_tools.length > 0
|
||||
? filterAllowedTools(registry, data.allowed_tools)
|
||||
@@ -277,6 +305,7 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
systemPrompt,
|
||||
toolDefs,
|
||||
maxTurns,
|
||||
maxOutputTokens,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -535,7 +564,7 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
// `model` stays qualified everywhere else (persistence, recipe
|
||||
// lookup at recipeIdFromModel(), capability gate).
|
||||
model: stripProviderPrefix(model),
|
||||
max_tokens: 4096,
|
||||
max_tokens: maxOutputTokens,
|
||||
system: [
|
||||
{ type: 'text', text: systemPrompt, cache_control: { type: 'ephemeral' } },
|
||||
] as any,
|
||||
@@ -628,7 +657,10 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
b.type === 'tool_use',
|
||||
);
|
||||
if (toolUses.length === 0) {
|
||||
stopReason = 'end_turn';
|
||||
// #2778: an output-cap hit is NOT end_turn — the text (and possibly a
|
||||
// dropped trailing tool_use block) is truncated. Surface it as its own
|
||||
// stop_reason instead of silently reporting a clean end_turn.
|
||||
stopReason = assistantMsg.stop_reason === 'max_tokens' ? 'max_tokens' : 'end_turn';
|
||||
// Concatenate text blocks as the final answer.
|
||||
finalText = blocks
|
||||
.filter(b => b.type === 'text' && typeof b.text === 'string')
|
||||
@@ -741,6 +773,24 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
}
|
||||
}
|
||||
|
||||
// #2778: a max_tokens stop with tool_use blocks means the API dropped an
|
||||
// incomplete trailing block (e.g. a large put_page body that overflowed
|
||||
// the cap). Tell the model so it re-issues the cut-off call (split, or
|
||||
// smaller pages) instead of assuming the write happened.
|
||||
if (assistantMsg.stop_reason === 'max_tokens') {
|
||||
toolResults.push({
|
||||
type: 'text',
|
||||
text: `[system] Your previous response hit the ${maxOutputTokens}-token output cap and was truncated; ` +
|
||||
`any tool call cut off by the cap was DROPPED and did not execute. Re-issue it, splitting large content if needed.`,
|
||||
} as ContentBlock);
|
||||
logSubagentHeartbeat({
|
||||
job_id: ctx.id,
|
||||
event: 'llm_call_completed',
|
||||
turn_idx: turnIdx,
|
||||
error: `stop_reason=max_tokens at cap ${maxOutputTokens}; truncation note injected`,
|
||||
});
|
||||
}
|
||||
|
||||
// 6. Append the synthesized user turn (tool_result wrappers) to the
|
||||
// conversation and persist it so replay picks it up.
|
||||
const userIdx = nextMessageIdx++;
|
||||
@@ -776,6 +826,8 @@ interface GatewayRunArgs {
|
||||
systemPrompt: string;
|
||||
toolDefs: ToolDef[];
|
||||
maxTurns: number;
|
||||
/** #2778: per-turn output-token cap (resolved by resolveMaxOutputTokens). */
|
||||
maxOutputTokens: number;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -793,7 +845,7 @@ interface GatewayRunArgs {
|
||||
* reconciler sees both shapes uniformly.
|
||||
*/
|
||||
async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResult> {
|
||||
const { engine, ctx, data, model, systemPrompt, toolDefs, maxTurns } = args;
|
||||
const { engine, ctx, data, model, systemPrompt, toolDefs, maxTurns, maxOutputTokens } = args;
|
||||
|
||||
// Map ToolDef → ChatToolDef (gateway shape). The gateway's chat() bridges
|
||||
// this to provider-specific tool definitions via the Vercel AI SDK.
|
||||
@@ -917,6 +969,7 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
|
||||
tools: chatTools,
|
||||
toolHandlers,
|
||||
maxTurns,
|
||||
maxTokens: maxOutputTokens,
|
||||
abortSignal: ctx.signal,
|
||||
cacheSystem,
|
||||
// ALWAYS pass replayState (even on fresh runs) so the gateway loop's
|
||||
|
||||
@@ -163,24 +163,36 @@ export class MinionQueue {
|
||||
if (opts?.maxWaiting !== undefined) {
|
||||
const maxWaiting = Math.max(1, Math.floor(opts.maxWaiting));
|
||||
const backpressureQueue = opts?.queue ?? 'default';
|
||||
// Multi-source scope: jobs of the same (name, queue) but different
|
||||
// data.sourceId are independent workstreams (per-source sync/cycle).
|
||||
// Counting them together made a waiting default-source sync swallow
|
||||
// every other source's freshness sync — a secondary source sat 29h stale
|
||||
// while dispatch logs showed its syncs "dispatched" (coalesced into
|
||||
// the default row). Key the lock and the count on sourceId when the
|
||||
// submission carries one; NULL keeps legacy single-scope behavior.
|
||||
const bpSourceId = typeof (data as Record<string, unknown> | undefined)?.sourceId === 'string'
|
||||
? (data as Record<string, unknown>).sourceId as string
|
||||
: null;
|
||||
await tx.executeRaw(
|
||||
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2))`,
|
||||
[jobName, backpressureQueue]
|
||||
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2 || ':' || coalesce($3, '')))`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
const waitingCountRows = await tx.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text AS count
|
||||
FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'`,
|
||||
[jobName, backpressureQueue]
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'
|
||||
AND ($3::text IS NULL OR data->>'sourceId' IS NOT DISTINCT FROM $3)`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
const waitingCount = parseInt(waitingCountRows[0]?.count ?? '0', 10);
|
||||
if (waitingCount >= maxWaiting) {
|
||||
const existingWaiting = await tx.executeRaw<Record<string, unknown>>(
|
||||
`SELECT * FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'
|
||||
AND ($3::text IS NULL OR data->>'sourceId' IS NOT DISTINCT FROM $3)
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT 1`,
|
||||
[jobName, backpressureQueue]
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
if (existingWaiting.length > 0) {
|
||||
const coalesced = rowToMinionJob(existingWaiting[0]);
|
||||
@@ -471,12 +483,34 @@ export class MinionQueue {
|
||||
});
|
||||
}
|
||||
|
||||
/** Re-queue a failed or dead job for retry. */
|
||||
/**
|
||||
* Re-queue a failed or dead job for retry.
|
||||
*
|
||||
* #2783: an explicit `jobs retry` is an operator asserting "run this
|
||||
* fresh" — so it clears `started_at` (re-stamped on re-claim via
|
||||
* `claim()`'s `COALESCE(started_at, now())`, `queue.ts:620`) and resets
|
||||
* `attempts_made`/`attempts_started` to 0. Without this, `started_at`
|
||||
* kept the ORIGINAL first-claim time, so `handleWallClockTimeouts()`
|
||||
* (anchored on `now() - started_at`, `queue.ts:729-749`) could measure
|
||||
* from long before the retry — a retry issued more than `timeout_ms * 2`
|
||||
* after the original claim was dead-lettered again in under a second,
|
||||
* with `attempts_made` already past `max_attempts`. This made retry
|
||||
* useless for exactly the case it exists for: recovering work after an
|
||||
* outage that outlasted the job's timeout.
|
||||
*
|
||||
* Also resets `stalled_counter` (Codex review): `handleStalled()`
|
||||
* dead-letters once `stalled_counter + 1 >= max_stalled` (`queue.ts:1190`).
|
||||
* A job dead-lettered BY stall exhaustion, left un-reset, would hit that
|
||||
* same threshold on its very first lock expiry after retry — a job
|
||||
* killed by 3 stalls doesn't get a fresh stall budget, contradicting
|
||||
* "run this fresh" the same way the unreset attempt counters did.
|
||||
*/
|
||||
async retryJob(id: number): Promise<MinionJob | null> {
|
||||
const rows = await this.engine.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET status = 'waiting', error_text = NULL,
|
||||
lock_token = NULL, lock_until = NULL, delay_until = NULL,
|
||||
finished_at = NULL, updated_at = now()
|
||||
finished_at = NULL, started_at = NULL, attempts_made = 0,
|
||||
attempts_started = 0, stalled_counter = 0, updated_at = now()
|
||||
WHERE id = $1 AND status IN ('failed', 'dead')
|
||||
RETURNING *`,
|
||||
[id]
|
||||
|
||||
@@ -27,6 +27,7 @@ import type { GBrainConfig } from '../../config.ts';
|
||||
import { operations } from '../../operations.ts';
|
||||
import type { Operation, OperationContext } from '../../operations.ts';
|
||||
import { paramDefToSchema } from '../../../mcp/tool-defs.ts';
|
||||
import { validateSourceId } from '../../utils.ts';
|
||||
import type { ToolCtx, ToolDef } from '../types.ts';
|
||||
|
||||
/**
|
||||
@@ -61,6 +62,12 @@ export const BRAIN_TOOL_ALLOWLIST: ReadonlySet<string> = new Set([
|
||||
'resolve_slugs',
|
||||
'get_ingest_log',
|
||||
'put_page',
|
||||
// #2778: the canonical timeline-write op. Fenced exactly like put_page —
|
||||
// operations.ts:enforceSubagentSlugFence confines the target slug to the
|
||||
// trusted-workspace allow-list (or the wiki/agents/<id>/ namespace) when
|
||||
// ctx.viaSubagent=true, so a subagent can only append timeline entries to
|
||||
// pages it could have written anyway.
|
||||
'add_timeline_entry',
|
||||
// v0.29 — Salience + Anomaly Detection. Both read-only. `get_recent_transcripts`
|
||||
// is intentionally NOT included: subagent calls always have ctx.remote=true,
|
||||
// and the v0.29 trust gate rejects remote callers — adding it here would be
|
||||
@@ -97,6 +104,7 @@ export const BRAIN_TOOL_USAGE_HINTS: Readonly<Record<string, string>> = {
|
||||
resolve_slugs: 'Resolve free-form entity names to canonical slugs (e.g. "Alice" → `people/alice-example`). Use before any tool that takes a slug if the user gave a name not a slug.',
|
||||
get_ingest_log: 'Read the brain ingestion log for diagnostic / verification queries.',
|
||||
put_page: 'Write a markdown page to the gbrain DATABASE (NOT the local filesystem). Page becomes searchable + linkable. Slug must match the agent\'s allowed namespace.',
|
||||
add_timeline_entry: 'Append a dated timeline entry to an existing page (the canonical timeline write). Use over rewriting the page body when recording a dated event. Slug must match the agent\'s allowed namespace.',
|
||||
get_recent_salience: 'Read pages ranked by emotional + activity salience over a recency window. Use for "what\'s been on my mind lately".',
|
||||
find_anomalies: 'Read cohort-level activity outliers (e.g. tag-cohort or type-cohort with unusual recent volume). Use for "what\'s unusual lately".',
|
||||
};
|
||||
@@ -194,6 +202,13 @@ export interface BuildBrainToolsOpts {
|
||||
* SubagentHandlerData.allowed_slug_prefixes via the handler.
|
||||
*/
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
/**
|
||||
* Brain source every tool-call OperationContext is scoped to (#1586).
|
||||
* Trusted (flows from SubagentHandlerData.source_id, which only
|
||||
* PROTECTED_JOB_NAMES-gated submitters can set); validated at build time.
|
||||
* Unset → legacy 'default'.
|
||||
*/
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
interface OpContextDeps {
|
||||
@@ -204,6 +219,7 @@ interface OpContextDeps {
|
||||
signal?: AbortSignal;
|
||||
brainId?: string;
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
@@ -217,7 +233,8 @@ function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
},
|
||||
dryRun: false,
|
||||
remote: true, // match MCP trust boundary for auto-link skip
|
||||
sourceId: 'default', // v0.34 D4: required; subagent tools default to host source
|
||||
// #1586: cycle-resolved source when provided; legacy host default else.
|
||||
sourceId: deps.sourceId ?? 'default',
|
||||
jobId: deps.jobId,
|
||||
subagentId: deps.subagentId,
|
||||
viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace
|
||||
@@ -241,6 +258,11 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
op => BRAIN_TOOL_ALLOWLIST.has(op.name) && filter.has(op.name),
|
||||
);
|
||||
|
||||
// #1586: fail fast on a malformed source id before any tool executes
|
||||
// (defense-in-depth — the seam is trusted, but the value round-trips
|
||||
// through the job payload).
|
||||
if (opts.sourceId !== undefined) validateSourceId(opts.sourceId);
|
||||
|
||||
return picked.map<ToolDef>(op => {
|
||||
const schema = op.name === 'put_page'
|
||||
? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes)
|
||||
@@ -270,6 +292,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
signal: ctx.signal,
|
||||
brainId: opts.brainId,
|
||||
allowedSlugPrefixes: opts.allowedSlugPrefixes,
|
||||
sourceId: opts.sourceId,
|
||||
});
|
||||
const params = (input && typeof input === 'object') ? input as Record<string, unknown> : {};
|
||||
return op.handler(opCtx, params);
|
||||
|
||||
@@ -200,6 +200,12 @@ export interface MinionJobContext {
|
||||
attempts_made: number;
|
||||
/** AbortSignal for cooperative cancellation (fires on timeout, cancel, pause, or lock loss). */
|
||||
signal: AbortSignal;
|
||||
/** Absolute wall-clock deadline (epoch ms) from the claim-time `timeout_at` stamp,
|
||||
* or null when the job has no per-job timeout. This is the DB's ground truth —
|
||||
* the same instant handleTimeouts() dead-letters against — so handlers that
|
||||
* spawn bounded sub-work (e.g. autopilot-cycle's subagent phases) can budget
|
||||
* from the REMAINING time instead of a fixed constant that may exceed it. */
|
||||
deadlineAtMs: number | null;
|
||||
/** AbortSignal that fires only on worker process SIGTERM/SIGINT. Handlers sensitive
|
||||
* to deploy restarts (e.g. the shell handler, which must run a SIGTERM → 5s → SIGKILL
|
||||
* sequence on its child) listen to this in addition to `signal`. Most handlers can
|
||||
@@ -411,6 +417,12 @@ export interface SubagentHandlerData {
|
||||
model?: string;
|
||||
/** Max assistant turns before the loop fails with stop_reason='max_turns'. */
|
||||
max_turns?: number;
|
||||
/**
|
||||
* Per-turn max output tokens (#2778). Resolution: this field →
|
||||
* `agent.max_output_tokens` config → 8192 default. The pre-#2778
|
||||
* hardcoded 4096 made pages >~12KB unwritable via put_page.
|
||||
*/
|
||||
max_tokens?: number;
|
||||
/**
|
||||
* Whitelist of tool names the agent may call. MUST be a subset of the
|
||||
* derived registry names — invalid entries are rejected at tool-dispatch
|
||||
@@ -449,6 +461,17 @@ export interface SubagentHandlerData {
|
||||
* and direct CLI submitters set it.
|
||||
*/
|
||||
allowed_slug_prefixes?: string[];
|
||||
/**
|
||||
* Brain source the subagent's tool calls are scoped to (#1586).
|
||||
*
|
||||
* When set, every tool-call `OperationContext.sourceId` uses this value
|
||||
* instead of the legacy 'default', so put_page writes land in the cycle's
|
||||
* resolved source. Same trust story as `allowed_slug_prefixes`:
|
||||
* PROTECTED_JOB_NAMES gates subagent submission, so only cycle.ts and
|
||||
* direct CLI submitters can set it. Validated via `validateSourceId` at
|
||||
* tool-registry build time.
|
||||
*/
|
||||
source_id?: string;
|
||||
/**
|
||||
* v0.41 Approach C: opt out of the auto-generated tool-usage preamble
|
||||
* that `buildSystemPrompt()` splices into `system`. Default behavior
|
||||
@@ -562,6 +585,7 @@ export type ContentBlock =
|
||||
export type SubagentStopReason =
|
||||
| 'end_turn' // Anthropic says end_turn and last message has no tool_use
|
||||
| 'max_turns' // hit max_turns budget before end_turn
|
||||
| 'max_tokens' // final turn hit the output-token cap — result text is TRUNCATED (#2778)
|
||||
| 'refusal' // detected via stop_reason + content shape
|
||||
| 'error'; // unrecoverable (empty response retry exhausted, etc.)
|
||||
|
||||
|
||||
@@ -900,15 +900,22 @@ export class MinionWorker extends EventEmitter {
|
||||
|
||||
// Per-job wall-clock timeout (timer-armed only if `timeout_ms` was
|
||||
// set on the job; the grace-evict pattern above now lives outside
|
||||
// this branch).
|
||||
// this branch). The delay derives from the claim-time `timeout_at`
|
||||
// stamp when present so this timer, the DB sweeper (handleTimeouts),
|
||||
// and the handler-visible `deadlineAtMs` all agree on ONE absolute
|
||||
// deadline instead of three clocks started at slightly different
|
||||
// instants.
|
||||
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
if (job.timeout_ms != null) {
|
||||
const delayMs = job.timeout_at != null
|
||||
? Math.max(0, job.timeout_at.getTime() - Date.now())
|
||||
: job.timeout_ms;
|
||||
timeoutTimer = setTimeout(() => {
|
||||
if (!abort.signal.aborted) {
|
||||
console.warn(`Job ${job.id} (${job.name}) hit per-job timeout (${job.timeout_ms}ms), aborting`);
|
||||
abort.abort(new Error('timeout'));
|
||||
}
|
||||
}, job.timeout_ms);
|
||||
}, delayMs);
|
||||
}
|
||||
|
||||
const promise = this.executeJob(job, lockToken, abort, lockTimer)
|
||||
@@ -964,6 +971,7 @@ export class MinionWorker extends EventEmitter {
|
||||
data: job.data,
|
||||
attempts_made: job.attempts_made,
|
||||
signal: abort.signal,
|
||||
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
|
||||
shutdownSignal: this.shutdownAbort.signal,
|
||||
updateProgress: async (progress: unknown) => {
|
||||
await this.queue.updateProgress(job.id, lockToken, progress);
|
||||
|
||||
@@ -55,7 +55,7 @@ export interface AgentClientBindings {
|
||||
* `redirect_uri` containing `,`) would be parsed by Postgres as MULTIPLE
|
||||
* array elements, smuggling values past validation. See CSO finding #5.
|
||||
*/
|
||||
function pgArray(arr: string[]): string {
|
||||
export function pgArray(arr: string[]): string {
|
||||
if (!arr || arr.length === 0) return '{}';
|
||||
const escaped = arr.map(s => `"${s.replace(/\\/g, '\\\\').replace(/"/g, '\\"')}"`);
|
||||
return `{${escaped.join(',')}}`;
|
||||
|
||||
+41
-31
@@ -193,6 +193,39 @@ export function matchesSlugAllowList(slug: string, prefixes: readonly string[]):
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Subagent slug-fence enforcement, shared by every mutating op a subagent
|
||||
* can reach (put_page, add_timeline_entry). FAIL-CLOSED: `viaSubagent=true`
|
||||
* enforces the check even if the dispatcher forgot to populate `subagentId`.
|
||||
*
|
||||
* - Trusted-workspace path (ctx.allowedSlugPrefixes set by cycle.ts under
|
||||
* PROTECTED_JOB_NAMES \u2014 MCP cannot reach it): slug must match the
|
||||
* allow-list globs.
|
||||
* - Legacy default: slug must live under `wiki/agents/<subagentId>/...`
|
||||
* (anchored, slash-boundary \u2014 `wiki/agents/12evil/*` can't impersonate
|
||||
* subagent 12).
|
||||
*/
|
||||
function enforceSubagentSlugFence(ctx: OperationContext, slug: string, opName: string): void {
|
||||
if (ctx.viaSubagent !== true) return;
|
||||
if (typeof ctx.subagentId !== 'number' || Number.isNaN(ctx.subagentId)) {
|
||||
throw new OperationError('permission_denied', `${opName} via subagent requires ctx.subagentId`);
|
||||
}
|
||||
const allowList = ctx.allowedSlugPrefixes;
|
||||
if (allowList && allowList.length > 0) {
|
||||
if (!matchesSlugAllowList(slug, allowList)) {
|
||||
throw new OperationError(
|
||||
'permission_denied',
|
||||
`${opName} slug '${slug}' is not within the trusted-workspace allow-list (${allowList.join(', ')})`
|
||||
);
|
||||
}
|
||||
} else {
|
||||
const prefix = `wiki/agents/${ctx.subagentId}/`;
|
||||
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
|
||||
throw new OperationError('permission_denied', `${opName} via subagent must write under '${prefix}...'`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Allowlist validator for uploaded file basenames. Rejects control chars, backslashes,
|
||||
* RTL overrides (\u202E), leading dot (hidden files) and leading dash (CLI flag confusion).
|
||||
@@ -784,37 +817,9 @@ const put_page: Operation = {
|
||||
}
|
||||
|
||||
// Subagent namespace enforcement (v0.15+). Runs BEFORE the dry-run
|
||||
// short-circuit so preview calls surface the same rejection. Confines
|
||||
// LLM-driven writes to wiki/agents/<subagentId>/... — no leading slash
|
||||
// (slug grammar rejects that), anchored, slash-boundary to defeat prefix
|
||||
// collisions like `wiki/agents/12evil/*` impersonating subagent 12.
|
||||
//
|
||||
// FAIL-CLOSED: `viaSubagent=true` enforces the check even if the
|
||||
// dispatcher forgot to populate `subagentId`. Agent-originated writes
|
||||
// without an owning subagent id are rejected outright.
|
||||
if (ctx.viaSubagent === true) {
|
||||
if (typeof ctx.subagentId !== 'number' || Number.isNaN(ctx.subagentId)) {
|
||||
throw new OperationError('permission_denied', 'put_page via subagent requires ctx.subagentId');
|
||||
}
|
||||
const allowList = ctx.allowedSlugPrefixes;
|
||||
if (allowList && allowList.length > 0) {
|
||||
// Trusted-workspace path: explicit allow-list bounds writes.
|
||||
// Set only by cycle.ts (synthesize/patterns) which submits subagent
|
||||
// jobs under PROTECTED_JOB_NAMES — MCP cannot reach this branch.
|
||||
if (!matchesSlugAllowList(slug, allowList)) {
|
||||
throw new OperationError(
|
||||
'permission_denied',
|
||||
`put_page slug '${slug}' is not within the trusted-workspace allow-list (${allowList.join(', ')})`
|
||||
);
|
||||
}
|
||||
} else {
|
||||
// Legacy default: agent-namespace confinement.
|
||||
const prefix = `wiki/agents/${ctx.subagentId}/`;
|
||||
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
|
||||
throw new OperationError('permission_denied', `put_page via subagent must write under '${prefix}...'`);
|
||||
}
|
||||
}
|
||||
}
|
||||
// short-circuit so preview calls surface the same rejection. See
|
||||
// enforceSubagentSlugFence for the fail-closed policy.
|
||||
enforceSubagentSlugFence(ctx, slug, 'put_page');
|
||||
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'put_page', slug: p.slug };
|
||||
// Skip embedding when the AI gateway has no embedding provider configured.
|
||||
@@ -2149,6 +2154,11 @@ const add_timeline_entry: Operation = {
|
||||
mutating: true,
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
// #2778: same fail-closed slug fence as put_page. add_timeline_entry is
|
||||
// subagent-allowlisted (brain-allowlist.ts), so timeline writes must be
|
||||
// confined to the same namespace/allow-list as page writes. Runs before
|
||||
// the dry-run short-circuit so preview calls surface the same rejection.
|
||||
enforceSubagentSlugFence(ctx, p.slug as string, 'add_timeline_entry');
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'add_timeline_entry', slug: p.slug };
|
||||
const date = p.date as string;
|
||||
// Reject anything that isn't a strict YYYY-MM-DD with year 1900-2199 and
|
||||
|
||||
+156
-10
@@ -23,7 +23,9 @@ import { runMigrations } from './migrate.ts';
|
||||
import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
|
||||
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
|
||||
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
|
||||
import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts';
|
||||
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
|
||||
@@ -54,7 +56,7 @@ import { GBrainError, PAGE_SORT_SQL, ENRICH_ORDER_SQL } from './types.ts';
|
||||
import { finalizeLastSeen } from './chronicle/last-seen.ts';
|
||||
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte } from './search/sql-ranking.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
|
||||
import {
|
||||
normalizeEngineColumn,
|
||||
buildVectorCastFragment,
|
||||
@@ -1034,7 +1036,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const ingestedAt = (sourceKind || sourceUri || ingestedVia) ? new Date().toISOString() : null;
|
||||
const { rows } = await this.db.query(
|
||||
`INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename, chunker_version, source_path, source_kind, source_uri, ingested_via, ingested_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, now(), $10::timestamptz, $11, $12, COALESCE($13, 1), $14, $15, $16, $17, $18::timestamptz)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, now(), $10::timestamptz, $11, $12, COALESCE($13, ${MARKDOWN_CHUNKER_VERSION}), $14, $15, $16, $17, $18::timestamptz)
|
||||
ON CONFLICT (source_id, slug) DO UPDATE SET
|
||||
type = EXCLUDED.type,
|
||||
page_kind = EXCLUDED.page_kind,
|
||||
@@ -1625,20 +1627,24 @@ export class PGLiteEngine implements BrainEngine {
|
||||
extraFilter += ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const keywordSql =
|
||||
`WITH ranked AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
-- v0.27.1: hide image rows from default text-keyword search so
|
||||
-- OCR text doesn't drown text-page hits. Image-similarity queries
|
||||
-- run a separate vector path on embedding_image.
|
||||
@@ -1649,10 +1655,140 @@ export class PGLiteEngine implements BrainEngine {
|
||||
${buildBestPerPagePoolCte('ranked')}
|
||||
SELECT * FROM best_per_page
|
||||
ORDER BY score DESC, page_id ASC, chunk_id ASC
|
||||
LIMIT $3 OFFSET $4`,
|
||||
params
|
||||
);
|
||||
LIMIT $3 OFFSET $4`;
|
||||
|
||||
let { rows } = await this.db.query(keywordSql, params);
|
||||
// D2 fix (fix/title-retrieval-arm): websearch AND semantics at chunk
|
||||
// grain mean one non-co-occurring token zeroes keyword recall. When the
|
||||
// strict query returns nothing, retry ONCE with OR-of-terms. Strict-AND
|
||||
// results always win when non-empty (no change for working queries).
|
||||
// Opt-in via SearchOpts.orFallback (Reviewer F1): only hybridSearch's
|
||||
// recall arm relaxes; precision consumers (countMentions,
|
||||
// link-extraction, eval) keep the strict-AND contract.
|
||||
if (rows.length === 0 && opts?.orFallback) {
|
||||
const orQuery = buildOrFallbackWebsearchQuery(query);
|
||||
if (orQuery) {
|
||||
const fallbackParams = [...params];
|
||||
fallbackParams[0] = orQuery;
|
||||
({ rows } = await this.db.query(keywordSql, fallbackParams));
|
||||
}
|
||||
}
|
||||
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
}
|
||||
|
||||
/**
|
||||
* fix/title-retrieval-arm (D1): page-grain title candidate arm. See the
|
||||
* BrainEngine interface doc for the full contract. Queries
|
||||
* pages.search_vector (title weight 'A' dominates ts_rank_cd by
|
||||
* construction) with the same page-grain filters the keyword arm applies
|
||||
* (type/types/excludeSlugs/date/source scoping, hard-excludes,
|
||||
* visibility), joined to one representative chunk per page. Applies the
|
||||
* same AND→OR recall fallback as searchKeyword. NO query-length gate —
|
||||
* long exact-title queries are the case this arm exists for.
|
||||
*
|
||||
* CJK queries fall through to websearch FTS here (a single-token CJK
|
||||
* query CAN exact-match a single-token CJK title); the richer CJK ILIKE
|
||||
* fallback stays keyword-arm-only.
|
||||
*/
|
||||
async searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
// language/symbolKind are chunk-grain code filters with no page-grain
|
||||
// meaning; a code-scoped query gets no title candidates rather than
|
||||
// rows that silently violate the caller's filter.
|
||||
if (opts?.language || opts?.symbolKind) return [];
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const detailLow = opts?.detail === 'low';
|
||||
|
||||
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
||||
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
||||
}
|
||||
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const params: unknown[] = [query, limit, offset];
|
||||
let extraFilter = '';
|
||||
if (opts?.type) {
|
||||
params.push(opts.type);
|
||||
extraFilter += ` AND p.type = $${params.length}`;
|
||||
}
|
||||
if (opts?.types && opts.types.length > 0) {
|
||||
params.push(opts.types);
|
||||
extraFilter += ` AND p.type = ANY($${params.length}::text[])`;
|
||||
}
|
||||
if (opts?.exclude_slugs?.length) {
|
||||
params.push(opts.exclude_slugs);
|
||||
extraFilter += ` AND p.slug != ALL($${params.length}::text[])`;
|
||||
}
|
||||
if (opts?.afterDate) {
|
||||
params.push(opts.afterDate);
|
||||
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
|
||||
}
|
||||
if (opts?.beforeDate) {
|
||||
params.push(opts.beforeDate);
|
||||
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
|
||||
}
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
params.push(opts.sourceIds);
|
||||
extraFilter += ` AND p.source_id = ANY($${params.length}::text[])`;
|
||||
} else if (opts?.sourceId) {
|
||||
params.push(opts.sourceId);
|
||||
extraFilter += ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
|
||||
// Page grain — one row per page by construction, so no best_per_page
|
||||
// pooling CTE is needed. The LEFT JOIN LATERAL picks the representative
|
||||
// chunk (compiled_truth first, then lowest chunk_index); COALESCEs keep
|
||||
// chunkless pages retrievable (the extreme D1 case: a title with no
|
||||
// body) with the alias-hop row shape (chunk_id 0, empty chunk_text).
|
||||
// Accepted limitations (Reviewer F5/F6): the synthetic chunkless row
|
||||
// inherits the compiled-truth RRF boost and dedups on empty chunk_text;
|
||||
// and detail='low' filters only the REPRESENTATIVE — pages without a
|
||||
// compiled_truth chunk still surface (unlike the keyword arm's filter).
|
||||
const titlesSql =
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
COALESCE(rep.id, 0) as chunk_id,
|
||||
COALESCE(rep.chunk_index, 0) as chunk_index,
|
||||
COALESCE(rep.chunk_text, '') as chunk_text,
|
||||
COALESCE(rep.chunk_source, 'compiled_truth') as chunk_source,
|
||||
ts_rank_cd(p.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM pages p
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT cc.id, cc.chunk_index, cc.chunk_text, cc.chunk_source
|
||||
FROM content_chunks cc
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.modality = 'text'
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
ORDER BY (cc.chunk_source = 'compiled_truth') DESC, cc.chunk_index ASC
|
||||
LIMIT 1
|
||||
) rep ON true
|
||||
WHERE p.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
|
||||
${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY score DESC, p.id ASC
|
||||
LIMIT $2 OFFSET $3`;
|
||||
|
||||
let { rows } = await this.db.query(titlesSql, params);
|
||||
if (rows.length === 0) {
|
||||
const orQuery = buildOrFallbackWebsearchQuery(query);
|
||||
if (orQuery) {
|
||||
const fallbackParams = [...params];
|
||||
fallbackParams[0] = orQuery;
|
||||
({ rows } = await this.db.query(titlesSql, fallbackParams));
|
||||
}
|
||||
}
|
||||
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
||||
}
|
||||
|
||||
@@ -1857,20 +1993,23 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// visibilityClause already declared above (v0.32.7: hoisted so CJK branch can reuse).
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2 OFFSET $3`,
|
||||
params
|
||||
@@ -3536,6 +3675,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
THEN ep.frontmatter->'event'->'who' ELSE '[]'::jsonb END
|
||||
) AS w(name) WHERE w.name = $1 OR w.name LIKE $2)))`,
|
||||
];
|
||||
// "Last seen" is a PAST relation: chronicle stores future events
|
||||
// (calendar-event is eligible), which must not read as "last seen".
|
||||
// Bound to <= asof/today, mirroring getOnThisDay's `te.date < target`.
|
||||
let seenThrough: string;
|
||||
if (opts?.asof) { params.push(opts.asof); seenThrough = `$${params.length}::date`; }
|
||||
else { seenThrough = `current_date`; }
|
||||
where.push(`te.date <= ${seenThrough}`);
|
||||
this.pushChronicleSource(where, params, opts);
|
||||
const result = await this.db.query(
|
||||
`SELECT te.date::text AS last_date, ep.slug AS last_event_slug
|
||||
|
||||
@@ -1022,6 +1022,12 @@ ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
|
||||
|
||||
-- #2704: compiled_truth (unbounded whole-page body) deliberately NOT
|
||||
-- indexed — overflows Postgres's 1MB tsvector cap on large pages.
|
||||
-- content_chunks.search_vector (chunk-grain, populated separately) is
|
||||
-- what searchKeyword() actually queries. See migrate.ts's v124 migration
|
||||
-- for the full rationale; keep in sync with that + reindex-search-vector.ts
|
||||
-- + schema-embedded.ts.
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
@@ -1033,7 +1039,6 @@ BEGIN
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
|
||||
+694
-372
File diff suppressed because it is too large
Load Diff
@@ -830,6 +830,12 @@ ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
|
||||
|
||||
-- Function to rebuild search_vector for a page
|
||||
-- #2704: compiled_truth (unbounded whole-page body) deliberately NOT
|
||||
-- indexed — overflows Postgres's 1MB tsvector cap on large pages.
|
||||
-- content_chunks.search_vector (chunk-grain, populated separately) is
|
||||
-- what searchKeyword() actually queries. See migrate.ts's v124 migration
|
||||
-- for the full rationale; keep in sync with that + reindex-search-vector.ts
|
||||
-- + pglite-schema.ts.
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS \$\$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
@@ -843,7 +849,6 @@ BEGIN
|
||||
-- Build weighted tsvector
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
|
||||
+116
-17
@@ -34,6 +34,7 @@ import { normalizeAlias } from './alias-normalize.ts';
|
||||
import { stampEvidence } from './evidence.ts';
|
||||
import { expandAnchors, hydrateChunks } from './two-pass.ts';
|
||||
import { enforceTokenBudget } from './token-budget.ts';
|
||||
import { warnOncePerProcess } from '../utils.ts';
|
||||
import { recordSearchTelemetry } from './telemetry.ts';
|
||||
import {
|
||||
weightsForIntent,
|
||||
@@ -740,6 +741,21 @@ export interface HybridSearchOpts extends SearchOpts {
|
||||
* a fresh per-call deadline. Not part of the public contract.
|
||||
*/
|
||||
_queryEmbedDeadline?: QueryEmbedDeadline;
|
||||
|
||||
/**
|
||||
* INTERNAL — cache-consult outcome threaded from `hybridSearchCached` into
|
||||
* the inner `hybridSearch` so the ONE telemetry record per search (emitted
|
||||
* by the inner function) carries the cache classification: 'miss' when the
|
||||
* semantic cache was consulted and had no row, 'disabled' when the consult
|
||||
* was skipped (cache off, walk/near-symbol/non-default-column/adaptive
|
||||
* skip, or the lookup embed failed). Folded into the RECORDED meta only —
|
||||
* `onMeta` payloads are unchanged. Direct `hybridSearch` callers leave it
|
||||
* undefined and keep recording with no cache field (they never consulted
|
||||
* the cache). The cache-HIT record is emitted by `hybridSearchCached`
|
||||
* itself, since the inner function never runs on a hit. Not part of the
|
||||
* public contract.
|
||||
*/
|
||||
_telemetryCacheStatus?: 'miss' | 'disabled';
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -917,6 +933,11 @@ export async function hybridSearch(
|
||||
// it never has to read config. Engines normalize string-or-descriptor
|
||||
// via normalizeEngineColumn; the descriptor path is the strict one.
|
||||
embeddingColumn: resolvedCol,
|
||||
// D2 fix (fix/title-retrieval-arm, Reviewer F1): the hybrid keyword arm
|
||||
// is a recall arm — opt in to the engine's AND→OR zero-recall fallback.
|
||||
// Direct searchKeyword consumers (countMentions, link-extraction, eval)
|
||||
// do NOT set this and keep the strict-AND contract.
|
||||
orFallback: true,
|
||||
};
|
||||
// Track what actually ran for the optional onMeta callback (v0.25.0).
|
||||
// Caller leaves onMeta undefined → these flags are computed but never
|
||||
@@ -943,7 +964,15 @@ export async function hybridSearch(
|
||||
// swallow — capture telemetry is best-effort
|
||||
}
|
||||
try {
|
||||
recordSearchTelemetry(engine, meta, { results_count: lastResultsCount, rank1_score: lastRank1Score });
|
||||
// #2952 — fold the cache-consult outcome (threaded by hybridSearchCached)
|
||||
// into the RECORDED meta only. None of the inner return paths set a
|
||||
// `cache` field themselves, so this is the sole source of the miss /
|
||||
// disabled classification; `onMeta` consumers above still receive the
|
||||
// meta unchanged (the cached wrapper emits its own merged meta to them).
|
||||
const recordedMeta = opts?._telemetryCacheStatus
|
||||
? { ...meta, cache: { status: opts._telemetryCacheStatus } }
|
||||
: meta;
|
||||
recordSearchTelemetry(engine, recordedMeta, { results_count: lastResultsCount, rank1_score: lastRank1Score });
|
||||
} catch {
|
||||
// swallow — telemetry must never break the search hot path.
|
||||
}
|
||||
@@ -967,8 +996,31 @@ export async function hybridSearch(
|
||||
const earlyModality = (opts?.crossModal && opts.crossModal !== 'auto')
|
||||
? opts.crossModal
|
||||
: (suggestions.suggestedModality ?? 'text');
|
||||
const keywordResults: SearchResult[] =
|
||||
earlyModality === 'image' ? [] : await engine.searchKeyword(query, searchOpts);
|
||||
// D1 fix (fix/title-retrieval-arm): page-grain title candidate arm,
|
||||
// fetched CONCURRENTLY with the keyword arm (Reviewer F7 — independent
|
||||
// engine queries). The chunk FTS vector never includes the page title, so
|
||||
// an exact-title query can be unretrievable by keyword — this arm queries
|
||||
// pages.search_vector (title weight 'A') directly. Runs regardless of
|
||||
// query token count: the alias hop (≤6-token guard) and the title-phrase
|
||||
// boost are re-rank-only, so LONG exact-title queries — where strict-AND
|
||||
// chunk FTS is weakest — need a candidate GENERATOR. Fail-open WITH
|
||||
// SIGNAL (Reviewer F2): a SQL error (e.g. a pre-search_vector brain)
|
||||
// degrades to no title candidates, but warns once per process so a
|
||||
// broken engine arm cannot ship dark.
|
||||
const [keywordResults, titleResults]: [SearchResult[], SearchResult[]] =
|
||||
earlyModality === 'image'
|
||||
? [[], []]
|
||||
: await Promise.all([
|
||||
engine.searchKeyword(query, searchOpts),
|
||||
engine.searchTitles(query, searchOpts).catch((err: unknown) => {
|
||||
warnOncePerProcess(
|
||||
'search-titles-arm-failed',
|
||||
`[gbrain] searchTitles arm failed (fail-open, title candidates skipped): ` +
|
||||
`${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
return [] as SearchResult[];
|
||||
}),
|
||||
]);
|
||||
|
||||
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
|
||||
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
|
||||
@@ -1046,14 +1098,16 @@ export async function hybridSearch(
|
||||
if (!isAvailable('embedding', providerProbe)) {
|
||||
// v0.43 — fuse the relational arm with keyword so typed-edge answers
|
||||
// survive on the no-embedding-provider path (the relational win is most
|
||||
// valuable exactly when vector is unavailable).
|
||||
// valuable exactly when vector is unavailable). The title arm fuses here
|
||||
// too — an exact-title lookup on a keyless install is precisely where
|
||||
// chunk-grain keyword FTS alone fails (D1).
|
||||
let noEmbedResults = keywordResults;
|
||||
if (relationalList.length > 0) {
|
||||
if (relationalList.length > 0 || titleResults.length > 0) {
|
||||
const fk = opts?.rrfK ?? RRF_K;
|
||||
noEmbedResults = rrfFusionWeighted(
|
||||
[{ list: keywordResults, k: fk }, { list: relationalList, k: fk }],
|
||||
detailResolved !== 'high',
|
||||
);
|
||||
const noEmbedLists = [{ list: keywordResults, k: fk }];
|
||||
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
|
||||
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
|
||||
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
|
||||
}
|
||||
if (noEmbedResults.length > 0) {
|
||||
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
|
||||
@@ -1280,14 +1334,15 @@ export async function hybridSearch(
|
||||
// post-fusion stages here too — without it, salience='on' silently
|
||||
// does nothing on embed failures.
|
||||
// v0.43: fuse the relational arm with keyword via RRF so typed-edge
|
||||
// answers survive even when vector is unavailable.
|
||||
// answers survive even when vector is unavailable. The title arm fuses
|
||||
// here too (same rationale as the no-embedding-provider path — D1).
|
||||
let fallbackResults = keywordResults;
|
||||
if (relationalList.length > 0) {
|
||||
if (relationalList.length > 0 || titleResults.length > 0) {
|
||||
const fk = opts?.rrfK ?? RRF_K;
|
||||
fallbackResults = rrfFusionWeighted(
|
||||
[{ list: keywordResults, k: fk }, { list: relationalList, k: fk }],
|
||||
detail !== 'high',
|
||||
);
|
||||
const fallbackLists = [{ list: keywordResults, k: fk }];
|
||||
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
|
||||
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
|
||||
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
|
||||
}
|
||||
if (fallbackResults.length > 0) {
|
||||
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
|
||||
@@ -1352,6 +1407,15 @@ export async function hybridSearch(
|
||||
{ list: keywordResults, k: keywordK },
|
||||
];
|
||||
|
||||
// D1 fix (fix/title-retrieval-arm) — title candidate arm as a third
|
||||
// weighted list. Fuses at the keyword arm's intent-effective k (same
|
||||
// lexical-evidence class, no new tunable). Mirrors the keyword list's
|
||||
// inclusion rules: fetch was gated on earlyModality, so no extra modality
|
||||
// check here. Empty for non-matching queries → pure no-op.
|
||||
if (titleResults.length > 0) {
|
||||
allLists.push({ list: titleResults, k: keywordK });
|
||||
}
|
||||
|
||||
// v0.43 — relational recall arm (fourth RRF arm), built above so it also
|
||||
// contributes on the keyword-only fallback path. Neutral weight (baseRrfK):
|
||||
// competes evenly with keyword/vector, not dominating. Empty for
|
||||
@@ -1744,8 +1808,17 @@ export async function hybridSearchCached(
|
||||
...(hit.meta?.embedding_column ? { embedding_column: hit.meta.embedding_column } : {}),
|
||||
...(hit.meta?.adaptive_return ? { adaptive_return: hit.meta.adaptive_return } : {}),
|
||||
...(hit.meta?.autocut ? { autocut: hit.meta.autocut } : {}),
|
||||
// Per-call budget: prefer the STORED budget record, which carries
|
||||
// the true dropped count from the write-time cut — the
|
||||
// re-application above ran on an already-cut set and reads
|
||||
// dropped=0 (same masking as the miss path's finalMeta). Safe
|
||||
// unconditionally: tokenBudget is folded into knobsHash (`tb=`),
|
||||
// so a hit only ever serves a lookup with the identical resolved
|
||||
// budget as the write — the outer pass can never cut further.
|
||||
// budgetMeta stays as the fallback for legacy rows stored without
|
||||
// a budget record.
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
? { token_budget: hit.meta?.token_budget ?? budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
@@ -1753,6 +1826,21 @@ export async function hybridSearchCached(
|
||||
} catch {
|
||||
// swallow — telemetry is best-effort
|
||||
}
|
||||
// #2952 — a cache hit never reaches the inner hybridSearch (the only
|
||||
// other telemetry site), so record the search HERE or it vanishes from
|
||||
// stats entirely (count, results, tokens, rank-1 — not just the hit
|
||||
// counter). Same rank-1 rule as the inner return paths. Tokens are
|
||||
// gated on the MODE-resolved budget, mirroring the inner paths' `if
|
||||
// (resolvedMode.tokenBudget > 0)` meta condition — otherwise a
|
||||
// tokenmax (budget-off) brain would record real tokens on hits but 0
|
||||
// on misses, skewing avg-tokens upward as the hit rate rises (codex).
|
||||
recordSearchTelemetry(engine, cachedMeta, {
|
||||
results_count: budgeted.length,
|
||||
...(resolvedForCache.tokenBudget && resolvedForCache.tokenBudget > 0
|
||||
? { tokens_estimate: budgetMeta.used }
|
||||
: {}),
|
||||
rank1_score: budgeted[0] ? (budgeted[0].base_score ?? budgeted[0].score) : undefined,
|
||||
});
|
||||
return budgeted;
|
||||
}
|
||||
}
|
||||
@@ -1768,6 +1856,10 @@ export async function hybridSearchCached(
|
||||
// v0.42.20.0 (Fix 3) — share the query-embed deadline so the inner embed
|
||||
// doesn't start a fresh 6s budget after the cache-lookup already spent it.
|
||||
_queryEmbedDeadline: queryEmbedDl,
|
||||
// #2952 — classify this search's telemetry record (emitted by the inner
|
||||
// function) with the cache-consult outcome. 'hit' already returned above,
|
||||
// so only miss/disabled reach this call.
|
||||
_telemetryCacheStatus: cacheStatus === 'disabled' ? 'disabled' : 'miss',
|
||||
onMeta: (m) => {
|
||||
innerMetaBox.current = m;
|
||||
// Do NOT call userOnMeta here — we'll emit a merged meta below
|
||||
@@ -1794,8 +1886,15 @@ export async function hybridSearchCached(
|
||||
...(innerMeta?.embedding_column ? { embedding_column: innerMeta.embedding_column } : {}),
|
||||
...(innerMeta?.adaptive_return ? { adaptive_return: innerMeta.adaptive_return } : {}),
|
||||
...(innerMeta?.autocut ? { autocut: innerMeta.autocut } : {}),
|
||||
// Per-call budget: prefer the INNER meta's budget record. The inner
|
||||
// hybridSearch already enforced the same resolved budget (per-call wins
|
||||
// in resolveSearchMode), so the re-application above sees an
|
||||
// already-cut set and its meta reads dropped=0 — masking the real cut
|
||||
// from onMeta consumers (the `dropped` under-report the restored
|
||||
// search-lite test caught). The outer pass stays as the enforcement
|
||||
// for the cache-HIT path, where no inner run exists.
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
? { token_budget: innerMeta?.token_budget ?? budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
|
||||
@@ -206,6 +206,51 @@ export function buildBestPerPagePoolCte(candidateCte: string): string {
|
||||
)`;
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// AND→OR keyword-recall fallback (fix/title-retrieval-arm, D2)
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Build a relaxed OR-of-terms websearch string for the keyword-arm recall
|
||||
* fallback.
|
||||
*
|
||||
* `websearch_to_tsquery('english', query)` joins unquoted terms with `&`
|
||||
* (AND). At chunk grain, one query token that doesn't co-occur in any
|
||||
* single chunk zeroes keyword recall with no fallback. When the strict
|
||||
* AND query returns zero rows, engines retry ONCE with the string this
|
||||
* builder returns — the same tokens joined with websearch's `OR` keyword,
|
||||
* which compiles to `|`.
|
||||
*
|
||||
* Why rebuild via websearch syntax instead of hand-assembling a tsquery:
|
||||
* websearch_to_tsquery never raises on malformed input, applies the same
|
||||
* stemming/stopword pipeline as the document side, and an all-stopword
|
||||
* token list degrades to an empty tsquery (matches nothing) instead of a
|
||||
* SQL error — the empty-tsquery guard comes free.
|
||||
*
|
||||
* Returns null when relaxation is pointless or unsafe:
|
||||
* - fewer than 2 tokens survive tokenization (OR of one term is the same
|
||||
* query as AND of one term);
|
||||
* - the raw query uses websearch OPERATORS (Reviewer F3): a `-term`
|
||||
* negation would be RESURRECTED as a positive OR term, and a quoted
|
||||
* phrase would degrade to a bag of words — both invert caller intent,
|
||||
* so operator queries get no fallback at all.
|
||||
* Tokenization splits on non-alphanumeric runs (Unicode-aware). Literal
|
||||
* OR/AND words are dropped so they can't be re-parsed as operators
|
||||
* mid-list.
|
||||
*/
|
||||
export function buildOrFallbackWebsearchQuery(query: string): string | null {
|
||||
// F3 operator guard: any double quote, or a dash LEADING a token
|
||||
// (whitespace/start boundary — interior hyphens like "foo-bar" are fine).
|
||||
if (query.includes('"') || /(^|\s)-\S/.test(query)) return null;
|
||||
const tokens = query
|
||||
.normalize('NFKC')
|
||||
.split(/[^\p{L}\p{N}]+/u)
|
||||
.filter(Boolean)
|
||||
.filter(t => { const u = t.toUpperCase(); return u !== 'OR' && u !== 'AND'; });
|
||||
if (tokens.length < 2) return null;
|
||||
return tokens.join(' OR ');
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// v0.29.1 — Recency component SQL builder
|
||||
// ============================================================
|
||||
|
||||
@@ -29,6 +29,12 @@ export interface SkilloptPhaseOpts {
|
||||
engine: BrainEngine;
|
||||
dryRun?: boolean;
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* issue #2860 — `gbrain dream --phase skillopt --once`. Bypasses the
|
||||
* `cycle.skillopt.enabled` feature flag for THIS call only; never reads
|
||||
* or writes config. Per-skill + brain-wide cost caps still apply.
|
||||
*/
|
||||
once?: boolean;
|
||||
}
|
||||
|
||||
export interface SkilloptPhaseResult {
|
||||
@@ -63,13 +69,19 @@ export async function runPhaseSkillopt(opts: SkilloptPhaseOpts): Promise<Skillop
|
||||
enabled = v === 'true';
|
||||
} catch { /* default OFF */ }
|
||||
if (!enabled) {
|
||||
return {
|
||||
phase: 'skillopt',
|
||||
status: 'skipped',
|
||||
duration_ms: Date.now() - start,
|
||||
summary: 'feature flag off (gbrain config set cycle.skillopt.enabled true to enable)',
|
||||
details: { reason: 'feature_flag_off' },
|
||||
};
|
||||
if (!opts.once) {
|
||||
return {
|
||||
phase: 'skillopt',
|
||||
status: 'skipped',
|
||||
duration_ms: Date.now() - start,
|
||||
summary: 'feature flag off (gbrain config set cycle.skillopt.enabled true to enable)',
|
||||
details: { reason: 'feature_flag_off' },
|
||||
};
|
||||
}
|
||||
process.stderr.write(
|
||||
'[dream] --once: cycle.skillopt.enabled is false but ' +
|
||||
'--phase skillopt --once forces this run (config untouched)\n',
|
||||
);
|
||||
}
|
||||
|
||||
// Per-skill + brain-wide cost caps.
|
||||
|
||||
+68
-1
@@ -45,6 +45,8 @@ import {
|
||||
parseRemoteUrl,
|
||||
cloneRepo,
|
||||
validateRepoState,
|
||||
isInsideGitRepo,
|
||||
hasTrackedContent,
|
||||
RemoteUrlError,
|
||||
GitOperationError,
|
||||
type RepoState,
|
||||
@@ -67,7 +69,8 @@ export type SourceOpErrorCode =
|
||||
| 'protected_id'
|
||||
| 'clone_dir_outside_gbrain'
|
||||
| 'symlink_escape'
|
||||
| 'unmanaged_path';
|
||||
| 'unmanaged_path'
|
||||
| 'not_a_git_repo';
|
||||
|
||||
export class SourceOpError extends Error {
|
||||
constructor(
|
||||
@@ -145,6 +148,13 @@ export interface AddSourceOpts {
|
||||
* Only honored when remoteUrl is set.
|
||||
*/
|
||||
cloneDir?: string;
|
||||
/**
|
||||
* Skip the #2707 git-repo validation on `localPath`. Opt-in escape hatch
|
||||
* for registering a path before it's git-initialized (e.g. an automated
|
||||
* pipeline that populates + `git init`s the directory after `sources add`
|
||||
* runs). Does NOT auto-`git init` anything — see `addSource` docstring.
|
||||
*/
|
||||
force?: boolean;
|
||||
}
|
||||
|
||||
export interface RemoveSourceOpts {
|
||||
@@ -157,6 +167,20 @@ export interface RemoveSourceOpts {
|
||||
|
||||
// ── Helpers ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* POSIX single-quote `arg` unless it's already shell-safe. #2707 codex round
|
||||
* 1: the `not_a_git_repo` remediation error prints a pasteable `git ...`
|
||||
* command built from the caller-supplied path — spaces, `$()`, backticks,
|
||||
* etc. must be inert literals when pasted, which double-quoting would not
|
||||
* guarantee (command substitution still runs inside "..."). Mirrors
|
||||
* `src/commands/connect.ts:shellQuote` (not imported — that file is a
|
||||
* commands/ caller of core/, not the other way around).
|
||||
*/
|
||||
function shellQuote(arg: string): string {
|
||||
if (/^[A-Za-z0-9_.:/@-]+$/.test(arg)) return arg;
|
||||
return `'${arg.replace(/'/g, "'\\''")}'`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate via the canonical regex from `source-id.ts` but rethrow as the
|
||||
* sources-ops-tagged error so `gbrain sources add` keeps its user-facing
|
||||
@@ -310,6 +334,18 @@ export function unownedHint(
|
||||
|
||||
// ── addSource ───────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* #2707: `--path` registration used to accept any existing directory with
|
||||
* zero git validation, deferring the failure to the first `gbrain sync`
|
||||
* ("Not inside a git repository: ..."). By the time that surfaces the
|
||||
* source has already been silently stale for however long nobody read the
|
||||
* sync logs. This is registration-time, fail-fast validation ONLY — it
|
||||
* never auto-`git init`s the directory (that would cross the consent
|
||||
* boundary #2967 established for sync-time self-heal: a `--path` source is
|
||||
* the user's own external directory, and gbrain must not mutate it without
|
||||
* explicit ask). Callers who want to register before git-init exists opt in
|
||||
* via `force: true` (CLI: `--force`).
|
||||
*/
|
||||
export async function addSource(
|
||||
engine: BrainEngine,
|
||||
opts: AddSourceOpts,
|
||||
@@ -437,6 +473,37 @@ export async function addSource(
|
||||
}
|
||||
} else {
|
||||
// ── Path B: --path or no path (existing behavior, pre-v0.28) ─────────
|
||||
// #2707: only validate when the path actually exists — a not-yet-created
|
||||
// path is a different (pre-existing, out of scope) failure mode, and
|
||||
// gating on existsSync keeps this a fail-fast check on the exact bug
|
||||
// report ("plain directory accepted, sync fails later") rather than a
|
||||
// broader "does this path exist" check nobody asked for.
|
||||
//
|
||||
// Both isInsideGitRepo AND hasTrackedContent must hold. isInsideGitRepo
|
||||
// alone lets through a `git init`ed-but-never-committed directory (fails
|
||||
// sync's "No commits in repo ..."), AND an empty-commit-then-untracked-
|
||||
// files directory (git resolves HEAD fine but the tree is empty — the
|
||||
// exact silent-staleness footgun #2707(c) describes: sync "succeeds"
|
||||
// importing nothing, then never notices the untracked files change).
|
||||
// hasTrackedContent's `ls-tree HEAD -- .` catches both (codex round 2).
|
||||
if (
|
||||
opts.localPath &&
|
||||
!opts.force &&
|
||||
existsSync(opts.localPath) &&
|
||||
(!isInsideGitRepo(opts.localPath) || !hasTrackedContent(opts.localPath))
|
||||
) {
|
||||
const q = shellQuote(opts.localPath);
|
||||
throw new SourceOpError(
|
||||
'not_a_git_repo',
|
||||
`"${opts.localPath}" is not a git repository with committed, tracked files ` +
|
||||
`(or a subdirectory of one). GBrain sync requires every --path source to ` +
|
||||
`be git-initialized, with the files actually committed — an empty commit ` +
|
||||
`is not enough (the walker reads through git objects, so untracked files ` +
|
||||
`stay invisible). Fix: \`git -C ${q} init && git -C ${q} add -A && ` +
|
||||
`git -C ${q} commit -m "initial import"\`, then re-run this command. To ` +
|
||||
`register anyway and git-init later, pass --force.`,
|
||||
);
|
||||
}
|
||||
const config: Record<string, unknown> = {};
|
||||
if (opts.federated !== null && opts.federated !== undefined) {
|
||||
config.federated = opts.federated;
|
||||
|
||||
+9
-4
@@ -219,7 +219,7 @@ function globToRegex(pattern: string): RegExp {
|
||||
return new RegExp(regex);
|
||||
}
|
||||
|
||||
function matchesAnyGlob(path: string, patterns?: string[]): boolean {
|
||||
export function matchesAnyGlob(path: string, patterns?: string[]): boolean {
|
||||
if (!patterns || patterns.length === 0) return false;
|
||||
const normalized = path.replace(/\\/g, '/');
|
||||
return patterns.some((pattern) => globToRegex(pattern).test(normalized));
|
||||
@@ -255,7 +255,12 @@ const PRUNE_DIR_NAMES = new Set<string>([
|
||||
// with the first-sync walker in commands/import.ts.
|
||||
'venv',
|
||||
'.raw',
|
||||
'ops',
|
||||
// NOTE (#2404): `'ops'` used to be in this list (a v0.2.0-era carve-out for
|
||||
// one brain layout). Matching the bare segment pruned EVERY user `ops/`
|
||||
// directory at any depth — sync silently deleted `ops/*` pages and never
|
||||
// imported `ops/*` files, while the bundled daily-task-manager skill
|
||||
// prescribes `ops/tasks` as its canonical page. `ops/` is ordinary content;
|
||||
// do NOT re-add it. Only generated/vendored trees belong here.
|
||||
]);
|
||||
|
||||
/**
|
||||
@@ -352,8 +357,8 @@ function classifySync(path: string, opts: SyncableOptions = {}): SyncableReason
|
||||
if (!isAllowedByStrategy(path, strategy)) return 'strategy';
|
||||
|
||||
// Skip every path segment that pruneDir would block walkers from descending
|
||||
// into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars,
|
||||
// `node_modules/` (latent bug fix), and `ops/` at any depth.
|
||||
// into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, and
|
||||
// vendor/generated trees (`node_modules/`, `vendor/`, …) at any depth.
|
||||
const segments = path.split('/');
|
||||
if (segments.some(p => !pruneDir(p))) return 'pruned-dir';
|
||||
|
||||
|
||||
+21
-3
@@ -454,13 +454,31 @@ export async function runThink(
|
||||
// Closes #952 (think over MCP returns "no LLM available").
|
||||
const client = opts.client ?? await tryBuildGatewayClient(modelUsed, { explicitModel: opts.modelExplicit });
|
||||
if (!client) {
|
||||
warnings.push('NO_ANTHROPIC_API_KEY');
|
||||
// Label the failure honestly: a missing key and an unusable model id are
|
||||
// different incidents with different fixes. Pre-fix EVERY null client was
|
||||
// stamped NO_ANTHROPIC_API_KEY, which sent operators chasing env/keychain
|
||||
// problems when the real cause was a model id the recipe didn't know
|
||||
// (e.g. a tier-configured model newer than the recipe list). The re-probe
|
||||
// is pure and cheap (no IO): same predicate tryBuildGatewayClient used.
|
||||
const probe = probeChatModel(normalizeModelId(modelUsed));
|
||||
const modelProblem = !probe.ok && probe.reason !== 'unavailable';
|
||||
warnings.push(
|
||||
modelProblem ? `MODEL_NOT_USABLE:${(probe as { reason: string }).reason}` : 'NO_ANTHROPIC_API_KEY',
|
||||
);
|
||||
const detail = !probe.ok ? probe.detail : '';
|
||||
const fix = !probe.ok && probe.fix ? ` Fix: ${probe.fix}` : '';
|
||||
// Degrade gracefully: return the gather without synthesis. Better than throwing.
|
||||
return {
|
||||
question: opts.question,
|
||||
answer: '(no LLM available — set ANTHROPIC_API_KEY or pass `client`)',
|
||||
answer: modelProblem
|
||||
? `(model "${modelUsed}" not usable — ${detail}${fix})`
|
||||
: '(no LLM available — set ANTHROPIC_API_KEY or pass `client`)',
|
||||
citations: [],
|
||||
gaps: ['no LLM available; gather succeeded but synthesis skipped'],
|
||||
gaps: [
|
||||
modelProblem
|
||||
? `model "${modelUsed}" not usable (${(probe as { reason: string }).reason}); gather succeeded but synthesis skipped`
|
||||
: 'no LLM available; gather succeeded but synthesis skipped',
|
||||
],
|
||||
pagesGathered: gather.pages.length,
|
||||
takesGathered: gather.takes.length,
|
||||
graphHits: gather.graphSlugs.length,
|
||||
|
||||
@@ -581,6 +581,13 @@ export interface Chunk {
|
||||
parent_symbol_path?: string[] | null;
|
||||
doc_comment?: string | null;
|
||||
symbol_name_qualified?: string | null;
|
||||
/**
|
||||
* v0.27.1 multimodal. Read side of ChunkInput.modality — must round-trip
|
||||
* through getChunks → embed-stale merge → upsertChunks or image rows get
|
||||
* reset to 'text' (EXCLUDED.modality on the upsert) and vanish from the
|
||||
* image search arm.
|
||||
*/
|
||||
modality?: 'text' | 'image';
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -974,6 +981,19 @@ export interface SearchOpts {
|
||||
* client) → `sourceIds`; otherwise `ctx.sourceId` (scalar) → `sourceId`.
|
||||
*/
|
||||
sourceIds?: string[];
|
||||
/**
|
||||
* fix/title-retrieval-arm (D2, Reviewer F1): opt-in AND→OR keyword-recall
|
||||
* fallback. When true, `searchKeyword` retries ONCE with OR-of-terms after
|
||||
* the strict websearch AND query returns zero rows (strict results always
|
||||
* win when non-empty). Default false/undefined = strict-AND only — the
|
||||
* pre-fix contract. hybridSearch opts in for its keyword arm; precision
|
||||
* consumers (enrichment countMentions, link-extraction resolution, eval
|
||||
* paths) MUST NOT set this: OR-matches would inflate mention counts and
|
||||
* relax link-candidate resolution ("John Smith" matching every John and
|
||||
* every Smith). `searchTitles` has its own page-grain fallback and
|
||||
* ignores this flag.
|
||||
*/
|
||||
orFallback?: boolean;
|
||||
/**
|
||||
* v0.27.1 / v0.36 (D11): target column for vector search. Two shapes:
|
||||
*
|
||||
|
||||
@@ -329,6 +329,7 @@ export function rowToChunk(row: Record<string, unknown>, includeEmbedding = fals
|
||||
parent_symbol_path: (row.parent_symbol_path as string[] | null | undefined) ?? null,
|
||||
doc_comment: (row.doc_comment as string | null | undefined) ?? null,
|
||||
symbol_name_qualified: (row.symbol_name_qualified as string | null | undefined) ?? null,
|
||||
modality: (row.modality as 'text' | 'image' | undefined) ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -27,6 +27,7 @@ import { randomBytes } from 'crypto';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts';
|
||||
import { isWriteTargetContained } from './path-confine.ts';
|
||||
import { isDurabilityHardened, commitWriteThroughFile } from './brain-repo-durability.ts';
|
||||
|
||||
/** Minimal logger surface — structurally compatible with operations.ts `Logger`. */
|
||||
export interface WriteThroughLogger {
|
||||
@@ -36,6 +37,13 @@ export interface WriteThroughLogger {
|
||||
export interface WriteThroughResult {
|
||||
written: boolean;
|
||||
path?: string;
|
||||
/**
|
||||
* True when the write was also committed to git (#2426). Only attempted on
|
||||
* repos hardened via `gbrain sources harden` (durability hook installed);
|
||||
* the hook then background-pushes the commit. Best-effort — a false/absent
|
||||
* value never blocks the write.
|
||||
*/
|
||||
committed?: boolean;
|
||||
/**
|
||||
* Non-error reasons the file was not written:
|
||||
* - no_repo_configured: the resolved target (source `local_path` or, for a
|
||||
@@ -157,7 +165,20 @@ export async function writePageThrough(
|
||||
throw writeErr;
|
||||
}
|
||||
|
||||
return { written: true, path: filePath };
|
||||
// #2426: on a durability-hardened repo (user ran `gbrain sources harden`),
|
||||
// commit the artifact so it reaches git — pre-fix, write-through content
|
||||
// stayed uncommitted forever: never pushed, `last_sync_at` frozen, and
|
||||
// silently deleted by a later `sync --full` delete-reconcile. The local
|
||||
// post-commit hook background-pushes the commit. Best-effort: a commit
|
||||
// failure never fails the write (the DB row + file are the durable sinks).
|
||||
let committed = false;
|
||||
try {
|
||||
if (isDurabilityHardened(writeRoot)) {
|
||||
committed = commitWriteThroughFile(writeRoot, filePath, slug);
|
||||
}
|
||||
} catch { /* best-effort */ }
|
||||
|
||||
return { written: true, path: filePath, ...(committed ? { committed } : {}) };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
opts.logger?.warn(`[write-through] failed for ${slug}: ${msg}`);
|
||||
|
||||
+6
-1
@@ -826,6 +826,12 @@ ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
|
||||
|
||||
-- Function to rebuild search_vector for a page
|
||||
-- #2704: compiled_truth (unbounded whole-page body) deliberately NOT
|
||||
-- indexed — overflows Postgres's 1MB tsvector cap on large pages.
|
||||
-- content_chunks.search_vector (chunk-grain, populated separately) is
|
||||
-- what searchKeyword() actually queries. See migrate.ts's v124 migration
|
||||
-- for the full rationale; keep in sync with that + reindex-search-vector.ts
|
||||
-- + pglite-schema.ts.
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
@@ -839,7 +845,6 @@ BEGIN
|
||||
-- Build weighted tsvector
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { queryAdminSources } from '../src/commands/serve-http.ts';
|
||||
import { buildSyncStatusReport } from '../src/commands/sync.ts';
|
||||
|
||||
/**
|
||||
* v0.41.29 Sources tab — `/admin/api/sources` endpoint SQL.
|
||||
*
|
||||
* The endpoint is a thin Express handler over `queryAdminSources` +
|
||||
* `buildSyncStatusReport`; the source-selection SQL is the load-bearing
|
||||
* surface (same pattern as test/admin-agents-spend.test.ts).
|
||||
*
|
||||
* Pinned behaviors:
|
||||
* - Excludes archived sources
|
||||
* - INCLUDES sources with null local_path (push-only brains: filtering
|
||||
* on local_path emptied the Sources tab + federation source-picker)
|
||||
* - JSONB config surfaces as an object, defaulting to {}
|
||||
* - Deterministic ORDER BY id
|
||||
* - buildSyncStatusReport accepts the rows (no disk I/O on null paths)
|
||||
*/
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
describe('queryAdminSources (/admin/api/sources SQL)', () => {
|
||||
it('includes push-only sources with null local_path', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config)
|
||||
VALUES ('push-only', 'push-only', NULL, '{}'::jsonb)`,
|
||||
);
|
||||
const sources = await queryAdminSources(engine);
|
||||
const ids = sources.map((s) => s.id);
|
||||
expect(ids).toContain('push-only');
|
||||
expect(sources.find((s) => s.id === 'push-only')!.local_path).toBe(null);
|
||||
});
|
||||
|
||||
it('excludes archived sources', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, archived) VALUES ('gone', 'gone', true)`,
|
||||
);
|
||||
const sources = await queryAdminSources(engine);
|
||||
expect(sources.map((s) => s.id)).not.toContain('gone');
|
||||
});
|
||||
|
||||
it('surfaces JSONB config as an object and orders by id', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config)
|
||||
VALUES ('bbb', 'bbb', '{"syncEnabled": true}'::jsonb),
|
||||
('aaa', 'aaa', '{}'::jsonb)`,
|
||||
);
|
||||
const sources = await queryAdminSources(engine);
|
||||
const ids = sources.map((s) => s.id);
|
||||
expect(ids.indexOf('aaa')).toBeLessThan(ids.indexOf('bbb'));
|
||||
expect(sources.find((s) => s.id === 'bbb')!.config).toEqual({ syncEnabled: true });
|
||||
expect(sources.find((s) => s.id === 'aaa')!.config).toEqual({});
|
||||
});
|
||||
|
||||
it('buildSyncStatusReport accepts the rows (null local_path does not throw)', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config)
|
||||
VALUES ('push-only', 'push-only', NULL, '{}'::jsonb)`,
|
||||
);
|
||||
const report = await buildSyncStatusReport(engine, await queryAdminSources(engine));
|
||||
expect(report.schema_version).toBe(1);
|
||||
const row = report.sources.find((s) => s.source_id === 'push-only');
|
||||
expect(row).toBeDefined();
|
||||
});
|
||||
});
|
||||
@@ -28,7 +28,7 @@
|
||||
* (excluding the OpenAI canonical fast-path recipe).
|
||||
*/
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, mock, test } from 'bun:test';
|
||||
import { afterAll, afterEach, beforeEach, describe, expect, mock, test } from 'bun:test';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
@@ -40,6 +40,15 @@ import {
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
import { AIConfigError, AITransientError } from '../../src/core/ai/errors.ts';
|
||||
|
||||
// The last test in this file leaves the gateway configured with a remote
|
||||
// provider + fake key and a REAL embed transport. Without a final reset,
|
||||
// that config leaks into whichever test file the shard runs next — the
|
||||
// first downstream embed then makes a live HTTP call (broke master shard 6
|
||||
// when #3022's new test file reshuffled shard composition). The bunfig
|
||||
// legacy-embedding preload only re-applies its default when the gateway is
|
||||
// UNCONFIGURED, so a configured-but-stale slot survives file boundaries.
|
||||
afterAll(() => resetGateway());
|
||||
|
||||
// --------- Test helpers ---------
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
/**
|
||||
* gbrain#2490 — gateway.chat() never caches a stable system prompt across
|
||||
* varying single-turn calls (page-summary, skillopt, enrich).
|
||||
*
|
||||
* Root cause: `chat()` passed `system` as a bare string and relied solely on
|
||||
* a CALL-LEVEL `providerOptions.anthropic.cacheControl`. On `ai@6` +
|
||||
* `@ai-sdk/anthropic@3.x`, that call-level marker is real — it's serialized
|
||||
* as a top-level `cache_control` field on the Anthropic request body, which
|
||||
* the Messages API resolves via its documented "auto-cache the LAST
|
||||
* cacheable block in the request" shorthand (see Anthropic's prompt-caching
|
||||
* docs). For a single-turn call with a stable system prompt and a DIFFERENT
|
||||
* user message every time, "the last cacheable block" is that ever-varying
|
||||
* user message — every call WRITES a fresh cache entry there and never
|
||||
* READS a prior one, so `cache_read_input_tokens` stays 0 forever even
|
||||
* though a `cache_control` breakpoint genuinely reaches Anthropic.
|
||||
*
|
||||
* Fix: ALSO pass `system` as a `SystemModelMessage` object (`{ role:
|
||||
* 'system', content, providerOptions }`) when caching is requested — the
|
||||
* shape `ai` documents specifically for attaching provider options to the
|
||||
* system block — and mark the last tool def's own `providerOptions` too
|
||||
* (mirrors the already-correct raw-SDK path in `subagent.ts`). The
|
||||
* call-level marker is KEPT (not removed): it's what gives `toolLoop()`'s
|
||||
* growing multi-turn conversation a rolling cache breakpoint on each turn's
|
||||
* tail, which the explicit system/tool markers alone don't provide.
|
||||
*
|
||||
* These tests pin the FIX by inspecting the exact args handed to the
|
||||
* `generateText` transport (via `__setGenerateTextTransportForTests`),
|
||||
* not by asserting on `providerOptions` alone — that field is exactly what
|
||||
* the bug made you believe was sufficient.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import {
|
||||
chat,
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setGenerateTextTransportForTests,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
|
||||
describe('gbrain#2490 — Anthropic cache breakpoint placement', () => {
|
||||
beforeEach(() => {
|
||||
resetGateway();
|
||||
__setGenerateTextTransportForTests(null);
|
||||
});
|
||||
|
||||
async function captureTransportArgs(
|
||||
opts: Partial<Parameters<typeof chat>[0]> = {},
|
||||
): Promise<any> {
|
||||
let captured: any;
|
||||
__setGenerateTextTransportForTests(async (args: any) => {
|
||||
captured = args;
|
||||
return {
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
} as any;
|
||||
});
|
||||
configureGateway({
|
||||
chat_model: 'anthropic:claude-sonnet-4-6',
|
||||
env: { ANTHROPIC_API_KEY: 'fake' },
|
||||
});
|
||||
await chat({
|
||||
model: 'anthropic:claude-sonnet-4-6',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
...opts,
|
||||
});
|
||||
return captured;
|
||||
}
|
||||
|
||||
test('cacheSystem:true puts a real breakpoint on the system block (SystemModelMessage, not a bare string)', async () => {
|
||||
const args = await captureTransportArgs({ system: 'You are a helpful assistant.', cacheSystem: true });
|
||||
|
||||
// The regression: `system` used to stay a bare string forever, which
|
||||
// carries no per-block `providerOptions` — no breakpoint could ever land.
|
||||
expect(typeof args.system).not.toBe('string');
|
||||
expect(args.system).toEqual({
|
||||
role: 'system',
|
||||
content: 'You are a helpful assistant.',
|
||||
providerOptions: { anthropic: { cacheControl: { type: 'ephemeral' } } },
|
||||
});
|
||||
});
|
||||
|
||||
test('cacheSystem:true ALSO keeps the call-level cache_control on top-level providerOptions (rolling-conversation cache for toolLoop)', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS', cacheSystem: true });
|
||||
|
||||
// Not removed: @ai-sdk/anthropic serializes this as the Anthropic API's
|
||||
// documented top-level "auto-cache the last cacheable block" shorthand,
|
||||
// which is what gives a growing multi-turn toolLoop() conversation a
|
||||
// rolling cache breakpoint on each turn's tail. The explicit
|
||||
// system-block marker (asserted above) is what actually fixes gbrain#2490
|
||||
// for single-turn callers — the two coexist, marking different blocks.
|
||||
expect(args.providerOptions?.anthropic?.cacheControl).toEqual({ type: 'ephemeral' });
|
||||
});
|
||||
|
||||
test('cacheSystem:true marks the LAST tool def with its own providerOptions.anthropic.cacheControl', async () => {
|
||||
const args = await captureTransportArgs({
|
||||
system: 'SYS',
|
||||
cacheSystem: true,
|
||||
tools: [
|
||||
{ name: 'search', description: 'search', inputSchema: { type: 'object', properties: {} } },
|
||||
{ name: 'put_page', description: 'put_page', inputSchema: { type: 'object', properties: {} } },
|
||||
],
|
||||
});
|
||||
|
||||
expect(args.tools.search.providerOptions).toBeUndefined();
|
||||
expect(args.tools.put_page.providerOptions).toEqual({
|
||||
anthropic: { cacheControl: { type: 'ephemeral' } },
|
||||
});
|
||||
});
|
||||
|
||||
test('cacheSystem:false (default) leaves system a byte-identical bare string — no behavior change', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS', cacheSystem: false });
|
||||
expect(args.system).toBe('SYS');
|
||||
expect(args.providerOptions).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem omitted entirely leaves system a byte-identical bare string — no behavior change', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS' });
|
||||
expect(args.system).toBe('SYS');
|
||||
expect(args.providerOptions).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem:true with no system prompt does not synthesize an empty cached system block', async () => {
|
||||
const args = await captureTransportArgs({ cacheSystem: true });
|
||||
expect(args.system).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem:true with no tools does not throw and leaves tools undefined', async () => {
|
||||
const args = await captureTransportArgs({ system: 'SYS', cacheSystem: true });
|
||||
expect(args.tools).toBeUndefined();
|
||||
});
|
||||
|
||||
test('cacheSystem:true on a non-Anthropic model is silently ignored (supports_prompt_cache=false)', async () => {
|
||||
let captured: any;
|
||||
__setGenerateTextTransportForTests(async (args: any) => {
|
||||
captured = args;
|
||||
return {
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
} as any;
|
||||
});
|
||||
configureGateway({
|
||||
chat_model: 'openai:gpt-4o-mini',
|
||||
env: { OPENAI_API_KEY: 'fake' },
|
||||
});
|
||||
await chat({
|
||||
model: 'openai:gpt-4o-mini',
|
||||
system: 'SYS',
|
||||
cacheSystem: true,
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
// Still a bare string — the recipe doesn't support prompt caching, so
|
||||
// useCache is false regardless of the caller's request.
|
||||
expect(captured.system).toBe('SYS');
|
||||
});
|
||||
|
||||
test('a configured cacheControl TTL override applies to every breakpoint, not just the call-level one', async () => {
|
||||
// Codex review finding: with three independently-hardcoded `{type:
|
||||
// 'ephemeral'}` markers, a `provider_chat_options.anthropic.cacheControl`
|
||||
// TTL override (e.g. `ttl: '1h'`) would only reach the call-level marker
|
||||
// via applyConfiguredChatProviderOptions()'s deep-merge — the system and
|
||||
// tool markers would stay implicit 5m, mixing TTLs across breakpoints in
|
||||
// the same request. Assert all three markers derive from ONE canonical
|
||||
// value instead.
|
||||
let captured: any;
|
||||
__setGenerateTextTransportForTests(async (args: any) => {
|
||||
captured = args;
|
||||
return {
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
} as any;
|
||||
});
|
||||
configureGateway({
|
||||
chat_model: 'anthropic:claude-sonnet-4-6',
|
||||
provider_chat_options: {
|
||||
anthropic: { cacheControl: { type: 'ephemeral', ttl: '1h' } },
|
||||
},
|
||||
env: { ANTHROPIC_API_KEY: 'fake' },
|
||||
});
|
||||
await chat({
|
||||
model: 'anthropic:claude-sonnet-4-6',
|
||||
system: 'SYS',
|
||||
cacheSystem: true,
|
||||
tools: [{ name: 'search', description: 'search', inputSchema: { type: 'object', properties: {} } }],
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
});
|
||||
|
||||
const expected = { type: 'ephemeral', ttl: '1h' };
|
||||
expect(captured.providerOptions?.anthropic?.cacheControl).toEqual(expected);
|
||||
expect((captured.system as any)?.providerOptions?.anthropic?.cacheControl).toEqual(expected);
|
||||
expect(captured.tools?.search?.providerOptions?.anthropic?.cacheControl).toEqual(expected);
|
||||
});
|
||||
});
|
||||
@@ -297,6 +297,14 @@ describe('chat touchpoint — provider_chat_options passthrough', () => {
|
||||
});
|
||||
|
||||
test('anthropic cacheControl survives provider_chat_options merging', async () => {
|
||||
// gbrain#2490: this call-level cacheControl is real (not a no-op) —
|
||||
// @ai-sdk/anthropic serializes it as the Anthropic API's documented
|
||||
// top-level "auto-cache the last cacheable block" shorthand. It's kept
|
||||
// alongside the fix (an explicit breakpoint on the system message's own
|
||||
// providerOptions — see test/ai/gateway-cache-breakpoint.test.ts) because
|
||||
// it's what gives toolLoop()'s growing multi-turn conversation a rolling
|
||||
// cache breakpoint on each turn's tail. See gateway.ts's `useCache` block
|
||||
// for the full explanation of why both markers are needed.
|
||||
const providerOptions = await captureProviderOptions({
|
||||
chat_model: 'anthropic:claude-sonnet-4-6',
|
||||
provider_chat_options: {
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* OpenAI `prompt_cache_key` routing hint (takeover of PR #2442's remaining
|
||||
* half, originally by @CoachRyanNguyen).
|
||||
*
|
||||
* OpenAI caches prompt prefixes automatically; a stable `prompt_cache_key`
|
||||
* keeps requests that share a prefix on the same inference engine, lifting the
|
||||
* automatic-cache hit rate. `chat()` derives one from the system prompt + tool
|
||||
* names for native-OpenAI models and passes it via
|
||||
* `providerOptions.openai.promptCacheKey` (which @ai-sdk/openai maps to the
|
||||
* request's `prompt_cache_key`).
|
||||
*
|
||||
* Pins:
|
||||
* - key derivation is stable (tool ORDER doesn't matter), sensitive to
|
||||
* system/tool-set changes, and absent without a system prompt
|
||||
* - the chat() wiring only fires for native-openai (anthropic/compat get
|
||||
* nothing), and config `provider_chat_options` overrides the derived key
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach } from 'bun:test';
|
||||
import {
|
||||
chat,
|
||||
configureGateway,
|
||||
openAIPromptCacheKey,
|
||||
resetGateway,
|
||||
__setGenerateTextTransportForTests,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
|
||||
describe('openAIPromptCacheKey — derivation', () => {
|
||||
test('same system + same tools → identical stable key (sticky routing)', () => {
|
||||
const a = openAIPromptCacheKey({ system: 'SYS', toolNames: ['search', 'put_page'] });
|
||||
const b = openAIPromptCacheKey({ system: 'SYS', toolNames: ['put_page', 'search'] });
|
||||
expect(a).toBe(b as string); // tool ORDER must not change the key
|
||||
expect(a).toMatch(/^gbrain:[0-9a-f]{32}$/);
|
||||
});
|
||||
|
||||
test('different system → different key', () => {
|
||||
const a = openAIPromptCacheKey({ system: 'SYS A', toolNames: [] });
|
||||
const b = openAIPromptCacheKey({ system: 'SYS B', toolNames: [] });
|
||||
expect(a).not.toBe(b as string);
|
||||
});
|
||||
|
||||
test('different tool set → different key', () => {
|
||||
const a = openAIPromptCacheKey({ system: 'SYS', toolNames: ['search'] });
|
||||
const b = openAIPromptCacheKey({ system: 'SYS', toolNames: ['search', 'put_page'] });
|
||||
expect(a).not.toBe(b as string);
|
||||
});
|
||||
|
||||
test('no system prompt → undefined (do not pin one-off requests)', () => {
|
||||
expect(openAIPromptCacheKey({ system: undefined, toolNames: ['search'] })).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('chat() wiring — prompt_cache_key per provider', () => {
|
||||
beforeEach(() => {
|
||||
resetGateway();
|
||||
__setGenerateTextTransportForTests(null);
|
||||
});
|
||||
|
||||
async function captureProviderOptions(
|
||||
config: Parameters<typeof configureGateway>[0],
|
||||
opts: Partial<Parameters<typeof chat>[0]> = {},
|
||||
): Promise<Record<string, any> | undefined> {
|
||||
let captured: Record<string, any> | undefined;
|
||||
__setGenerateTextTransportForTests(async (args: any) => {
|
||||
captured = args.providerOptions;
|
||||
return {
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
} as any;
|
||||
});
|
||||
configureGateway(config);
|
||||
await chat({
|
||||
model: config.chat_model ?? 'anthropic:claude-sonnet-4-6',
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
...opts,
|
||||
});
|
||||
return captured;
|
||||
}
|
||||
|
||||
test('native-openai with a system prompt → providerOptions.openai.promptCacheKey', async () => {
|
||||
const providerOptions = await captureProviderOptions(
|
||||
{ chat_model: 'openai:gpt-4o-mini', env: { OPENAI_API_KEY: 'fake' } },
|
||||
{ system: 'SYS' },
|
||||
);
|
||||
expect(providerOptions?.openai?.promptCacheKey).toMatch(/^gbrain:[0-9a-f]{32}$/);
|
||||
});
|
||||
|
||||
test('native-openai without a system prompt → no providerOptions at all', async () => {
|
||||
const providerOptions = await captureProviderOptions(
|
||||
{ chat_model: 'openai:gpt-4o-mini', env: { OPENAI_API_KEY: 'fake' } },
|
||||
);
|
||||
expect(providerOptions).toBeUndefined();
|
||||
});
|
||||
|
||||
test('native-anthropic never gets an openai promptCacheKey', async () => {
|
||||
const providerOptions = await captureProviderOptions(
|
||||
{ chat_model: 'anthropic:claude-sonnet-4-6', env: { ANTHROPIC_API_KEY: 'fake' } },
|
||||
{ system: 'SYS' },
|
||||
);
|
||||
expect(providerOptions?.openai).toBeUndefined();
|
||||
});
|
||||
|
||||
test('openai-compatible (deepseek) never gets promptCacheKey (provider ignores providerOptions.openai)', async () => {
|
||||
const providerOptions = await captureProviderOptions(
|
||||
{ chat_model: 'deepseek:deepseek-chat', env: { DEEPSEEK_API_KEY: 'fake' } },
|
||||
{ system: 'SYS' },
|
||||
);
|
||||
expect(providerOptions?.openai).toBeUndefined();
|
||||
});
|
||||
|
||||
test('config provider_chat_options overrides the derived key', async () => {
|
||||
const providerOptions = await captureProviderOptions(
|
||||
{
|
||||
chat_model: 'openai:gpt-4o-mini',
|
||||
env: { OPENAI_API_KEY: 'fake' },
|
||||
provider_chat_options: { openai: { promptCacheKey: 'session-42' } },
|
||||
},
|
||||
{ system: 'SYS' },
|
||||
);
|
||||
expect(providerOptions?.openai?.promptCacheKey).toBe('session-42');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* Mistral recipe smoke.
|
||||
*
|
||||
* The load-bearing assertion here is the negative one: mistral-embed rejects
|
||||
* every dimension parameter with HTTP 400, so dimsProviderOptions() must emit
|
||||
* no dimension field for it. Same contract as voyage-4-nano, pinned the same
|
||||
* way (see the negative regression assertion in test/ai/gateway.test.ts).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { getRecipe } from '../../src/core/ai/recipes/index.ts';
|
||||
import { defaultResolveAuth } from '../../src/core/ai/gateway.ts';
|
||||
import { assertTouchpoint } from '../../src/core/ai/model-resolver.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
import { dimsProviderOptions } from '../../src/core/ai/dims.ts';
|
||||
import { lookupEmbeddingPrice } from '../../src/core/embedding-pricing.ts';
|
||||
|
||||
describe('recipe: mistral', () => {
|
||||
test('registered with expected OpenAI-compatible shape', () => {
|
||||
const r = getRecipe('mistral');
|
||||
expect(r).toBeDefined();
|
||||
expect(r!.id).toBe('mistral');
|
||||
expect(r!.tier).toBe('openai-compat');
|
||||
expect(r!.implementation).toBe('openai-compatible');
|
||||
expect(r!.base_url_default).toBe('https://api.mistral.ai/v1');
|
||||
expect(r!.auth_env?.required).toEqual(['MISTRAL_API_KEY']);
|
||||
});
|
||||
|
||||
test('embedding touchpoint pins the measured 1024 dims and 64K batch ceiling', () => {
|
||||
const e = getRecipe('mistral')!.touchpoints.embedding;
|
||||
expect(e).toBeDefined();
|
||||
expect(e!.models).toContain('mistral-embed');
|
||||
expect(e!.default_dims).toBe(1024);
|
||||
// Measured: a 65,286-token batch is accepted, 66,960 returns 400 code 3210.
|
||||
expect(e!.max_batch_tokens).toBe(65_536);
|
||||
// chars_per_token is a DIVISOR in splitByTokenBudget(), so a lower value
|
||||
// is the conservative direction. The module default of 4 is an English
|
||||
// assumption and overshoots on denser prose.
|
||||
expect(e!.chars_per_token).toBe(2);
|
||||
});
|
||||
|
||||
test('NEGATIVE: no dimension parameter is emitted for mistral-embed', () => {
|
||||
// Mistral rejects both spellings:
|
||||
// {"dimensions": N} -> 400 extra_forbidden
|
||||
// {"output_dimension": N} -> 400 "does not support output_dimension"
|
||||
// If a future change adds mistral-embed to a flexible-dim allowlist in
|
||||
// dims.ts, this assertion fails before it reaches users as a 400 on every
|
||||
// embed call.
|
||||
expect(dimsProviderOptions('openai-compatible', 'mistral-embed', 1024)).toBeUndefined();
|
||||
expect(dimsProviderOptions('openai-compatible', 'mistral-embed-2312', 1024)).toBeUndefined();
|
||||
});
|
||||
|
||||
test('embedding models resolve to a known price', () => {
|
||||
// An unknown price makes the embedding spend cap fail closed.
|
||||
expect(lookupEmbeddingPrice('mistral:mistral-embed').kind).toBe('known');
|
||||
expect(lookupEmbeddingPrice('mistral:mistral-embed-2312').kind).toBe('known');
|
||||
});
|
||||
|
||||
test('chat and expansion touchpoints accept their configured models', () => {
|
||||
const r = getRecipe('mistral')!;
|
||||
expect(r.touchpoints.chat!.supports_tools).toBe(true);
|
||||
expect(r.touchpoints.chat!.supports_subagent_loop).toBe(false);
|
||||
expect(() => assertTouchpoint(r, 'chat', 'mistral-small-latest')).not.toThrow();
|
||||
expect(() => assertTouchpoint(r, 'expansion', 'ministral-3b-latest')).not.toThrow();
|
||||
expect(() => assertTouchpoint(r, 'embedding', 'mistral-embed')).not.toThrow();
|
||||
});
|
||||
|
||||
test('codestral-embed is deliberately absent (1536 dims would mix under a 1024 declaration)', () => {
|
||||
const e = getRecipe('mistral')!.touchpoints.embedding!;
|
||||
expect(e.models).not.toContain('codestral-embed');
|
||||
expect(e.models).not.toContain('codestral-embed-2505');
|
||||
});
|
||||
|
||||
test('default auth: MISTRAL_API_KEY set -> Bearer token', () => {
|
||||
const r = getRecipe('mistral')!;
|
||||
const auth = defaultResolveAuth(r, { MISTRAL_API_KEY: 'fake-mistral-key' }, 'embedding');
|
||||
expect(auth.headerName).toBe('Authorization');
|
||||
expect(auth.token).toBe('Bearer fake-mistral-key');
|
||||
});
|
||||
|
||||
test('default auth: missing MISTRAL_API_KEY -> AIConfigError', () => {
|
||||
const r = getRecipe('mistral')!;
|
||||
expect(() => defaultResolveAuth(r, {}, 'embedding')).toThrow(AIConfigError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* Moonshot/Kimi local recipe smoke.
|
||||
*
|
||||
* This pins the governed production exception GBrain-Local-003: GBrain can
|
||||
* route configured Kimi chat/expansion IDs through Moonshot's OpenAI-compatible
|
||||
* endpoint without treating `moonshot` as an unknown provider.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { getRecipe } from '../../src/core/ai/recipes/index.ts';
|
||||
import { defaultResolveAuth } from '../../src/core/ai/gateway.ts';
|
||||
import { assertTouchpoint } from '../../src/core/ai/model-resolver.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
|
||||
describe('recipe: moonshot', () => {
|
||||
test('registered with expected OpenAI-compatible shape', () => {
|
||||
const r = getRecipe('moonshot');
|
||||
expect(r).toBeDefined();
|
||||
expect(r!.id).toBe('moonshot');
|
||||
expect(r!.tier).toBe('openai-compat');
|
||||
expect(r!.implementation).toBe('openai-compatible');
|
||||
expect(r!.base_url_default).toBe('https://api.moonshot.ai/v1');
|
||||
expect(r!.auth_env?.required).toEqual(['MOONSHOT_API_KEY']);
|
||||
});
|
||||
|
||||
test('chat and expansion touchpoints include Kimi K2.7 Code', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(r.touchpoints.chat).toBeDefined();
|
||||
expect(r.touchpoints.expansion).toBeDefined();
|
||||
expect(r.touchpoints.chat!.models).toContain('kimi-k2.7-code');
|
||||
expect(r.touchpoints.expansion!.models).toContain('kimi-k2.7-code');
|
||||
expect(r.touchpoints.chat!.supports_tools).toBe(true);
|
||||
expect(r.touchpoints.chat!.supports_subagent_loop).toBe(false);
|
||||
});
|
||||
|
||||
test('configured Kimi model is accepted for chat and expansion', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(() => assertTouchpoint(r, 'chat', 'kimi-k2.7-code')).not.toThrow();
|
||||
expect(() => assertTouchpoint(r, 'expansion', 'kimi-k2.7-code')).not.toThrow();
|
||||
});
|
||||
|
||||
test('default auth: MOONSHOT_API_KEY set -> Bearer token', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
const auth = defaultResolveAuth(r, { MOONSHOT_API_KEY: 'fake-moonshot-key' }, 'chat');
|
||||
expect(auth.headerName).toBe('Authorization');
|
||||
expect(auth.token).toBe('Bearer fake-moonshot-key');
|
||||
});
|
||||
|
||||
test('default auth: missing MOONSHOT_API_KEY -> AIConfigError', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(() => defaultResolveAuth(r, {}, 'chat')).toThrow(AIConfigError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,89 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
dimsProviderOptions,
|
||||
nvidiaEmbeddingDimOptions,
|
||||
supportsNvidiaEmbeddingDimension,
|
||||
} from '../../src/core/ai/dims.ts';
|
||||
import { getRecipe, RECIPES } from '../../src/core/ai/recipes/index.ts';
|
||||
import { nvidia } from '../../src/core/ai/recipes/nvidia.ts';
|
||||
import { defaultResolveAuth } from '../../src/core/ai/gateway.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
|
||||
describe('recipe: nvidia', () => {
|
||||
test('registered with OpenAI-compatible NIM endpoint', () => {
|
||||
expect(RECIPES.has('nvidia')).toBe(true);
|
||||
expect(getRecipe('nvidia')).toBe(nvidia);
|
||||
expect(nvidia.id).toBe('nvidia');
|
||||
expect(nvidia.tier).toBe('openai-compat');
|
||||
expect(nvidia.implementation).toBe('openai-compatible');
|
||||
expect(nvidia.base_url_default).toBe('https://integrate.api.nvidia.com/v1');
|
||||
});
|
||||
|
||||
test('auth flows through defaultResolveAuth — NVIDIA_API_KEY as bearer token', () => {
|
||||
// IRON RULE: only Azure overrides resolveAuth. NVIDIA is plain
|
||||
// Authorization Bearer, so the recipe must NOT declare its own resolver;
|
||||
// defaultResolveAuth derives the header from auth_env.required.
|
||||
expect(nvidia.resolveAuth).toBeUndefined();
|
||||
expect(nvidia.auth_env?.required).toEqual(['NVIDIA_API_KEY']);
|
||||
expect(defaultResolveAuth(nvidia, { NVIDIA_API_KEY: 'fake-nvidia' }, 'embedding')).toEqual({
|
||||
headerName: 'Authorization',
|
||||
token: 'Bearer fake-nvidia',
|
||||
});
|
||||
expect(() => defaultResolveAuth(nvidia, {}, 'chat')).toThrow(AIConfigError);
|
||||
});
|
||||
|
||||
test('chat touchpoint declares Nemotron 3 Super without subagent-loop claims', () => {
|
||||
const chat = nvidia.touchpoints.chat!;
|
||||
expect(chat.models).toContain('nvidia/nemotron-3-super-120b-a12b');
|
||||
expect(chat.supports_tools).toBe(false);
|
||||
expect(chat.supports_subagent_loop).toBe(false);
|
||||
expect(chat.max_context_tokens).toBe(128000);
|
||||
});
|
||||
|
||||
test('embedding touchpoint declares tested NVIDIA models and natural dimensions', () => {
|
||||
const e = nvidia.touchpoints.embedding!;
|
||||
expect(e.models).toContain('nvidia/nv-embedqa-e5-v5');
|
||||
expect(e.models).toContain('nvidia/llama-nemotron-embed-1b-v2');
|
||||
expect(e.models).toContain('nvidia/nv-embed-v1');
|
||||
expect(e.models).toContain('nvidia/nv-embedcode-7b-v1');
|
||||
expect(e.default_dims).toBe(1024);
|
||||
expect(e.dims_options).toEqual([1024, 2048, 4096]);
|
||||
expect(e.max_batch_tokens).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('aliases allow short model names while preserving NVIDIA catalog ids', () => {
|
||||
expect(nvidia.aliases?.['nv-embedqa-e5-v5']).toBe('nvidia/nv-embedqa-e5-v5');
|
||||
expect(nvidia.aliases?.['llama-nemotron-embed-1b-v2']).toBe('nvidia/llama-nemotron-embed-1b-v2');
|
||||
expect(nvidia.aliases?.['nemotron-3-super']).toBe('nvidia/nemotron-3-super-120b-a12b');
|
||||
expect(nvidia.aliases?.['nemotron-3-super-120b-a12b']).toBe('nvidia/nemotron-3-super-120b-a12b');
|
||||
});
|
||||
|
||||
test('dimsProviderOptions emits passage input_type by default for NVIDIA embeddings', () => {
|
||||
expect(dimsProviderOptions('openai-compatible', 'nvidia/nv-embedqa-e5-v5', 1024)).toEqual({
|
||||
openaiCompatible: { input_type: 'passage' },
|
||||
});
|
||||
});
|
||||
|
||||
test('dimsProviderOptions maps query/document inputType for NVIDIA embeddings', () => {
|
||||
expect(dimsProviderOptions('openai-compatible', 'nvidia/nv-embedqa-e5-v5', 1024, 'query')).toEqual({
|
||||
openaiCompatible: { input_type: 'query' },
|
||||
});
|
||||
expect(dimsProviderOptions('openai-compatible', 'nvidia/nv-embedqa-e5-v5', 1024, 'document')).toEqual({
|
||||
openaiCompatible: { input_type: 'passage' },
|
||||
});
|
||||
});
|
||||
|
||||
test('llama-nemotron supports a 1280d Matryoshka dimension override', () => {
|
||||
expect(nvidiaEmbeddingDimOptions('nvidia/llama-nemotron-embed-1b-v2')).toContain(1280);
|
||||
expect(supportsNvidiaEmbeddingDimension('nvidia/llama-nemotron-embed-1b-v2', 1280)).toBe(true);
|
||||
expect(dimsProviderOptions('openai-compatible', 'nvidia/llama-nemotron-embed-1b-v2', 1280, 'query')).toEqual({
|
||||
openaiCompatible: { input_type: 'query', dimensions: 1280 },
|
||||
});
|
||||
});
|
||||
|
||||
test('fixed-dim NVIDIA models omit dimensions because they reject overrides', () => {
|
||||
const opts = dimsProviderOptions('openai-compatible', 'nvidia/nv-embedqa-e5-v5', 1024, 'query');
|
||||
expect(opts).toEqual({ openaiCompatible: { input_type: 'query' } });
|
||||
expect(JSON.stringify(opts)).not.toContain('dimensions');
|
||||
});
|
||||
});
|
||||
@@ -35,6 +35,20 @@ describe('Anthropic recipe model IDs', () => {
|
||||
expect(anthropic.aliases?.['claude-sonnet-4-6-20250929']).toBe('claude-sonnet-4-6');
|
||||
});
|
||||
|
||||
it('current-generation models are listed for chat (Fable 5 / Opus 4.8 / Sonnet 5)', () => {
|
||||
// Regression guard for the tier-config incident: a brain with
|
||||
// `models.tier.deep = anthropic:claude-opus-4-8` had think/auto_think
|
||||
// silently degrade because the recipe list stopped at Opus 4.7.
|
||||
const chatModels = anthropic.touchpoints?.chat?.models ?? [];
|
||||
expect(chatModels).toContain('claude-fable-5');
|
||||
expect(chatModels).toContain('claude-opus-4-8');
|
||||
expect(chatModels).toContain('claude-sonnet-5');
|
||||
});
|
||||
|
||||
it('Sonnet 5 is listed for expansion', () => {
|
||||
expect(anthropic.touchpoints?.expansion?.models ?? []).toContain('claude-sonnet-5');
|
||||
});
|
||||
|
||||
it('all listed models follow naming conventions', () => {
|
||||
const allModels = [
|
||||
...(anthropic.touchpoints?.chat?.models ?? []),
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user