Compare commits

...
Author SHA1 Message Date
Garry Tan e22b8fb555 Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	VERSION
#	llms-full.txt
#	package.json
#	src/commands/sync.ts
2026-04-29 22:49:21 -07:00
Garry TanandWintermute 1e73e93344 v0.22.12 feat: structured error code summary for sync --skip-failed (closes #500) (#518)
* feat: structured error code summary for sync --skip-failed (#500)

When sync encounters per-file failures, the blocked/skip-failed messages
now include a breakdown by error code (SLUG_MISMATCH, YAML_PARSE, etc.)
instead of just a raw count. This makes it immediately obvious *why*
files failed without requiring manual investigation.

Changes:
- Add classifyErrorCode() — maps error messages to ParseValidationCode
- Add summarizeFailuresByCode() — groups failures into sorted code summary
- SyncFailure now carries a 'code' field (backfilled on acknowledge)
- acknowledgeSyncFailures() returns AcknowledgeResult {count, summary}
- sync blocked + skip-failed messages show code breakdown
- doctor sync_failures check shows code breakdown for both unacked and historical
- 12 new tests for classifyErrorCode, summarizeFailuresByCode, and structured returns

Before:
  Sync blocked: 2688 file(s) failed to parse.

After:
  Sync blocked: 2688 file(s) failed to parse:
    SLUG_MISMATCH: 2685
    YAML_DUPLICATE_KEY: 3

Closes #500

* test(sync): broaden classifier regexes and pin coverage with 6 new unit tests

Eng review of PR #501 found two ship-blocking gaps in the classifier:

1. Four real production error sites in src/core/import-file.ts emit strings
   that bucketed to UNKNOWN — exactly the silent-systemic-failure pattern
   that motivated #500 in the first place. Add two regex lines:
     FILE_TOO_LARGE       — covers import-file.ts:199, 352, 401
     SYMLINK_NOT_ALLOWED  — covers import-file.ts:347

2. Three existing classifier regexes (MISSING_OPEN, MISSING_CLOSE,
   EMPTY_FRONTMATTER) only matched the literal code-name prefix. The actual
   message strings emitted by markdown.ts:159-244 (e.g. "Frontmatter must
   start with --- on the first non-empty line") wouldn't match. Broaden
   each to match production message text. NESTED_QUOTES already worked.

Add 6 unit tests pinning the contract between markdown.ts/import-file.ts
strings and the classifier regex set. If anyone reworks a validator
message, both sides have to move together — the test fails loudly otherwise.

Test count: 22 → 28 in test/sync-failures.test.ts, all green.

* test(e2e): add failure-loop E2E for sync --skip-failed (issue #500 ship-blocker)

The full code path (record → classify → block → skip → doctor render →
second cycle) had only mocked-JSONL unit coverage. For a hotfix that
changes user-visible CLI output and the doctor surface, that's thin.

One comprehensive E2E test covers the loop:
  1. First sync of clean repo — succeeds, bookmark advances
  2. Add file with bad slug — sync returns 'blocked_by_failures',
     bookmark stays put, JSONL has 1 unacked entry coded SLUG_MISMATCH
  3. --skip-failed — bookmark advances past the bad commit, entry
     transitions to acknowledged, AcknowledgeResult.summary aggregates
  4. Second broken file (different path, same code) — sync blocks again,
     1 acked + 1 unacked, dedup honors path identity
  5. --skip-failed again — both acked, summary correctly counts 2

Hermetic on a developer machine: saves ~/.gbrain/sync-failures.jsonl
before the test, restores it after. Doctor rendering verified by calling
the same primitives doctor.ts uses (loadSyncFailures + summarizeFailuresByCode)
rather than runDoctor() — runDoctor is a CLI entrypoint with stdout/exit
side effects that truncate the test mid-flow.

E2E count: 13 → 14 in test/e2e/sync.test.ts. All 14 pass under real
Postgres + pgvector (gbrain-test-pg/pgvector:pg16).

* v0.22.12: structured error code summary for sync --skip-failed

Closes issue #500. PR #501 by @wintermute is the foundation (cherry-picked
as c356ea4 — classifier, doctor breakdown, AcknowledgeResult shape, 12 unit
tests). This release adds:

- Classifier coverage for FILE_TOO_LARGE + SYMLINK_NOT_ALLOWED (the four
  size/symlink rejection sites in import-file.ts that bucketed to UNKNOWN).
- Three regex breadths (MISSING_OPEN, MISSING_CLOSE, EMPTY_FRONTMATTER)
  matching actual markdown.ts validator messages, not just the literal
  code-name prefix.
- 6 new unit tests pinning literal production strings.
- 1 comprehensive E2E test exercising the full failure loop.

Total v0.22.12 diff: ~340 lines on top of PR #501. Backward-compatible —
pre-v0.22.12 JSONL entries get classified at acknowledge time.

* chore: regenerate llms-full.txt for v0.22.12 CLAUDE.md changes

CI regen-drift guard caught that llms-full.txt was stale after the v0.22.12
CLAUDE.md annotation updates (sync.ts, doctor.ts, sync-failures.test.ts,
e2e/sync.test.ts entries). Per CLAUDE.md "Auto-derived" rule: run
`bun run build:llms` after any release ship that touches Key Files
annotations. The bundle reflects current docs state.

llms.txt unchanged (curated index doesn't index those entries).
llms-full.txt: 308192 bytes.

test/build-llms.test.ts now passes 7/7 (was 6/7 in CI).

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
2026-04-29 22:34:04 -07:00
52f9581966 v0.22.11 feat: storage tiering — db_tracked vs db_only directories (#494)
* feat: storage tiering — git-tracked vs supabase-only directories

Brain repos scaling to 200K+ files. Bulk data (tweets, articles, transcripts)
bloats git repos and slows operations. New storage config in gbrain.yml lets
users declare git-tracked and supabase-only directories.

Changes:
- New config: storage.git_tracked and storage.supabase_only in gbrain.yml
- gbrain sync auto-manages .gitignore for supabase-only paths
- gbrain export --restore-only restores missing supabase-only files from DB
- New gbrain storage status command shows tier breakdown
- Config validation warns on conflicts
- 8 tests passing, full docs at docs/storage-tiering.md

Backward compatible — systems without gbrain.yml work unchanged.

* feat: add getDefaultSourcePath() typed accessor (step 1/15)

Single source of truth for "what brain repo are we operating against?"
Replaces ad-hoc raw SQL in storage.ts:38 (Issue #3 of eng review). Used by
both gbrain storage status and gbrain export --restore-only.

Returns null on miss, throws on DB error. Composes with the existing
resolveSourceId chain so it honors --source flag / GBRAIN_SOURCE env /
.gbrain-source dotfile / longest-prefix CWD match / brain-level default.

4 new test cases covering happy path, missing local_path, DB error
propagation, and CWD-prefix resolution priority.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: replace gray-matter with dedicated YAML parser (step 2/15)

The original storage-config.ts called gray-matter on a delimiter-less YAML
file. Gray-matter only parses YAML inside `---` frontmatter blocks; without
delimiters, it returns `{data: {}}`. Result: loadStorageConfig() always
returned null, the entire feature was a silent no-op for every user.

Original eng review's P0 confidence-9 finding (Issue #1).

Replaces gray-matter with a small dedicated parser for the gbrain.yml shape
(top-level `storage:` section, two array-valued nested keys). Yaml-lite was
considered first, but its flat key:value design doesn't handle nested
arrays. The dedicated parser is ~50 lines and trades expressiveness for
zero-dep, predictable parsing of a file format we control.

Adds the Issue #1B sanity warning (locked B): when gbrain.yml exists but
has no storage section (or empty arrays), warn once-per-process so the
user sees their config didn't take. The single test that would have caught
the original P0 — write a real gbrain.yml, call loadStorageConfig, assert
non-null — now exists.

Also tightens loadStorageConfig per D36: distinguishes "absent" (silent
null) from "unreadable" (throws). The previous code silently swallowed
read errors, hiding broken installs.

8 new test cases: real-disk happy path, comments + blank lines, quoted
values, missing storage section warning, empty section warning,
once-per-process warning suppression, unreadable file behavior, and the
existing helper tests (validation, tier matching, edge cases) all still
pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: rename storage keys to db_tracked/db_only (step 3/15)

The vendor-specific names "supabase_only" and "git_tracked" hardcoded a
backend (Supabase) into the config schema. gbrain ships two engines —
PGLite and Postgres-via-Supabase. The canonical distinction is "lives in
the brain DB only" vs "lives in the brain DB and on disk under git." Both
work on either engine.

Renamed throughout (Issue #4 of eng review):
  git_tracked    → db_tracked
  supabase_only  → db_only
  isGitTracked() → isDbTracked()
  isSupabaseOnly() → isDbOnly()
  StorageTier 'git_tracked'/'supabase_only' → 'db_tracked'/'db_only'

Backward compatibility (D3 lock):
  loadStorageConfig accepts both shapes. Loader resolution order per the
  eng-review pass-2 finding: parse YAML → if canonical keys present use
  them, else if deprecated keys present map to canonical AND emit
  once-per-process deprecation warning → THEN run validation.
  Validation always sees the canonical shape so error messages reference
  db_tracked/db_only regardless of which keys the user wrote.

  The deprecation warning suggests `gbrain doctor --fix` for an automated
  rename (D72 — fix path lands in step 7).

  When both shapes coexist in one file, canonical wins and a stronger
  warning fires ("deprecated keys ignored — remove them").

Aliases isGitTracked/isSupabaseOnly kept for now to avoid churning the
sync.ts / export.ts / storage.ts call sites in this commit; they'll be
removed in a follow-up step. Storage.ts's tier-bucket initializers and
output strings updated. ASCII output replaces unicode box-drawing per D10.

gbrain.yml example file updated to canonical keys with explanatory
comments.

2 new test cases: deprecated-key fallback (asserts both shapes load
correctly with warning), canonical-wins-over-deprecated (asserts the
"both shapes coexist" path).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: add slugPrefix to PageFilters with engine-side filter (step 4/15)

Issue #13 of the eng review: storage.ts and export.ts loaded every page
in the brain (limit: 1_000_000) to check tier membership. On the 200K-page
brains this feature targets, that's the wall-clock and memory landmine
the feature exists to fix.

Adds an optional `slugPrefix` field to PageFilters. Both engines implement
it as `WHERE slug LIKE prefix || '%' ESCAPE '\'`, with literal escaping of
LIKE metacharacters (%, _, \) so user-supplied prefixes like `media/x/`
are treated as exact string prefixes.

Performance: the (source_id, slug) UNIQUE constraint on the pages table
gives both engines a btree index that supports LIKE-prefix range scans.
An EXPLAIN on Postgres confirms the index range scan rather than a seq
scan. PGLite has the same index shape via pglite-schema.ts.

Consumers updated:
  - export.ts: --slug-prefix flag now goes engine-side (no in-memory
    .filter(...)). The --restore-only path queries each db_only directory
    with slugPrefix in a loop instead of one full-table scan, with seen-set
    deduplication and disk-existence check inline.
  - storage.ts: keeps the full-scan path because storage-status needs the
    "unspecified" bucket count, which can't be computed without enumerating
    every page. Comment notes that step 5 (single-walk filesystem scan)
    will reduce per-page disk syscall cost.

2 new test cases on PGLiteEngine: slugPrefix happy path (3 tier dirs,
asserts only matching slugs return) and metacharacter escape regression
(asserts safe/ doesn't match unrelated slugs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf: single-walk filesystem scan via walkBrainRepo() (step 5/15)

Issue #14 of the eng review: storage.ts called existsSync + statSync
per-page in a synchronous loop. On a 200K-page brain that's 400K syscalls
serialized. Wall-clock landmine.

Adds src/core/disk-walk.ts with walkBrainRepo(repoPath) — one recursive
readdirSync walk, builds a Map<slug, {size, mtimeMs}>. Storage.ts looks
up each DB page in the map (O(1)) instead of stat-checking on demand.
Slug derivation matches the pages-table convention: people/alice.md on
disk becomes people/alice as the map key.

Skipped during walk:
  - dot-directories (.git, .gbrain, .vscode, etc) — not part of the brain
    namespace
  - node_modules — guards against accidentally walking into imported repos
  - non-.md files (sidecar JSON, binaries) — tracked by the brain through
    the files table, not by slug

Reusable: future commands (gbrain doctor's storage_tiering check, the
optional autopilot tier-fix path) get the same walk for free.

9 new test cases: empty dir, nonexistent dir, top-level files, nested
dirs, dot-dir skipping, node_modules skipping, non-.md filtering, size
capture, mtimeMs capture.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: path-segment matching for tier directories (step 6/15)

Issue #5 + D6 of the eng review: tier matching used slug.startsWith(dir),
which falsely matches 'media/xerox/foo' against 'media/x' if a user wrote
the directory without a trailing slash.

The new matcher requires the configured directory to end with `/` and
treats it as a canonical path-segment ancestor:

  media/x/   matches  media/x/tweet-1       ✓
  media/x/   doesn't  media/xerox/foo       ✗
  media/x    refused  media/x/tweet-1       (matcher requires trailing /)

Non-canonical input (no trailing slash) is refused outright. Step 7's
auto-normalizing validator converts user-written 'media/x' → 'media/x/'
on load, so the matcher never sees non-canonical input from real configs.
The behavior tested here is the strict matcher's contract.

Regression test pins the media/xerox collision case explicitly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: auto-normalize trailing-slash, throw on tier overlap (step 7/15)

D7+D8 of the eng review: validation was warnings-only. Users miss warnings.
Now:

  - Cosmetic: missing trailing slash auto-corrected, one-time info note
    showing what changed ("normalized 2 storage paths: 'people' →
    'people/', 'media/x' → 'media/x/'"). Once-per-process to keep noise low.

  - Semantic: same directory in both tiers throws StorageConfigError.
    Ambiguous routing — does media/ win as db_tracked or db_only? — is a
    real bug the user must fix. Caller propagates to the CLI for a clean
    exit-1 with actionable message.

loadStorageConfig now applies normalize+validate after merging deprecated
keys, so the path-segment matcher (step 6) only ever sees canonical
trailing-slash directories.

The pure validateStorageConfig kept for callers who want the warnings list
without the auto-fix side effects (gbrain doctor's reporting path).

2 new test cases: auto-normalize round-trip with warning text assertion,
overlap throws StorageConfigError.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: wire manageGitignore into runSync, only on success (step 8/15)

Issue #2 of the eng review: manageGitignore was defined and never
invoked. Docs claimed "auto-managed by gbrain" — false. Users hit a
.gitignore that never updated and committed db_only directories anyway.

Wire-up: runSync now calls manageGitignore after each successful
performSync return, in both watch and one-shot modes.

Eng review pass-2 finding #1: skip on dry_run AND blocked_by_failures
status. A sync that aborted partway has stale state; mutating .gitignore
based on a partially-loaded config invites drift. Failure-skip test
added (uses .gitignore-as-a-directory to simulate write failure;
asserts warning fired and disk wasn't corrupted).

Hardened manageGitignore itself with three additional behaviors:

  - GBRAIN_NO_GITIGNORE=1 escape hatch (D23) for shared-repo setups
    where a maintainer wants gbrain to leave .gitignore alone.

  - Submodule detection (D49). When repoPath/.git is a regular file
    (gitdir: ... pointer), the repo is a git submodule. Submodule
    .gitignore changes don't survive parent submodule updates, so we
    skip with an actionable warning ("add db_only directories to your
    parent repo's .gitignore manually").

  - Graceful failure (D9). Read errors, write errors, and
    StorageConfigError (overlap from step 7) all log a warning and
    return — sync's primary job (moving data) shouldn't die because of
    a side-effect on .gitignore.

manageGitignore is now exported (previously private) so the
storage-sync test file can hit it directly without spinning up sync.

9 new test cases: no-op without gbrain.yml, no-op with empty db_only,
happy-path append, idempotency (run twice, single entry), preservation
of user-written rules, GBRAIN_NO_GITIGNORE skip, submodule skip,
.git-directory normal path, write-failure graceful warning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: D5 resolution chain for --restore-only and storage status (step 9/15)

D5 of the eng review: gbrain export --restore-only without --repo
silently fell through to the regular export path, dumping every page in
the database to the wrong directory. Hard regression risk.

Now exits 1 with an actionable message when --restore-only has no
--repo AND no configured default source. Resolution order:
  1. Explicit --repo flag
  2. Typed sources.getDefault() (reuses step 1's accessor)
  3. Hard error — never fall through to cwd

storage.ts:38 also bypassed BrainEngine with raw SQL and a bare
try/catch (Issue #3 + Issue #9). Replaced with the same typed
getDefaultSourcePath() — single source of truth, errors propagate
cleanly to the user, no silent cwd fallback.

Regular export (no --restore-only) keeps its current behavior per D26:
exports include everything, --repo is optional.

4 new test cases on PGLite in-memory:
  - hard-errors with no --repo + no default
  - explicit --repo wins
  - falls back to sources default local_path
  - non-restore export does not require --repo

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: split storage.ts into pure data + JSON + human formatters (step 10/15)

Issue #10 of the eng review: getStorageStatus and runStorageStatus mixed
data gathering, JSON serialization, and human-readable output in one
function. Hard to test, hard to reuse, mismatched the orphans.ts pattern
that CLAUDE.md cites as the precedent.

Now three pure functions + a thin dispatcher:

  getStorageStatus(engine, repoPath) — async, returns StorageStatusResult.
    Side effects: engine.listPages + one walkBrainRepo (Issue #14).
    Exported so MCP exposure (D14) and gbrain doctor (D13) can consume the
    same data without re-running the loop.

  formatStorageStatusJson(result) — pure, returns indented JSON. Stable
    contract on the StorageStatusResult shape, suitable for orchestrators.

  formatStorageStatusHuman(result) — pure, returns ASCII text (D10 — no
    unicode box-drawing). Composable into other commands later.

  runStorageStatus(engine, args) — thin dispatcher: parses --repo /
    --json, calls getStorageStatus, picks a formatter, prints.

8 new test cases on the formatters: JSON parse round-trip, null-config
fallback, missing-files capped at 10 with rollup, ASCII-only assertion
(D10 regression guard), warnings inline, configuration listing, disk-
usage block omitted when zero bytes.

The StorageStatusResult interface is now exported as a public type, so
gbrain doctor's storage_tiering check can build its own findings from
the same shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* types: distinct PageCountsByTier and DiskUsageByTier (step 11/15)

Issue #11 of the eng review: pagesByTier (page counts) and
diskUsageByTier (byte totals) shared the same structural type
(Record<StorageTier, number>). Both are tier-keyed numeric maps but
carry semantically different units. A future bug that swaps them at a
call site (e.g., displaying disk bytes where the count belongs) wouldn't
trip the compiler.

Replaced with distinct nominal types via a brand field. Structurally
identical at runtime (no overhead) but compile-time disjoint —
TypeScript catches accidental cross-assignment.

  PageCountsByTier   { db_tracked, db_only, unspecified } : numbers (count)
  DiskUsageByTier    { db_tracked, db_only, unspecified } : numbers (bytes)

Both initialized in getStorageStatus, both threaded into
StorageStatusResult, both consumed by formatStorageStatusHuman /
formatStorageStatusJson without further changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: PGLite soft-warn + full lifecycle test (step 12/15)

D4: storage tiering on PGLite is a partial feature. The "DB" the pages
live in IS the local file gbrain uses for everything else, so "db_only"
has no real offload effect. The .gitignore management still helps
(keeps bulk content out of git history), so we warn and proceed —
not refuse.

Two warning sites (once-per-process each via module-local flags):
  - storage status: warns at runStorageStatus entry
  - sync: warns inside manageGitignore when engineKind='pglite' and
    config has db_only entries

Both phrased actionably ("To get full tiering, migrate to Postgres
with `gbrain migrate --to supabase`").

manageGitignore signature now takes an optional `engineKind` param.
runSync passes engine.kind. Stand-alone callers (tests, future
gbrain doctor --fix path) can omit it.

New test: test/storage-pglite.test.ts — D8 + D4 lifecycle. 6 cases:
engine.kind assertion, getStorageStatus loading gbrain.yml + reporting
tier counts, manageGitignore PGLite-warn (once per process), Postgres
no-warn, slugPrefix on PGLite, end-to-end (config + putPage + status
+ gitignore).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: add trailing-newline CI guard (step 14/15)

Issue #7 of the eng review: all four new files in the original
storage-tiering branch lacked POSIX trailing newlines. Linters complain,
git diffs phantom-flag every future edit. We've been adding newlines as
each file landed; this commit catches the regression class.

scripts/check-trailing-newline.sh:
  - sibling to check-jsonb-pattern.sh / check-progress-to-stdout.sh per
    CLAUDE.md's CI guard pattern
  - portable to bash 3.2 (macOS default; no mapfile, no associative arrays)
  - covers src/**, test/**, gbrain.yml, top-level *.md
  - reports each missing file by path and exits 1

Wired into `bun run test` between progress-to-stdout and typecheck.

Also fixed docs/storage-tiering.md (pre-existing missing newline from
the original branch — caught by the new guard on first run).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.23.0 — VERSION, CHANGELOG, README, CLAUDE.md, storage-tiering.md (step 15/15)

VERSION → 0.23.0 (minor bump for new feature surface).

CHANGELOG entry in Garry voice with the canonical format:
  - Two-line bold headline ("Storage tiering, finally working...")
  - Lead paragraph naming what was broken before and what users get now
  - "Numbers that matter" before/after table for the 6 things that
    actually changed
  - "What this means for your brain" closer
  - "To take advantage of v0.23.0" self-repair block (per CLAUDE.md
    convention) — 6 numbered steps users can follow
  - Itemized changes split into critical fixes / new+renamed surface /
    architecture cleanup / tests + CI guards

CLAUDE.md "Key files" gains four new entries: storage-config.ts,
disk-walk.ts, the v0.23.0 storage.ts shape, and gbrain.yml itself.

README.md gains a new "Storage tiering" section between Skillify and
Getting Data In with the canonical example + commands + link to the
full guide.

docs/storage-tiering.md rewritten end-to-end with canonical key names
(db_tracked / db_only), v0.23.0 hardening details (idempotency,
submodule detection, GBRAIN_NO_GITIGNORE, dry-run gating), the
resolution chain for --restore-only, the auto-normalize +
throw-on-overlap validator, and the PGLite engine note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: e2e Postgres lifecycle for storage tiering (step 16/16)

Per the v0.23.0 plan: full lifecycle E2E against real Postgres.

  - engine.kind === 'postgres' assertion
  - Full lifecycle: write 4 pages (1 db_tracked, 2 db_only, 1 unspecified)
    → getStorageStatus reports correct tier counts → human formatter
    renders → manageGitignore writes managed block → idempotency check
    → getDefaultSourcePath() resolves the configured local_path.
  - Container restart simulation: 2 db_only pages in DB, files missing
    on disk → status.missingFiles.length === 2 → slugPrefix engine
    filter on Postgres returns exactly the tier slugs.
  - slugPrefix index-based range scan regression: 50 media/x/* + 50
    people/p-* pages → slugPrefix='media/x/' returns exactly 50.
  - getDefaultSourcePath returns null when default source has no
    local_path (the hard-error path that replaces the original silent
    cwd fallback).
  - manageGitignore on Postgres engine does NOT emit the PGLite
    soft-warn (cross-engine assertion).

Skips gracefully when DATABASE_URL is unset, per CLAUDE.md E2E pattern.
Run via: DATABASE_URL=... bun test test/e2e/storage-tiering.test.ts

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump version 0.23.0 → 0.22.9

Reverts the minor bump back to a patch-style version on the v0.22 line.
Storage tiering ships within the v0.22.x train alongside the recent
fix waves. Updates VERSION, package.json, CHANGELOG header + body refs,
CLAUDE.md Key files annotations, README.md section heading, and the
docs/storage-tiering.md backward-compat note.

* chore: bump version 0.22.9 → 0.22.11

Sibling workspaces claimed v0.22.10 in the queue. This branch advances
to v0.22.11 to keep the version monotonic on master.

Updates VERSION, package.json, CHANGELOG header + body refs, CLAUDE.md
Key files annotations, README.md section heading, and the
docs/storage-tiering.md backward-compat note.

* fix: address Codex pre-landing review findings (4 fixes)

Codex found 4 real issues during pre-landing review of v0.22.11 diff:

[P0] export --restore-only fell through to full export when
storageConfig was null (no gbrain.yml present). On older or
misconfigured brains, the recovery command would silently dump the
entire database. src/commands/export.ts now refuses with an actionable
error before any page query fires — matches the D5 lock spirit
("never silently fall through").

[P1] manageGitignore wire-up only fired when --repo was passed
explicitly. performSync resolves the repo from sync.repo_path or
sources.local_path, so the common `gbrain sync` path (after
setup, no flag) never updated .gitignore. src/commands/sync.ts now
uses the same source-resolver chain as the rest of /ship: opts.repoPath
→ getDefaultSourcePath → null. Fires in both watch and one-shot modes.

[P2] getDefaultSourcePath only consulted sources.local_path, missing
the legacy global sync.repo_path config key that pre-v0.18 brains use.
Added a fallback to engine.getConfig('sync.repo_path') when the
sources row has NULL local_path. Pre-v0.18 brains now work without
forcing a `gbrain sources add . --path .` migration.

[P2] sync --all multi-source loop never called manageGitignore even
though src.local_path was already known. Each source now gets its own
gitignore update on successful sync.

Tests:
  - test/storage-export.test.ts: replaced the old "falls through to
    full export" test with one that asserts the new refusal path
    (storage-tiering config required for --restore-only).
  - test/source-resolver.test.ts: added a fallback test exercising the
    legacy sync.repo_path code path for pre-v0.18 brains.
  - All 78 storage-tiering tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms.txt + llms-full.txt for v0.22.11

Per CLAUDE.md: "Run `bun run build:llms` after adding a new doc."
The README's new Storage tiering section + the rewritten
docs/storage-tiering.md changed the inlined bundle. test/build-llms.test.ts
catches the drift and was failing on master pre-regen.

* fix: typecheck error in disk-walk.ts (CI #73350475897)

tsc --noEmit failed in CI because ReturnType<typeof readdirSync> with
withFileTypes:true picks an overload union that includes
Dirent<Buffer<ArrayBufferLike>>. Strict tsc treats entry.name as Buffer,
so .startsWith / .endsWith / string comparisons all blew up.

Annotate the variable as Dirent[] (string-based) and cast through unknown,
matching the pattern sync.ts already uses for its own filesystem walk.
Same runtime behavior; clean typecheck.

Tests still 9/9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:21:07 -07:00
Garry Tan 2a9feb859f Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	VERSION
#	llms-full.txt
#	package.json
2026-04-29 22:18:26 -07:00
5d9dc4393e v0.22.10 fix: autopilot-cycle handler forwards job.data.phases to runCycle (#521)
* fix: autopilot-cycle handler forwards job.data.phases to runCycle

The autopilot-cycle handler always ran ALL_PHASES regardless of job data.
This caused production stalls when the embed phase had a large backlog
(17K+ stale chunks) that exceeded the 30-minute job timeout. Every 5-min
cycle would start, hit the embed wall, stall, and get force-killed —
creating an infinite stall loop that kept the queue perpetually unhealthy.

The fix validates job.data.phases against ALL_PHASES (preventing injection)
and forwards the selected phases to runCycle(). Callers can now submit
fast cycles (lint+backlinks+sync+extract) on a 5-min cron and run embed
separately with a longer timeout during off-peak hours.

If phases is omitted, not an array, or filters to empty, behavior is
unchanged (all phases run).

Tests: 4 new cases covering phase restriction, invalid name filtering,
empty array fallback, and non-array type safety.

* test: widen autopilot-cycle handler-block window for phases-passthrough

The regression guard sliced the first 500 chars after `worker.register('autopilot-cycle'`
and asserted `signal: job.signal` was present. The phase-validation block added in
787ec7de pushed the signal arg past that boundary, so CI test shard 3 failed even
though the handler still propagates the signal correctly. Bump the window to 2000.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.22.10)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: sync release notes for v0.22.10

Note autopilot-cycle phases passthrough fix on the src/commands/jobs.ts
key-files annotation so future readers know the handler honors
job.data.phases (validated against ALL_PHASES) as of v0.22.10.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt for v0.22.10 CLAUDE.md update

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:13:05 -07:00
Garry Tan 15b9316dbf Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	TODOS.md
#	VERSION
#	package.json
2026-04-29 11:37:09 -07:00
08746b06d2 v0.22.9 feat: structured error code summary for sync --skip-failed (#501)
* feat: structured error code summary for sync --skip-failed (#500)

When sync encounters per-file failures, the blocked/skip-failed messages
now include a breakdown by error code (SLUG_MISMATCH, YAML_PARSE, etc.)
instead of just a raw count. This makes it immediately obvious *why*
files failed without requiring manual investigation.

Changes:
- Add classifyErrorCode() — maps error messages to ParseValidationCode
- Add summarizeFailuresByCode() — groups failures into sorted code summary
- SyncFailure now carries a 'code' field (backfilled on acknowledge)
- acknowledgeSyncFailures() returns AcknowledgeResult {count, summary}
- sync blocked + skip-failed messages show code breakdown
- doctor sync_failures check shows code breakdown for both unacked and historical
- 12 new tests for classifyErrorCode, summarizeFailuresByCode, and structured returns

Before:
  Sync blocked: 2688 file(s) failed to parse.

After:
  Sync blocked: 2688 file(s) failed to parse:
    SLUG_MISMATCH: 2685
    YAML_DUPLICATE_KEY: 3

Closes #500

* fix: eng-review fixes for sync error-code classification

- Reorder classifyErrorCode() so DB-layer errors (DB_DUPLICATE_KEY,
  STATEMENT_TIMEOUT) check BEFORE YAML patterns. Postgres "duplicate key
  value violates unique constraint" no longer mislabels as YAML_DUPLICATE_KEY.
- Rewrite MISSING_OPEN/MISSING_CLOSE/EMPTY_FRONTMATTER/NULL_BYTES/NESTED_QUOTES
  regexes to match the canonical messages emitted by collectValidationErrors()
  in src/core/markdown.ts. Previous patterns (e.g. /missing.*open/i) never
  fired because the upstream throw site emits prose ("File is empty...",
  "No closing --- delimiter found"), not the code name.
- Extract formatCodeBreakdown() helper that accepts either raw failures or
  pre-summarized {code, count}[] input. Replaces 3 duplicate inline builders
  in src/commands/sync.ts.
- 15 new tests (37/37 pass on test/sync-failures.test.ts):
  - DB vs YAML duplicate-key disambiguation (3 cases)
  - Canonical-message coverage for the 5 frontmatter codes (7 cases)
  - acknowledgeSyncFailures() legacy-entry backfill branch (2 cases)
  - formatCodeBreakdown() dual-input shape (3 cases)
- TODOS.md: file 3 follow-ups (P2 plumb structured ParseValidationCode;
  P0-at-ship CHANGELOG migration note for AcknowledgeResult; P3 concurrent-
  safe ack of sync-failures.jsonl).

Eng-review plan: ~/.claude/plans/then-codex-synchronous-toucan.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.22.9)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: 16-core runner + 4-way matrix shard for test job

The unit test suite ran 22m17s on ubuntu-latest (2-core/7GB) because:
- 187 test files run with bun test parallelism bounded by core count
- 23 of those files spin up a fresh PGLiteEngine + initSchema in beforeEach,
  paying ~22s WASM cold-start per test on the small runner

This commit fixes the runner side:
- runs-on: ubuntu-latest-16-cores (16 vCPU / 64 GB RAM)
- strategy.matrix.shard splits 4 parallel jobs, each running ~40 of 158 unit
  test files. Single-file wall-time floor is ~3 min after the test refactor,
  so 4 shards × 16 cores hits the floor quickly without wasting cores past it.
- pre-test gates (typecheck, check-jsonb, check-progress, check-wasm) only run
  on shard 1 — they're not test files and don't benefit from sharding.

scripts/test-shard.sh partitions test files by stable FNV-1a hash mod N. Same
file always lands in the same shard, so retries are reproducible. Pure shell,
portable to bash 3.2 (macOS) and bash 5.x (CI). Excludes test/e2e/ which runs
via bun run test:e2e separately and needs DATABASE_URL.

Also: ignore .claude/ harness state files (scheduled_tasks.lock etc) instead
of just .claude/skills/.

Cost: ~$0.19/run vs $0 (public repo, default runner is free). At 50 PRs/month
that's ~$10/month for ~5x faster CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: refactor top-3 PGLite-heavy files to share one engine per file

Three test files were spinning up a fresh PGLiteEngine + connect + initSchema
in beforeEach. PGLite WASM cold-start is ~22s on the small CI runner; doing
this per test multiplied wall-time across the suite. The 3 files alone
accounted for ~6.5 min of the 22m CI run (177s + 132s + 87s).

Refactor: move PGLite setup to beforeAll (one engine per file), wipe data
in beforeEach via the new test/helpers/reset-pglite.ts helper.

The reset helper:
- TRUNCATEs every public table CASCADE, including sources (so tests that
  register their own sources don't leak rows into the next test).
- Re-seeds the default source row that pages.source_id's DEFAULT FKs against.
  Without this, the next page insert would fail FK validation.
- Preserves schema_version so migration helpers don't think the brain is on v0.

Files refactored:
- test/extract-incremental.test.ts (8 tests, was 177s on CI)
- test/brain-writer.test.ts (16 tests; only the scanBrainSources block uses
  PGLite, was 132s on CI)
- test/sync.test.ts (37 tests; only the performSync dry-run block uses PGLite,
  was 87s on CI)

All 61 tests still pass locally. The remaining 20 PGLite-heavy files use the
same beforeEach anti-pattern; this commit only refactors the proven worst
offenders. Sweep the rest in a follow-up if CI numbers indicate it's worth it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: fall back to ubuntu-latest for matrix shard

The ubuntu-latest-16-cores label requires a provisioned larger-runner pool in
repo/org settings. Without that setup, jobs queue indefinitely waiting for a
runner that doesn't exist (verified: 4 shards stuck in 'queued' status with
empty runner_name for 5+ min).

Drop back to the default 2-core ubuntu-latest. The 4-way matrix shard still
delivers ~5-6x speedup via parallelism alone — 4 jobs running in parallel,
each handling ~40 of 158 unit test files. Cost stays $0 (default runner is
free for public repos).

If we ever provision a larger-runner pool, flip this label back to
ubuntu-latest-16-cores. The matrix + sharder will use the bigger boxes
unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 10:12:10 -07:00
Garry TanandClaude Opus 4.7 f739de5521 chore: regenerate llms-full.txt for v0.22.13 doc updates
CI's build-llms generator test failed because llms-full.txt was stale
relative to the README + CLAUDE.md updates this PR added (--workers
flag in the IMPORT section, sync-concurrency.ts/db-lock.ts/sync.ts
entries in the Key files section).

Per CLAUDE.md: "Run \`bun run build:llms\` after adding a new doc."
The test test/build-llms.test.ts:67 verifies committed bundles match
generator output — now they do again.

llms.txt was already in sync (no curated config additions); only
llms-full.txt needed the regen.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 08:23:44 -07:00
Garry TanandClaude Opus 4.7 02d585c0a4 chore: bump version slot to v0.22.13
VERSION 0.22.10 → 0.22.13. Master moved to 0.22.8 plus claimed slots
0.22.9-0.22.12 in sibling workspaces; 0.22.13 is the next free slot for
this PR's parallel-sync hardening work.

Updated all v0.22.10 references in CHANGELOG.md (release header +
self-repair block), TODOS.md (D-PR490-1 follow-up tag), CLAUDE.md
(Key files entries + tests + commands subsection), and the inline
v0.22.10 markers in src/core/sync-concurrency.ts, src/core/db-lock.ts,
src/commands/sync.ts, src/commands/import.ts, src/commands/jobs.ts,
test/sync-parallel.test.ts, test/e2e/sync-parallel.test.ts.

No behavioral change. CHANGELOG header rewrite, content unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 20:05:17 -07:00
Garry Tan e573fa6988 Merge remote-tracking branch 'origin/master' into feat/parallel-sync
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	VERSION
#	package.json
2026-04-28 19:56:25 -07:00
Garry Tan ff6320e552 Merge remote-tracking branch 'origin/master' into pr-490
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	TODOS.md
#	VERSION
#	package.json
2026-04-28 08:53:54 -07:00
Garry TanandClaude Opus 4.7 36c750bbec docs: update CLAUDE.md + README for v0.22.10 sync hardening
CLAUDE.md:
- New "Key files" entries for src/core/sync-concurrency.ts and
  src/core/db-lock.ts (both v0.22.10).
- New "Key files" entry for src/commands/sync.ts (covers the lock,
  head-drift gate, engine.kind discriminator, vanished-file failure
  capture, parallel branch wiring).
- Updated src/commands/jobs.ts entry with v0.22.10 sourceId
  resolution + autoConcurrency policy + noEmbed contract.
- Added test/sync-concurrency.test.ts and test/sync-parallel.test.ts
  to the unit-test list with case counts.
- Added test/e2e/sync-parallel.test.ts to the E2E section with the
  SYNC_PARALLEL_BENCH grep marker for CHANGELOG quoting.
- Added "Key commands added in v0.22.10" section: gbrain sync --workers,
  gbrain import --workers (parseWorkers validation).

README.md: added --workers flag to the IMPORT section's gbrain sync
and gbrain import lines, with the >100-file auto-parallelize note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:47:19 -07:00
Garry TanandClaude Opus 4.7 7f2c81f929 chore: v0.22.10 release notes + sync follow-up TODO
VERSION + package.json + bun.lock: 0.22.5/0.22.6 → 0.22.10. Repo had
existing drift between VERSION and package.json on master; this commit
brings them back in sync at the bumped value.

CHANGELOG.md: v0.22.10 entry replaces the unfinished v0.23.0 stub from
PR #490's original commit. Voice-rule clean (no em dashes, no AI
vocabulary), real benchmark numbers from the new E2E test
(serial=289ms parallel(4)=221ms speedup=1.31x), additive worker-pool
note (A3), 'To take advantage of v0.22.10' self-repair block per
CLAUDE.md convention.

TODOS.md: A4 follow-up filed — plumb resolved database_url through
SyncOpts so performSync / performFullSync / import.ts don't each call
loadConfig() separately. Deferred to a future patch; not on the
v0.22.10 critical path.

Patch (not minor) framing held even though new CLI surface lands here;
release-notes prose names the behavior change explicitly so users know
to read them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:43:07 -07:00
Garry TanandClaude Opus 4.7 1353366b5f test: e2e parallel sync against real Postgres + benchmark
DATABASE_URL-gated E2E coverage that PGLite-only tests can't reach:

T2 — happy path: 60 files imported at concurrency=4, all 60 pages land
in the DB, with a pg_stat_activity probe before/after to confirm worker
engines (4 × 2 connections) actually disconnected.

P4 — benchmark: 120-file fixture, serial vs concurrency=4 timing.
Emits a single-line `SYNC_PARALLEL_BENCH 120 files | serial=Xms |
parallel(4)=Yms | speedup=Zx` so the CHANGELOG can quote a real
number instead of an unbacked '~4×' claim. Asserts parallel <=
serial * 1.5 to allow for noisy CI but fail genuine regressions.

Skips gracefully when DATABASE_URL is unset (consistent with the rest
of test/e2e/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:54 -07:00
Garry TanandClaude Opus 4.7 93ae40dd3a fix: jobs.ts sync handler — resolve sourceId, autoConcurrency
CODEX-1: resolve sourceId at handler entry by looking up sources.local_path.
Mirrors cycle.ts:480's autopilot-cycle fix (PR #475). Without this, every
Minion sync job on a multi-source brain reads global config.sync.last_commit
instead of the per-source anchor, which on a regularly-GC'd repo can drop
out of git history and trigger 30-min full reimports every cycle.

The handler accepts an optional sourceId job param for callers that want
to override; falls back to the resolveSourceForDir lookup when absent.

CODEX-4: replace the hardcoded concurrency=4 default with the shared
autoConcurrency policy. Behavior is now consistent between CLI sync,
the Minion handler, and the autopilot cycle's sync phase. Jobs that
request a specific concurrency via job.data.concurrency still win.

noEmbed default stays at true — embed is a separate job (submit
gbrain embed --stale, OR rely on the autopilot cycle's embed phase).
The doc comment makes that contract explicit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:43 -07:00
Garry TanandClaude Opus 4.7 8fcd2737bf fix: import.ts — engine.kind discriminator, worker try/finally, parseWorkers
A1: replace the config?.engine === 'pglite' string sniff with
engine.kind === 'pglite' to match sync.ts and the v0.13.1 contract.

A2: wrap worker engine creation + the parallel loop in try/finally so
disconnects always fire — same pattern as sync.ts. Worker engines now
push onto an array as they connect (rather than Promise.all) so the
finally block can clean up partial-connect state.

Q2: route --workers parsing through the shared parseWorkers() helper.
parseInt-with-no-validation is gone — '0', '-3', 'foo', '1.5' now exit
with a clear error message instead of silently falling through.

Q3: drop the config!.database_url! non-null assertion; fall back to
serial when database_url is unset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:30 -07:00
Garry TanandClaude Opus 4.7 b23f24f91b fix: harden performSync — writer lock, head-drift gate, engine.kind
CODEX-2: wrap performSync body in a gbrain-sync DB lock so two concurrent
syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot
both read last_commit, both write it unconditionally, and let the last
writer win. cycle.ts continues to hold gbrain-cycle for its broader scope;
the two ids nest cleanly.

CODEX-3: capture git HEAD at sync entry, re-rev-parse after the import
phase, refuse to advance last_commit if HEAD drifted (someone ran
git checkout / git pull mid-sync). Vanished files now go into failedFiles
instead of silent-skip — same gating mechanism, no more bookmark advance
past unimported work.

A1: replace both PGLite detection sites with engine.kind === 'pglite'.
The constructor.name sniff is gone (breaks under bundling) and so is the
inconsistent config?.engine string check.

A2: connect worker engines serially into an array, run inside try/finally
so disconnect always fires — even on partial connect failure, OOM, or
mid-import abort. Prior Promise.all(...disconnect) leaked the 8 worker
connections on any panic path.

Q1: explicit --workers / opts.concurrency now bypasses the >50-file floor.
User opt-in beats the auto-path safety net.

Q3: drop the config!.database_url! non-null assertions; fall back to serial
when database_url is unset instead of crashing on TypeError.

Q4: worker-count banner moves from console.log to console.error so stdout
stays clean for --json output.

test/sync-parallel.test.ts — 7 cases over PGLite covering the bookmark
gate under concurrency request, the head-drift gate, vanished-file
failure capture, PGLite-stays-serial, and the writer-lock contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:21 -07:00
Garry TanandClaude Opus 4.7 10d96545a4 feat: shared concurrency policy + db-lock primitive
src/core/sync-concurrency.ts — single source of truth for autoConcurrency()
+ parseWorkers() + shouldRunParallel() + constants. Replaces three drifted
call-site policies (performSync, performFullSync, jobs handler).

src/core/db-lock.ts — generic tryAcquireDbLock(engine, lockId, ttlMinutes)
over the existing gbrain_cycle_locks table. Parameterized lock id so
performSync (gbrain-sync) can nest cleanly under cycle.ts (gbrain-cycle)
without deadlock.

test/sync-concurrency.test.ts — 17 cases covering PGLite-forces-serial,
explicit override clamping, auto-path threshold, parseWorkers validation
(rejects 0, negatives, NaN, decimals, trailing chars).

No consumers yet; subsequent commits wire sync.ts, import.ts, and jobs.ts
to use these helpers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 08:42:02 -07:00
root 6b2f3bc321 feat: parallel sync — bounded concurrent imports (#489)
gbrain sync --concurrency N (alias --workers N) parallelizes the import
phase using per-worker Postgres engine instances with an atomic queue
index (same proven pattern as gbrain import --workers N).

Auto-concurrency: when a sync touches >100 files and the user didn't
explicitly set --concurrency, defaults to 4 workers. Small incremental
syncs (<50 files) stay serial. Full syncs auto-detect Postgres and
default to 4 workers.

Minion sync handler defaults to concurrency=4, configurable via job
params: {"concurrency": 8}.

Delete and rename phases remain serial (order-dependent, fast).
PGLite falls back to serial automatically (single-connection engine).

Changes:
- src/commands/sync.ts: SyncOpts.concurrency, parallel import loop in
  performSync incremental path, --workers passthrough in performFullSync
- src/commands/jobs.ts: sync handler accepts concurrency param (default 4)
- CHANGELOG.md: v0.23.0 parallel sync entry

All 37 existing sync tests pass. Typecheck clean.
2026-04-28 06:43:12 +00:00
50 changed files with 5176 additions and 105 deletions
+12 -1
View File
@@ -21,11 +21,22 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
test:
# ubuntu-latest is free 2-core/7GB. Larger runners (16-cores, etc.) require
# a provisioned runner pool in repo settings. Falling back to default keeps
# the matrix shard speedup (~5-6x via parallelism) at zero cost.
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: latest
- run: bun install
- run: bun run test
- name: Pre-test gates (shard 1 only — they're not test files)
if: matrix.shard == 1
run: scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck
- name: Run test shard ${{ matrix.shard }}/4
run: scripts/test-shard.sh ${{ matrix.shard }} 4
+1
View File
@@ -17,3 +17,4 @@ eval/data/world-v1/world.html
# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life)
eval/data/amara-life-v1/_cache/
.claude/
+293 -2
View File
@@ -2,6 +2,299 @@
All notable changes to GBrain will be documented in this file.
## [0.22.13] - 2026-04-28
**Sync got faster, and the bookmark stopped lying.**
**Parallel imports, a real writer lock, and a head-drift gate that catches the worst race.**
The headline is `gbrain sync --workers N`: per-worker Postgres engines with an atomic queue index, same pattern as `gbrain import --workers N`. On a 7,000-page brain that used to take 25+ minutes, the import phase now runs across 4 workers by default. The reproducible benchmark in `test/e2e/sync-parallel.test.ts` shows `parallel(4)` finishing 1.3× faster than serial on a 120-file fixture against local Postgres (`serial=289ms parallel(4)=221ms`). The speedup grows on larger brains and slower-roundtrip databases (Supabase, remote PgBouncer) because the worker setup cost amortizes over more files. But the bigger story is that the sync writer is finally exclusive across processes, and the `last_commit` bookmark refuses to advance when git HEAD has drifted out from under us. The silent-skip-then-advance pathology has survived every prior sync hardening pass. It is dead now.
### What you can do now
- `gbrain sync --workers 4` (alias `--concurrency 4`) parallelizes the import phase. Each worker holds 2 connections, so total Postgres connections during the parallel phase is `workers * 2` plus your caller's pool. At the default of 4 workers and a 10-connection caller pool, that's up to 18 connections, well under PgBouncer's `max_client_conn` default of 100 but worth knowing on tight Supabase tiers.
- **Auto-concurrency:** if you don't pass `--workers`, sync uses 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless, since it's a single-connection engine.
- **Full sync** routes through the same path. First syncs on large brains parallelize automatically.
- **Minion `sync` jobs** also use the new `autoConcurrency()` policy. Behavior is now consistent between CLI sync, the Minion handler, and the autopilot cycle's sync phase. (`noEmbed` defaults to `true` in the jobs handler. Submit `gbrain embed --stale` as a separate job when needed, or rely on the autopilot cycle's embed phase.)
- **`--workers` validation is loud now.** `--workers 0`, `--workers -3`, `--workers foo`, `--workers 1.5` all exit with an error message. The prior behavior silently fell through to auto-concurrency (4 workers), the opposite of what you typed.
### Correctness fixes you didn't have to ask for
- **Cross-process writer lock.** Two `gbrain sync` calls (manual + autopilot, two terminals, two Conductor workspaces) used to read the same `last_commit`, both write it, and let the last writer win. The new `gbrain-sync` row in `gbrain_cycle_locks` serializes the writer window. Same-process reentrance from the autopilot cycle handler was already covered by the broader `gbrain-cycle` lock; sync's lock is narrower and runs underneath it.
- **Head-drift gate.** If `git checkout` or `git pull` runs in your worktree mid-sync (Conductor sibling workspace, ad-hoc terminal), the captured `headCommit` no longer matches HEAD when sync finishes. `last_commit` no longer advances in that case. The next sync re-walks the diff against the new HEAD instead of silently moving the bookmark past unimported work.
- **Vanished files now block bookmark advance.** A file the diff said exists at `headCommit` but is gone from disk used to register as a benign skip. It now goes into `failedFiles` and gates `last_commit` the same way a parse failure does.
- **Per-source bookmark for Minion `sync` jobs.** The job handler now resolves `sourceId` from the repo path (mirrors the autopilot cycle's `cycle.ts` fix from PR #475). On multi-source brains, this prevents the 30-min full-reimport-every-cycle behavior caused by reading the global `config.sync.last_commit` anchor when the per-source row would have been correct.
- **Worker connection cleanup.** Worker engines now disconnect inside `try/finally`, even on partial connect failure or mid-import error. The prior `Promise.all(...disconnect)` ran outside any try/finally, so panic-path leaks never released the 8 worker connections.
- **Engine detection unified.** Both PGLite-detection sites in sync.ts now use `engine.kind === 'pglite'` (the discriminator added in v0.13.1). The `engine.constructor.name === 'PGLiteEngine'` sniff is gone, since it broke under bundling and was inconsistent with the other site's `config.engine` string check.
### What this means for you
If you run autopilot on a 7,000-page Postgres brain, your sync cycle gets faster on day one with no flags. If you have ever felt the bookmark "skip past" work that didn't import, you'll stop seeing it. If you have multiple Conductor workspaces poking the same brain, you'll either wait politely on the writer lock or get a clear "another sync is in progress" error. None of this requires a config change.
## To take advantage of v0.22.13
`gbrain upgrade` should do this automatically. If you want to use the new flags right now:
1. **For a one-off speed win on a large brain:**
```bash
gbrain sync --workers 4
```
Or for incremental syncs that touch >100 files, just run `gbrain sync`. Auto-concurrency fires.
2. **For your autopilot cycle:** no action. The Minion `sync` handler picks up the new auto-concurrency policy automatically.
3. **Verify the writer lock is working:**
```bash
gbrain sync &
gbrain sync # second call will say "Another sync is in progress" or wait
```
4. **If sync ever errors with "Another sync is in progress" and stays stuck:** the lock is in `gbrain_cycle_locks` with id `gbrain-sync` and a 30-minute TTL. If a worker crashed without releasing, the next acquirer takes over once the TTL expires. To unstick faster:
```sql
DELETE FROM gbrain_cycle_locks WHERE id = 'gbrain-sync';
```
5. **If anything looks wrong,** file an issue: https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
- `src/commands/sync.ts`: `performSync` now wraps body in a `gbrain-sync` DB lock; `--workers` honored regardless of file count when explicit; head-drift gate after import phase; engine.kind detection; try/finally around worker engines; banner moved to stderr.
- `src/commands/import.ts`: `engine.kind === 'pglite'` discriminator; try/finally around worker engines; shared `parseWorkers()` for `--workers` validation.
- `src/commands/jobs.ts`: sync handler resolves `sourceId` via `sources.local_path` lookup; concurrency routed through `autoConcurrency()`; `noEmbed: true` default documented.
- `src/core/sync-concurrency.ts` (new): `autoConcurrency()` + `parseWorkers()` + constants. One source of truth for the concurrency policy that previously lived in three call sites.
- `src/core/db-lock.ts` (new): generic `tryAcquireDbLock(engine, lockId)` over the existing `gbrain_cycle_locks` table. Reused by performSync. cycle.ts continues to use its own ID `gbrain-cycle` so the two locks nest cleanly.
- `test/sync-concurrency.test.ts` (new): 17 cases covering autoConcurrency thresholds, shouldRunParallel gates, parseWorkers validation.
- `test/sync-parallel.test.ts` (new): PGLite-routed coverage of the bookmark gate under concurrency request, the head-drift gate, the writer-lock contract, and PGLite-stays-serial.
- `test/e2e/sync-parallel.test.ts` (new): DATABASE_URL-gated Postgres E2E. 60-file happy path with `pg_stat_activity` leak probe, plus a 120-file serial-vs-parallel benchmark that prints `SYNC_PARALLEL_BENCH ...` for CHANGELOG quoting.
### For contributors
- `BrainEngine.kind` is now the canonical PGLite/Postgres discriminator. Avoid `engine.constructor.name === '...'` (breaks under bundling) and `config.engine === '...'` (inconsistent with the engine actually in use).
- The `gbrain_cycle_locks` table is now multi-purpose. The id column distinguishes lock scopes: `gbrain-cycle` for the cycle, `gbrain-sync` for the sync writer. Future locks should pick distinct ids and reuse `tryAcquireDbLock`.
- `parseWorkers()` is the canonical CLI flag parser for `--workers`. Use it instead of inline `parseInt`.
## [0.22.12] - 2026-04-29
**`sync --skip-failed` now classifies file-size and symlink rejections instead of bucketing them as UNKNOWN.**
**Plus a full end-to-end test for the failure loop.**
v0.22.9 shipped the headline classifier work: code-grouped breakdowns at sync time,
DB-vs-YAML disambiguation, doctor surfaces both unacked and historical entries with
`[CODE=N]` lines. v0.22.12 closes the last two coverage gaps that v0.22.9 left on
the table:
- **FILE_TOO_LARGE** now covers the three real production sites in
`src/core/import-file.ts:199, 352, 401` ("Content too large", "File too large",
"Code file too large"). On v0.22.9 these all bucketed as UNKNOWN — the same
silent-systemic-failure pattern that motivated the original issue.
- **SYMLINK_NOT_ALLOWED** covers `src/core/import-file.ts:347` ("Skipping symlink").
Security-relevant rejection that operators should see.
- **End-to-end failure-loop test** in `test/e2e/sync.test.ts` exercises the full
chain: broken file → sync blocks with grouped breakdown → `--skip-failed`
advances bookmark with grouped acknowledgement → second broken file → second
cycle. PostgreSQL-backed; verifies bookmark gating, JSONL state, dedup, and
summary aggregation. v0.22.9's coverage was unit-tests-only.
Twelve total error codes ship in the classifier:
`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `DB_DUPLICATE_KEY`,
`MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`,
`NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`,
`SYMLINK_NOT_ALLOWED`. Anything the regex set doesn't recognize falls through
as `UNKNOWN`.
### What this means for you
If your brain rejects oversized files or symlinks, you now see those rejections
in the doctor breakdown and at sync time grouped by code, instead of as
`UNKNOWN`. Run `gbrain upgrade`. No manual action required.
### Itemized changes
#### Added
- `FILE_TOO_LARGE` classifier code covering `src/core/import-file.ts:199, 352, 401`.
- `SYMLINK_NOT_ALLOWED` classifier code covering `src/core/import-file.ts:347`.
- Two new unit tests in `test/sync-failures.test.ts` pinning the new codes against
literal production message strings (`File too large (N bytes)`, `Skipping symlink: ...`).
- `test/e2e/sync.test.ts` — new failure-loop test exercising broken-file → block →
`--skip-failed` → second cycle. Hermetic on developer machines (saves+restores
the user's real `~/.gbrain/sync-failures.jsonl`).
## To take advantage of v0.22.12
No manual action required. Run `gbrain upgrade`. The new `FILE_TOO_LARGE` and
`SYMLINK_NOT_ALLOWED` classifier codes apply on the next `gbrain sync`.
## [0.22.11] - 2026-04-27
**Storage tiering, finally working. Brains scaling past 100K files stop bloating git.**
The original storage-tiering branch shipped two silent bugs (gray-matter on YAML returned empty data; `manageGitignore` was defined and never invoked) so the feature was a no-op for every user who tried it. v0.22.11 rewrites the broken bits, hardens the surface, and adds proper test coverage. If you have a brain repo north of 100K files where bulk machine-generated content (tweets, articles, transcripts) is the size driver, this is the release that pulls it out of git without losing any data.
Configure tiering in `gbrain.yml` at the brain repo root:
```yaml
storage:
db_tracked:
- people/
- companies/
- deals/
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
`gbrain sync` then auto-manages your `.gitignore` for `db_only` directories so bulk content stops landing in commits. `gbrain export --restore-only` repopulates missing `db_only` files from the database (container restart, fresh clone, accidental rm). `gbrain storage status` shows the breakdown — counts, disk usage, missing files.
### The numbers that matter
200K-page brain, half tweets and articles. Before v0.22.11:
| Metric | Before | After | Δ |
|--------|--------|-------|---|
| `gbrain.yml` actually loads | no (silent null) | yes | feature works |
| `.gitignore` auto-manages | no (function never called) | yes | docs match reality |
| `--restore-only` without `--repo` | silent full export | hard error | no data-loss footgun |
| `media/xerox` matched against `media/x` | yes (collision) | no | path-segment matching |
| Per-page disk syscalls during status | ~400K (existsSync + statSync) | ~one per dir + one stat per .md | single-walk scan |
| Validation surfaces overlap | warning only | throws StorageConfigError | semantic error caught |
### What this means for your brain
If you've been reading the storage-tiering docs and waiting for the feature to actually do something: it does now. If you're already over 50K files: configure `gbrain.yml`, run `gbrain sync`, watch `.gitignore` update itself, watch your next clone get faster.
## To take advantage of v0.22.11
1. Add a `storage:` section to `gbrain.yml` at your brain repo root with `db_tracked` and `db_only` arrays. The directory paths must end with `/` (the validator auto-normalizes if you forget, with a one-time info note).
2. Run `gbrain sync`. It updates `.gitignore` automatically on success.
3. Run `gbrain storage status` to see the tier breakdown and any missing `db_only` files.
4. If files are missing on disk (e.g., after a container restart): `gbrain export --restore-only --repo /path/to/brain`.
5. If you previously had `git_tracked` / `supabase_only` keys: they still load, with a once-per-process deprecation warning. Rename to `db_tracked` / `db_only` at your convenience.
6. On PGLite: tiering has limited effect (the "DB" is your local file). The `.gitignore` housekeeping still helps. A one-time soft-warn explains.
If anything looks off, file an issue at <https://github.com/garrytan/gbrain/issues> with `gbrain doctor` output and the contents of your `gbrain.yml`.
### Itemized changes
#### Critical fixes
- **YAML parser swap**: replaced `gray-matter` with a dedicated YAML reader for the `gbrain.yml` shape. The original code called `matter()` on a delimiter-less file, which always returned `{data: {}}``loadStorageConfig` returned null on every install. The dedicated parser handles top-level `storage:` plus nested array-valued keys, with comment + blank-line tolerance. Once-per-process sanity warning when `gbrain.yml` exists but has no `storage:` section.
- **`manageGitignore` actually runs now**: wired into `runSync` after every successful sync (skipped on dry-run, blocked-by-failures, and unhandled errors). Idempotent. Detects git submodule context (`.git` is a file, not a directory) and skips with an actionable warning. Honors `GBRAIN_NO_GITIGNORE=1` for shared-repo setups.
- **No more silent `--restore-only` footgun**: `gbrain export --restore-only` without `--repo` now resolves through a typed `getDefaultSourcePath()` accessor (sources table → null → hard error). Never falls through to the current directory. Never silently re-exports your entire database into the wrong place.
#### New + renamed surface
- **Canonical key names**: `db_tracked` / `db_only` replace the vendor-baked `git_tracked` / `supabase_only`. The deprecated keys still load, with a once-per-process warning suggesting `gbrain doctor --fix` for an automated rename. Canonical wins when both shapes coexist.
- **Engine-side `slugPrefix` filter**: `PageFilters.slugPrefix` lands on both engines as `WHERE slug LIKE prefix || '%'` with literal-escape of LIKE metacharacters. Uses the existing `(source_id, slug)` UNIQUE btree index for range scans. Powers `gbrain export --restore-only` per-tier queries and `gbrain export --slug-prefix`.
- **Single-walk filesystem scan**: `src/core/disk-walk.ts` exposes `walkBrainRepo(repoPath)` that returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Replaces the per-page `existsSync + statSync` loop in `gbrain storage status` (~400K syscalls on a 200K-page brain → tens).
- **Path-segment matching**: tier directory matcher requires trailing `/` and treats the slash as a path separator. `media/x/` does not match `media/xerox/foo`. Validator (`normalizeAndValidateStorageConfig`) auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap.
#### Architecture cleanup
- `src/commands/storage.ts` split into pure data + JSON formatter + human formatter + thin dispatcher, matching the `orphans.ts` precedent. `getStorageStatus` is exported for `gbrain doctor` integration. ASCII-only output (no unicode box-drawing) for cross-platform terminal compatibility.
- Distinct nominal types `PageCountsByTier` and `DiskUsageByTier` so accidental swaps between page counts and byte totals are compile-time errors.
- PGLite soft-warn on storage tiering (D4): the feature is partial on PGLite (the "DB" is your local file), but `.gitignore` housekeeping still helps. Once-per-process warning explains and proceeds.
#### Tests + CI guards
- New unit tests across `test/storage-config.test.ts`, `test/storage-sync.test.ts`, `test/storage-status.test.ts`, `test/storage-export.test.ts`, `test/storage-pglite.test.ts`, `test/disk-walk.test.ts`. Plus extensions to `test/source-resolver.test.ts` and `test/pglite-engine.test.ts`. The single-line test that would have caught the original gray-matter P0 (write a real `gbrain.yml`, call `loadStorageConfig`, assert non-null) now exists.
- New CI guard `scripts/check-trailing-newline.sh` (sibling to the existing jsonb-pattern + progress-to-stdout guards). Wired into `bun run test`. Fixed pre-existing missing newline in `docs/storage-tiering.md`.
### For contributors
- The eng-review path forward is documented in `~/.claude/plans/lets-take-a-look-ticklish-pizza.md` (15 numbered defects + D1-D8 abstraction calls). Every commit on this branch maps to one numbered step in the plan.
## [0.22.10] - 2026-04-30
**`gbrain jobs submit autopilot-cycle --params '{"phases":["lint","backlinks"]}'` now actually runs only those phases.**
If you ever submitted an `autopilot-cycle` job with a `phases:` array hoping to skip embed for a fast cycle, you got the full 6-phase cycle anyway. The handler in `src/commands/jobs.ts` was calling `runCycle(...)` without forwarding `job.data.phases`, so per-cycle phase selection was silently ignored.
This release wires the array through. The handler imports `ALL_PHASES` from `src/core/cycle.ts`, builds a `Set` for O(1) validation, and filters the caller's `phases` array against it before forwarding to `runCycle`. Invalid phase names get dropped (no injection surface — `ALL_PHASES` is the authoritative list). Empty arrays and non-array values fall back to the default (run all phases), preserving the prior behavior for callers who didn't ask for selective phases.
### What this means for you
If you've been using `gbrain jobs submit autopilot-cycle --params '{"phases":[...]}'` for triage cycles (e.g. `["lint","backlinks"]` for a fast structural sweep, skipping the slow embed phase), you'll now see those cycles take seconds instead of minutes. The CLI surface didn't change — only the worker's handler now respects the `phases` it was already accepting.
### Itemized changes
#### Fixed
- `autopilot-cycle` minion handler in `src/commands/jobs.ts` now forwards `job.data.phases` to `runCycle()`. Previously the handler accepted the array via `MinionJobInput.params` but discarded it before dispatch.
- Phase names validated against `ALL_PHASES` from `src/core/cycle.ts`. Filter is exhaustive: array → filtered, non-array → undefined (default), filtered-to-empty → no `phases` key in opts (also default).
#### Tests
- 4 new test cases in `test/handlers.test.ts` under `autopilot-cycle handler — phase passthrough`: valid phases forwarded, invalid names filtered, empty array falls back to all-phases, non-array `phases` value ignored. Pin both the contract and the fallback semantics.
- `test/cycle-abort.test.ts` regression-guard window widened from 500 → 2000 chars so the source-level `signal: job.signal` check finds the line after the new validation block was added between `worker.register('autopilot-cycle', ...)` and the `runCycle(...)` call. Pure test fix; the handler still propagates the abort signal correctly.
## [0.22.9] - 2026-04-29
**Sync failures now tell you why, not just how many.**
**`gbrain sync --skip-failed` and `gbrain doctor` group failures by error code, so 2,685 silent SLUG_MISMATCH files don't hide behind a single count.**
Before this release, when sync hit per-file parse errors the only signal was a number:
```
Sync blocked: 2688 file(s) failed to parse. Fix the YAML frontmatter...
```
That count is useless when you're staring at 2,688 files and don't know what's wrong. On a real 81K-page brain, 2,685 of those turned out to be `SLUG_MISMATCH` from a posterous import — a single root cause hiding behind a giant number. It took manual `cat ~/.gbrain/sync-failures.jsonl | jq` to figure that out.
After:
```
Sync blocked: 2688 file(s) failed to parse:
SLUG_MISMATCH: 2685
YAML_DUPLICATE_KEY: 3
Fix the YAML frontmatter in the files above and re-run, or use 'gbrain sync --skip-failed' to acknowledge and move on.
# gbrain sync --skip-failed
Acknowledged 2688 failure(s) and advancing past them:
SLUG_MISMATCH: 2685
YAML_DUPLICATE_KEY: 3
```
`gbrain doctor` shows the same breakdown for unacknowledged AND historical entries:
```
[WARN] sync_failures: 2688 unacknowledged sync failure(s) [SLUG_MISMATCH=2685, YAML_DUPLICATE_KEY=3].
[OK] sync_failures: 500544 historical sync failure(s), all acknowledged [SLUG_MISMATCH=2685, ...].
```
The classifier knows the canonical messages from `collectValidationErrors()` in `src/core/markdown.ts` (8 frontmatter codes), Postgres unique-constraint violations (`DB_DUPLICATE_KEY`), statement-timeout errors (`STATEMENT_TIMEOUT`), invalid UTF-8, and YAML duplicates. DB-layer errors check before YAML-layer ones — so a Postgres `duplicate key value violates unique constraint` no longer mislabels as a YAML duplicate. Unrecognized errors fall through to `UNKNOWN`.
### What this means for you
If `gbrain sync` blocks with parse failures, the breakdown tells you what to fix first. SLUG_MISMATCH is one fix-pattern (frontmatter says one slug, path says another); YAML_PARSE is a different one (malformed YAML); STATEMENT_TIMEOUT means a DB timeout, not a parse problem. You stop staring at counts and start fixing root causes.
### For contributors
`acknowledgeSyncFailures()` in `src/core/sync.ts` now returns `{count, summary}` instead of `number`. If you import this directly from `gbrain/sync`, replace `n` with `result.count` and use `result.summary` (an `Array<{code, count}>`) for the new code-grouped breakdown. The function is reachable via the package exports map; this is a deliberate, non-shimmed breaking change. There is a new `formatCodeBreakdown()` helper in the same module that accepts either raw failures or pre-summarized input — use it instead of building breakdown strings inline.
### Itemized changes
#### Added
- `classifyErrorCode(errorMsg)` in `src/core/sync.ts` — best-effort error-code extraction from sync failure messages. Codes: `SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `NESTED_QUOTES`, `DB_DUPLICATE_KEY`, `STATEMENT_TIMEOUT`, `INVALID_UTF8`, `UNKNOWN`.
- `summarizeFailuresByCode(failures)` — groups failures by code and returns a sorted `Array<{code, count}>`.
- `formatCodeBreakdown(input)` — renders a multi-line `code: count` string from either raw failures or a pre-computed summary. Single helper, two input shapes.
- `code?: string` field on the `SyncFailure` JSONL row in `~/.gbrain/sync-failures.jsonl`. Populated at write-time so the classifier runs once per failure, not on every load.
- `AcknowledgeResult` interface as the new return shape of `acknowledgeSyncFailures()`.
- 15 new test cases in `test/sync-failures.test.ts`: DB-vs-YAML duplicate-key disambiguation, canonical-message coverage for all 7 frontmatter codes, `acknowledgeSyncFailures()` legacy-entry backfill branch, `formatCodeBreakdown()` dual-input shape.
#### Changed
- `gbrain sync` blocked-message: now lists code breakdown above the fix instructions (both incremental and full-sync paths).
- `gbrain sync --skip-failed` ack message: now lists what was skipped, grouped by code.
- `gbrain doctor` `sync_failures` check: warn-and-ok messages both include `[code=count, ...]` breakdown.
- `recordSyncFailures()` now stores `code` alongside `error` so downstream readers don't re-classify.
- `acknowledgeSyncFailures()` backfills `code` on legacy rows that predate the field — upgrade-safe for users with existing `~/.gbrain/sync-failures.jsonl`.
- DB-layer error patterns (`DB_DUPLICATE_KEY`, `STATEMENT_TIMEOUT`) check BEFORE YAML patterns in the classifier, so Postgres errors don't get YAML-labeled.
- Frontmatter regex patterns rewritten to match canonical messages from `collectValidationErrors()` (`File is empty...`, `No closing --- delimiter found`, `Frontmatter block is empty`) instead of aspirational code-token strings (`missing.*open`) that never appeared in practice.
Closes #500.
## [0.22.8] - 2026-04-28
## **Doctor stops timing out on Supabase. Integrity scan finishes in ~6s, multi-source brains get correct counts.**
@@ -150,8 +443,6 @@ Then point Claude Desktop, claude.ai/code, or any MCP client at `http://your-tun
If anything breaks: `gbrain doctor`, `~/.gbrain/upgrade-errors.jsonl` (if present), and please file an issue at https://github.com/garrytan/gbrain/issues with both.
## [0.22.6.1] - 2026-04-26
**Old brains can upgrade again.**
+19 -3
View File
@@ -32,8 +32,12 @@ strict behavior when unset.
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
@@ -88,7 +92,7 @@ strict behavior when unset.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts``gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
- `src/commands/jobs.ts``gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts``gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts``gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
@@ -101,10 +105,13 @@ strict behavior when unset.
- `src/commands/repair-jsonb.ts``gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts``gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/integrity.ts``gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts``gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
@@ -216,6 +223,10 @@ Key commands added in v0.14.3 (fix wave):
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
Key commands added in v0.22.13 (PR #490):
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
## Testing
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
@@ -271,6 +282,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases),
@@ -291,6 +305,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
- `test/e2e/integrity-batch.test.ts` (v0.22.8) — parity tests for `scanIntegrity`'s batch-load fast path vs sequential. Four cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins the codex-caught multi-source overcounting regression.
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
- `test/e2e/sync.test.ts` (v0.22.12 — `--skip-failed` failure-loop test, alongside the existing 13 happy-path tests): exercises the full chain — broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic on a developer machine. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format. This is the integration test that proves the v0.22.12 chain holds together — unit tests cover the pure functions in isolation, this covers the integration.
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
@@ -299,6 +314,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
+29 -2
View File
@@ -360,6 +360,30 @@ accumulate rows across separate single-skill installs instead of overwriting eac
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
and the anti-patterns it catches.
## Storage tiering: keep bulk content out of git (v0.22.11)
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
becomes the size driver, declare which directories belong in git and which live in the database only.
```yaml
# gbrain.yml at the brain repo root
storage:
db_tracked:
- people/
- companies/
- deals/
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
repopulates missing files from the database (container restart, fresh clone, accidental rm).
`gbrain storage status` shows the tier breakdown.
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
## Getting Data In
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
@@ -615,8 +639,11 @@ SEARCH
gbrain query <question> Hybrid search (vector + keyword + RRF)
IMPORT
gbrain import <dir> [--no-embed] Import markdown (idempotent)
gbrain sync [--repo <path>] Git-to-brain incremental sync
gbrain import <dir> [--no-embed] [--workers N]
Import markdown (idempotent)
gbrain sync [--repo <path>] [--workers N]
Git-to-brain incremental sync
(>100-file diffs auto-parallelize 4 workers on Postgres)
gbrain export [--dir ./out/] Export to markdown
FILES
+129
View File
@@ -1,5 +1,134 @@
# TODOS
## sync (v0.22.13 follow-up — PR #490 review)
### D-PR490-1 — Plumb resolved `database_url` through `SyncOpts`
**Priority:** P3
**What:** Add `database_url?: string` (or a richer `resolvedConnection` shape) to
`SyncOpts` and have the caller (`runSync`, the cycle handler, the jobs handler)
populate it from the active engine instead of having `performSync` /
`performFullSync` / `import.ts` each call `loadConfig()` separately. Today every
sync run hits the config file three times.
**Why:** v0.18 multi-source brains can in principle run different sources against
different `database_url` endpoints (or different per-source overrides via
`sources.config_jsonb`). Right now `loadConfig()` returns the global config, and
that always matches the engine in practice — but the convention papers over a
real divergence the moment someone wants per-source connection settings. Folding
the resolution into `SyncOpts` makes the worker-engine creation in `sync.ts` and
`import.ts` deterministic from `SyncOpts` alone.
**Pros:**
- Removes 3 redundant `loadConfig()` calls per sync.
- Makes `performSync` / `performFullSync` side-effect-free with respect to the
on-disk config file.
- Sets up for per-source `database_url` overrides without further refactor.
- Makes the v0.22.13 belt-and-suspenders fallback (PR #490 Q3) cleaner — no
more `!config?.database_url` short-circuit inside the parallel branch.
**Cons:**
- API-shape change to `SyncOpts` (mild; not externally exported).
- Touching three callers (`runSync`, jobs handler, `cycle.ts` `runPhaseSync`).
- Only worth doing when paired with a per-source override story; otherwise
it's just plumbing.
**Context:** Surfaced during the PR #490 plan-eng-review (parallel sync).
Deferred because it isn't on the v0.22.13 critical path. The same pattern would
benefit the cycle handler and the autopilot daemon. See the plan-eng-review
decisions log: A4 = "Defer; file as TODO."
**Depends on / blocked by:** Nothing structural. Best paired with the v0.18
per-source `config_jsonb` work if/when that lands.
## sync error-code classification (PR #501 follow-ups)
### Plumb structured `ParseValidationCode` through `ImportResult`
**Priority:** P2
**What:** Replace the regex-on-error-message path in `src/core/sync.ts:classifyErrorCode`
with a structured `code` field threaded through `ImportResult` from the parse layer.
Three changes:
1. `src/core/import-file.ts:362` — call `parseMarkdown(content, relativePath, { validate: true, expectedSlug })`
so `parsed.errors[0].code` is populated.
2. `src/core/import-file.ts` — add `code?: string` to `ImportResult`. Promote the
structured code (or `'SLUG_MISMATCH'` when the existing expectedSlug check trips)
into the result envelope alongside `error`.
3. `src/commands/sync.ts:488` — extend `failedFiles` shape with `code?: string`.
`recordSyncFailures` already accepts the field; the only thing missing is the
capture site populating it.
4. `src/core/sync.ts:classifyErrorCode` — keep as a fallback for un-coded errors
(DB exceptions, generic catches). Primary path reads the structured code.
**Why:** The repo already has `ParseValidationCode` + `ParseValidationError` in
`src/core/markdown.ts:5-18`, and three other consumers (`src/commands/lint.ts:72`,
`src/commands/frontmatter.ts:148`, `src/core/brain-writer.ts:314`) read structured
errors directly. Sync is the outlier — it calls `parseMarkdown` without validation
and reverse-engineers codes via regex. PR #501 shipped that regex out of pragmatism;
this TODO removes ~50% of `classifyErrorCode` and eliminates a class of false-positives.
**Pros:**
- One source of truth for parse codes (the enum in `markdown.ts`).
- Eliminates regex fragility — adding a new validation code in `markdown.ts`
automatically flows to sync without a new regex.
- Closes the case where canonical messages (`File is empty...`, `No closing ---...`)
don't match aspirational regex patterns.
**Cons:** Touches `ImportResult` interface, which ripples through `src/commands/import.ts:105`,
`src/commands/sync.ts:498-510`, `src/core/cycle.ts`, brain-writer reconciler.
**Context:** PR #501 documented this as P3 in the eng review at
`~/.claude/plans/then-codex-synchronous-toucan.md`. Codex's outside-voice review
agreed independently. The fix is small — ~50 lines including tests + downstream
call sites — and it's the correct architectural endpoint.
**Effort:** M (human: ~2 hr / CC: ~20 min).
**Depends on / blocked by:** Nothing.
### CHANGELOG migration note for `acknowledgeSyncFailures()` shape change
**Priority:** P0 — required at /ship time
**What:** When PR #501 ships, the release CHANGELOG entry MUST include this
`### For contributors` block:
```markdown
### For contributors
`acknowledgeSyncFailures()` now returns `{count, summary}` instead of `number`.
If you import this directly from `gbrain/sync`, replace `n` with `result.count`
and use `result.summary` for the new code-grouped breakdown.
```
**Why:** The function is exported from `src/core/sync.ts:433` and reachable via
the package exports map. External TS consumers (gbrain-evals, host agent forks)
that imported it got `number` and now get an object — silent type break.
**Effort:** XS (human: ~1 min). Just don't forget.
**Depends on / blocked by:** PR #501 ship.
### Concurrent-safe ack of `~/.gbrain/sync-failures.jsonl`
**Priority:** P3
**What:** Two concurrent `gbrain sync` runs hitting `acknowledgeSyncFailures()`
can clobber each other. The function does a whole-file `writeFileSync` rewrite
(`src/core/sync.ts:433-455`); `recordSyncFailures()` does independent
`appendFileSync` (`src/core/sync.ts:395-416`). Concurrent ack + append can lose rows.
**Why:** Pre-existing — predates PR #501. Real risk only on autopilot setups where
multiple sync invocations might overlap (rare today, more likely as multi-source
sync matures).
**Fix sketch:** Atomic rename pattern (write to `sync-failures.jsonl.tmp`, then
`renameSync`) plus a file lock for the read-modify-write cycle. Or move the
acknowledged-set to the DB.
**Effort:** S (human: ~1 hr / CC: ~10 min).
**Depends on / blocked by:** Nothing.
## test-infra
### Parallel-load timeout flake on v0.21 PGLite-heavy tests
+1 -1
View File
@@ -1 +1 @@
0.22.8
0.22.13
+13 -10
View File
@@ -20,6 +20,7 @@
},
"devDependencies": {
"@types/bun": "latest",
"bun-types": "^1.3.13",
"typescript": "^5.6.0",
},
},
@@ -220,7 +221,7 @@
"@types/bun": ["@types/bun@1.3.11", "", { "dependencies": { "bun-types": "1.3.11" } }, "sha512-5vPne5QvtpjGpsGYXiFyycfpDF2ECyPcTSsFBMa0fraoxiQyMJ3SmuQIGhzPg2WJuWxVBoxWJ2kClYTcw/4fAg=="],
"@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"@types/node-fetch": ["@types/node-fetch@2.6.13", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],
@@ -242,7 +243,7 @@
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
"bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"bun-types": ["bun-types@1.3.13", "", { "dependencies": { "@types/node": "*" } }, "sha512-QXKeHLlOLqQX9LgYaHJfzdBaV21T63HhFJnvuRCcjZiaUDpbs5ED1MgxbMra71CsryN/1dAoXuJJJwIv/2drVA=="],
"bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="],
@@ -466,7 +467,7 @@
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
"undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
@@ -488,30 +489,32 @@
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
"@anthropic-ai/sdk/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@aws-crypto/util/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="],
"@types/node-fetch/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"bun-types/@types/node": ["@types/node@25.5.2", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-tO4ZIRKNC+MDWV4qKVZe3Ql/woTnmHDr5JD8UI5hn2pwBrHEwOEMZK7WlNb5RKB6EoJ02gwmQS9OrjuFnZYdpg=="],
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@anthropic-ai/sdk/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="],
"@types/node-fetch/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"bun-types/@types/node/undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="],
"form-data/mime-types/mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
"openai/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
"@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
"@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="],
+210
View File
@@ -0,0 +1,210 @@
# Storage Tiering: db-tracked vs db-only directories
## Overview
GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.
> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning. Run `gbrain doctor --fix` for an automated rename when that path lands.
## Configuration
Add a `storage` section to your `gbrain.yml` file in the brain repository root:
```yaml
storage:
# Directories that are version-controlled (human-edited, committed to git).
db_tracked:
- people/
- companies/
- deals/
- concepts/
- yc/
- ideas/
- projects/
# Directories persisted via the brain database only (bulk machine-generated
# content). Written to disk as a local cache but not committed to git;
# `gbrain sync` auto-manages .gitignore for these paths. `gbrain export
# --restore-only` repopulates missing files from the database.
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
Path requirements:
- Each directory must end with `/` for canonical form. The validator auto-normalizes missing trailing slashes (one-time info note shows what changed).
- A directory cannot appear in both tiers — that's a tier-overlap error and `loadStorageConfig` throws `StorageConfigError`. Edit `gbrain.yml` to remove the overlap and try again.
## Behavior Changes
### 1. `gbrain sync` — automatic .gitignore management
When storage configuration is present, `gbrain sync` automatically manages `.gitignore` entries on every successful sync:
- Adds missing `db_only` directory patterns to `.gitignore`.
- Idempotent — re-running adds no duplicate entries.
- Stable comment header so the managed block is grep-able.
- Skipped on `--dry-run` (don't mutate disk in preview mode).
- Skipped on `blocked_by_failures` status (sync state is inconsistent).
- Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
- Skipped entirely when `GBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone).
- Failures (write permission denied, etc.) are caught and logged, never crash sync.
Example `.gitignore` addition:
```gitignore
# Auto-managed by gbrain (db_only directories)
media/x/
media/articles/
meetings/transcripts/
```
### 2. `gbrain export --restore-only` — repopulate missing db_only files
```bash
# Restore only missing db_only files from the database.
gbrain export --restore-only --repo /path/to/brain
# Filter by page type.
gbrain export --restore-only --type media --repo /path/to/brain
# Filter by slug prefix.
gbrain export --restore-only --slug-prefix media/x/ --repo /path/to/brain
# Combine filters.
gbrain export --restore-only --type media --slug-prefix media/x/ --repo /path/to/brain
```
The `--restore-only` flag:
- Resolves repoPath via the chain `--repo` → typed `sources.getDefault()` → hard error.
Never falls through to the current directory.
- Only exports pages that match `db_only` patterns AND are missing from disk.
- Ideal for container restart recovery and fresh clones.
### 3. `gbrain storage status` — storage-tier health dashboard
```bash
# Human-readable status.
gbrain storage status --repo /path/to/brain
# JSON output for scripts and orchestrators.
gbrain storage status --repo /path/to/brain --json
```
Output includes:
- Total page counts by storage tier.
- Disk usage breakdown by tier.
- Missing files that need restoration (top 10 shown; full list in `--json`).
- Configuration validation warnings.
- Current tier directory listing.
Example output:
```
Storage Status
==============
Repository: /data/brain
Total pages: 15,243
Storage Tiers:
-------------
DB tracked: 2,156 pages
DB only: 12,887 pages
Unspecified: 200 pages
Disk Usage:
-----------
DB tracked: 45.2 MB
DB only: 2.1 GB
Missing Files (need restore):
-----------------------------
media/x/tweet-1234567890
media/x/tweet-0987654321
... and 47 more
Use: gbrain export --restore-only --repo "/data/brain"
Configuration:
--------------
DB tracked directories:
- people/
- companies/
- deals/
DB-only directories:
- media/x/
- media/articles/
- meetings/transcripts/
```
## Validation
`loadStorageConfig` runs `normalizeAndValidateStorageConfig` after parsing:
- Auto-fixes (silent, with one-time info note showing what changed):
- Missing trailing `/` is added: `'media/x'``'media/x/'`.
- Throws `StorageConfigError` (caller sees a clean exit-1 with actionable message):
- Same directory in both `db_tracked` and `db_only` (ambiguous routing).
## Use cases
### Brain repository scaling
Perfect for brain repositories crossing 50K-200K+ files where:
- Core knowledge (people, companies, deals) remains git-tracked.
- Bulk data (tweets, articles, transcripts) moves to db_only.
- Development stays fast with smaller git repos.
- Full data remains available via the database.
### Container-based deployments
Essential for ephemeral container environments:
- Git repo contains only essential files.
- Container restarts don't lose db_only data.
- `gbrain export --restore-only` quickly restores bulk files when needed.
- Local disk acts as a cache layer.
### Multi-environment consistency
Enables consistent data access across environments:
- Development: small git clone, restore bulk data on demand.
- Production: full dataset via the database, selective local caching.
- CI/CD: fast tests with git-tracked data only.
## Migration strategy
1. **Assess current repository**: use `gbrain storage status` to understand current distribution.
2. **Plan directory structure**: identify which directories should be db_tracked vs db_only.
3. **Create `gbrain.yml`**: add storage configuration to the repository root.
4. **Test with dry-run**: `gbrain sync --dry-run` to verify behavior; `.gitignore` is NOT touched on dry-run.
5. **Run a real sync**: `gbrain sync` updates `.gitignore` automatically on success.
6. **Verify restore**: test `gbrain export --restore-only --repo .` against a small db_only directory.
## Best practices
- **Directory naming**: end storage paths with `/` (canonical form). The validator normalizes if you forget.
- **Start small**: begin with clearly machine-generated directories in `db_only`.
- **Address validation errors**: tier overlap is an error, not a warning. Fix it before sync.
- **Test restore**: regularly test `--restore-only` in staging environments.
- **Document decisions**: comment your `gbrain.yml` to explain tier choices.
## PGLite engine note
On the PGLite engine (gbrain's local-only embedded Postgres), the "DB" your db_only pages live in IS the local file gbrain uses for everything else. The `.gitignore` housekeeping still helps (keeps bulk content out of git history), but the offload-to-DB promise is technically vacuous. A once-per-process soft-warn explains when the engine is detected. To get full tiering, migrate to Postgres with `gbrain migrate --to supabase`.
## Compatibility
- **Backward compatible**: systems without `gbrain.yml` work unchanged.
- **Progressive enhancement**: add configuration when needed.
- **Database unchanged**: all data remains in Postgres regardless of tier.
- **Existing workflows**: all existing `sync` and `export` behavior preserved.
- **Deprecated keys**: `git_tracked` / `supabase_only` still load with a once-per-process warning.
+18
View File
@@ -0,0 +1,18 @@
storage:
# Directories that are version-controlled — human-curated, edited by hand.
db_tracked:
- people/
- companies/
- deals/
- concepts/
- yc/
- ideas/
- projects/
# Directories persisted via the brain database only — bulk machine-generated
# content. .gitignored automatically by `gbrain sync`. Restorable from the DB
# via `gbrain export --restore-only`.
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
+48 -5
View File
@@ -111,8 +111,12 @@ strict behavior when unset.
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
@@ -167,7 +171,7 @@ strict behavior when unset.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase).
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path.
@@ -180,10 +184,13 @@ strict behavior when unset.
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2).
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
@@ -295,6 +302,10 @@ Key commands added in v0.14.3 (fix wave):
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
Key commands added in v0.22.13 (PR #490):
- `gbrain sync --workers N` (alias `--concurrency N`) — parallelize the import phase using per-worker Postgres engines (small pool of 2 each) with an atomic queue index. Auto-concurrency: defaults to 4 workers when the diff exceeds 100 files. Smaller diffs stay serial. Explicit `--workers` always wins (even on a 30-file diff). PGLite forces serial regardless. Validation rejects `0`, negatives, non-integers loud (replaces the prior silent fall-through to auto-concurrency).
- `gbrain import --workers N` — same `parseWorkers()` validation as sync; same try/finally worker-engine cleanup. Behavior surface unchanged.
## Testing
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
@@ -350,6 +361,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
`test/sync-concurrency.test.ts` (v0.22.13 PR #490: 17 cases covering `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping, `shouldRunParallel()` Q1 explicit-bypasses-floor contract, and `parseWorkers()` validation that rejects `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars),
`test/sync-parallel.test.ts` (v0.22.13 PR #490: PGLite-routed coverage of the bookmark gate under concurrency request, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract — 7 cases),
`test/sync-failures.test.ts` (v0.22.12: 28 cases pinning `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts:159-244` and `import-file.ts:199, 347, 352, 401`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` AcknowledgeResult shape + backfill on pre-v0.22.12 entries),
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases),
@@ -370,6 +384,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
- `test/e2e/integrity-batch.test.ts` (v0.22.8) — parity tests for `scanIntegrity`'s batch-load fast path vs sequential. Four cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins the codex-caught multi-source overcounting regression.
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
- `test/e2e/sync.test.ts` (v0.22.12 — `--skip-failed` failure-loop test, alongside the existing 13 happy-path tests): exercises the full chain — broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic on a developer machine. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format. This is the integration test that proves the v0.22.12 chain holds together — unit tests cover the pure functions in isolation, this covers the integration.
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
@@ -378,6 +393,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review.
- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset.
- `test/e2e/sync-parallel.test.ts` (v0.22.13 PR #490) — DATABASE_URL-gated. T2: 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). P4: 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx` for CHANGELOG quoting. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
@@ -1638,6 +1654,30 @@ accumulate rows across separate single-skill installs instead of overwriting eac
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
and the anti-patterns it catches.
## Storage tiering: keep bulk content out of git (v0.22.11)
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
becomes the size driver, declare which directories belong in git and which live in the database only.
```yaml
# gbrain.yml at the brain repo root
storage:
db_tracked:
- people/
- companies/
- deals/
db_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
repopulates missing files from the database (container restart, fresh clone, accidental rm).
`gbrain storage status` shows the tier breakdown.
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
## Getting Data In
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
@@ -1893,8 +1933,11 @@ SEARCH
gbrain query <question> Hybrid search (vector + keyword + RRF)
IMPORT
gbrain import <dir> [--no-embed] Import markdown (idempotent)
gbrain sync [--repo <path>] Git-to-brain incremental sync
gbrain import <dir> [--no-embed] [--workers N]
Import markdown (idempotent)
gbrain sync [--repo <path>] [--workers N]
Git-to-brain incremental sync
(>100-file diffs auto-parallelize 4 workers on Postgres)
gbrain export [--dir ./out/] Export to markdown
FILES
+4 -2
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.22.8",
"version": "0.22.13",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -32,8 +32,9 @@
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
"build:schema": "bash scripts/build-schema.sh",
"build:llms": "bun run scripts/build-llms.ts",
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
@@ -63,6 +64,7 @@
},
"devDependencies": {
"@types/bun": "latest",
"bun-types": "^1.3.13",
"typescript": "^5.6.0"
},
"trustedDependencies": [
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# CI guard: every text file under src/, test/, and the repo root .yml/.md
# files must end with a newline. POSIX-noncompliant trailing data shows up
# as a phantom diff on every future edit and trips most linters.
#
# Sibling to scripts/check-progress-to-stdout.sh and
# scripts/check-jsonb-pattern.sh per CLAUDE.md's CI guard pattern.
# Wired into `bun run test` via package.json's `test` script.
set -euo pipefail
# Files to check: anything tracked under src/ + test/ that's a code/text file.
# Also the top-level *.yml + *.md the repo controls. Portable to bash 3.2
# (macOS default) — no mapfile, no associative arrays.
files=$(
git ls-files \
'src/**/*.ts' 'src/**/*.js' 'src/**/*.json' 'src/**/*.sql' 'src/**/*.md' \
'test/**/*.ts' 'test/**/*.js' 'test/**/*.json' 'test/**/*.md' \
'gbrain.yml' '*.md' \
2>/dev/null | sort -u
)
missing=""
total=0
while IFS= read -r f; do
[ -n "$f" ] || continue
[ -f "$f" ] || continue
[ -s "$f" ] || continue
total=$((total + 1))
if [ -n "$(tail -c 1 "$f")" ]; then
missing="${missing} $f"$'\n'
fi
done <<< "$files"
if [ -n "$missing" ]; then
echo "ERROR: the following files are missing a trailing newline:" >&2
printf '%s' "$missing" >&2
echo >&2
echo "Fix: append a newline. e.g. \`printf '\\n' >> <file>\` or your editor's" >&2
echo "'final newline' setting (most editors do this automatically)." >&2
exit 1
fi
echo "trailing-newline check: ok ($total files)"
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Partition unit test files into N shards by stable hash and run one shard.
#
# Usage: scripts/test-shard.sh <shard-index> <total-shards>
# shard-index: 1-based (1..N)
# total-shards: positive integer
#
# E2E tests under test/e2e/ are excluded — they need DATABASE_URL and run via
# bun run test:e2e separately.
#
# Stable partitioning: a file's shard is `(hash(path) % N) + 1`. Same file
# lands in the same shard on every run, regardless of how many other files
# exist, so retries are reproducible. Hash is FNV-1a — pure shell, no jq.
set -euo pipefail
if [ "$#" -ne 2 ]; then
echo "usage: scripts/test-shard.sh <shard-index> <total-shards>" >&2
exit 1
fi
SHARD_INDEX="$1"
TOTAL_SHARDS="$2"
if ! [[ "$SHARD_INDEX" =~ ^[0-9]+$ ]] || ! [[ "$TOTAL_SHARDS" =~ ^[0-9]+$ ]]; then
echo "error: shard index and total must be positive integers" >&2
exit 1
fi
if [ "$SHARD_INDEX" -lt 1 ] || [ "$SHARD_INDEX" -gt "$TOTAL_SHARDS" ]; then
echo "error: shard index $SHARD_INDEX out of range 1..$TOTAL_SHARDS" >&2
exit 1
fi
cd "$(dirname "$0")/.."
# Find all unit test files, deterministic order. Excludes test/e2e/.
# Portable: avoid `mapfile` (bash 4+) so this runs on macOS bash 3.2 too.
FILES=()
while IFS= read -r line; do
FILES+=("$line")
done < <(find test -name '*.test.ts' -not -path 'test/e2e/*' | sort)
if [ "${#FILES[@]}" -eq 0 ]; then
echo "no test files found under test/" >&2
exit 1
fi
# FNV-1a 32-bit hash of a string — implemented in pure bash so we don't depend
# on python/openssl/etc on the runner. Output is decimal.
fnv1a() {
local str="$1"
local h=2166136261 # FNV offset basis
local i ord
for (( i=0; i<${#str}; i++ )); do
ord=$(printf '%d' "'${str:$i:1}")
h=$(( (h ^ ord) & 0xFFFFFFFF ))
h=$(( (h * 16777619) & 0xFFFFFFFF ))
done
echo "$h"
}
SHARD_FILES=()
for f in "${FILES[@]}"; do
hash=$(fnv1a "$f")
bucket=$(( hash % TOTAL_SHARDS + 1 ))
if [ "$bucket" -eq "$SHARD_INDEX" ]; then
SHARD_FILES+=("$f")
fi
done
echo "shard $SHARD_INDEX/$TOTAL_SHARDS: ${#SHARD_FILES[@]}/${#FILES[@]} files"
if [ "${#SHARD_FILES[@]}" -eq 0 ]; then
echo "warning: shard $SHARD_INDEX has no files (rehash or reduce shard count)" >&2
exit 0
fi
exec bun test --timeout=60000 "${SHARD_FILES[@]}"
+10 -1
View File
@@ -19,7 +19,7 @@ for (const op of operations) {
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth']);
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth']);
async function main() {
// Parse global flags (--quiet / --progress-json / --progress-interval)
@@ -530,6 +530,11 @@ async function handleCliOnly(command: string, args: string[]) {
await runSources(engine, args);
break;
}
case 'storage': {
const { runStorage } = await import('./commands/storage.ts');
await runStorage(engine, args);
break;
}
case 'code-def': {
const { runCodeDef } = await import('./commands/code-def.ts');
await runCodeDef(engine, args);
@@ -645,6 +650,8 @@ IMPORT/EXPORT
sync --watch [--interval N] Continuous sync (loops until stopped)
sync --install-cron Install persistent sync daemon
export [--dir ./out/] Export to markdown
export --restore-only [--repo <p>] Restore missing supabase-only files
[--type T] [--slug-prefix S] With optional filters
FILES
files list [slug] List stored files
@@ -726,6 +733,8 @@ ADMIN
features [--json] [--auto-fix] Scan usage + recommend unused features
autopilot [--repo] [--interval N] Self-maintaining brain daemon
config [show|get|set] <key> [val] Brain config
storage status [--repo <path>] Storage tier status and health
[--json] (git-tracked vs supabase-only)
serve MCP server (stdio)
call <tool> '<json>' Raw tool invocation
version Version info
+8 -4
View File
@@ -249,25 +249,29 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
// Without this doctor check, users see "sync blocked" and have no
// surface showing which files to fix.
try {
const { unacknowledgedSyncFailures, loadSyncFailures } = await import('../core/sync.ts');
const { unacknowledgedSyncFailures, loadSyncFailures, summarizeFailuresByCode } = await import('../core/sync.ts');
const unacked = unacknowledgedSyncFailures();
const all = loadSyncFailures();
if (unacked.length > 0) {
const codeSummary = summarizeFailuresByCode(unacked);
const codeBreakdown = codeSummary.map(s => `${s.code}=${s.count}`).join(', ');
const preview = unacked.slice(0, 3).map(f => `${f.path} (${f.error.slice(0, 60)})`).join('; ');
checks.push({
name: 'sync_failures',
status: 'warn',
message:
`${unacked.length} unacknowledged sync failure(s). ${preview}` +
`${unacked.length} unacknowledged sync failure(s) [${codeBreakdown}]. ${preview}` +
`${unacked.length > 3 ? `, and ${unacked.length - 3} more` : ''}. ` +
`Fix the file(s) and re-run 'gbrain sync', or use 'gbrain sync --skip-failed' to acknowledge.`,
});
} else if (all.length > 0) {
// Acknowledged-only: informational, not a warning.
// Acknowledged-only: show code breakdown for visibility.
const ackedSummary = summarizeFailuresByCode(all);
const ackedBreakdown = ackedSummary.map(s => `${s.code}=${s.count}`).join(', ');
checks.push({
name: 'sync_failures',
status: 'ok',
message: `${all.length} historical sync failure(s), all acknowledged.`,
message: `${all.length} historical sync failure(s), all acknowledged [${ackedBreakdown}].`,
});
}
} catch {
+97 -4
View File
@@ -1,16 +1,105 @@
import { writeFileSync, mkdirSync } from 'fs';
import { writeFileSync, mkdirSync, existsSync } from 'fs';
import { join, dirname } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { serializeMarkdown } from '../core/markdown.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { loadStorageConfig, isDbOnly } from '../core/storage-config.ts';
import { getDefaultSourcePath } from '../core/source-resolver.ts';
import type { PageType } from '../core/types.ts';
export async function runExport(engine: BrainEngine, args: string[]) {
const dirIdx = args.indexOf('--dir');
const outDir = dirIdx !== -1 ? args[dirIdx + 1] : './export';
const pages = await engine.listPages({ limit: 100000 });
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
const repoIdx = args.indexOf('--repo');
const explicitRepoPath = repoIdx !== -1 ? args[repoIdx + 1] : null;
const typeIdx = args.indexOf('--type');
const typeFilter = typeIdx !== -1 ? (args[typeIdx + 1] as PageType) : undefined;
const slugPrefixIdx = args.indexOf('--slug-prefix');
const slugPrefix = slugPrefixIdx !== -1 ? args[slugPrefixIdx + 1] : undefined;
const restoreOnly = args.includes('--restore-only');
// Resolution chain (D5): explicit --repo → typed sources.getDefault() →
// hard-error for restore-only paths (never fall through to cwd).
// For non-restore exports, repoPath stays null because regular export
// doesn't need a brain repo to run (D26 — exports include everything).
let repoPath: string | null = explicitRepoPath;
if (restoreOnly && !repoPath) {
repoPath = await getDefaultSourcePath(engine);
if (!repoPath) {
console.error(
`Error: gbrain export --restore-only requires --repo <path> or a configured\n` +
`default source with a local_path. Run \`gbrain sources list\` to inspect\n` +
`sources, or pass --repo explicitly.`,
);
process.exit(1);
}
}
// Load storage configuration if repo path is provided
const storageConfig = repoPath ? loadStorageConfig(repoPath) : null;
// D5 + Codex P0: refuse --restore-only when there's no storage config to
// scope the restore. Without storageConfig, the selective filter (db_only
// pages missing on disk) can't run, and falling through to the full
// listPages export silently dumps the entire DB. Catch this before any
// page query fires.
if (restoreOnly && !storageConfig) {
console.error(
`Error: gbrain export --restore-only requires a storage tiering config\n` +
`(gbrain.yml with a "storage:" section) at ${repoPath}/gbrain.yml.\n` +
`Without it, there's nothing to scope the restore to.\n` +
`Run \`gbrain storage status\` to inspect the current configuration.`,
);
process.exit(1);
}
// Build filters. slugPrefix is engine-side (Issue #13) — no in-memory
// post-filter, no full-table load.
const filters: import('../core/types.ts').PageFilters = { limit: 100000 };
if (typeFilter) filters.type = typeFilter;
if (slugPrefix) filters.slugPrefix = slugPrefix;
let pages: import('../core/types.ts').Page[];
// Restore-only path: query each db_only directory with slugPrefix instead
// of loading every page in the brain. On a 200K-page brain where 95% is
// db_only, this is roughly the same load — but on brains where only 5K
// out of 200K are db_only, this is a ~40x reduction.
if (restoreOnly && repoPath && storageConfig) {
const seen = new Set<string>();
pages = [];
for (const dir of storageConfig.db_only) {
const tierFilters: import('../core/types.ts').PageFilters = {
...filters,
slugPrefix: filters.slugPrefix
? // If user passed --slug-prefix, only include tier dirs that start with it.
(dir.startsWith(filters.slugPrefix) ? dir : undefined)
: dir,
};
if (!tierFilters.slugPrefix) continue;
const tierPages = await engine.listPages(tierFilters);
for (const p of tierPages) {
if (seen.has(p.slug)) continue;
seen.add(p.slug);
if (!isDbOnly(p.slug, storageConfig)) continue; // belt-and-suspenders
const filePath = join(repoPath, p.slug + '.md');
if (existsSync(filePath)) continue;
pages.push(p);
}
}
} else {
pages = await engine.listPages(filters);
}
if (restoreOnly) {
console.log(`Restoring ${pages.length} db_only pages to ${outDir}/`);
} else {
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
}
// Progress on stderr so stdout stays clean for scripts parsing counts.
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
@@ -52,5 +141,9 @@ export async function runExport(engine: BrainEngine, args: string[]) {
progress.finish();
// Stdout summary preserved so scripts that grep for "Exported N pages" keep working.
console.log(`Exported ${exported} pages to ${outDir}/`);
if (restoreOnly) {
console.log(`Restored ${exported} pages to ${outDir}/`);
} else {
console.log(`Exported ${exported} pages to ${outDir}/`);
}
}
+55 -28
View File
@@ -34,7 +34,17 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
const jsonOutput = args.includes('--json');
const workersIdx = args.indexOf('--workers');
const workersArg = workersIdx !== -1 ? args[workersIdx + 1] : null;
const workerCount = workersArg ? parseInt(workersArg, 10) : 1;
// v0.22.13 (PR #490 Q2): shared parseWorkers helper rejects bad input
// (--workers 0, -3, "foo") with a loud error instead of silently falling
// through to 1. Mirrors sync.ts's flag handling.
const { parseWorkers } = await import('../core/sync-concurrency.ts');
let workerCount: number;
try {
workerCount = parseWorkers(workersArg ?? undefined) ?? 1;
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
}
// Find dir: first non-flag arg that isn't a value for --workers
const flagValues = new Set<number>();
if (workersIdx !== -1) flagValues.add(workersIdx + 1);
@@ -141,40 +151,57 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
}
if (actualWorkers > 1) {
// Parallel: create per-worker engine instances with small pool
// PGLite is single-connection, so parallel workers are only for Postgres
// v0.22.13 (PR #490 A1 + Q3): use engine.kind discriminator (not config.engine
// string sniff) and fall back to serial when database_url is unset. Both
// checks belt-and-suspenders so we never crash on a null assertion.
const config = loadConfig();
if (config?.engine === 'pglite') {
// PGLite: sequential import through single engine
if (engine.kind === 'pglite' || !config?.database_url) {
for (const file of files) {
await processFile(engine, file);
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerEngines = await Promise.all(
Array.from({ length: actualWorkers }, async () => {
const eng = new PostgresEngine();
await eng.connect({ database_url: config!.database_url!, poolSize: workerPoolSize });
return eng;
})
);
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const databaseUrl = config.database_url;
// Thread-safe queue: use an atomic index counter instead of array.shift()
let queueIndex = 0;
await Promise.all(workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= files.length) break;
await processFile(eng, files[idx]);
// v0.22.13 (PR #490 A2): connect workers serially so a partial failure
// leaves us with the connected ones already pushed onto workerEngines
// for the finally-block cleanup. The prior Promise.all could leak any
// engine that connected before another's connect() rejected.
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
try {
for (let i = 0; i < actualWorkers; i++) {
const eng = new PostgresEngine();
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
workerEngines.push(eng);
}
// Thread-safe queue: atomic index counter (JS is single-threaded; the
// read-then-increment happens between awaits so no lock is needed).
let queueIndex = 0;
await Promise.all(workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= files.length) break;
await processFile(eng, files[idx]);
}
}));
} finally {
// v0.22.13 (PR #490 A2): try/finally guarantees cleanup even when the
// worker loop throws. Each disconnect is best-effort — one failing
// disconnect must not strand the others.
await Promise.all(
workerEngines.map(e =>
e.disconnect().catch((err: unknown) =>
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
),
),
);
}
}));
await Promise.all(workerEngines.map(e => e.disconnect()));
} // end else (postgres parallel)
} else {
// Sequential: use the provided engine
+42 -1
View File
@@ -864,8 +864,40 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
const { performSync } = await import('./sync.ts');
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
const noPull = !!job.data.noPull;
// noEmbed defaults to true (embed is a separate job — submit `embed --stale`
// after sync, OR run via the autopilot cycle which has its own embed phase).
// Caller can opt in by passing { noEmbed: false } in job params.
const noEmbed = job.data.noEmbed !== false;
const result = await performSync(engine, { repoPath, noPull, noEmbed });
// v0.22.13 (PR #490 CODEX-1): resolve sourceId from job param OR by looking
// up the sources row for repoPath. Mirrors cycle.ts:480 — without this, a
// multi-source brain reads the global config.sync.last_commit anchor
// instead of sources.last_commit, which on a regularly-GC'd repo can drop
// out of git history and trigger 30-min full reimports every cycle.
let sourceId: string | undefined =
typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
if (!sourceId && repoPath) {
try {
const rows = await engine.executeRaw<{ id: string }>(
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
[repoPath],
);
sourceId = rows[0]?.id;
} catch {
// sources table may not exist on very old brains — fall through to
// global config.sync.* anchor in performSync.
}
}
// v0.22.13 (PR #490 CODEX-4): route concurrency through the shared
// autoConcurrency helper instead of hardcoded 4. PGLite engines stay
// serial (forced 1); explicit job param wins; auto path defaults are
// applied inside performSync against the resolved file count.
const concurrencyOverride = typeof job.data.concurrency === 'number'
? job.data.concurrency
: undefined;
const result = await performSync(engine, {
repoPath, sourceId, noPull, noEmbed,
concurrency: concurrencyOverride,
});
return result;
});
@@ -948,10 +980,19 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
? job.data.repoPath
: (await engine.getConfig('sync.repo_path')) ?? '.';
// Allow callers to select phases via job data (e.g. skip embed for
// fast cycles). Validates against ALL_PHASES to prevent injection.
const { ALL_PHASES } = await import('../core/cycle.ts');
const validPhases = new Set(ALL_PHASES);
const requestedPhases = Array.isArray(job.data.phases)
? (job.data.phases as string[]).filter(p => validPhases.has(p as any))
: undefined;
const report = await runCycle(engine, {
brainDir: repoPath,
pull: true, // autopilot daemon opts into git pull
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
yieldBetweenPhases: async () => {
// Yield to the event loop so worker lock-renewal can fire.
await new Promise<void>(r => setImmediate(r));
+245
View File
@@ -0,0 +1,245 @@
import { join } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { loadStorageConfig, validateStorageConfig, getStorageTier } from '../core/storage-config.ts';
import type { StorageConfig, StorageTier } from '../core/storage-config.ts';
import { walkBrainRepo, type DiskFileEntry } from '../core/disk-walk.ts';
import { getDefaultSourcePath } from '../core/source-resolver.ts';
/**
* Distinct nominal types for the two tier-keyed numeric maps. Both shapes
* are `Record<StorageTier, number>` structurally but they carry
* semantically different units (page COUNT vs disk BYTES). Distinct types
* make accidental swaps a compile-time error rather than a silent display
* bug. Issue #11 of the eng review.
*/
export type PageCountsByTier = Record<StorageTier, number> & { __brand?: 'page-counts' };
export type DiskUsageByTier = Record<StorageTier, number> & { __brand?: 'disk-bytes' };
/**
* Pure-data result of a storage-status query. No side effects, no I/O
* beyond the engine call and one filesystem walk. Consumed by both the
* JSON formatter and the human formatter; kept narrow so it's a stable
* MCP/scripting contract (D14: storage_status is read-only MCP-exposed).
*/
export interface StorageStatusResult {
config: StorageConfig | null;
repoPath: string | null;
totalPages: number;
pagesByTier: PageCountsByTier;
missingFiles: Array<{ slug: string; expectedPath: string }>;
diskUsageByTier: DiskUsageByTier;
warnings: string[];
}
// ── Dispatcher ────────────────────────────────────────────
export async function runStorage(engine: BrainEngine, args: string[]): Promise<void> {
const subcommand = args[0];
if (!subcommand || subcommand === 'status') {
await runStorageStatus(engine, args.slice(1));
return;
}
console.error(`Unknown storage subcommand: ${subcommand}`);
console.error('Available subcommands: status');
process.exit(1);
}
async function runStorageStatus(engine: BrainEngine, args: string[]): Promise<void> {
warnIfPGLite(engine);
// Resolution chain (D5, Issue #3): explicit --repo → typed accessor → null.
// No cwd fallback. The original silent footgun is dead.
let repoPath: string | null = null;
const repoIdx = args.indexOf('--repo');
if (repoIdx !== -1 && args[repoIdx + 1]) {
repoPath = args[repoIdx + 1];
} else {
repoPath = await getDefaultSourcePath(engine);
}
const result = await getStorageStatus(engine, repoPath);
if (args.includes('--json')) {
console.log(formatStorageStatusJson(result));
return;
}
console.log(formatStorageStatusHuman(result));
}
/**
* D4: storage tiering on PGLite is a partial feature. The "DB" the pages
* live in IS the local file gbrain uses for everything else, so "db_only"
* has no real offload effect. The .gitignore management still helps
* (keeps bulk content out of git history), so we warn but proceed.
*
* Once-per-process via a module-local flag sub-commands invoked from a
* single CLI run share the same warning.
*/
let _pgliteWarned = false;
function warnIfPGLite(engine: BrainEngine): void {
if (_pgliteWarned) return;
if (engine.kind !== 'pglite') return;
_pgliteWarned = true;
console.warn(
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
`local database file regardless of tier. The .gitignore management still ` +
`keeps bulk content out of git history. To get full tiering, migrate to ` +
`Postgres with \`gbrain migrate --to supabase\`.`,
);
}
/** Reset for tests. */
export function __resetPGLiteWarn(): void {
_pgliteWarned = false;
}
// ── Pure data ─────────────────────────────────────────────
/**
* Compute the storage status against the given engine + brain repo path.
*
* Side-effect-free apart from the engine.listPages call and one recursive
* filesystem walk. Pure for testability formatters are tested separately.
*
* Returns null `config` when no gbrain.yml is present at repoPath. In that
* case pagesByTier is all zeros for db_tracked/db_only and totals roll up
* into unspecified.
*/
export async function getStorageStatus(
engine: BrainEngine,
repoPath: string | null,
): Promise<StorageStatusResult> {
const config = repoPath ? loadStorageConfig(repoPath) : null;
const warnings = config ? validateStorageConfig(config) : [];
const pagesByTier: PageCountsByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
const diskUsageByTier: DiskUsageByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
const missingFiles: Array<{ slug: string; expectedPath: string }> = [];
// Single recursive walk of the brain repo (Issue #14). Replaces per-page
// existsSync+statSync — was ~400K syscalls on 200K-page brains, now ~one
// per directory + one stat per .md file, plus O(1) lookups below.
const fileMap: Map<string, DiskFileEntry> = repoPath ? walkBrainRepo(repoPath) : new Map();
const pages = await engine.listPages({ limit: 1_000_000 });
for (const page of pages) {
const tier = config ? getStorageTier(page.slug, config) : 'unspecified';
pagesByTier[tier]++;
if (!repoPath) continue;
const entry = fileMap.get(page.slug);
if (entry) {
diskUsageByTier[tier] += entry.size;
} else if (config && tier === 'db_only') {
missingFiles.push({ slug: page.slug, expectedPath: join(repoPath, page.slug + '.md') });
}
}
return {
config,
repoPath,
totalPages: pages.length,
pagesByTier,
missingFiles,
diskUsageByTier,
warnings,
};
}
// ── JSON formatter ────────────────────────────────────────
/**
* Serialize StorageStatusResult to a stable JSON contract. Indented for
* human readability; agents/orchestrators can parse with a standard
* JSON.parse. Schema is the StorageStatusResult interface above.
*/
export function formatStorageStatusJson(result: StorageStatusResult): string {
return JSON.stringify(result, null, 2);
}
// ── Human formatter ───────────────────────────────────────
/**
* Render StorageStatusResult to ASCII text suitable for terminal output.
* D10 lock: ASCII separators only universally portable. No unicode
* box-drawing.
*/
export function formatStorageStatusHuman(result: StorageStatusResult): string {
const lines: string[] = [];
lines.push('Storage Status');
lines.push('==============');
lines.push('');
if (!result.config) {
lines.push('No gbrain.yml configuration found.');
if (result.repoPath) lines.push(`Checked: ${result.repoPath}/gbrain.yml`);
lines.push('');
lines.push('All pages are stored in git by default.');
lines.push(`Total pages: ${result.totalPages}`);
return lines.join('\n');
}
lines.push(`Repository: ${result.repoPath}`);
lines.push(`Total pages: ${result.totalPages}`);
lines.push('');
lines.push('Storage Tiers:');
lines.push('-------------');
lines.push(`DB tracked: ${result.pagesByTier.db_tracked.toLocaleString()} pages`);
lines.push(`DB only: ${result.pagesByTier.db_only.toLocaleString()} pages`);
lines.push(`Unspecified: ${result.pagesByTier.unspecified.toLocaleString()} pages`);
if (result.diskUsageByTier.db_tracked > 0 || result.diskUsageByTier.db_only > 0) {
lines.push('');
lines.push('Disk Usage:');
lines.push('-----------');
if (result.diskUsageByTier.db_tracked > 0) {
lines.push(`DB tracked: ${formatBytes(result.diskUsageByTier.db_tracked)}`);
}
if (result.diskUsageByTier.db_only > 0) {
lines.push(`DB only: ${formatBytes(result.diskUsageByTier.db_only)}`);
}
if (result.diskUsageByTier.unspecified > 0) {
lines.push(`Unspecified: ${formatBytes(result.diskUsageByTier.unspecified)}`);
}
}
if (result.missingFiles.length > 0) {
lines.push('');
lines.push('Missing Files (need restore):');
lines.push('-----------------------------');
for (const missing of result.missingFiles.slice(0, 10)) {
lines.push(` ${missing.slug}`);
}
if (result.missingFiles.length > 10) {
lines.push(` ... and ${result.missingFiles.length - 10} more`);
}
lines.push('');
lines.push(`Use: gbrain export --restore-only --repo "${result.repoPath}"`);
}
if (result.warnings.length > 0) {
lines.push('');
lines.push('Warnings:');
lines.push('---------');
for (const warning of result.warnings) lines.push(` ! ${warning}`);
}
lines.push('');
lines.push('Configuration:');
lines.push('--------------');
lines.push('DB tracked directories:');
for (const dir of result.config.db_tracked) lines.push(` - ${dir}`);
lines.push('');
lines.push('DB-only directories:');
for (const dir of result.config.db_only) lines.push(` - ${dir}`);
return lines.join('\n');
}
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B';
const k = 1024;
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
}
+365 -14
View File
@@ -1,9 +1,8 @@
import { existsSync } from 'fs';
import { existsSync, readFileSync, writeFileSync, statSync, readdirSync } from 'fs';
import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { importFile } from '../core/import-file.ts';
import { readFileSync, statSync, readdirSync } from 'fs';
import { createInterface } from 'readline';
import {
buildSyncManifest,
@@ -12,6 +11,7 @@ import {
recordSyncFailures,
unacknowledgedSyncFailures,
acknowledgeSyncFailures,
formatCodeBreakdown,
} from '../core/sync.ts';
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts';
@@ -19,6 +19,15 @@ import { errorFor, serializeError } from '../core/errors.ts';
import type { SyncManifest } from '../core/sync.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { loadConfig } from '../core/config.ts';
import {
autoConcurrency,
shouldRunParallel,
parseWorkers,
} from '../core/sync-concurrency.ts';
import { tryAcquireDbLock, SYNC_LOCK_ID } from '../core/db-lock.ts';
import { loadStorageConfig } from '../core/storage-config.ts';
import { getDefaultSourcePath } from '../core/source-resolver.ts';
export interface SyncResult {
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
@@ -157,6 +166,27 @@ export interface SyncOpts {
sourceId?: string;
/** Multi-repo: sync strategy override (markdown, code, auto). */
strategy?: 'markdown' | 'code' | 'auto';
/**
* Number of parallel workers for the import phase. When > 1, each worker
* gets its own small Postgres connection pool and files are dispatched via
* an atomic queue index (same pattern as `import --workers N`).
*
* Deletes and renames remain serial (order-dependent).
* Default: undefined auto-concurrency picks (`src/core/sync-concurrency.ts`).
*
* v0.22.13 (PR #490 Q1): when this is explicitly set, the >50-file floor
* is bypassed explicit user intent beats the auto-path safety net.
*/
concurrency?: number;
/**
* Internal: skip acquiring the gbrain-sync DB lock. Set by the cycle
* handler (cycle.ts) which already holds gbrain-cycle and therefore
* already serializes against other cycle runs. CLI sync, jobs handler,
* and any external caller leave this undefined so they take the lock.
*
* v0.22.13 (PR #490 CODEX-2). Not part of the public CLI surface.
*/
skipLock?: boolean;
}
function git(repoPath: string, ...args: string[]): string {
@@ -250,6 +280,39 @@ async function writeChunkerVersion(
}
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
// CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two
// concurrent syncs can otherwise read the same last_commit anchor, both
// write last_commit unconditionally, and the last writer wins — including
// regressing the bookmark backwards. cycle.ts already takes gbrain-cycle
// for its broader scope; performSync (called from cycle, jobs handler,
// and CLI) takes gbrain-sync just for the writer window. The two ids
// nest cleanly: cycle holds gbrain-cycle, calls performSync, performSync
// takes gbrain-sync. Other callers serialize on gbrain-sync against
// each other AND against the cycle's sync phase.
//
// skipLock is reserved for callers that already serialize via another
// mechanism (none in v0.22.13; reserved for future).
let lockHandle: { release: () => Promise<void> } | null = null;
if (!opts.skipLock) {
lockHandle = await tryAcquireDbLock(engine, SYNC_LOCK_ID);
if (!lockHandle) {
throw new Error(
`Another sync is in progress (lock ${SYNC_LOCK_ID} held). ` +
`Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`,
);
}
}
try {
return await performSyncInner(engine, opts);
} finally {
if (lockHandle) {
try { await lockHandle.release(); } catch { /* best-effort release */ }
}
}
}
async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
// Resolve repo path
const repoPath = opts.repoPath || await readSyncAnchor(engine, opts.sourceId, 'repo_path');
if (!repoPath) {
@@ -486,21 +549,41 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
// gate `sync.last_commit` advancement and record recoverable errors.
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
const addsAndMods = [...filtered.added, ...filtered.modified];
// v0.22.13 (PR #490 Q5): one source of truth for the concurrency decision.
// engine.kind === 'pglite' → forced 1; explicit opts.concurrency wins;
// auto path returns DEFAULT_PARALLEL_WORKERS only when fileCount > 100.
const explicitConcurrency = opts.concurrency !== undefined;
const effectiveConcurrency = autoConcurrency(engine, addsAndMods.length, opts.concurrency);
const runParallel = shouldRunParallel(effectiveConcurrency, addsAndMods.length, explicitConcurrency);
if (addsAndMods.length > 0) {
progress.start('sync.imports', addsAndMods.length);
for (const path of addsAndMods) {
const filePath = join(repoPath, path);
// Core import logic shared by serial and parallel paths.
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
const syncRepoPath = repoPath!;
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
const filePath = join(syncRepoPath, path);
if (!existsSync(filePath)) {
// CODEX-3 (v0.22.13): a file the diff said exists at headCommit but
// is gone from disk means the working tree has drifted (someone ran
// `git checkout` / `git reset` mid-sync, or the file was deleted
// post-diff). Record as a failure so last_commit does NOT advance —
// the silent-skip-then-advance pathology was the bug.
failedFiles.push({
path,
error: 'file vanished mid-sync (working tree drifted from headCommit)',
});
progress.tick(1, `skip:${path}`);
continue;
return;
}
try {
const result = await importFile(engine, filePath, path, { noEmbed });
const result = await importFile(eng, filePath, path, { noEmbed });
if (result.status === 'imported') {
chunksCreated += result.chunks;
pagesAffected.push(result.slug);
} else if (result.status === 'skipped' && (result as any).error) {
// importFile returned a non-throw skip with a reason.
failedFiles.push({ path, error: String((result as any).error) });
}
} catch (e: unknown) {
@@ -510,9 +593,98 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
}
progress.tick(1, path);
}
if (runParallel) {
// A1 (v0.22.13): use engine.kind discriminator instead of config?.engine
// string compare or constructor.name sniff. Q3: belt-and-suspenders fall
// back to serial when database_url is unset, so we never crash on a null
// assertion if config is missing.
const config = loadConfig();
if (engine.kind === 'pglite' || !config?.database_url) {
for (const path of addsAndMods) {
await importOnePath(engine, path);
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerCount = Math.min(effectiveConcurrency, addsAndMods.length);
const databaseUrl = config.database_url;
// Q4 (v0.22.13): banner on stderr so stdout stays clean for --json.
console.error(` Parallel sync: ${workerCount} workers for ${addsAndMods.length} files`);
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
try {
// Connect workers one-by-one rather than Promise.all so a partial
// failure leaves us with the connected ones in workerEngines for
// the finally-block cleanup. The original code lost track of
// already-connected engines on any one failure.
for (let i = 0; i < workerCount; i++) {
const eng = new PostgresEngine();
await eng.connect({ database_url: databaseUrl, poolSize: workerPoolSize });
workerEngines.push(eng);
}
// Atomic queue index — JS is single-threaded; the read-then-increment
// happens between awaits, so no lock is needed.
let queueIndex = 0;
await Promise.all(
workerEngines.map(async (eng) => {
while (true) {
const idx = queueIndex++;
if (idx >= addsAndMods.length) break;
await importOnePath(eng, addsAndMods[idx]);
}
}),
);
} finally {
// A2 (v0.22.13): try/finally guarantees connection cleanup even when
// the worker loop throws (partial connect failure, OOM, mid-import
// signal). Each disconnect is best-effort — one worker failing to
// disconnect must not strand the others.
await Promise.all(
workerEngines.map((e) =>
e.disconnect().catch((err: unknown) =>
console.error(` worker disconnect failed: ${err instanceof Error ? err.message : String(err)}`),
),
),
);
}
}
} else {
// Serial path (small auto diffs or explicit --workers 1).
for (const path of addsAndMods) {
await importOnePath(engine, path);
}
}
progress.finish();
}
// CODEX-3 (v0.22.13): head-drift gate. If git HEAD moved during the import
// window (someone ran `git checkout` or `git pull` in another terminal /
// sibling Conductor workspace), the chunks we just imported reflect a
// different tree than `headCommit` claims. Refuse to advance last_commit
// so the next sync re-walks against the new HEAD. The lock from CODEX-2
// prevents *this* gbrain process from stepping on itself; this gate
// catches drift caused by external `git` commands the lock cannot see.
try {
const currentHead = git(repoPath, 'rev-parse', 'HEAD');
if (currentHead !== headCommit) {
failedFiles.push({
path: '<head>',
error: `git HEAD drifted during sync: captured ${headCommit.slice(0, 8)}, now ${currentHead.slice(0, 8)}`,
});
}
} catch (e) {
// rev-parse failure is itself a drift signal (worktree disappeared).
failedFiles.push({
path: '<head>',
error: `git HEAD verification failed: ${e instanceof Error ? e.message : String(e)}`,
});
}
const elapsed = Date.now() - start;
// Bug 9 — gate the sync bookmark on success. If any per-file parse
@@ -522,9 +694,13 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
// current set, --retry-failed re-parses before running the normal sync.
if (failedFiles.length > 0) {
recordSyncFailures(failedFiles, headCommit);
// Emit structured summary grouped by error code so the operator
// can see *why* files failed, not just how many.
const codeBreakdown = formatCodeBreakdown(failedFiles);
if (!opts.skipFailed) {
console.error(
`\nSync blocked: ${failedFiles.length} file(s) failed to parse. ` +
`\nSync blocked: ${failedFiles.length} file(s) failed to parse:\n` +
`${codeBreakdown}\n\n` +
`Fix the YAML frontmatter in the files above and re-run, or use ` +
`'gbrain sync --skip-failed' to acknowledge and move on.`,
);
@@ -547,8 +723,11 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
}
// --skip-failed: acknowledge the now-recorded set and proceed.
const acked = acknowledgeSyncFailures();
if (acked > 0) {
console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
if (acked.count > 0) {
console.error(
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
`${formatCodeBreakdown(acked.summary)}`,
);
}
}
@@ -644,10 +823,18 @@ async function performFullSync(
};
}
console.log(`Running full import of ${repoPath}...`);
// v0.22.13 (PR #490 A1 + Q5): full sync is always "large" by definition
// (entire working tree). Auto-concurrency fires unconditionally for Postgres;
// PGLite stays serial because its engine is single-connection. Routes the
// policy through autoConcurrency() so it stays consistent with incremental
// sync and the jobs handler.
const FULL_SYNC_LARGE_MARKER = Number.MAX_SAFE_INTEGER;
const fullConcurrency = autoConcurrency(engine, FULL_SYNC_LARGE_MARKER, opts.concurrency);
console.log(`Running full import of ${repoPath}${fullConcurrency > 1 ? ` (${fullConcurrency} workers)` : ''}...`);
const { runImport } = await import('./import.ts');
const importArgs = [repoPath];
if (opts.noEmbed) importArgs.push('--no-embed');
if (fullConcurrency > 1) importArgs.push('--workers', String(fullConcurrency));
const result = await runImport(engine, importArgs, { commit: headCommit });
// Bug 9 — gate the full-sync bookmark on success. runImport already
@@ -656,9 +843,11 @@ async function performFullSync(
// the sync module owns the last_commit write. Respect the same gate.
if (result.failures.length > 0) {
recordSyncFailures(result.failures, headCommit);
const codeBreakdown = formatCodeBreakdown(result.failures);
if (!opts.skipFailed) {
console.error(
`\nFull sync blocked: ${result.failures.length} file(s) failed. ` +
`\nFull sync blocked: ${result.failures.length} file(s) failed:\n` +
`${codeBreakdown}\n\n` +
`Fix the YAML in those files and re-run, or use '--skip-failed'.`,
);
await engine.setConfig('sync.last_run', new Date().toISOString());
@@ -675,7 +864,12 @@ async function performFullSync(
};
}
const acked = acknowledgeSyncFailures();
if (acked > 0) console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
if (acked.count > 0) {
console.error(
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
`${formatCodeBreakdown(acked.summary)}`,
);
}
}
// Persist sync state so next sync is incremental (C1 fix: was missing).
@@ -728,6 +922,17 @@ export async function runSync(engine: BrainEngine, args: string[]) {
const jsonOut = args.includes('--json');
const yesFlag = args.includes('--yes');
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
// of silently falling through to auto-concurrency or NaN. Loud failure beats
// a 4-worker spawn from a typo.
let concurrency: number | undefined;
try {
concurrency = parseWorkers(concurrencyStr);
} catch (e) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
}
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
@@ -820,10 +1025,18 @@ export async function runSync(engine: BrainEngine, args: string[]) {
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
sourceId: src.id,
strategy: cfg.strategy,
concurrency,
};
try {
const result = await performSync(engine, repoOpts);
printSyncResult(result);
// Codex P2: --all loop must also manage .gitignore per-source. Without
// this, multi-source users who rely on `gbrain sync --all` never get
// the advertised db_only ignore rules unless they sync each repo
// individually.
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
manageGitignore(src.local_path!, engine.kind);
}
} catch (e: unknown) {
console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
}
@@ -831,7 +1044,7 @@ export async function runSync(engine: BrainEngine, args: string[]) {
return;
}
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg };
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, sourceId, strategy: strategyArg, concurrency };
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
// flags so the sync picks them up as fresh work. The actual re-attempt
@@ -850,6 +1063,18 @@ export async function runSync(engine: BrainEngine, args: string[]) {
if (!watch) {
const result = await performSync(engine, opts);
printSyncResult(result);
// Issue #2 + eng-review pass-2 finding #1 + Codex P1: manage .gitignore ONLY
// on successful sync. Skip on dry-run (don't mutate disk in preview mode)
// and blocked_by_failures (sync state is inconsistent — defer .gitignore
// until next clean run). Resolve the effective repo path so the wire-up
// fires in the common case where the user runs `gbrain sync` without
// passing --repo every time.
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
if (effectiveRepoPath) {
manageGitignore(effectiveRepoPath, engine.kind);
}
}
return;
}
@@ -865,6 +1090,14 @@ export async function runSync(engine: BrainEngine, args: string[]) {
const ts = new Date().toISOString().slice(11, 19);
console.log(`[${ts}] Synced: +${result.added} ~${result.modified} -${result.deleted} R${result.renamed}`);
}
// Same gate as non-watch: only manage .gitignore on successful sync.
// Same repo-resolution path so watch mode catches the implicit-resolved case.
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
if (effectiveRepoPath) {
manageGitignore(effectiveRepoPath, engine.kind);
}
}
} catch (e: unknown) {
consecutiveErrors++;
const msg = e instanceof Error ? e.message : String(e);
@@ -878,6 +1111,124 @@ export async function runSync(engine: BrainEngine, args: string[]) {
}
}
/**
* Auto-manage .gitignore entries for db_only directories.
*
* Caller invokes ONLY on successful sync this function trusts that the
* sync's data state is consistent. See `runSync` for the gating logic.
*
* Idempotent: re-running adds no duplicate entries. The managed block has
* a stable comment header so it's grep-able and editable.
*
* Skipped (with actionable warning) when:
* - GBRAIN_NO_GITIGNORE=1 D23 escape hatch for shared-repo setups
* - The repo is a git submodule (`.git` is a file not a directory)
* D49 lock; submodule .gitignore changes don't survive parent updates
*
* On PGLite (D4): emits a once-per-process soft-warn explaining that
* tiering has limited effect but still manages the .gitignore so the
* config-present user gets the gitignore housekeeping.
*
* Failures (write permission denied, EROFS, etc.) are caught, warned, and
* swallowed (D9 lock). Sync's primary job is moving data; .gitignore
* management is a side effect don't kill the main job for the side effect.
*/
let _pgliteTierWarned = false;
export function __resetPGLiteTierWarn(): void {
_pgliteTierWarned = false;
}
export function manageGitignore(
repoPath: string,
engineKind?: 'pglite' | 'postgres',
): void {
if (process.env.GBRAIN_NO_GITIGNORE === '1') {
return;
}
// D49: submodule detection. In a submodule, `.git` is a regular file
// (containing `gitdir: ../path/to/parent.git/modules/x`), not a directory.
const dotGit = join(repoPath, '.git');
if (existsSync(dotGit)) {
try {
if (statSync(dotGit).isFile()) {
console.warn(
`Note: skipping .gitignore management — ${repoPath} is a git submodule. ` +
`Add db_only directories to your parent repo's .gitignore manually.`,
);
return;
}
} catch {
// proceed; can't tell, default to managing
}
}
let storageConfig;
try {
storageConfig = loadStorageConfig(repoPath);
} catch (error) {
// StorageConfigError (overlap) or read error — surface, don't manage.
console.warn(
`Skipped .gitignore update: ${error instanceof Error ? error.message : String(error)}`,
);
return;
}
if (!storageConfig || storageConfig.db_only.length === 0) {
return;
}
// D4 soft-warn: storage tiering has limited effect on PGLite, but the
// .gitignore housekeeping still helps. Warn once per process; proceed.
if (engineKind === 'pglite' && !_pgliteTierWarned) {
_pgliteTierWarned = true;
console.warn(
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
`local database file regardless of tier. Managing .gitignore anyway.`,
);
}
const gitignorePath = join(repoPath, '.gitignore');
let gitignoreContent = '';
if (existsSync(gitignorePath)) {
try {
gitignoreContent = readFileSync(gitignorePath, 'utf-8');
} catch (error) {
console.warn(
`Could not read ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
`skipping .gitignore update. Add db_only directories manually.`,
);
return;
}
}
const existingLines = new Set(gitignoreContent.split('\n').map((line) => line.trim()));
const linesToAdd: string[] = [];
for (const dir of storageConfig.db_only) {
if (!existingLines.has(dir) && !existingLines.has(`/${dir}`)) {
linesToAdd.push(dir);
}
}
if (linesToAdd.length === 0) return;
if (gitignoreContent && !gitignoreContent.endsWith('\n')) {
gitignoreContent += '\n';
}
gitignoreContent += '\n# Auto-managed by gbrain (db_only directories)\n';
gitignoreContent += linesToAdd.join('\n') + '\n';
try {
writeFileSync(gitignorePath, gitignoreContent);
} catch (error) {
console.warn(
`Could not update ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
`please add db_only directories manually:\n ${linesToAdd.join('\n ')}`,
);
}
}
function printSyncResult(result: SyncResult) {
switch (result.status) {
case 'up_to_date':
+140
View File
@@ -0,0 +1,140 @@
/**
* Generic DB-backed lock primitive.
*
* Reuses the gbrain_cycle_locks table (id PK + holder_pid + ttl_expires_at)
* with a parameterized lock id. Both `gbrain-cycle` (the broad cycle lock)
* and `gbrain-sync` (performSync's writer lock) live here.
*
* Why not pg_advisory_xact_lock: it is session-scoped, and PgBouncer
* transaction pooling drops session state between calls. This row-based
* lock survives PgBouncer because it's plain INSERT/UPDATE/DELETE with
* a TTL fallback (a crashed holder's row times out).
*
* Why a separate table-row per lock id rather than reusing the cycle lock:
* the cycle lock is broader (covers every phase). performSync's write-window
* is narrower. If performSync reused the cycle lock and the cycle handler
* called performSync, the inner acquire would deadlock against itself. Two
* lock ids let callers nest cleanly: cycle holds gbrain-cycle for its run;
* performSync (called from anywhere cycle, jobs handler, CLI) takes
* gbrain-sync just for the write window.
*
* v0.22.13 added in PR #490 to fix CODEX-2 (no cross-process lock for
* direct sync paths). The cycle path was already protected.
*/
import { hostname } from 'os';
import type { BrainEngine } from './engine.ts';
export interface DbLockHandle {
id: string;
release: () => Promise<void>;
refresh: () => Promise<void>;
}
/** Default TTL: 30 minutes, same as cycle lock. */
const DEFAULT_TTL_MINUTES = 30;
/**
* Try to acquire a named DB lock.
*
* Returns a handle on success. Returns `null` if another live holder has
* the lock (its row exists and ttl_expires_at is in the future).
*
* The acquire is upsert-style:
* INSERT ... ON CONFLICT (id) DO UPDATE
* ... WHERE existing.ttl_expires_at < NOW()
* RETURNING id
*
* Empty RETURNING means the existing row is still live. An expired holder
* (worker crashed without releasing) is auto-superseded by the UPDATE
* branch.
*/
export async function tryAcquireDbLock(
engine: BrainEngine,
lockId: string,
ttlMinutes: number = DEFAULT_TTL_MINUTES,
): Promise<DbLockHandle | null> {
const pid = process.pid;
const host = hostname();
// Engine-agnostic: prefer the engine's raw escape hatch (`sql` for postgres-js,
// `db.query` for PGLite). Mirrors cycle.ts's pattern so behavior stays identical.
const maybePG = engine as unknown as { sql?: (...args: unknown[]) => Promise<unknown> };
const maybePGLite = engine as unknown as {
db?: { query: (sql: string, params?: unknown[]) => Promise<{ rows: unknown[] }> };
};
if (engine.kind === 'postgres' && maybePG.sql) {
const sql = maybePG.sql as any;
const ttl = `${ttlMinutes} minutes`;
const rows: Array<{ id: string }> = await sql`
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES (${lockId}, ${pid}, ${host}, NOW(), NOW() + ${ttl}::interval)
ON CONFLICT (id) DO UPDATE
SET holder_pid = ${pid},
holder_host = ${host},
acquired_at = NOW(),
ttl_expires_at = NOW() + ${ttl}::interval
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id
`;
if (rows.length === 0) return null;
return {
id: lockId,
refresh: async () => {
await sql`
UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + ${ttl}::interval
WHERE id = ${lockId} AND holder_pid = ${pid}
`;
},
release: async () => {
await sql`
DELETE FROM gbrain_cycle_locks
WHERE id = ${lockId} AND holder_pid = ${pid}
`;
},
};
}
if (engine.kind === 'pglite' && maybePGLite.db) {
const db = maybePGLite.db;
const ttl = `${ttlMinutes} minutes`;
const { rows } = await db.query(
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES ($1, $2, $3, NOW(), NOW() + $4::interval)
ON CONFLICT (id) DO UPDATE
SET holder_pid = $2,
holder_host = $3,
acquired_at = NOW(),
ttl_expires_at = NOW() + $4::interval
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id`,
[lockId, pid, host, ttl],
);
if (rows.length === 0) return null;
return {
id: lockId,
refresh: async () => {
await db.query(
`UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + $1::interval
WHERE id = $2 AND holder_pid = $3`,
[ttl, lockId, pid],
);
},
release: async () => {
await db.query(
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
[lockId, pid],
);
},
};
}
throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`);
}
/** Lock id for performSync's writer window. Distinct from gbrain-cycle so the
* cycle handler can hold gbrain-cycle while performSync (called from inside
* the cycle) acquires gbrain-sync. */
export const SYNC_LOCK_ID = 'gbrain-sync';
+83
View File
@@ -0,0 +1,83 @@
/**
* Recursive filesystem walk into a slug Stats map.
*
* Replaces per-page `existsSync` + `statSync` syscall storms (Issue #14 of
* the v0.22.3 eng review). On a 200K-page brain the per-page approach was
* 400K syscalls in a synchronous loop; this walk is one syscall per directory
* plus one stat per file, then O(1) Map lookups for everything downstream.
*
* The slug key is the on-disk path relative to the brain repo, with the
* trailing `.md` stripped, matching how pages are stored: `people/alice.md`
* on disk becomes `people/alice` as a slug.
*
* Skipped entries:
* - `.git/`, `node_modules/`, and dot-directories generally not part of
* the brain's page namespace. Speeds up walks significantly on dirty
* working copies.
* - Files that don't end in `.md`. Sidecar JSON, raw binary attachments,
* etc. are tracked by the brain but not via slugs.
*/
import { readdirSync, statSync, type Stats, type Dirent } from 'fs';
import { join } from 'path';
export interface DiskFileEntry {
size: number;
mtimeMs: number;
}
/**
* Walk `repoPath` and return a Map of slug file metadata for every `.md`
* file. Skips dot-directories. Synchronous (matches the call-site shape and
* the io pattern of stat-heavy scans).
*
* @param repoPath Absolute path to the brain repo root.
* @returns Map keyed by slug (no `.md` suffix). Empty map if repoPath
* doesn't exist or contains no markdown files.
*/
export function walkBrainRepo(repoPath: string): Map<string, DiskFileEntry> {
const result = new Map<string, DiskFileEntry>();
function recurse(dirPath: string, slugPrefix: string): void {
// Annotate as Dirent[] explicitly: ReturnType<typeof readdirSync> with
// withFileTypes:true picks an overload union that includes
// Dirent<Buffer<ArrayBufferLike>>, which makes entry.name a Buffer in
// strict tsc mode. Cast to the string-based Dirent[] (same shape sync.ts
// uses for its own filesystem walk).
let entries: Dirent[];
try {
entries = readdirSync(dirPath, { withFileTypes: true }) as unknown as Dirent[];
} catch {
return; // unreadable directory — skip silently
}
for (const entry of entries) {
// Skip dot-directories (.git, .gbrain, .vscode, etc) and node_modules.
if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;
const childPath = join(dirPath, entry.name);
if (entry.isDirectory()) {
recurse(childPath, slugPrefix ? `${slugPrefix}/${entry.name}` : entry.name);
continue;
}
if (!entry.isFile()) continue;
if (!entry.name.endsWith('.md')) continue;
let stats: Stats;
try {
stats = statSync(childPath);
} catch {
continue; // race: file deleted between readdir and stat
}
const slug = slugPrefix
? `${slugPrefix}/${entry.name.slice(0, -3)}`
: entry.name.slice(0, -3);
result.set(slug, { size: stats.size, mtimeMs: stats.mtimeMs });
}
}
recurse(repoPath, '');
return result;
}
+7
View File
@@ -294,6 +294,13 @@ export class PGLiteEngine implements BrainEngine {
params.push(filters.updated_after);
where.push(`p.updated_at > $${params.length}::timestamptz`);
}
// slugPrefix uses the (source_id, slug) UNIQUE btree for index range scans.
// Escape LIKE metacharacters so the user prefix is treated as a literal.
if (filters?.slugPrefix) {
const escaped = filters.slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%';
params.push(escaped);
where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`);
}
const whereSql = where.length > 0 ? `WHERE ${where.join(' AND ')}` : '';
params.push(limit, offset);
+7 -1
View File
@@ -334,11 +334,17 @@ export class PostgresEngine implements BrainEngine {
const tagJoin = filters?.tag ? sql`JOIN tags t ON t.page_id = p.id` : sql``;
const tagCondition = filters?.tag ? sql`AND t.tag = ${filters.tag}` : sql``;
const updatedCondition = updatedAfter ? sql`AND p.updated_at > ${updatedAfter}::timestamptz` : sql``;
// slugPrefix uses the (source_id, slug) UNIQUE btree index for range scans.
// Escape LIKE metacharacters so the user prefix is treated as a literal.
const slugPrefix = filters?.slugPrefix;
const slugCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
const rows = await sql`
SELECT p.* FROM pages p
${tagJoin}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
`;
+36
View File
@@ -132,6 +132,42 @@ async function assertSourceExists(engine: BrainEngine, id: string): Promise<void
}
}
/**
* Get the local_path of the resolved source (per the resolveSourceId chain).
*
* Returns the on-disk brain repo path for the source the user is currently
* operating against. Used by `gbrain storage status` and `gbrain export
* --restore-only` to find the brain repo without raw SQL or bare try/catch.
*
* Resolution order:
* 1. `sources.local_path` for the resolved source id (multi-source v0.18+ path)
* 2. Legacy global `sync.repo_path` config key (pre-v0.18 default-source brains)
* 3. null
*
* @returns local_path string, or null if no path is configured anywhere.
* @throws If DB error occurs (does NOT silently swallow). Callers handle
* the null case to provide their own fallback (typically a hard error
* telling the user to pass --repo).
*/
export async function getDefaultSourcePath(
engine: BrainEngine,
cwd: string = process.cwd(),
): Promise<string | null> {
const sourceId = await resolveSourceId(engine, null, cwd);
const rows = await engine.executeRaw<{ local_path: string | null }>(
`SELECT local_path FROM sources WHERE id = $1`,
[sourceId],
);
if (rows[0]?.local_path) return rows[0].local_path;
// Legacy fallback: pre-v0.18 brains stored the repo path in the global
// config table under sync.repo_path. The sources table exists but its
// local_path is NULL for the seeded 'default' row. Fall back so storage
// tiering works without forcing a `gbrain sources add . --path .` migration.
const legacyPath = await engine.getConfig('sync.repo_path');
return legacyPath ?? null;
}
/** Exposed for tests. */
export const __testing = {
readDotfileWalk,
+377
View File
@@ -0,0 +1,377 @@
import { readFileSync, existsSync } from 'fs';
import { join } from 'path';
/**
* Storage tier configuration loaded from gbrain.yml.
*
* The canonical key names are `db_tracked` and `db_only` (engine-agnostic).
* The deprecated keys `git_tracked` and `supabase_only` are still read for
* backward compatibility but emit a once-per-process deprecation warning.
* Sunset: future release will reject the deprecated names.
*/
export interface StorageConfig {
db_tracked: string[];
db_only: string[];
}
export type StorageTier = 'db_tracked' | 'db_only' | 'unspecified';
/** Recognized YAML keys (canonical and deprecated). */
const STORAGE_KEYS = new Set([
'db_tracked', 'db_only',
'git_tracked', 'supabase_only', // deprecated aliases
]);
/**
* Parse the gbrain.yml shape: a top-level `storage:` section with up to four
* array-valued nested keys (canonical `db_tracked` / `db_only` plus the
* deprecated aliases `git_tracked` / `supabase_only`).
*
* Intentionally narrow. Does NOT handle the full YAML spec only the file
* shape gbrain controls. Trades expressiveness for zero-dep parsing and
* predictable behavior. Returns null if the file has no `storage:` section
* (so callers can distinguish "no config" from "empty config").
*
* Replaces gray-matter, which silently returned `{data: {}}` on
* delimiter-less YAML and broke the entire feature on every install.
* The defect that prompted this rewrite: storage-config.ts:24 in the
* pre-v0.22.3 implementation.
*
* Returns the raw key map. The caller (loadStorageConfig) is responsible
* for normalizing deprecated keys canonical, emitting deprecation
* warnings, and merging if both old and new keys appear.
*/
type RawStorage = {
db_tracked?: string[];
db_only?: string[];
git_tracked?: string[];
supabase_only?: string[];
};
function parseStorageYaml(content: string): RawStorage | null {
const lines = content.split('\n').map((line) => line.replace(/\r$/, ''));
let inStorage = false;
let currentList: keyof RawStorage | null = null;
const raw: RawStorage = {};
let sawStorage = false;
for (const line of lines) {
// Strip comments. Conservative: drop trailing `# ...` and full-line `#`.
const noComment = line.replace(/\s+#.*$/, '').replace(/^#.*$/, '');
if (noComment.trim() === '') continue;
// Top-level key (no leading whitespace).
if (!noComment.startsWith(' ') && !noComment.startsWith('\t')) {
const colon = noComment.indexOf(':');
if (colon === -1) continue;
const key = noComment.slice(0, colon).trim();
if (key === 'storage') {
inStorage = true;
sawStorage = true;
currentList = null;
continue;
}
inStorage = false;
currentList = null;
continue;
}
if (!inStorage) continue;
const indented = noComment.replace(/^\s+/, '');
if (indented.startsWith('-')) {
if (!currentList) continue;
const value = indented.slice(1).trim().replace(/^["']|["']$/g, '');
if (value) {
if (!raw[currentList]) raw[currentList] = [];
raw[currentList]!.push(value);
}
continue;
}
const colon = indented.indexOf(':');
if (colon === -1) continue;
const key = indented.slice(0, colon).trim();
if (STORAGE_KEYS.has(key)) {
currentList = key as keyof RawStorage;
// Inline empty list: `db_only: []`.
const remainder = indented.slice(colon + 1).trim();
if (remainder === '[]' && !raw[currentList]) {
raw[currentList] = [];
}
continue;
}
currentList = null;
}
if (!sawStorage) return null;
return raw;
}
/**
* Normalize raw parsed keys into canonical StorageConfig shape.
*
* Resolution order (per plan eng-review pass 2 finding #2):
* 1. If canonical keys present, use them.
* 2. Else if deprecated keys present, map to canonical AND emit a
* once-per-process deprecation warning suggesting `gbrain doctor --fix`.
* 3. If both are present, canonical wins. Deprecated keys are ignored
* with a stronger warning (the user is mid-migration).
*
* Validation (validateStorageConfig) always runs against the canonical
* shape, so error messages reference `db_only` / `db_tracked` regardless
* of which keys the user wrote.
*/
let _deprecationWarned = false;
function normalizeStorageConfig(raw: RawStorage): StorageConfig {
const hasCanonical = Boolean(raw.db_tracked || raw.db_only);
const hasDeprecated = Boolean(raw.git_tracked || raw.supabase_only);
if (hasDeprecated && !_deprecationWarned) {
_deprecationWarned = true;
const which = [
raw.git_tracked ? '`git_tracked`' : null,
raw.supabase_only ? '`supabase_only`' : null,
].filter(Boolean).join(' and ');
if (hasCanonical) {
console.warn(
`Warning: ${which} in gbrain.yml is deprecated and ignored ` +
`(canonical keys db_tracked/db_only are present). ` +
`Remove the deprecated keys, or run \`gbrain doctor --fix\`.`,
);
} else {
console.warn(
`Warning: ${which} in gbrain.yml is deprecated. ` +
`Rename to db_tracked / db_only — see docs/storage-tiering.md. ` +
`Run \`gbrain doctor --fix\` for an automated rename.`,
);
}
}
if (hasCanonical) {
return {
db_tracked: raw.db_tracked ?? [],
db_only: raw.db_only ?? [],
};
}
return {
db_tracked: raw.git_tracked ?? [],
db_only: raw.supabase_only ?? [],
};
}
/**
* Load gbrain.yml configuration from the brain repository root.
*
* Returns null when:
* - repoPath is null/undefined
* - gbrain.yml doesn't exist at the repo root
* - gbrain.yml exists but has no `storage:` section (with sanity warning)
*
* Throws when:
* - gbrain.yml exists but is unreadable (permission denied, etc.) D36 lock:
* fail loud rather than silently disable the feature.
*
* Logs a console.warn (once per process) when:
* - File parses but `storage:` section is empty or missing Issue #1 lock:
* surface "your config didn't take" rather than silently no-op.
*/
let _missingStorageWarned = false;
export function loadStorageConfig(repoPath?: string | null): StorageConfig | null {
if (!repoPath) return null;
const yamlPath = join(repoPath, 'gbrain.yml');
if (!existsSync(yamlPath)) return null;
// Read failure is a real error (not a "feature not configured" signal).
// Throwing here lets the caller decide whether to crash or fall back.
const content = readFileSync(yamlPath, 'utf-8');
let raw: RawStorage | null;
try {
raw = parseStorageYaml(content);
} catch (error) {
console.warn(
`Warning: Failed to parse gbrain.yml: ${error instanceof Error ? error.message : String(error)}`,
);
return null;
}
// No storage section at all → null (with sanity warning).
if (raw === null) {
if (!_missingStorageWarned) {
_missingStorageWarned = true;
console.warn(
`Warning: ${yamlPath} exists but has no storage configuration. ` +
`Add a "storage:" section with db_tracked / db_only arrays, ` +
`or remove gbrain.yml to suppress this warning.`,
);
}
return null;
}
const merged = normalizeStorageConfig(raw);
// Empty storage section → return as-is but warn.
if (merged.db_tracked.length === 0 && merged.db_only.length === 0) {
if (!_missingStorageWarned) {
_missingStorageWarned = true;
console.warn(
`Warning: ${yamlPath} exists but has no storage configuration. ` +
`Add a "storage:" section with db_tracked / db_only arrays, ` +
`or remove gbrain.yml to suppress this warning.`,
);
}
return merged;
}
// Normalize cosmetic issues + throw on semantic overlap (D7).
// Throws StorageConfigError on overlap — propagates to the caller.
return normalizeAndValidateStorageConfig(merged);
}
export class StorageConfigError extends Error {
constructor(message: string) {
super(message);
this.name = 'StorageConfigError';
}
}
/**
* Validate storage configuration for conflicts and issues.
* Returns warning strings; callers decide how to surface them.
*
* Always runs against the canonical (db_tracked / db_only) shape error
* messages reference canonical names regardless of which keys the user
* wrote in gbrain.yml.
*
* Pure: does not mutate. For the auto-normalize behavior (D7), see
* `normalizeAndValidateStorageConfig` below.
*/
export function validateStorageConfig(config: StorageConfig): string[] {
const warnings: string[] = [];
const trackedSet = new Set(config.db_tracked);
for (const path of config.db_only) {
if (trackedSet.has(path)) {
warnings.push(`Directory "${path}" appears in both db_tracked and db_only`);
}
}
const allPaths = [...config.db_tracked, ...config.db_only];
for (const path of allPaths) {
if (!path.endsWith('/')) {
warnings.push(`Directory path "${path}" should end with "/" for consistency`);
}
}
return warnings;
}
/**
* Auto-normalize and strict-validate per D7+D8.
*
* 1. Cosmetic fixups are applied silently with a one-time info message
* naming what changed:
* - missing trailing `/` is added
* The message helps the user learn the canonical form without nagging.
* 2. Semantic problems THROW (don't return warnings):
* - same directory in both tiers (ambiguous routing)
*
* Caller passes a fresh raw config; this returns the normalized shape that
* the rest of the code (matcher, sync, etc.) sees.
*/
let _normalizationInfoEmitted = false;
export function normalizeAndValidateStorageConfig(input: StorageConfig): StorageConfig {
const normalize = (paths: string[]): { normalized: string[]; changed: string[] } => {
const normalized: string[] = [];
const changed: string[] = [];
for (const p of paths) {
if (p.endsWith('/')) {
normalized.push(p);
} else {
normalized.push(p + '/');
changed.push(`"${p}" → "${p}/"`);
}
}
return { normalized, changed };
};
const tracked = normalize(input.db_tracked);
const dbonly = normalize(input.db_only);
const allChanged = [...tracked.changed, ...dbonly.changed];
if (allChanged.length > 0 && !_normalizationInfoEmitted) {
_normalizationInfoEmitted = true;
console.warn(
`Note: normalized ${allChanged.length} storage path(s) in gbrain.yml — ` +
`${allChanged.join(', ')}. Add trailing "/" to suppress this note.`,
);
}
// Semantic check: overlap between tiers throws. Ambiguous routing.
const trackedSet = new Set(tracked.normalized);
for (const path of dbonly.normalized) {
if (trackedSet.has(path)) {
throw new StorageConfigError(
`gbrain.yml: directory "${path}" appears in both db_tracked and db_only — ` +
`pick one tier. Edit gbrain.yml to remove the overlap.`,
);
}
}
return { db_tracked: tracked.normalized, db_only: dbonly.normalized };
}
/**
* Path-segment match: a slug belongs to a tier directory iff the directory
* is a complete path-segment ancestor of the slug. `media/x/` matches
* `media/x/foo` but NOT `media/xerox/foo` eliminates the prefix-collision
* class of bug (Issue #5 of the eng review, D6 lock).
*
* Strict: requires the configured directory to end with `/`. The validator
* (per D7+D8) auto-normalizes input so the matcher only ever sees canonical
* trailing-`/` directories.
*/
function matchesTierDir(slug: string, dir: string): boolean {
if (!dir.endsWith('/')) return false; // not normalized — matcher refuses
// slug must equal dir's bare prefix OR start with the trailing-slash form.
// Example: dir = 'media/x/' matches 'media/x/anything' but not 'media/x'
// or 'media/xerox'. (A slug that exactly equals 'media/x' is a directory-
// level entry the brain doesn't write.)
return slug.startsWith(dir);
}
export function isDbTracked(slug: string, config: StorageConfig): boolean {
return config.db_tracked.some((dir) => matchesTierDir(slug, dir));
}
export function isDbOnly(slug: string, config: StorageConfig): boolean {
return config.db_only.some((dir) => matchesTierDir(slug, dir));
}
export function getStorageTier(slug: string, config: StorageConfig): StorageTier {
if (isDbTracked(slug, config)) return 'db_tracked';
if (isDbOnly(slug, config)) return 'db_only';
return 'unspecified';
}
// ── Deprecated aliases — to be removed in a future release ────────
// Kept so existing callers (storage.ts, export.ts) compile during the
// step-by-step refactor. Will be deleted once those call sites migrate
// to the canonical names.
export const isGitTracked = isDbTracked;
export const isSupabaseOnly = isDbOnly;
/** Reset once-per-process warning flags. Test-only. */
export function __resetMissingStorageWarning(): void {
_missingStorageWarned = false;
_deprecationWarned = false;
_normalizationInfoEmitted = false;
}
+101
View File
@@ -0,0 +1,101 @@
/**
* Shared concurrency policy for sync + import + jobs paths.
*
* Three callers used to embed three different policies:
* - performSync (incremental): >100 files 4 workers
* - performFullSync: Postgres 4 workers
* - jobs.ts sync handler: hardcoded 4
*
* They drift over time and confuse users ("why does my sync not parallelize?"
* is a different answer in each path). This module is one source of truth.
*
* v0.22.13 extracted as part of the parallel-sync hardening (PR #490).
*/
import type { BrainEngine } from './engine.ts';
/** Threshold above which auto-concurrency fires for incremental sync paths. */
export const AUTO_CONCURRENCY_FILE_THRESHOLD = 100;
/** Minimum file count below which the parallel branch is skipped even when
* auto-concurrency would otherwise fire. Prevents spawning workers for trivial
* diffs where setup cost exceeds parallelism gains. Only consulted on the
* auto path; explicit `--workers N` bypasses this. */
export const PARALLEL_FILE_FLOOR = 50;
/** Default worker count when auto-concurrency fires. */
export const DEFAULT_PARALLEL_WORKERS = 4;
/**
* Resolve effective worker count for a sync/import operation.
*
* Inputs:
* - engine.kind: 'pglite' always returns 1 (single-connection)
* - override: caller's explicit --workers / opts.concurrency value
* - fileCount: size of the work batch
*
* Rules:
* - PGLite always 1 (the engine is single-connection regardless)
* - explicit override respect it (clamped to >=1)
* - auto path DEFAULT_PARALLEL_WORKERS when fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD, else 1
*
* Note: this function does NOT consult PARALLEL_FILE_FLOOR. The floor is a
* caller-side gate that decides whether to take the parallel code path even
* when the worker count is > 1. It only applies to the auto path; explicit
* --workers bypasses the floor entirely (per Q1 in PR #490).
*/
export function autoConcurrency(
engine: BrainEngine,
fileCount: number,
override?: number,
): number {
if (engine.kind === 'pglite') return 1;
if (override !== undefined) return Math.max(1, override);
return fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD
? DEFAULT_PARALLEL_WORKERS
: 1;
}
/**
* Decide whether the parallel code path should run.
*
* - workers <= 1 never parallel
* - workers > 1 + explicit override always parallel (user opted in,
* respect them even on small diffs Q1 in PR #490)
* - workers > 1 + auto path parallel only when fileCount > PARALLEL_FILE_FLOOR
*/
export function shouldRunParallel(
workers: number,
fileCount: number,
explicit: boolean,
): boolean {
if (workers <= 1) return false;
if (explicit) return true;
return fileCount > PARALLEL_FILE_FLOOR;
}
/**
* Parse a `--workers N` / `--concurrency N` CLI argument value.
*
* Returns:
* - undefined when the flag was not provided
* - a positive integer when the flag was provided with a valid value
*
* Throws on:
* - non-integer ("foo", "1.5", "")
* - zero or negative ("0", "-3")
* - NaN / Infinity
*
* Q2 in PR #490: the prior parseInt-with-no-validation accepted `--workers 0`
* and silently fell through to auto-concurrency (4 workers), the opposite of
* what the user typed. Fail loud instead.
*/
export function parseWorkers(s: string | undefined): number | undefined {
if (s === undefined) return undefined;
const n = parseInt(s, 10);
if (!Number.isFinite(n) || n < 1 || String(n) !== s.trim()) {
throw new Error(
`--workers must be a positive integer, got: ${JSON.stringify(s)}`,
);
}
return n;
}
+108 -6
View File
@@ -307,6 +307,8 @@ import { createHash as _createHash } from 'crypto';
export interface SyncFailure {
path: string;
error: string;
/** Structured error code extracted from the error message. */
code?: string;
commit: string;
line?: number;
ts: string;
@@ -314,6 +316,91 @@ export interface SyncFailure {
acknowledged_at?: string;
}
/**
* Best-effort extraction of a structured error code from a sync failure
* message. Matches known ParseValidationCode patterns (SLUG_MISMATCH,
* YAML_PARSE, etc.) and common DB / timeout errors. Returns 'UNKNOWN'
* when no pattern matches.
*
* Order matters: DB-layer errors are checked BEFORE YAML-layer ones so
* Postgres `duplicate key value violates unique constraint` doesn't get
* mislabeled as a YAML duplicate-key. Frontmatter patterns key off the
* canonical messages emitted by `collectValidationErrors()` in markdown.ts.
*/
export function classifyErrorCode(errorMsg: string): string {
// SLUG_MISMATCH: thrown by importFromFile() at src/core/import-file.ts:374.
if (/slug.*does not match|SLUG_MISMATCH/i.test(errorMsg)) return 'SLUG_MISMATCH';
// DB-layer errors come BEFORE the YAML duplicate-key check. Postgres unique-
// constraint violations contain "duplicate key" but are not a YAML problem.
if (/duplicate key value violates unique constraint|DB_DUPLICATE_KEY/i.test(errorMsg)) {
return 'DB_DUPLICATE_KEY';
}
if (/canceling statement due to statement timeout|STATEMENT_TIMEOUT/i.test(errorMsg)) {
return 'STATEMENT_TIMEOUT';
}
// YAML / frontmatter patterns. These match either the canonical message
// strings in src/core/markdown.ts (collectValidationErrors) or the literal
// ParseValidationCode token, so they fire whether the caller stores the
// message or just the code.
if (/YAML parse failed|YAML_PARSE/i.test(errorMsg)) return 'YAML_PARSE';
if (/YAMLException|duplicated mapping key|YAML_DUPLICATE_KEY/i.test(errorMsg)) {
return 'YAML_DUPLICATE_KEY';
}
if (/File is empty or whitespace-only|Frontmatter must start with ---|MISSING_OPEN/i.test(errorMsg)) {
return 'MISSING_OPEN';
}
if (/No closing --- delimiter|Heading at line .* found inside frontmatter|MISSING_CLOSE/i.test(errorMsg)) {
return 'MISSING_CLOSE';
}
if (/Frontmatter block is empty|EMPTY_FRONTMATTER/i.test(errorMsg)) return 'EMPTY_FRONTMATTER';
if (/Content contains null bytes|NULL_BYTES|null byte/i.test(errorMsg)) return 'NULL_BYTES';
if (/Nested double quotes|NESTED_QUOTES/i.test(errorMsg)) return 'NESTED_QUOTES';
// Generic fallbacks.
if (/invalid UTF-?8|INVALID_UTF8/i.test(errorMsg)) return 'INVALID_UTF8';
// v0.22.12 additions: covers the four real production sites in src/core/import-file.ts
// (lines 199, 347, 352, 401) that previously bucketed to UNKNOWN.
if (/file too large|content too large|FILE_TOO_LARGE/i.test(errorMsg)) return 'FILE_TOO_LARGE';
if (/skipping symlink|symlink|SYMLINK_NOT_ALLOWED/i.test(errorMsg)) return 'SYMLINK_NOT_ALLOWED';
return 'UNKNOWN';
}
/** Group failures by error code and return a sorted summary. */
export function summarizeFailuresByCode(
failures: Array<{ error: string; code?: string }>,
): Array<{ code: string; count: number }> {
const counts: Record<string, number> = {};
for (const f of failures) {
const code = f.code ?? classifyErrorCode(f.error);
counts[code] = (counts[code] ?? 0) + 1;
}
return Object.entries(counts)
.sort(([, a], [, b]) => b - a)
.map(([code, count]) => ({ code, count }));
}
/**
* Format a code-grouped summary as a human-readable multi-line string for
* stderr / doctor output. Accepts either raw failures (which are summarized
* internally) or an already-summarized `{code, count}[]` shape (the return
* value of `summarizeFailuresByCode` or `AcknowledgeResult.summary`).
* Returns an empty string when the input is empty.
*/
export function formatCodeBreakdown(
input: Array<{ error: string; code?: string }> | Array<{ code: string; count: number }>,
): string {
// Distinguish by shape: summary entries have a numeric `count`. Empty array
// returns '' from either branch — both paths produce a 0-length join.
const summary =
input.length > 0 && typeof (input[0] as { count?: unknown }).count === 'number'
? (input as Array<{ code: string; count: number }>)
: summarizeFailuresByCode(input as Array<{ error: string; code?: string }>);
return summary.map(s => ` ${s.code}: ${s.count}`).join('\n');
}
function _failuresDir(): string {
return _joinPath(_homedir(), '.gbrain');
}
@@ -370,6 +457,7 @@ export function recordSyncFailures(
const entry: SyncFailure = {
path: f.path,
error: f.error,
code: classifyErrorCode(f.error),
commit,
line: f.line,
ts: now,
@@ -380,28 +468,42 @@ export function recordSyncFailures(
}
}
export interface AcknowledgeResult {
count: number;
summary: Array<{ code: string; count: number }>;
}
/**
* Mark all unacknowledged failures as acknowledged. Used by
* `gbrain sync --skip-failed`. Returns the number newly acknowledged.
* `gbrain sync --skip-failed`. Returns count and a structured summary
* grouped by error code so the operator can see *why* files were skipped.
*
* We do not delete acknowledged entries stay as historical record so
* doctor can still show them under a "previously skipped" bucket.
*/
export function acknowledgeSyncFailures(): number {
export function acknowledgeSyncFailures(): AcknowledgeResult {
const entries = loadSyncFailures();
if (entries.length === 0) return 0;
if (entries.length === 0) return { count: 0, summary: [] };
const now = new Date().toISOString();
let changed = 0;
const newlyAcked: SyncFailure[] = [];
const updated = entries.map(e => {
if (e.acknowledged) return e;
changed++;
return { ...e, acknowledged: true, acknowledged_at: now };
// Backfill code for entries that predate the code field.
const code = e.code ?? classifyErrorCode(e.error);
const acked = { ...e, code, acknowledged: true, acknowledged_at: now };
newlyAcked.push(acked);
return acked;
});
if (changed === 0) return 0;
if (changed === 0) return { count: 0, summary: [] };
_mkdirSync(_failuresDir(), { recursive: true });
const fd = require('fs').writeFileSync;
fd(syncFailuresPath(), updated.map(e => JSON.stringify(e)).join('\n') + '\n');
return changed;
return {
count: changed,
summary: summarizeFailuresByCode(newlyAcked),
};
}
/** Return only unacknowledged failures. */
+8
View File
@@ -45,6 +45,14 @@ export interface PageFilters {
offset?: number;
/** ISO date string (YYYY-MM-DD or full ISO timestamp). Filter to pages updated_at > value. */
updated_after?: string;
/**
* Prefix-match filter on slug. Implemented as `WHERE slug LIKE prefix || '%'`
* in both engines so it uses the (source_id, slug) UNIQUE constraint's btree
* index for efficient range scans on large brains. Used by storage-tiering
* commands (gbrain storage status, gbrain export --restore-only) to scope
* queries to a tier directory without loading every page into memory.
*/
slugPrefix?: string;
}
// Chunks
+14 -4
View File
@@ -1,4 +1,4 @@
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
import { describe, expect, test, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync, symlinkSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
@@ -9,6 +9,7 @@ import {
BrainWriterError,
} from '../src/core/brain-writer.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
const fence = '---';
@@ -115,15 +116,24 @@ describe('scanBrainSources (PGLite)', () => {
let tmp: string;
let engine: PGLiteEngine;
beforeEach(async () => {
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-scan-'));
// One PGLite per file — beforeEach wipes data only. PGLite cold-start is
// ~20s on CI; sharing one engine across 6 tests in this block saves ~2 min.
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterEach(async () => {
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-scan-'));
});
afterEach(() => {
rmSync(tmp, { recursive: true, force: true });
});
+1 -1
View File
@@ -108,7 +108,7 @@ describe('autopilot-cycle handler contract (v0.20.5)', () => {
// This is a source-level regression guard
const handlerBlock = jobsSource.slice(
jobsSource.indexOf("worker.register('autopilot-cycle'"),
jobsSource.indexOf("worker.register('autopilot-cycle'") + 500,
jobsSource.indexOf("worker.register('autopilot-cycle'") + 2000,
);
expect(handlerBlock).toContain('signal: job.signal');
+92
View File
@@ -0,0 +1,92 @@
/**
* Tests for src/core/disk-walk.ts single-walk filesystem scan.
*
* Replaces the per-page existsSync+statSync syscall storm in storage.ts
* (Issue #14 of the v0.22.3 eng review).
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { walkBrainRepo } from '../src/core/disk-walk.ts';
let tmp: string;
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-walk-test-'));
});
afterEach(() => {
rmSync(tmp, { recursive: true, force: true });
});
function write(relPath: string, content: string): void {
const full = join(tmp, relPath);
mkdirSync(join(full, '..'), { recursive: true });
writeFileSync(full, content);
}
describe('walkBrainRepo', () => {
test('returns empty map for empty directory', () => {
expect(walkBrainRepo(tmp).size).toBe(0);
});
test('returns empty map for nonexistent directory', () => {
expect(walkBrainRepo(join(tmp, 'does-not-exist')).size).toBe(0);
});
test('finds top-level .md files keyed by slug (no .md suffix)', () => {
write('alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(result.has('alice')).toBe(true);
expect(result.get('alice')!.size).toBeGreaterThan(0);
});
test('walks nested directories and produces slash-joined slugs', () => {
write('people/alice.md', '# Alice');
write('media/x/tweet-1.md', 'tweet');
write('media/articles/post-1.md', 'post');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(
new Set(['people/alice', 'media/x/tweet-1', 'media/articles/post-1']),
);
});
test('skips dot-directories (.git, .gbrain, .vscode)', () => {
write('.git/HEAD', 'ref: refs/heads/main');
write('.gbrain/config.json', '{}');
write('.vscode/settings.json', '{}');
write('people/alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
});
test('skips node_modules', () => {
write('node_modules/foo/bar.md', 'noise');
write('people/alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
});
test('ignores non-.md files', () => {
write('people/alice.md', '# Alice');
write('people/alice.json', '{}');
write('people/photo.png', 'binary');
const result = walkBrainRepo(tmp);
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
});
test('captures size from stat', () => {
const content = '# Alice\n'.repeat(100);
write('people/alice.md', content);
const result = walkBrainRepo(tmp);
expect(result.get('people/alice')!.size).toBe(content.length);
});
test('captures mtimeMs', () => {
write('people/alice.md', '# Alice');
const result = walkBrainRepo(tmp);
expect(result.get('people/alice')!.mtimeMs).toBeGreaterThan(0);
});
});
+273
View File
@@ -0,0 +1,273 @@
/**
* E2E test for storage tiering Postgres-only.
*
* Per the v0.23.0 plan: full lifecycle. Container restart simulation:
* write pages via Postgres, delete files from disk, run gbrain export
* --restore-only, assert files restored. Real .gitignore round-trip.
* Real source-resolver path through getDefaultSourcePath().
*
* Skips gracefully when DATABASE_URL is unset (per CLAUDE.md E2E pattern).
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { setupDB, teardownDB, getEngine, hasDatabase, getConn } from './helpers.ts';
import {
getStorageStatus,
formatStorageStatusHuman,
__resetPGLiteWarn,
} from '../../src/commands/storage.ts';
import { manageGitignore, __resetPGLiteTierWarn } from '../../src/commands/sync.ts';
import { getDefaultSourcePath } from '../../src/core/source-resolver.ts';
import { __resetMissingStorageWarning } from '../../src/core/storage-config.ts';
if (!hasDatabase()) {
describe('storage-tiering E2E', () => {
test.skip('DATABASE_URL not set — skipping E2E', () => {});
});
} else {
describe('storage-tiering E2E (Postgres lifecycle)', () => {
let tmp: string;
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
await teardownDB();
});
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-e2e-storage-'));
__resetMissingStorageWarning();
__resetPGLiteWarn();
__resetPGLiteTierWarn();
});
function cleanup(): void {
rmSync(tmp, { recursive: true, force: true });
}
function writeGbrainYml(): void {
writeFileSync(
join(tmp, 'gbrain.yml'),
`storage:
db_tracked:
- people/
db_only:
- media/x/
- media/articles/
`,
);
}
test('engine.kind is postgres', () => {
try {
expect(getEngine().kind).toBe('postgres');
} finally {
cleanup();
}
});
test('full lifecycle: write pages → status reports tiers → manage .gitignore → restore-only path', async () => {
try {
const engine = getEngine();
// Truncate sources + pages so this test has a clean slate.
const conn = getConn();
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
await conn.unsafe(
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
[tmp],
);
writeGbrainYml();
// Seed 4 pages: 1 db_tracked, 2 db_only, 1 unspecified.
await engine.putPage('people/alice', {
type: 'person',
title: 'Alice',
compiled_truth: 'Alice is a founder.',
timeline: '',
});
await engine.putPage('media/x/tweet-1', {
type: 'media',
title: 'Tweet 1',
compiled_truth: 'tweet body',
timeline: '',
});
await engine.putPage('media/x/tweet-2', {
type: 'media',
title: 'Tweet 2',
compiled_truth: 'tweet body 2',
timeline: '',
});
await engine.putPage('random/note', {
type: 'note',
title: 'Random',
compiled_truth: 'random',
timeline: '',
});
// Storage status reports tier counts correctly.
const status = await getStorageStatus(engine, tmp);
expect(status.totalPages).toBe(4);
expect(status.pagesByTier.db_tracked).toBe(1);
expect(status.pagesByTier.db_only).toBe(2);
expect(status.pagesByTier.unspecified).toBe(1);
// Human formatter renders without errors.
const out = formatStorageStatusHuman(status);
expect(out).toContain('DB tracked: 1 pages');
expect(out).toContain('DB only: 2 pages');
// .gitignore management: empty .gitignore → managed block written.
manageGitignore(tmp, 'postgres');
const gitignore = readFileSync(join(tmp, '.gitignore'), 'utf-8');
expect(gitignore).toContain('# Auto-managed by gbrain');
expect(gitignore).toContain('media/x/');
expect(gitignore).toContain('media/articles/');
// Idempotency: second run adds nothing new.
manageGitignore(tmp, 'postgres');
const gitignore2 = readFileSync(join(tmp, '.gitignore'), 'utf-8');
const xCount = (gitignore2.match(/^media\/x\/$/gm) || []).length;
expect(xCount).toBe(1);
// Source resolution finds the local_path we registered.
const resolvedPath = await getDefaultSourcePath(engine);
expect(resolvedPath).toBe(tmp);
} finally {
cleanup();
}
});
test('container restart simulation: db_only files missing on disk are restorable from DB', async () => {
try {
const engine = getEngine();
const conn = getConn();
// Fresh slate.
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
await conn.unsafe(
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
[tmp],
);
writeGbrainYml();
// Write some db_only pages to the database.
await engine.putPage('media/x/tweet-1', {
type: 'media',
title: 'Tweet 1',
compiled_truth: 'tweet body 1',
timeline: '',
});
await engine.putPage('media/x/tweet-2', {
type: 'media',
title: 'Tweet 2',
compiled_truth: 'tweet body 2',
timeline: '',
});
// Simulate "files were on disk, but the container restarted."
// Storage status: missingFiles should list them.
const status = await getStorageStatus(engine, tmp);
expect(status.pagesByTier.db_only).toBe(2);
expect(status.missingFiles.length).toBe(2);
// Verify slugPrefix engine filter (Issue #13) works on Postgres for
// the prefix that --restore-only would use.
const tierPages = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
expect(tierPages.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
// Source-default path resolution returns the configured local_path
// (the typed accessor that replaces the original raw-SQL try/catch
// in storage.ts:38).
const path = await getDefaultSourcePath(engine);
expect(path).toBe(tmp);
} finally {
cleanup();
}
});
test('slugPrefix filter on Postgres uses index-based range scan (regression for Issue #13)', async () => {
try {
const engine = getEngine();
const conn = getConn();
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
await conn.unsafe(`INSERT INTO sources (id, name) VALUES ('default', 'Default')`);
// Seed enough data to make a difference between scan types.
for (let i = 0; i < 50; i++) {
await engine.putPage(`media/x/item-${i}`, {
type: 'media',
title: `Item ${i}`,
compiled_truth: 'x',
timeline: '',
});
}
for (let i = 0; i < 50; i++) {
await engine.putPage(`people/p-${i}`, {
type: 'person',
title: `Person ${i}`,
compiled_truth: 'x',
timeline: '',
});
}
// Prefix query should return exactly 50 (people not included).
const xResults = await engine.listPages({ slugPrefix: 'media/x/', limit: 200 });
expect(xResults.length).toBe(50);
for (const p of xResults) {
expect(p.slug.startsWith('media/x/')).toBe(true);
}
// Path-segment risk: slugPrefix 'media/x' (no /) would match
// 'media/xerox' if any existed. The engine treats slugPrefix as a
// literal string prefix; trailing-/ semantics are the matcher's
// responsibility (storage-config.ts).
const looseResults = await engine.listPages({ slugPrefix: 'media/x', limit: 200 });
expect(looseResults.length).toBe(50); // no media/xerox/* exists yet
} finally {
cleanup();
}
});
test('hard-error path: storage status without local_path or --repo gets null repoPath', async () => {
try {
const engine = getEngine();
const conn = getConn();
await conn.unsafe(`TRUNCATE sources CASCADE`);
// Default source with NO local_path.
await conn.unsafe(
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', NULL)`,
);
const path = await getDefaultSourcePath(engine);
expect(path).toBeNull();
} finally {
cleanup();
}
});
test('manageGitignore on Postgres engine does NOT emit PGLite warning', async () => {
try {
writeGbrainYml();
const warnings: string[] = [];
const orig = console.warn;
console.warn = (...a: unknown[]) => warnings.push(a.map(String).join(' '));
try {
manageGitignore(tmp, 'postgres');
} finally {
console.warn = orig;
}
expect(warnings.filter((w) => /limited effect on PGLite/.test(w))).toEqual([]);
} finally {
cleanup();
}
});
});
}
+167
View File
@@ -0,0 +1,167 @@
/**
* E2E test for parallel sync against real Postgres.
*
* T2 happy path: 60-file sync at concurrency=4 against PostgresEngine
* actually constructs N worker engines, imports correctly, and does
* not leak connections (probe pg_stat_activity before/after).
* P4 benchmark: serial vs concurrency=4 timing on the same fixture so
* the v0.22.13 CHANGELOG can quote a real number instead of "~4×".
*
* Gated on DATABASE_URL. Run via:
* docker run -d --name gbrain-test-pg -e POSTGRES_USER=postgres \
* -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test \
* -p 5435:5432 pgvector/pgvector:pg16
* DATABASE_URL=postgresql://postgres:postgres@localhost:5435/gbrain_test \
* bun test test/e2e/sync-parallel.test.ts
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { execSync } from 'child_process';
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E sync-parallel tests (DATABASE_URL not set)');
}
function seedRepo(repoPath: string, fileCount: number): string {
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
mkdirSync(join(repoPath, 'people'), { recursive: true });
for (let i = 0; i < fileCount; i++) {
writeFileSync(join(repoPath, `people/p${i}.md`), [
'---',
'type: person',
`title: Person ${i}`,
'---',
'',
`Person ${i} body — some text long enough to chunk.`,
`Iteration index ${i}, generated by sync-parallel E2E.`,
].join('\n'));
}
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
return execSync('git rev-parse HEAD', { cwd: repoPath, encoding: 'utf-8' }).trim();
}
async function activeConnections(): Promise<number> {
const conn = getConn();
const rows = await conn.unsafe(`
SELECT count(*) AS n FROM pg_stat_activity
WHERE datname = current_database()
AND state IS NOT NULL
`) as Array<{ n: string }>;
return parseInt(rows[0]?.n ?? '0', 10);
}
describeE2E('E2E sync-parallel: T2 happy path + leak probe', () => {
let repoPath: string;
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
await teardownDB();
});
test('60-file Postgres sync at concurrency=4 imports all + no connection leak', async () => {
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-e2e-par-'));
seedRepo(repoPath, 60);
const before = await activeConnections();
const { performSync } = await import('../../src/commands/sync.ts');
const engine = getEngine();
const result = await performSync(engine, {
repoPath,
noPull: true,
noEmbed: true,
concurrency: 4,
});
// First sync routes through performFullSync (delegates to runImport which
// also accepts --workers); status is 'first_sync'.
expect(result.status).toBe('first_sync');
const after = await activeConnections();
// Allow some slack — the helper engine + sync's normal pool stay open.
// Worker engines (4 × 2 = 8 connections) MUST have closed; if they
// hadn't, after - before would be at least 8.
expect(after - before).toBeLessThan(4);
// Verify pages are actually in the DB (via raw SQL — engine API also works).
const conn = getConn();
const pageRows = await conn.unsafe(
`SELECT count(*) AS n FROM pages WHERE slug LIKE 'people/p%'`,
) as Array<{ n: string }>;
const count = parseInt(pageRows[0]?.n ?? '0', 10);
expect(count).toBe(60);
}, 60_000);
});
describeE2E('E2E sync-parallel: P4 benchmark serial vs concurrency=4', () => {
let repoSerial: string;
let repoParallel: string;
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
if (repoSerial) rmSync(repoSerial, { recursive: true, force: true });
if (repoParallel) rmSync(repoParallel, { recursive: true, force: true });
await teardownDB();
});
test('120-file benchmark: report serial and parallel wall-clock', async () => {
// Two separate repos so neither sync's chunks bleed into the other.
repoSerial = mkdtempSync(join(tmpdir(), 'gbrain-bench-serial-'));
repoParallel = mkdtempSync(join(tmpdir(), 'gbrain-bench-parallel-'));
seedRepo(repoSerial, 120);
seedRepo(repoParallel, 120);
const { performSync } = await import('../../src/commands/sync.ts');
const engine = getEngine();
// Truncate between runs to keep the benchmark honest.
const conn = getConn();
const t1 = Date.now();
await performSync(engine, {
repoPath: repoSerial,
noPull: true,
noEmbed: true,
concurrency: 1,
});
const serialMs = Date.now() - t1;
// Wipe pages before second run so neither one is "incremental".
await conn.unsafe(`TRUNCATE pages CASCADE`);
await conn.unsafe(`TRUNCATE config CASCADE`);
const t2 = Date.now();
await performSync(engine, {
repoPath: repoParallel,
noPull: true,
noEmbed: true,
concurrency: 4,
});
const parallelMs = Date.now() - t2;
const speedup = (serialMs / parallelMs).toFixed(2);
// Emit as a single line stdout consumers can grep for.
console.log(`SYNC_PARALLEL_BENCH 120 files | serial=${serialMs}ms | parallel(4)=${parallelMs}ms | speedup=${speedup}x`);
// Soft assertion: parallel must not be slower than serial. The actual
// speedup ratio depends heavily on Postgres latency profile and is what
// the CHANGELOG quotes — don't gate the test on a specific multiplier.
expect(parallelMs).toBeLessThanOrEqual(serialMs * 1.5); // +50% slack for noisy CI
}, 120_000);
});
+162 -2
View File
@@ -10,10 +10,10 @@
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, unlinkSync } from 'fs';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, unlinkSync, existsSync, readFileSync } from 'fs';
import { join } from 'path';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { tmpdir, homedir } from 'os';
import {
hasDatabase, setupDB, teardownDB, getEngine,
} from './helpers.ts';
@@ -394,3 +394,163 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
expect(page!.title).toBe('Draft Meeting Notes');
});
});
/**
* E2E: --skip-failed loop with structured error code summary.
*
* Closes the v0.22.12 ship-blocker gap from issue #500 the whole code path
* (record classify block skip doctor render second cycle) had only
* mocked-JSONL unit coverage. This is the integration test that proves the
* chain holds together with a real Postgres engine, real git history, and
* real frontmatter validation.
*
* Owns its own repo + sync-failures.jsonl lifecycle so it can't leak state
* into the shared describeE2E above. Saves and restores the user's real
* ~/.gbrain/sync-failures.jsonl so running E2E on a developer machine
* doesn't trash their local sync state.
*/
describeE2E('E2E: sync --skip-failed structured summary loop (v0.22.12, issue #500)', () => {
let repoPath: string;
const realFailuresPath = join(homedir(), '.gbrain', 'sync-failures.jsonl');
let savedFailuresContent: string | null = null;
beforeAll(async () => {
await setupDB();
// Save+clear the real ~/.gbrain/sync-failures.jsonl so the test starts from
// a known-empty state. Restored in afterAll. This file is per-machine, NOT
// per-repo, so we have to be defensive about a developer running this
// suite on their actual brain machine.
if (existsSync(realFailuresPath)) {
savedFailuresContent = readFileSync(realFailuresPath, 'utf-8');
unlinkSync(realFailuresPath);
}
// Fresh git repo with one valid file. Mirrors createTestRepo above but
// scoped to this describe block.
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-skipfailed-e2e-'));
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
mkdirSync(join(repoPath, 'people'), { recursive: true });
writeFileSync(join(repoPath, 'people/alice.md'), [
'---', 'type: person', 'title: Alice', '---', '', 'Body.',
].join('\n'));
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
});
afterAll(async () => {
await teardownDB();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
// Restore the user's real sync-failures.jsonl, if any.
if (savedFailuresContent !== null) {
mkdirSync(join(homedir(), '.gbrain'), { recursive: true });
writeFileSync(realFailuresPath, savedFailuresContent);
} else if (existsSync(realFailuresPath)) {
// Test wrote one but there was none before. Clean up.
unlinkSync(realFailuresPath);
}
});
test('full --skip-failed loop: blocks on bad file, skip advances bookmark, doctor shows code breakdown', async () => {
const { performSync } = await import('../../src/commands/sync.ts');
const { loadSyncFailures, summarizeFailuresByCode } = await import('../../src/core/sync.ts');
const engine = getEngine();
// Step 1: First sync of the clean repo — should succeed.
let result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
expect(result.status).toBe('first_sync');
const firstCommit = await engine.getConfig('sync.last_commit');
expect(firstCommit).toBeTruthy();
// Step 2: Add a broken file — frontmatter slug doesn't match path-derived slug.
// The file path is people/bob.md so the path-derived slug is "people/bob",
// but we declare slug: "wrong-slug" in frontmatter. import-file.ts:368-377
// raises "Frontmatter slug ... does not match path-derived slug ..." which
// classifier hits as SLUG_MISMATCH.
writeFileSync(join(repoPath, 'people/bob.md'), [
'---', 'type: person', 'title: Bob', 'slug: wrong-slug', '---', '', 'Body.',
].join('\n'));
execSync('git add -A && git commit -m "add broken bob"', { cwd: repoPath, stdio: 'pipe' });
// Step 3: Sync should block. Bookmark must NOT advance.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
expect(result.status).toBe('blocked_by_failures');
const afterBlockedCommit = await engine.getConfig('sync.last_commit');
expect(afterBlockedCommit).toBe(firstCommit); // bookmark stuck at the pre-broken commit
// JSONL has one unacked entry with code SLUG_MISMATCH.
let failures = loadSyncFailures();
expect(failures.length).toBe(1);
expect(failures[0].code).toBe('SLUG_MISMATCH');
expect(failures[0].acknowledged).toBeFalsy();
// Group summary aggregates correctly across the unacked set.
expect(summarizeFailuresByCode(failures)).toEqual([{ code: 'SLUG_MISMATCH', count: 1 }]);
// Step 4: Run with skipFailed — bookmark advances, entry gets acked.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true, skipFailed: true });
expect(result.status).toBe('synced');
const afterSkipCommit = await engine.getConfig('sync.last_commit');
expect(afterSkipCommit).not.toBe(firstCommit); // bookmark moved past the broken commit
failures = loadSyncFailures();
expect(failures.length).toBe(1);
expect(failures[0].acknowledged).toBe(true);
expect(typeof failures[0].acknowledged_at).toBe('string');
// Step 5: Verify what doctor would render for the historical entry.
// We call the same primitives doctor's `sync_failures` check uses
// (src/commands/doctor.ts:252-275) — loadSyncFailures + summarizeFailuresByCode —
// and assert the rendering string. Directly invoking runDoctor() here is a CLI
// entrypoint with stdout/exit side effects that would truncate this test mid-flow.
{
const all = loadSyncFailures();
const ackedSummary = summarizeFailuresByCode(all);
const ackedBreakdown = ackedSummary.map(s => `${s.code}=${s.count}`).join(', ');
// This is the literal string interpolation doctor.ts:271-274 produces.
const doctorMessage = `${all.length} historical sync failure(s), all acknowledged [${ackedBreakdown}].`;
expect(doctorMessage).toContain('SLUG_MISMATCH=1');
expect(doctorMessage).toContain('1 historical');
}
// Step 6: Add a second broken file — this one with a different failure code
// (also SLUG_MISMATCH but on a different file) so the JSONL has 2 entries
// with DIFFERENT paths but the same code. This proves both: per-file dedup
// honors path identity, and summary aggregation sums across files.
//
// We'd ideally test a different code class here, but the sync path uses
// parseMarkdown WITHOUT {validate:true}, so the markdown.ts validation
// codes (MISSING_OPEN/CLOSE, NESTED_QUOTES, EMPTY_FRONTMATTER, NULL_BYTES)
// don't naturally surface — they'd need {validate:true} plumbed in. That
// plumbing is the v0.22.13+ follow-up. For v0.22.12, two SLUG_MISMATCH
// entries from different files still proves the dedup + aggregation chain.
writeFileSync(join(repoPath, 'people/carol.md'), [
'---', 'type: person', 'title: Carol', 'slug: also-wrong-slug', '---', '', 'Body.',
].join('\n'));
execSync('git add -A && git commit -m "add carol with bad slug"', { cwd: repoPath, stdio: 'pipe' });
// Step 7: Sync blocks again on the new failure. Old entry stays acked.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
expect(result.status).toBe('blocked_by_failures');
failures = loadSyncFailures();
expect(failures.length).toBe(2);
const acked = failures.filter(f => f.acknowledged);
const unacked = failures.filter(f => !f.acknowledged);
expect(acked.length).toBe(1);
expect(acked[0].code).toBe('SLUG_MISMATCH');
expect(acked[0].path).toContain('bob');
expect(unacked.length).toBe(1);
expect(unacked[0].code).toBe('SLUG_MISMATCH');
expect(unacked[0].path).toContain('carol');
// Step 8: Skip again — both entries acked, summary aggregates the count.
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true, skipFailed: true });
expect(result.status).toBe('synced');
failures = loadSyncFailures();
expect(failures.length).toBe(2);
expect(failures.every(f => f.acknowledged)).toBe(true);
const finalSummary = summarizeFailuresByCode(failures);
expect(finalSummary).toEqual([{ code: 'SLUG_MISMATCH', count: 2 }]);
});
});
+15 -4
View File
@@ -7,28 +7,39 @@
*
* All tests use PGLite/in-memory no DB connection required.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { runExtractCore } from '../src/commands/extract.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import type { BrainEngine } from '../src/core/engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
// One PGLite per file (beforeAll), wipe data per test (beforeEach).
// PGLite cold-start dominates wall-time; sharing the engine across all tests
// in this file cuts ~22s × 8 tests = ~3 min on CI.
let engine: PGLiteEngine;
let tempDir: string;
beforeEach(async () => {
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' });
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
tempDir = mkdtempSync(join(tmpdir(), 'gbrain-extract-test-'));
mkdirSync(join(tempDir, 'people'), { recursive: true });
mkdirSync(join(tempDir, 'companies'), { recursive: true });
});
afterEach(async () => {
await engine.disconnect();
afterEach(() => {
rmSync(tempDir, { recursive: true, force: true });
});
+102
View File
@@ -120,3 +120,105 @@ describe('autopilot-cycle handler — partial failure does NOT throw', () => {
}
}, 30_000);
});
describe('autopilot-cycle handler — phase passthrough', () => {
test('job.data.phases restricts which phases run', async () => {
const fs = await import('fs');
const { execSync } = await import('child_process');
const { tmpdir } = await import('os');
const { join } = await import('path');
const dir = fs.mkdtempSync(join(tmpdir(), 'gbrain-phase-pass-'));
try {
execSync('git init', { cwd: dir, stdio: 'pipe' });
execSync('git config user.email test@example.com', { cwd: dir, stdio: 'pipe' });
execSync('git config user.name Test', { cwd: dir, stdio: 'pipe' });
execSync('git commit --allow-empty -m init', { cwd: dir, stdio: 'pipe' });
const handler = (worker as any).handlers.get('autopilot-cycle');
// Request only lint and sync — embed should NOT appear
const result = await handler({
data: { repoPath: dir, phases: ['lint', 'sync'] },
signal: { aborted: false } as any,
job: { id: 10, name: 'autopilot-cycle' } as any,
});
expect(result).toBeDefined();
const report = (result as any).report;
expect(report).toBeDefined();
const phaseNames = report.phases.map((p: any) => p.phase);
expect(phaseNames).toContain('lint');
expect(phaseNames).toContain('sync');
// Phases NOT requested must be absent
expect(phaseNames).not.toContain('embed');
expect(phaseNames).not.toContain('extract');
expect(phaseNames).not.toContain('backlinks');
expect(phaseNames).not.toContain('orphans');
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
}, 30_000);
test('invalid phase names in job.data.phases are filtered out', async () => {
const fs = await import('fs');
const { execSync } = await import('child_process');
const { tmpdir } = await import('os');
const { join } = await import('path');
const dir = fs.mkdtempSync(join(tmpdir(), 'gbrain-phase-invalid-'));
try {
execSync('git init', { cwd: dir, stdio: 'pipe' });
execSync('git config user.email test@example.com', { cwd: dir, stdio: 'pipe' });
execSync('git config user.name Test', { cwd: dir, stdio: 'pipe' });
execSync('git commit --allow-empty -m init', { cwd: dir, stdio: 'pipe' });
const handler = (worker as any).handlers.get('autopilot-cycle');
// Mix valid and bogus names — only 'lint' should survive filtering
const result = await handler({
data: { repoPath: dir, phases: ['lint', 'BOGUS', 'rm -rf /'] },
signal: { aborted: false } as any,
job: { id: 11, name: 'autopilot-cycle' } as any,
});
const report = (result as any).report;
const phaseNames = report.phases.map((p: any) => p.phase);
expect(phaseNames).toContain('lint');
expect(phaseNames).not.toContain('BOGUS');
expect(phaseNames.length).toBe(1);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
}, 30_000);
test('empty phases array falls back to all phases (same as no phases)', async () => {
const handler = (worker as any).handlers.get('autopilot-cycle');
// Empty array should fall through to ALL_PHASES (same as omitting phases)
const result = await handler({
data: { repoPath: '/definitely-does-not-exist-for-phase-test', phases: [] },
signal: { aborted: false } as any,
job: { id: 12, name: 'autopilot-cycle' } as any,
});
const report = (result as any).report;
// With all phases, filesystem phases fail on missing dir
const phaseNames = report.phases.map((p: any) => p.phase);
expect(phaseNames).toContain('lint');
expect(phaseNames).toContain('backlinks');
expect(phaseNames).toContain('sync');
}, 30_000);
test('non-array phases value is ignored (falls back to all)', async () => {
const handler = (worker as any).handlers.get('autopilot-cycle');
// String instead of array — should be ignored
const result = await handler({
data: { repoPath: '/definitely-does-not-exist-for-phase-test', phases: 'lint' },
signal: { aborted: false } as any,
job: { id: 13, name: 'autopilot-cycle' } as any,
});
const report = (result as any).report;
const phaseNames = report.phases.map((p: any) => p.phase);
// Should have all phases since the string was ignored
expect(phaseNames).toContain('lint');
expect(phaseNames).toContain('sync');
expect(phaseNames).toContain('embed');
}, 30_000);
});
+43
View File
@@ -0,0 +1,43 @@
/**
* Wipe per-test data on a connected PGLite engine without dropping the schema.
* Used by tests that share one engine across the file (beforeAll) and need a
* clean slate per test (beforeEach).
*
* Why this exists: PGLite WASM cold-start + initSchema() is ~20s on CI runners.
* Spinning up a fresh engine per test (the prior beforeEach pattern) multiplies
* that across every test in every file. Sharing one engine and wiping data
* is two orders of magnitude faster.
*
* Implementation:
* 1. TRUNCATE every public table CASCADE, including `sources` (so tests
* that register their own sources don't leak rows into the next test).
* 2. Re-seed the default source row that pages.source_id's DEFAULT FKs
* against. Without this, the next page insert would fail FK validation.
* 3. Preserve `schema_version` it carries the migration ledger that
* initSchema() populates; wiping it would make migration helpers think
* the brain is on v0.
*
* Identifier-quoted defensively against pathological table names.
*/
import type { PGLiteEngine } from '../../src/core/pglite-engine.ts';
const PRESERVE_TABLES = new Set(['schema_version']);
export async function resetPgliteState(engine: PGLiteEngine): Promise<void> {
const rows = await engine.executeRaw<{ tablename: string }>(
`SELECT tablename FROM pg_tables WHERE schemaname='public'`,
);
const targets = rows
.map(r => r.tablename)
.filter(name => !PRESERVE_TABLES.has(name));
if (targets.length === 0) return;
const quoted = targets.map(t => `"${t.replace(/"/g, '""')}"`).join(', ');
await engine.executeRaw(`TRUNCATE ${quoted} RESTART IDENTITY CASCADE`);
// Re-seed the default source row that initSchema() inserts. Mirrors the
// INSERT in src/core/pglite-schema.ts so the FK target survives reset.
await engine.executeRaw(
`INSERT INTO sources (id, name, config)
VALUES ('default', 'default', '{"federated": true}'::jsonb)
ON CONFLICT (id) DO NOTHING`,
);
}
+31
View File
@@ -101,6 +101,37 @@ describe('PGLiteEngine: Pages', () => {
expect(tagged[0].slug).toBe('test/tagged');
});
test('listPages with slugPrefix filter (Issue #13)', async () => {
await truncateAll();
await engine.putPage('media/x/tweet-1', { ...testPage, type: 'concept' });
await engine.putPage('media/x/tweet-2', { ...testPage, type: 'concept' });
await engine.putPage('media/articles/post-1', { ...testPage, type: 'concept' });
await engine.putPage('people/alice', { ...testPage, type: 'person' });
const xOnly = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
expect(xOnly.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
const allMedia = await engine.listPages({ slugPrefix: 'media/', limit: 100 });
expect(allMedia.length).toBe(3);
// Path-segment risk: 'media/x' (no trailing /) would also match 'media/xerox'.
// The matcher in storage-config.ts is responsible for trailing-/ semantics
// (step 6); the engine treats slugPrefix as a literal string prefix.
expect((await engine.listPages({ slugPrefix: 'media/x', limit: 100 })).length).toBe(2);
});
test('listPages slugPrefix escapes LIKE metacharacters', async () => {
await truncateAll();
await engine.putPage('safe/foo', { ...testPage, type: 'concept' });
// A user prefix containing % or _ would otherwise match unintended slugs
// if not escaped. We can't easily insert a slug with % in it (most slugs
// are url-safe), but we can confirm the escape logic doesn't break the
// happy path.
const result = await engine.listPages({ slugPrefix: 'safe/', limit: 10 });
expect(result.length).toBe(1);
expect(result[0].slug).toBe('safe/foo');
});
test('resolveSlugs exact match', async () => {
await engine.putPage('test/exact', testPage);
const slugs = await engine.resolveSlugs('test/exact');
+98 -1
View File
@@ -14,7 +14,7 @@ import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { resolveSourceId, __testing } from '../src/core/source-resolver.ts';
import { resolveSourceId, getDefaultSourcePath, __testing } from '../src/core/source-resolver.ts';
import type { BrainEngine } from '../src/core/engine.ts';
// ── Stub engine ────────────────────────────────────────────
@@ -174,6 +174,103 @@ describe('resolveSourceId priority 6 — fallback', () => {
});
});
// ── getDefaultSourcePath ───────────────────────────────────
describe('getDefaultSourcePath', () => {
function makeStubWithPaths(
registeredSources: string[],
sourcePaths: Record<string, string | null>,
defaultKey: string | null,
): BrainEngine {
return {
kind: 'pglite',
executeRaw: async <T>(sql: string, params?: unknown[]): Promise<T[]> => {
if (sql.includes('SELECT id FROM sources WHERE id = $1')) {
const target = params?.[0];
return (registeredSources.includes(target as string)
? [{ id: target } as unknown as T]
: []);
}
if (sql.includes('SELECT local_path FROM sources WHERE id = $1')) {
const target = params?.[0] as string;
if (target in sourcePaths) {
return [{ local_path: sourcePaths[target] } as unknown as T];
}
return [];
}
if (sql.includes('SELECT id, local_path FROM sources')) {
return Object.entries(sourcePaths)
.filter(([_, p]) => p !== null)
.map(([id, local_path]) => ({ id, local_path }) as unknown as T);
}
return [];
},
getConfig: async (key: string) => (key === 'sources.default' ? defaultKey : null),
} as unknown as BrainEngine;
}
test('returns local_path of resolved default source', async () => {
const engine = makeStubWithPaths(['default'], { default: '/path/to/brain' }, null);
const path = await getDefaultSourcePath(engine, '/random/dir');
expect(path).toBe('/path/to/brain');
});
test('returns null when source has no local_path', async () => {
const engine = makeStubWithPaths(['default'], { default: null }, null);
const path = await getDefaultSourcePath(engine, '/random/dir');
expect(path).toBeNull();
});
test('throws on DB error (does not silently swallow)', async () => {
const engine = {
kind: 'pglite',
executeRaw: async () => {
throw new Error('connection refused');
},
getConfig: async () => null,
} as unknown as BrainEngine;
await expect(getDefaultSourcePath(engine, '/random/dir')).rejects.toThrow(/connection refused/);
});
test('falls back to legacy sync.repo_path config when sources.local_path is null', async () => {
// Pre-v0.18 brains: 'default' source exists but local_path is NULL; the
// repo path lives in the global config table under sync.repo_path.
const engine = {
kind: 'pglite',
executeRaw: async <T>(sql: string, params?: unknown[]): Promise<T[]> => {
if (sql.includes('SELECT id FROM sources WHERE id = $1')) {
return [{ id: params?.[0] } as unknown as T];
}
if (sql.includes('SELECT local_path FROM sources WHERE id = $1')) {
return [{ local_path: null } as unknown as T];
}
if (sql.includes('SELECT id, local_path FROM sources')) {
return [];
}
return [];
},
getConfig: async (key: string) => {
if (key === 'sources.default') return null;
if (key === 'sync.repo_path') return '/legacy/brain/path';
return null;
},
} as unknown as BrainEngine;
const path = await getDefaultSourcePath(engine, '/random/dir');
expect(path).toBe('/legacy/brain/path');
});
test('respects source resolution chain (registered local_path wins over default)', async () => {
// CWD is inside /custom/path → wiki source matches by path → wiki's local_path returned.
const engine = makeStubWithPaths(
['default', 'wiki'],
{ default: '/default/path', wiki: '/custom/path' },
'default',
);
const path = await getDefaultSourcePath(engine, '/custom/path/sub');
expect(path).toBe('/custom/path');
});
});
// ── Regex validation ───────────────────────────────────────
describe('SOURCE_ID_RE', () => {
+336
View File
@@ -0,0 +1,336 @@
import { test, expect, describe, beforeEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import {
validateStorageConfig,
isGitTracked,
isSupabaseOnly,
getStorageTier,
loadStorageConfig,
normalizeAndValidateStorageConfig,
StorageConfigError,
__resetMissingStorageWarning,
} from '../src/core/storage-config.ts';
import type { StorageConfig } from '../src/core/storage-config.ts';
describe('Storage Configuration', () => {
const testConfig: StorageConfig = {
db_tracked: ['people/', 'companies/', 'deals/'],
db_only: ['media/x/', 'media/articles/', 'meetings/transcripts/'],
};
describe('validateStorageConfig', () => {
test('should return no warnings for valid config', () => {
const warnings = validateStorageConfig(testConfig);
expect(warnings).toEqual([]);
});
test('should warn about overlap between db_tracked and db_only', () => {
const invalidConfig: StorageConfig = {
db_tracked: ['people/', 'media/'],
db_only: ['media/', 'articles/'],
};
const warnings = validateStorageConfig(invalidConfig);
expect(warnings).toContain('Directory "media/" appears in both db_tracked and db_only');
});
test('should warn about paths not ending with /', () => {
const invalidConfig: StorageConfig = {
db_tracked: ['people', 'companies/'],
db_only: ['media/x/', 'articles'],
};
const warnings = validateStorageConfig(invalidConfig);
expect(warnings).toContain('Directory path "people" should end with "/" for consistency');
expect(warnings).toContain('Directory path "articles" should end with "/" for consistency');
});
});
describe('Storage tier detection', () => {
test('identifies db-tracked pages', () => {
expect(isGitTracked('people/john-doe', testConfig)).toBe(true);
expect(isGitTracked('companies/acme-corp', testConfig)).toBe(true);
expect(isGitTracked('deals/series-a', testConfig)).toBe(true);
});
test('identifies db-only pages', () => {
expect(isSupabaseOnly('media/x/tweet-123', testConfig)).toBe(true);
expect(isSupabaseOnly('media/articles/blog-post', testConfig)).toBe(true);
expect(isSupabaseOnly('meetings/transcripts/standup', testConfig)).toBe(true);
});
test('returns false for non-matching paths', () => {
expect(isGitTracked('media/x/tweet-123', testConfig)).toBe(false);
expect(isSupabaseOnly('people/john-doe', testConfig)).toBe(false);
});
test('correctly determines storage tier (canonical names)', () => {
expect(getStorageTier('people/john-doe', testConfig)).toBe('db_tracked');
expect(getStorageTier('media/x/tweet-123', testConfig)).toBe('db_only');
expect(getStorageTier('projects/random-thing', testConfig)).toBe('unspecified');
});
test('handles prefix edge cases', () => {
expect(isGitTracked('people', testConfig)).toBe(false);
expect(isGitTracked('people/', testConfig)).toBe(true);
expect(isGitTracked('peoplex/test', testConfig)).toBe(false);
expect(isSupabaseOnly('mediax/test', testConfig)).toBe(false);
});
test('normalizeAndValidateStorageConfig auto-adds trailing slash silently with info note', () => {
__resetMissingStorageWarning();
const warnings: string[] = [];
const orig = console.warn;
console.warn = (...a: unknown[]) => { warnings.push(a.map(String).join(' ')); };
try {
const out = normalizeAndValidateStorageConfig({
db_tracked: ['people', 'companies/'],
db_only: ['media/x'],
});
expect(out.db_tracked).toEqual(['people/', 'companies/']);
expect(out.db_only).toEqual(['media/x/']);
expect(warnings.length).toBe(1);
expect(warnings[0]).toMatch(/normalized.*"people".*"people\/".*"media\/x".*"media\/x\/"/);
} finally {
console.warn = orig;
}
});
test('normalizeAndValidateStorageConfig throws on tier overlap', () => {
__resetMissingStorageWarning();
expect(() =>
normalizeAndValidateStorageConfig({
db_tracked: ['media/'],
db_only: ['media/'],
}),
).toThrow(StorageConfigError);
});
test('regression — media/xerox does NOT match media/x (path-segment matcher)', () => {
// Without path-segment matching, slug.startsWith('media/x') would falsely
// match 'media/xerox/foo'. The new matcher requires trailing '/'; if the
// user's config has 'media/x' (no slash), the matcher refuses to match —
// the validator's auto-normalize (step 7) ensures canonical input.
const collisionConfig: StorageConfig = {
db_tracked: [],
db_only: ['media/x/'], // canonical, with trailing slash
};
expect(isSupabaseOnly('media/xerox/something', collisionConfig)).toBe(false);
expect(isSupabaseOnly('media/x/tweet-1', collisionConfig)).toBe(true);
// Non-canonical input (no trailing slash) is refused by the matcher.
const noSlashConfig: StorageConfig = {
db_tracked: [],
db_only: ['media/x'],
};
expect(isSupabaseOnly('media/xerox/foo', noSlashConfig)).toBe(false);
expect(isSupabaseOnly('media/x/tweet-1', noSlashConfig)).toBe(false);
});
});
});
describe('loadStorageConfig — real-disk loader', () => {
let tmp: string;
let originalWarn: typeof console.warn;
let warnings: string[];
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-storage-test-'));
__resetMissingStorageWarning();
warnings = [];
originalWarn = console.warn;
console.warn = (...args: unknown[]) => {
warnings.push(args.map(String).join(' '));
};
});
function cleanup(): void {
console.warn = originalWarn;
rmSync(tmp, { recursive: true, force: true });
}
test('returns null when repoPath is missing', () => {
try {
expect(loadStorageConfig(undefined)).toBeNull();
expect(loadStorageConfig(null)).toBeNull();
expect(loadStorageConfig('')).toBeNull();
} finally {
cleanup();
}
});
test('returns null when gbrain.yml does not exist', () => {
try {
expect(loadStorageConfig(tmp)).toBeNull();
expect(warnings).toEqual([]);
} finally {
cleanup();
}
});
test('loads canonical gbrain.yml — the test that would have caught the original gray-matter P0', () => {
try {
const yaml = `# Brain storage tiering config
storage:
db_tracked:
- people/
- companies/
- deals/
db_only:
- media/x/
- media/articles/
`;
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
const config = loadStorageConfig(tmp);
expect(config).not.toBeNull();
expect(config!.db_tracked).toEqual(['people/', 'companies/', 'deals/']);
expect(config!.db_only).toEqual(['media/x/', 'media/articles/']);
expect(warnings).toEqual([]);
} finally {
cleanup();
}
});
test('handles inline comments and blank lines', () => {
try {
const yaml = `
storage:
db_tracked:
- people/ # human-curated
- companies/
db_only:
- media/x/ # bulk tweets
`;
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
const config = loadStorageConfig(tmp);
expect(config!.db_tracked).toEqual(['people/', 'companies/']);
expect(config!.db_only).toEqual(['media/x/']);
} finally {
cleanup();
}
});
test('strips quoted values', () => {
try {
const yaml = `storage:
db_tracked:
- "people/"
- 'companies/'
db_only: []
`;
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
const config = loadStorageConfig(tmp);
expect(config!.db_tracked).toEqual(['people/', 'companies/']);
} finally {
cleanup();
}
});
test('reads deprecated keys (git_tracked / supabase_only) with once-per-process warning', () => {
try {
const yaml = `storage:
git_tracked:
- people/
supabase_only:
- media/x/
`;
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
const config = loadStorageConfig(tmp);
expect(config!.db_tracked).toEqual(['people/']);
expect(config!.db_only).toEqual(['media/x/']);
expect(warnings.some((w) => /deprecated/.test(w))).toBe(true);
// Second call: no second deprecation warning (once-per-process).
const before = warnings.length;
loadStorageConfig(tmp);
const newWarnings = warnings.slice(before);
expect(newWarnings.filter((w) => /deprecated/.test(w))).toEqual([]);
} finally {
cleanup();
}
});
test('canonical keys win over deprecated keys when both present', () => {
try {
const yaml = `storage:
db_tracked:
- new-people/
git_tracked:
- old-people/
db_only:
- new-media/
supabase_only:
- old-media/
`;
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
const config = loadStorageConfig(tmp);
expect(config!.db_tracked).toEqual(['new-people/']);
expect(config!.db_only).toEqual(['new-media/']);
// Stronger deprecation warning when both shapes coexist.
expect(warnings.some((w) => /deprecated.*ignored/.test(w))).toBe(true);
} finally {
cleanup();
}
});
test('warns once when gbrain.yml exists but storage section is missing', () => {
try {
writeFileSync(join(tmp, 'gbrain.yml'), 'something_else: foo\n');
const config = loadStorageConfig(tmp);
expect(config).toBeNull();
expect(warnings.length).toBe(1);
expect(warnings[0]).toMatch(/no storage configuration/);
// Second call: no additional warning (once-per-process).
loadStorageConfig(tmp);
expect(warnings.length).toBe(1);
} finally {
cleanup();
}
});
test('warns when storage section is empty', () => {
try {
const yaml = `storage:
db_tracked: []
db_only: []
`;
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
const config = loadStorageConfig(tmp);
// Empty config is returned (not null) but warning fires.
expect(config).not.toBeNull();
expect(config!.db_tracked).toEqual([]);
expect(config!.db_only).toEqual([]);
const noConfigWarnings = warnings.filter((w) => /no storage configuration/.test(w));
expect(noConfigWarnings.length).toBe(1);
} finally {
cleanup();
}
});
test('throws on unreadable gbrain.yml (permission denied) — does not silently disable feature', () => {
try {
const yamlPath = join(tmp, 'gbrain.yml');
writeFileSync(yamlPath, 'storage:\n db_tracked:\n - x/\n');
// Simulate unreadable: chmod 000. May not work on all CI; skip if not supported.
const fs = require('fs');
fs.chmodSync(yamlPath, 0o000);
try {
// On systems where chmod 000 actually denies read, this throws.
// On systems where root can still read (CI containers), the read succeeds
// and the test is a no-op assertion.
try {
fs.readFileSync(yamlPath, 'utf-8');
// Read succeeded — skip strict assertion.
} catch {
expect(() => loadStorageConfig(tmp)).toThrow();
}
} finally {
fs.chmodSync(yamlPath, 0o644);
}
} finally {
cleanup();
}
});
});
+146
View File
@@ -0,0 +1,146 @@
/**
* Tests for export.ts --restore-only resolution chain step 9 of v0.22.3.
*
* D5: --repo sources.getDefault() hard error. Never fall through to
* cwd. Issue #9: bare try/catch removed from storage.ts:37.
*
* Tests use PGLite in-memory and a captured-output approach (process.exit
* is intercepted) to verify the resolution chain produces the right
* repoPath OR the right error.
*/
import { describe, test, expect, beforeEach, afterEach, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runExport } from '../src/commands/export.ts';
import { __resetMissingStorageWarning } from '../src/core/storage-config.ts';
let engine: PGLiteEngine;
let tmp: string;
let outDir: string;
let exitCode: number | null;
let originalExit: typeof process.exit;
let originalErr: typeof console.error;
let originalLog: typeof console.log;
let stderr: string[];
let stdout: string[];
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-export-test-'));
outDir = join(tmp, 'out');
exitCode = null;
stderr = [];
stdout = [];
__resetMissingStorageWarning();
originalExit = process.exit;
process.exit = ((code?: number) => {
exitCode = code ?? 0;
throw new Error(`__test_exit__:${code}`);
}) as typeof process.exit;
originalErr = console.error;
console.error = (...args: unknown[]) => {
stderr.push(args.map(String).join(' '));
};
originalLog = console.log;
console.log = (...args: unknown[]) => {
stdout.push(args.map(String).join(' '));
};
// Reset DB state between tests
const tables = ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages', 'sources'];
for (const t of tables) {
await (engine as unknown as { db: { exec(sql: string): Promise<unknown> } }).db.exec(`DELETE FROM ${t}`);
}
// Recreate the default source (the schema seed but truncated above).
await engine.executeRaw(
`INSERT INTO sources (id, name) VALUES ('default', 'Default') ON CONFLICT DO NOTHING`,
);
});
afterEach(() => {
process.exit = originalExit;
console.error = originalErr;
console.log = originalLog;
rmSync(tmp, { recursive: true, force: true });
});
async function tryRunExport(args: string[]): Promise<void> {
try {
await runExport(engine, args);
} catch (e) {
// Swallow only the test-exit sentinel; rethrow others for visibility.
if (!(e instanceof Error && e.message.startsWith('__test_exit__:'))) {
throw e;
}
}
}
describe('export --restore-only resolution chain (D5)', () => {
test('hard-errors when --restore-only has no --repo and no default source path', async () => {
// sources.default has no local_path (the seeded shape).
await tryRunExport(['--dir', outDir, '--restore-only']);
expect(exitCode).toBe(1);
expect(stderr.join('\n')).toMatch(/requires --repo|configured default source/);
});
test('uses explicit --repo when provided', async () => {
// Make a brain repo with gbrain.yml that has empty db_only — so we
// exit through the "0 pages to restore" path without needing real data.
writeFileSync(
join(tmp, 'gbrain.yml'),
`storage:
db_tracked: []
db_only: []
`,
);
await tryRunExport(['--dir', outDir, '--restore-only', '--repo', tmp]);
expect(exitCode).toBeNull(); // no exit
expect(stdout.some((line) => line.includes('Restoring 0'))).toBe(true);
});
test('falls back to sources default local_path when --repo absent', async () => {
// Configure default source path, write a real gbrain.yml so the storage
// config check passes — without gbrain.yml the Codex-P0 guard correctly
// refuses --restore-only (no storage config to scope to).
await engine.executeRaw(`UPDATE sources SET local_path = $1 WHERE id = 'default'`, [tmp]);
writeFileSync(
join(tmp, 'gbrain.yml'),
`storage:\n db_tracked: []\n db_only:\n - media/x/\n`,
);
await tryRunExport(['--dir', outDir, '--restore-only']);
expect(exitCode).toBeNull(); // resolution succeeded
});
test('refuses --restore-only when no storage config is present (Codex P0)', async () => {
// Default source has a path but no gbrain.yml. Without a storage config,
// --restore-only would silently fall through to a full export — exactly
// the silent-footgun D5 was supposed to prevent.
await engine.executeRaw(`UPDATE sources SET local_path = $1 WHERE id = 'default'`, [tmp]);
await tryRunExport(['--dir', outDir, '--restore-only']);
expect(exitCode).toBe(1);
expect(stderr.join('\n')).toMatch(/storage tiering config|gbrain\.yml/);
});
test('non-restore export does NOT require --repo (D26)', async () => {
// Regular export works without --repo since it dumps everything from DB.
// Pages table is empty → exports 0 pages, no error.
await tryRunExport(['--dir', outDir]);
expect(exitCode).toBeNull();
expect(stdout.some((line) => line.includes('Exporting 0'))).toBe(true);
});
});
+171
View File
@@ -0,0 +1,171 @@
/**
* PGLite lifecycle test for storage tiering D8 + D4 of v0.22.3.
*
* Per the plan: "the full PGLite lifecycle for D8's both-engines requirement.
* gbrain.yml load gbrain storage status soft-warn message present
* manageGitignore happy-path on a tmp dir. PGLite-specific path for the
* slugPrefix filter."
*
* In-memory PGLite, no Docker, no DATABASE_URL. Runs instantly in CI.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import {
getStorageStatus,
formatStorageStatusHuman,
__resetPGLiteWarn,
} from '../src/commands/storage.ts';
import { manageGitignore, __resetPGLiteTierWarn } from '../src/commands/sync.ts';
import { __resetMissingStorageWarning } from '../src/core/storage-config.ts';
let engine: PGLiteEngine;
let tmp: string;
let warnings: string[];
let originalWarn: typeof console.warn;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-pglite-test-'));
__resetMissingStorageWarning();
__resetPGLiteWarn();
__resetPGLiteTierWarn();
warnings = [];
originalWarn = console.warn;
console.warn = (...args: unknown[]) => {
warnings.push(args.map(String).join(' '));
};
// Reset DB between tests.
const tables = ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages', 'sources'];
for (const t of tables) {
await (engine as unknown as { db: { exec(sql: string): Promise<unknown> } }).db.exec(`DELETE FROM ${t}`);
}
await engine.executeRaw(
`INSERT INTO sources (id, name) VALUES ('default', 'Default') ON CONFLICT DO NOTHING`,
);
});
function cleanup(): void {
console.warn = originalWarn;
rmSync(tmp, { recursive: true, force: true });
}
function writeGbrainYml(): void {
writeFileSync(
join(tmp, 'gbrain.yml'),
`storage:
db_tracked:
- people/
db_only:
- media/x/
`,
);
}
describe('Storage tiering on PGLite — full lifecycle (D8 + D4)', () => {
test('engine.kind is pglite', () => {
try {
expect(engine.kind).toBe('pglite');
} finally {
cleanup();
}
});
test('getStorageStatus loads gbrain.yml and reports tier counts', async () => {
try {
writeGbrainYml();
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
await engine.putPage('media/x/tweet-1', { type: 'concept', title: 'Tweet', compiled_truth: '', timeline: '' });
await engine.putPage('media/x/tweet-2', { type: 'concept', title: 'Tweet 2', compiled_truth: '', timeline: '' });
await engine.putPage('random/note', { type: 'concept', title: 'Random', compiled_truth: '', timeline: '' });
const result = await getStorageStatus(engine, tmp);
expect(result.totalPages).toBe(4);
expect(result.pagesByTier.db_tracked).toBe(1);
expect(result.pagesByTier.db_only).toBe(2);
expect(result.pagesByTier.unspecified).toBe(1);
expect(result.config!.db_only).toEqual(['media/x/']);
} finally {
cleanup();
}
});
test('manageGitignore on PGLite emits the D4 soft-warn (once per process)', () => {
try {
writeGbrainYml();
manageGitignore(tmp, 'pglite');
expect(warnings.some((w) => /limited effect on PGLite/.test(w))).toBe(true);
expect(existsSync(join(tmp, '.gitignore'))).toBe(true);
expect(readFileSync(join(tmp, '.gitignore'), 'utf-8')).toContain('media/x/');
// Second call: no second warning (once-per-process).
const before = warnings.length;
manageGitignore(tmp, 'pglite');
const newWarnings = warnings.slice(before).filter((w) => /limited effect on PGLite/.test(w));
expect(newWarnings).toEqual([]);
} finally {
cleanup();
}
});
test('manageGitignore on Postgres does NOT emit the PGLite warning', () => {
try {
writeGbrainYml();
manageGitignore(tmp, 'postgres');
expect(warnings.filter((w) => /limited effect on PGLite/.test(w))).toEqual([]);
} finally {
cleanup();
}
});
test('slugPrefix engine filter works on PGLite (Issue #13)', async () => {
try {
await engine.putPage('media/x/tweet-1', { type: 'concept', title: 'T1', compiled_truth: '', timeline: '' });
await engine.putPage('media/x/tweet-2', { type: 'concept', title: 'T2', compiled_truth: '', timeline: '' });
await engine.putPage('media/articles/post-1', { type: 'concept', title: 'A1', compiled_truth: '', timeline: '' });
const xOnly = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
expect(xOnly.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
} finally {
cleanup();
}
});
test('end-to-end: gbrain.yml + putPage + storage status + .gitignore', async () => {
try {
writeGbrainYml();
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
await engine.putPage('media/x/tweet-1', { type: 'concept', title: 'T1', compiled_truth: '', timeline: '' });
// Status reads tier counts correctly.
const status = await getStorageStatus(engine, tmp);
expect(status.config).not.toBeNull();
expect(status.pagesByTier.db_only).toBe(1);
// Render to human output without errors.
const out = formatStorageStatusHuman(status);
expect(out).toContain('DB only: 1 pages');
// .gitignore management produces a managed block.
manageGitignore(tmp, 'pglite');
const gitignore = readFileSync(join(tmp, '.gitignore'), 'utf-8');
expect(gitignore).toContain('# Auto-managed by gbrain');
expect(gitignore).toContain('media/x/');
} finally {
cleanup();
}
});
});
+111
View File
@@ -0,0 +1,111 @@
/**
* Tests for storage-status formatters step 10 of v0.22.3.
*
* Issue #10 + D14: split storage.ts into pure data + JSON formatter +
* human formatter (matching orphans.ts). Formatters are now pure
* functions; this test file pins their output contracts.
*/
import { describe, test, expect } from 'bun:test';
import {
formatStorageStatusJson,
formatStorageStatusHuman,
type StorageStatusResult,
} from '../src/commands/storage.ts';
const baseResult: StorageStatusResult = {
config: {
db_tracked: ['people/', 'companies/'],
db_only: ['media/x/', 'media/articles/'],
},
repoPath: '/data/brain',
totalPages: 12500,
pagesByTier: { db_tracked: 2156, db_only: 10100, unspecified: 244 },
missingFiles: [],
diskUsageByTier: { db_tracked: 45_200_000, db_only: 2_100_000_000, unspecified: 0 },
warnings: [],
};
describe('formatStorageStatusJson', () => {
test('produces parseable JSON of the StorageStatusResult shape', () => {
const out = formatStorageStatusJson(baseResult);
const parsed = JSON.parse(out);
expect(parsed.repoPath).toBe('/data/brain');
expect(parsed.totalPages).toBe(12500);
expect(parsed.pagesByTier.db_only).toBe(10100);
expect(parsed.config.db_tracked).toEqual(['people/', 'companies/']);
});
test('handles null config (no gbrain.yml present)', () => {
const out = formatStorageStatusJson({ ...baseResult, config: null, totalPages: 5 });
const parsed = JSON.parse(out);
expect(parsed.config).toBeNull();
expect(parsed.totalPages).toBe(5);
});
});
describe('formatStorageStatusHuman', () => {
test('shows tier counts and disk usage when config present', () => {
const out = formatStorageStatusHuman(baseResult);
expect(out).toContain('Storage Status');
expect(out).toContain('Repository: /data/brain');
expect(out).toContain('Total pages: 12500');
expect(out).toContain('DB tracked: 2,156 pages');
expect(out).toContain('DB only: 10,100 pages');
expect(out).toContain('Unspecified: 244 pages');
});
test('shows ASCII separators only — no unicode (D10)', () => {
const out = formatStorageStatusHuman(baseResult);
expect(out).not.toContain('─'); // U+2500 box drawing
expect(out).not.toContain('•'); // U+2022 bullet
expect(out).toContain('-------------'); // ASCII fallback
});
test('shows fallback message when config is null', () => {
const out = formatStorageStatusHuman({ ...baseResult, config: null });
expect(out).toContain('No gbrain.yml configuration found.');
expect(out).toContain('All pages are stored in git by default.');
});
test('shows missing-files block when list is non-empty, capped at 10', () => {
const missing = Array.from({ length: 25 }, (_, i) => ({
slug: `media/x/tweet-${i}`,
expectedPath: `/data/brain/media/x/tweet-${i}.md`,
}));
const out = formatStorageStatusHuman({ ...baseResult, missingFiles: missing });
expect(out).toContain('Missing Files (need restore):');
expect(out).toContain('media/x/tweet-0');
expect(out).toContain('media/x/tweet-9'); // 10th
expect(out).not.toContain('media/x/tweet-10'); // 11th truncated
expect(out).toContain('and 15 more');
expect(out).toContain('gbrain export --restore-only --repo "/data/brain"');
});
test('shows configuration listing for both tiers', () => {
const out = formatStorageStatusHuman(baseResult);
expect(out).toContain('DB tracked directories:');
expect(out).toContain(' - people/');
expect(out).toContain(' - companies/');
expect(out).toContain('DB-only directories:');
expect(out).toContain(' - media/x/');
expect(out).toContain(' - media/articles/');
});
test('shows warnings inline when present', () => {
const out = formatStorageStatusHuman({
...baseResult,
warnings: ['Directory path "people" should end with "/" for consistency'],
});
expect(out).toContain('Warnings:');
expect(out).toContain('! Directory path "people"');
});
test('omits disk-usage block when both tiers report 0 bytes', () => {
const out = formatStorageStatusHuman({
...baseResult,
diskUsageByTier: { db_tracked: 0, db_only: 0, unspecified: 0 },
});
expect(out).not.toContain('Disk Usage:');
});
});
+144
View File
@@ -0,0 +1,144 @@
/**
* Tests for sync.ts manageGitignore() step 8 of v0.22.3 storage tiering.
*
* Issue #2: function was defined but never invoked. Now wired into runSync
* after a successful sync (skips on dry_run / blocked_by_failures / failure).
*
* Tests cover: happy path, idempotency, GBRAIN_NO_GITIGNORE escape hatch,
* submodule detection, write-error graceful degradation, and the "no
* config no-op" path.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, mkdirSync, rmSync, writeFileSync, readFileSync, existsSync, chmodSync, symlinkSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { manageGitignore } from '../src/commands/sync.ts';
import { __resetMissingStorageWarning } from '../src/core/storage-config.ts';
let tmp: string;
let warnings: string[];
let originalWarn: typeof console.warn;
let originalEnv: string | undefined;
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'gbrain-mgi-test-'));
__resetMissingStorageWarning();
warnings = [];
originalWarn = console.warn;
console.warn = (...args: unknown[]) => {
warnings.push(args.map(String).join(' '));
};
originalEnv = process.env.GBRAIN_NO_GITIGNORE;
delete process.env.GBRAIN_NO_GITIGNORE;
});
afterEach(() => {
console.warn = originalWarn;
if (originalEnv === undefined) delete process.env.GBRAIN_NO_GITIGNORE;
else process.env.GBRAIN_NO_GITIGNORE = originalEnv;
// Restore permissions for cleanup.
try {
chmodSync(tmp, 0o755);
} catch {
/* ignore */
}
rmSync(tmp, { recursive: true, force: true });
});
function writeStorageConfig(): void {
writeFileSync(
join(tmp, 'gbrain.yml'),
`storage:
db_tracked:
- people/
db_only:
- media/x/
- media/articles/
`,
);
}
describe('manageGitignore', () => {
test('no-op when gbrain.yml is absent', () => {
manageGitignore(tmp);
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
expect(warnings).toEqual([]);
});
test('no-op when storage config has empty db_only', () => {
writeFileSync(
join(tmp, 'gbrain.yml'),
`storage:
db_tracked:
- people/
db_only: []
`,
);
manageGitignore(tmp);
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
});
test('appends db_only directories to .gitignore — happy path', () => {
writeStorageConfig();
manageGitignore(tmp);
const content = readFileSync(join(tmp, '.gitignore'), 'utf-8');
expect(content).toContain('# Auto-managed by gbrain');
expect(content).toContain('media/x/');
expect(content).toContain('media/articles/');
});
test('idempotent — running twice does NOT duplicate entries', () => {
writeStorageConfig();
manageGitignore(tmp);
manageGitignore(tmp);
const content = readFileSync(join(tmp, '.gitignore'), 'utf-8');
const xCount = (content.match(/^media\/x\/$/gm) || []).length;
const articlesCount = (content.match(/^media\/articles\/$/gm) || []).length;
expect(xCount).toBe(1);
expect(articlesCount).toBe(1);
});
test('preserves user-written .gitignore entries', () => {
writeStorageConfig();
writeFileSync(join(tmp, '.gitignore'), '# my own rules\n*.swp\nnode_modules/\n');
manageGitignore(tmp);
const content = readFileSync(join(tmp, '.gitignore'), 'utf-8');
expect(content).toContain('# my own rules');
expect(content).toContain('*.swp');
expect(content).toContain('node_modules/');
expect(content).toContain('media/x/');
});
test('GBRAIN_NO_GITIGNORE=1 skips entirely', () => {
writeStorageConfig();
process.env.GBRAIN_NO_GITIGNORE = '1';
manageGitignore(tmp);
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
});
test('skips with actionable warning when repo is a git submodule', () => {
writeStorageConfig();
// Submodule: .git is a file containing `gitdir: ...` instead of a directory.
writeFileSync(join(tmp, '.git'), 'gitdir: ../.git/modules/sub\n');
manageGitignore(tmp);
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
expect(warnings.some((w) => /submodule/.test(w))).toBe(true);
});
test('proceeds when .git is a directory (regular repo)', () => {
writeStorageConfig();
mkdirSync(join(tmp, '.git'));
manageGitignore(tmp);
expect(existsSync(join(tmp, '.gitignore'))).toBe(true);
expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]);
});
test('warns and skips when .gitignore write fails (read-only filesystem simulation)', () => {
writeStorageConfig();
// Create a .gitignore as a directory — write to that path will fail with EISDIR.
mkdirSync(join(tmp, '.gitignore'));
manageGitignore(tmp);
expect(warnings.some((w) => /Could not (read|update)/.test(w))).toBe(true);
});
});
+123
View File
@@ -0,0 +1,123 @@
/**
* Unit tests for the shared concurrency-policy helper. Covers:
*
* - Q5: autoConcurrency() returns correct counts for PGLite, explicit
* override, auto path above/below threshold.
* - Q1: shouldRunParallel() respects explicit opt-in even on small diffs.
* - Q2/T3: parseWorkers() throws on bad CLI input (0, -3, "foo", "1.5").
*
* These exist because the prior policy was duplicated across three call sites
* (performSync, performFullSync, jobs handler) with subtle differences.
* Centralized helper + tests prevents the next drift.
*/
import { describe, expect, test } from 'bun:test';
import {
autoConcurrency,
shouldRunParallel,
parseWorkers,
AUTO_CONCURRENCY_FILE_THRESHOLD,
PARALLEL_FILE_FLOOR,
DEFAULT_PARALLEL_WORKERS,
} from '../src/core/sync-concurrency.ts';
import type { BrainEngine } from '../src/core/engine.ts';
// Minimal engine stub — autoConcurrency only reads .kind.
function engineOfKind(kind: 'postgres' | 'pglite'): BrainEngine {
return { kind } as unknown as BrainEngine;
}
describe('autoConcurrency', () => {
test('PGLite always serial (single connection)', () => {
expect(autoConcurrency(engineOfKind('pglite'), 1000)).toBe(1);
expect(autoConcurrency(engineOfKind('pglite'), 1000, 8)).toBe(1);
expect(autoConcurrency(engineOfKind('pglite'), 0)).toBe(1);
});
test('Postgres + explicit override wins', () => {
expect(autoConcurrency(engineOfKind('postgres'), 5, 4)).toBe(4);
expect(autoConcurrency(engineOfKind('postgres'), 5, 1)).toBe(1);
expect(autoConcurrency(engineOfKind('postgres'), 5, 16)).toBe(16);
});
test('Postgres explicit 0 clamped to 1 (paranoia — parseWorkers should reject first)', () => {
expect(autoConcurrency(engineOfKind('postgres'), 100, 0)).toBe(1);
expect(autoConcurrency(engineOfKind('postgres'), 100, -5)).toBe(1);
});
test('Postgres + auto path: under threshold serial', () => {
expect(autoConcurrency(engineOfKind('postgres'), 50)).toBe(1);
expect(autoConcurrency(engineOfKind('postgres'), AUTO_CONCURRENCY_FILE_THRESHOLD)).toBe(1);
});
test('Postgres + auto path: above threshold parallel', () => {
expect(autoConcurrency(engineOfKind('postgres'), AUTO_CONCURRENCY_FILE_THRESHOLD + 1)).toBe(DEFAULT_PARALLEL_WORKERS);
expect(autoConcurrency(engineOfKind('postgres'), 7000)).toBe(DEFAULT_PARALLEL_WORKERS);
});
test('full-sync large marker fires parallel for Postgres', () => {
expect(autoConcurrency(engineOfKind('postgres'), Number.MAX_SAFE_INTEGER)).toBe(DEFAULT_PARALLEL_WORKERS);
});
});
describe('shouldRunParallel', () => {
test('serial when worker count <= 1', () => {
expect(shouldRunParallel(1, 1000, false)).toBe(false);
expect(shouldRunParallel(1, 1000, true)).toBe(false);
expect(shouldRunParallel(0, 1000, true)).toBe(false);
});
test('Q1: explicit opt-in beats the file-count floor', () => {
// User typed --workers 4 with 30 files. Prior behavior: silently serial.
// New behavior: respect the user.
expect(shouldRunParallel(4, 30, /*explicit*/ true)).toBe(true);
expect(shouldRunParallel(2, 1, true)).toBe(true);
});
test('auto path honors PARALLEL_FILE_FLOOR', () => {
// No explicit opt-in: use the floor as the gate.
expect(shouldRunParallel(4, PARALLEL_FILE_FLOOR, false)).toBe(false);
expect(shouldRunParallel(4, PARALLEL_FILE_FLOOR + 1, false)).toBe(true);
expect(shouldRunParallel(4, 0, false)).toBe(false);
});
});
describe('parseWorkers (Q2)', () => {
test('undefined input → undefined output', () => {
expect(parseWorkers(undefined)).toBeUndefined();
});
test('positive integer accepted', () => {
expect(parseWorkers('1')).toBe(1);
expect(parseWorkers('4')).toBe(4);
expect(parseWorkers('128')).toBe(128);
});
test('zero rejected (the original silent footgun)', () => {
expect(() => parseWorkers('0')).toThrow(/positive integer/);
});
test('negative rejected', () => {
expect(() => parseWorkers('-3')).toThrow(/positive integer/);
expect(() => parseWorkers('-1')).toThrow(/positive integer/);
});
test('non-numeric rejected', () => {
expect(() => parseWorkers('foo')).toThrow(/positive integer/);
expect(() => parseWorkers('')).toThrow(/positive integer/);
});
test('non-integer (decimal) rejected', () => {
// parseInt("1.5") returns 1, but "1.5" !== "1" so we reject.
expect(() => parseWorkers('1.5')).toThrow(/positive integer/);
});
test('integer with trailing chars rejected', () => {
// parseInt("4abc") returns 4 silently; we want loud failure.
expect(() => parseWorkers('4abc')).toThrow(/positive integer/);
});
test('whitespace tolerated (since CLI parsers may pass the literal)', () => {
// " 4 " trims to "4" which equals String(4). Accepted.
expect(parseWorkers(' 4 ')).toBe(4);
});
});
+291 -4
View File
@@ -76,18 +76,19 @@ describe('Bug 9 — sync-failures JSONL helpers', () => {
{ path: 'b.md', error: 'err2' },
], 'commit1');
const n = acknowledgeSyncFailures();
expect(n).toBe(2);
const result = acknowledgeSyncFailures();
expect(result.count).toBe(2);
expect(result.summary.length).toBeGreaterThan(0);
const after = loadSyncFailures();
expect(after.every(e => e.acknowledged === true)).toBe(true);
expect(after.every(e => typeof e.acknowledged_at === 'string')).toBe(true);
// Second ack: nothing new to mark.
expect(acknowledgeSyncFailures()).toBe(0);
expect(acknowledgeSyncFailures().count).toBe(0);
// Adding a fresh failure then ack: only the new one flips.
recordSyncFailures([{ path: 'c.md', error: 'err3' }], 'commit2');
expect(acknowledgeSyncFailures()).toBe(1);
expect(acknowledgeSyncFailures().count).toBe(1);
expect(loadSyncFailures().length).toBe(3);
expect(loadSyncFailures().every(e => e.acknowledged === true)).toBe(true);
});
@@ -158,3 +159,289 @@ describe('Bug 9 — sync.ts CLI flag wiring', () => {
expect(source).toContain('recordSyncFailures');
});
});
describe('classifyErrorCode — error message to code mapping', () => {
test('classifies SLUG_MISMATCH from error message', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode(
'Frontmatter slug "my-friend-mike" does not match path-derived slug "2008-03-20-my-friend-mike"'
)).toBe('SLUG_MISMATCH');
});
test('classifies YAML_PARSE from error message', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('YAML parse failed: unexpected colon in title')).toBe('YAML_PARSE');
});
test('classifies YAML_DUPLICATE_KEY', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('YAMLException: duplicated mapping key')).toBe('YAML_DUPLICATE_KEY');
});
test('classifies STATEMENT_TIMEOUT', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('canceling statement due to statement timeout')).toBe('STATEMENT_TIMEOUT');
});
test('classifies NULL_BYTES', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('invalid UTF-8: null byte at position 3770')).toBe('NULL_BYTES');
});
test('classifies INVALID_UTF8', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('invalid UTF-8 sequence at position 500')).toBe('INVALID_UTF8');
});
test('classifies FILE_TOO_LARGE across all three production sites', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
// src/core/import-file.ts:352 — OS-level file size on disk
expect(classifyErrorCode('File too large (8432105 bytes)')).toBe('FILE_TOO_LARGE');
// src/core/import-file.ts:199 — content size limit (5MB cap)
expect(classifyErrorCode('Content too large (6000000 bytes, max 5000000). Split the content into smaller files or remove large embedded assets.')).toBe('FILE_TOO_LARGE');
// src/core/import-file.ts:401 — code file size cap
expect(classifyErrorCode('Code file too large (8000000 bytes)')).toBe('FILE_TOO_LARGE');
});
test('classifies SYMLINK_NOT_ALLOWED from import-file.ts symlink rejection', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('Skipping symlink: /path/to/link.md')).toBe('SYMLINK_NOT_ALLOWED');
});
test('returns UNKNOWN for unrecognized errors', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('something completely different')).toBe('UNKNOWN');
});
});
describe('summarizeFailuresByCode — grouped summary', () => {
test('groups failures by classified code', async () => {
const { summarizeFailuresByCode } = await import('../src/core/sync.ts');
const summary = summarizeFailuresByCode([
{ error: 'Frontmatter slug "a" does not match path-derived slug "b"' },
{ error: 'Frontmatter slug "c" does not match path-derived slug "d"' },
{ error: 'YAML parse failed: bad colon' },
{ error: 'something unknown' },
]);
expect(summary).toEqual([
{ code: 'SLUG_MISMATCH', count: 2 },
{ code: 'YAML_PARSE', count: 1 },
{ code: 'UNKNOWN', count: 1 },
]);
});
test('respects pre-classified code field', async () => {
const { summarizeFailuresByCode } = await import('../src/core/sync.ts');
const summary = summarizeFailuresByCode([
{ error: 'anything', code: 'SLUG_MISMATCH' },
{ error: 'anything', code: 'SLUG_MISMATCH' },
{ error: 'anything', code: 'YAML_PARSE' },
]);
expect(summary).toEqual([
{ code: 'SLUG_MISMATCH', count: 2 },
{ code: 'YAML_PARSE', count: 1 },
]);
});
test('returns empty array for no failures', async () => {
const { summarizeFailuresByCode } = await import('../src/core/sync.ts');
expect(summarizeFailuresByCode([])).toEqual([]);
});
});
describe('acknowledgeSyncFailures — structured return', () => {
test('returns count and code summary', async () => {
const { recordSyncFailures, acknowledgeSyncFailures } = await import('../src/core/sync.ts');
recordSyncFailures([
{ path: 'a.md', error: 'Frontmatter slug "x" does not match path-derived slug "y"' },
{ path: 'b.md', error: 'Frontmatter slug "p" does not match path-derived slug "q"' },
{ path: 'c.md', error: 'YAML parse failed: bad' },
], 'commit1');
const result = acknowledgeSyncFailures();
expect(result.count).toBe(3);
expect(result.summary).toEqual([
{ code: 'SLUG_MISMATCH', count: 2 },
{ code: 'YAML_PARSE', count: 1 },
]);
});
});
describe('recordSyncFailures — code field', () => {
test('records classified code alongside error message', async () => {
const { recordSyncFailures, loadSyncFailures } = await import('../src/core/sync.ts');
recordSyncFailures([
{ path: 'a.md', error: 'Frontmatter slug "x" does not match path-derived slug "y"' },
], 'commit1');
const entries = loadSyncFailures();
expect(entries[0].code).toBe('SLUG_MISMATCH');
});
});
// classifyErrorCode disambiguates Postgres unique-constraint errors from
// YAML duplicate-key errors. Pre-fix, every "duplicate.*key" string mapped
// to YAML_DUPLICATE_KEY, which mislabels DB-layer failures during sync.
describe('classifyErrorCode — DB vs YAML duplicate-key disambiguation', () => {
test('Postgres unique-constraint violation classifies as DB_DUPLICATE_KEY', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode(
'duplicate key value violates unique constraint "pages_slug_key"'
)).toBe('DB_DUPLICATE_KEY');
});
test('YAML duplicated mapping key still classifies as YAML_DUPLICATE_KEY', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('YAMLException: duplicated mapping key "title"'))
.toBe('YAML_DUPLICATE_KEY');
});
test('DB pattern is checked BEFORE YAML so DB errors are not mislabeled', async () => {
// Both patterns historically matched /duplicate.*key/i — order matters now.
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode(
'duplicate key value violates unique constraint on table "pages"'
)).toBe('DB_DUPLICATE_KEY');
expect(classifyErrorCode(
'duplicate key value violates unique constraint on table "pages"'
)).not.toBe('YAML_DUPLICATE_KEY');
});
});
// classifyErrorCode matches the canonical messages emitted by
// collectValidationErrors() in src/core/markdown.ts. Pre-fix, the regexes
// keyed off "missing open" / "missing close" / "empty frontmatter" — none
// of which are produced upstream. Today these all classify correctly.
describe('classifyErrorCode — canonical message coverage', () => {
test('MISSING_OPEN matches "File is empty or whitespace-only"', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode(
'File is empty or whitespace-only; expected frontmatter starting with ---'
)).toBe('MISSING_OPEN');
});
test('MISSING_OPEN matches "Frontmatter must start with ---"', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode(
'Frontmatter must start with --- on the first non-empty line'
)).toBe('MISSING_OPEN');
});
test('MISSING_CLOSE matches "No closing --- delimiter"', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('No closing --- delimiter found')).toBe('MISSING_CLOSE');
});
test('MISSING_CLOSE matches "Heading at line N found inside frontmatter"', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode(
'Heading at line 5 found inside frontmatter zone (closing --- comes after)'
)).toBe('MISSING_CLOSE');
});
test('EMPTY_FRONTMATTER matches "Frontmatter block is empty"', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('Frontmatter block is empty')).toBe('EMPTY_FRONTMATTER');
});
test('NULL_BYTES matches "Content contains null bytes"', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('Content contains null bytes (likely binary corruption)'))
.toBe('NULL_BYTES');
});
test('NESTED_QUOTES matches "Nested double quotes"', async () => {
const { classifyErrorCode } = await import('../src/core/sync.ts');
expect(classifyErrorCode('Nested double quotes in YAML value at line 3'))
.toBe('NESTED_QUOTES');
});
});
// acknowledgeSyncFailures backfills `code` on legacy entries that were
// recorded before the code field existed (~/.gbrain/sync-failures.jsonl
// from pre-PR brains). Without this branch, upgraded users see "UNKNOWN"
// for every previously-recorded failure even when the message is parseable.
describe('acknowledgeSyncFailures — backfill on legacy entries', () => {
test('backfills code on entries that predate the code field', async () => {
const { acknowledgeSyncFailures, loadSyncFailures, syncFailuresPath } =
await import('../src/core/sync.ts');
// Hand-write a legacy entry with no `code` field. Mimics a pre-PR
// ~/.gbrain/sync-failures.jsonl row that exists on real upgrades.
const { mkdirSync } = await import('fs');
const { dirname } = await import('path');
mkdirSync(dirname(syncFailuresPath()), { recursive: true });
writeFileSync(
syncFailuresPath(),
JSON.stringify({
path: 'a.md',
error: 'Frontmatter slug "x" does not match path-derived slug "y"',
commit: 'old',
ts: '2025-01-01T00:00:00Z',
}) + '\n',
);
const result = acknowledgeSyncFailures();
expect(result.count).toBe(1);
expect(result.summary).toEqual([{ code: 'SLUG_MISMATCH', count: 1 }]);
const after = loadSyncFailures();
expect(after).toHaveLength(1);
expect(after[0].code).toBe('SLUG_MISMATCH');
expect(after[0].acknowledged).toBe(true);
});
test('preserves existing code field; never reclassifies', async () => {
const { acknowledgeSyncFailures, loadSyncFailures, syncFailuresPath } =
await import('../src/core/sync.ts');
const { mkdirSync } = await import('fs');
const { dirname } = await import('path');
mkdirSync(dirname(syncFailuresPath()), { recursive: true });
// Pre-classified entry — should NOT be re-run through classifier.
writeFileSync(
syncFailuresPath(),
JSON.stringify({
path: 'a.md',
error: 'some message that would otherwise classify as UNKNOWN',
code: 'CUSTOM_CODE',
commit: 'x',
ts: '2025-01-01T00:00:00Z',
}) + '\n',
);
const result = acknowledgeSyncFailures();
expect(result.summary).toEqual([{ code: 'CUSTOM_CODE', count: 1 }]);
expect(loadSyncFailures()[0].code).toBe('CUSTOM_CODE');
});
});
// formatCodeBreakdown is the DRY helper used by both the failures-array
// path (sync.ts blocked-by-failures + full-sync stderr) and the pre-summarized
// AcknowledgeResult.summary path (--skip-failed ack message). One renderer,
// two input shapes.
describe('formatCodeBreakdown — dual input shape', () => {
test('renders raw failures by classifying internally', async () => {
const { formatCodeBreakdown } = await import('../src/core/sync.ts');
const out = formatCodeBreakdown([
{ error: 'Frontmatter slug "a" does not match path-derived slug "b"' },
{ error: 'Frontmatter slug "c" does not match path-derived slug "d"' },
{ error: 'YAML parse failed: bad' },
]);
expect(out).toBe(' SLUG_MISMATCH: 2\n YAML_PARSE: 1');
});
test('renders pre-summarized {code, count} input directly', async () => {
const { formatCodeBreakdown } = await import('../src/core/sync.ts');
const out = formatCodeBreakdown([
{ code: 'SLUG_MISMATCH', count: 5 },
{ code: 'YAML_PARSE', count: 2 },
]);
expect(out).toBe(' SLUG_MISMATCH: 5\n YAML_PARSE: 2');
});
test('returns empty string for empty input', async () => {
const { formatCodeBreakdown } = await import('../src/core/sync.ts');
expect(formatCodeBreakdown([])).toBe('');
});
});
+257
View File
@@ -0,0 +1,257 @@
/**
* Parallel-sync regression tests (PGLite, in-memory).
*
* T1 sync.last_commit failure-gate under concurrency=4 request.
* T4 PGLite + concurrency=4 stays serial (no crash, no PostgresEngine
* construction). Tightens the engine.kind guard introduced in
* v0.22.13 (PR #490 A1).
* CODEX-3 head-drift gate: when git HEAD moves between performSync's
* capture and its post-import re-check, last_commit must NOT advance.
*
* PGLite forces concurrency=1 internally regardless of the requested value,
* which is the *whole point* of T4 but the bookmark-gate logic
* (failedFiles don't advance) is engine-agnostic, so PGLite is fine for
* the T1 + CODEX-3 contracts. A separate Postgres E2E covers worker-engine
* construction directly.
*/
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { execSync } from 'child_process';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
function git(repo: string, ...args: string[]): string {
return execSync(`git ${args.join(' ')}`, { cwd: repo, encoding: 'utf-8' }).trim();
}
function seedRepoWithMarkdown(repoPath: string, fileCount: number): string {
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.name "Test"', { cwd: repoPath, stdio: 'pipe' });
mkdirSync(join(repoPath, 'people'), { recursive: true });
for (let i = 0; i < fileCount; i++) {
writeFileSync(join(repoPath, `people/p${i}.md`), [
'---',
'type: person',
`title: Person ${i}`,
'---',
'',
`This is person ${i}.`,
].join('\n'));
}
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
return git(repoPath, 'rev-parse', 'HEAD');
}
describe('sync-parallel: PGLite + concurrency=4 (T4)', () => {
let engine: PGLiteEngine;
let repoPath: string;
beforeEach(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-par-'));
});
afterEach(async () => {
await engine.disconnect();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
});
test('PGLite + concurrency=4 + 60 files: imports all without crashing', async () => {
seedRepoWithMarkdown(repoPath, 60);
const { performSync } = await import('../src/commands/sync.ts');
const result = await performSync(engine, {
repoPath,
noPull: true,
noEmbed: true,
concurrency: 4,
});
// First sync routes through performFullSync, returning 'first_sync'.
expect(result.status).toBe('first_sync');
// PGLite stayed single-connection; if the parallel branch had tried to
// construct PostgresEngine without database_url, this test would crash.
});
test('PGLite + explicit concurrency=4 + 30 files (below floor): still safe', async () => {
// Q1 path: explicit opt-in beats the >50 floor. PGLite forces serial
// anyway (engine.kind), so the test is that nothing crashes and the
// sync advances correctly.
seedRepoWithMarkdown(repoPath, 30);
const { performSync } = await import('../src/commands/sync.ts');
const result = await performSync(engine, {
repoPath,
noPull: true,
noEmbed: true,
concurrency: 4,
});
expect(result.status).toBe('first_sync');
});
});
describe('sync-parallel: bookmark gate under concurrency request (T1)', () => {
let engine: PGLiteEngine;
let repoPath: string;
beforeEach(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-gate-'));
});
afterEach(async () => {
await engine.disconnect();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
});
test('clean parallel sync advances last_commit', async () => {
const initialHead = seedRepoWithMarkdown(repoPath, 5);
const { performSync } = await import('../src/commands/sync.ts');
await performSync(engine, {
repoPath,
noPull: true,
noEmbed: true,
concurrency: 4,
});
const lastCommit = await engine.getConfig('sync.last_commit');
expect(lastCommit).toBe(initialHead);
});
test('failure-injection blocks last_commit advance', async () => {
// First sync: clean state.
const firstHead = seedRepoWithMarkdown(repoPath, 5);
const { performSync } = await import('../src/commands/sync.ts');
await performSync(engine, {
repoPath, noPull: true, noEmbed: true,
});
const lastAfterFirst = await engine.getConfig('sync.last_commit');
expect(lastAfterFirst).toBe(firstHead);
// Now add a malformed file (broken YAML frontmatter — closing --- missing
// means the parser hits a real failure that importFile reports).
writeFileSync(join(repoPath, 'people/broken.md'), [
'---',
'type: person',
'title: Broken', // intentionally no closing ---
'this line is body but parser thinks it is YAML',
].join('\n'));
execSync('git add -A && git commit -m "add broken"', { cwd: repoPath, stdio: 'pipe' });
const secondHead = git(repoPath, 'rev-parse', 'HEAD');
expect(secondHead).not.toBe(firstHead);
// Second sync: should record failure and NOT advance the bookmark.
const result = await performSync(engine, {
repoPath, noPull: true, noEmbed: true, concurrency: 4,
});
// Only fail the test when the parser actually rejected the broken file.
// Some YAML parsers are permissive; if so this test exercises the
// happy path AND the assertion below (lastCommit advanced) holds.
if (result.status === 'blocked_by_failures') {
const lastAfterBroken = await engine.getConfig('sync.last_commit');
expect(lastAfterBroken).toBe(firstHead); // unchanged — gate held
expect(result.failedFiles ?? 0).toBeGreaterThan(0);
} else {
// If the parser was permissive, at least confirm the bookmark moved.
const lastAfterBroken = await engine.getConfig('sync.last_commit');
expect(lastAfterBroken).toBe(secondHead);
}
});
});
describe('sync-parallel: head-drift gate (CODEX-3)', () => {
let engine: PGLiteEngine;
let repoPath: string;
beforeEach(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-drift-'));
});
afterEach(async () => {
await engine.disconnect();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
});
test('static-HEAD sync advances last_commit (control)', async () => {
const head = seedRepoWithMarkdown(repoPath, 3);
const { performSync } = await import('../src/commands/sync.ts');
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
expect(await engine.getConfig('sync.last_commit')).toBe(head);
});
test('vanished-mid-sync file produces a failedFiles entry', async () => {
// First sync: clean state for incremental.
seedRepoWithMarkdown(repoPath, 3);
const { performSync } = await import('../src/commands/sync.ts');
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
// Add a file, commit, then delete the file from disk WITHOUT amending the
// commit — diff says it exists at HEAD, but the file is gone. This is the
// "checkout/race deleted my file mid-sync" simulation.
writeFileSync(join(repoPath, 'people/will-vanish.md'), [
'---', 'type: person', 'title: Vanish', '---', '', 'body',
].join('\n'));
execSync('git add -A && git commit -m "add vanish"', { cwd: repoPath, stdio: 'pipe' });
rmSync(join(repoPath, 'people/will-vanish.md'));
const result = await performSync(engine, {
repoPath, noPull: true, noEmbed: true,
});
// Per CODEX-3 (v0.22.13): vanished files now go into failedFiles
// (prior behavior was a benign skip, which let last_commit advance).
expect(result.status).toBe('blocked_by_failures');
expect(result.failedFiles ?? 0).toBeGreaterThan(0);
});
});
describe('sync-parallel: writer lock prevents reentrance (CODEX-2)', () => {
let engine: PGLiteEngine;
let repoPath: string;
beforeEach(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-lock-'));
});
afterEach(async () => {
await engine.disconnect();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
});
test('two parallel performSync calls in same process: second waits or fails fast', async () => {
seedRepoWithMarkdown(repoPath, 5);
const { performSync } = await import('../src/commands/sync.ts');
// Same-process concurrent calls: PGLite serializes engine ops via its
// exclusive transaction mutex, but the writer-lock is the right barrier.
// We verify that one call completes (the lock holder) and any concurrent
// call either completes after (lock released) or surfaces the
// "Another sync is in progress" error.
const promise1 = performSync(engine, { repoPath, noPull: true, noEmbed: true });
let secondError: unknown = null;
try {
// Tiny delay so promise1 captures the lock first.
await new Promise((r) => setTimeout(r, 10));
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
} catch (e) {
secondError = e;
}
await promise1;
// Either: (a) second call completed after first released, both succeeded
// OR (b) second call hit the lock-busy error path. Either is correct.
if (secondError) {
const msg = secondError instanceof Error ? secondError.message : String(secondError);
expect(msg).toMatch(/Another sync is in progress|lock|gbrain-sync/i);
}
});
});
+13 -4
View File
@@ -1,10 +1,11 @@
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
import { buildSyncManifest, isSyncable, pathToSlug } from '../src/core/sync.ts';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
import { join } from 'path';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
describe('buildSyncManifest', () => {
test('parses A/M/D entries from single commit', () => {
@@ -204,11 +205,20 @@ describe('performSync dry-run never writes', () => {
let engine: PGLiteEngine;
let repoPath: string;
beforeEach(async () => {
// One PGLite per file — beforeEach wipes data only. Each test still gets a
// fresh git repo via mkdtempSync, but skips the ~20s PGLite cold-start.
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-sync-dryrun-'));
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
execSync('git config user.email "test@test.com"', { cwd: repoPath, stdio: 'pipe' });
@@ -233,8 +243,7 @@ describe('performSync dry-run never writes', () => {
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
});
afterEach(async () => {
await engine.disconnect();
afterEach(() => {
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
});