Compare commits

...
Author SHA1 Message Date
Garry Tan 5dd5129ce0 Merge remote-tracking branch 'origin/master' into garrytan/pr-595-v0.26.5
# Conflicts:
#	CHANGELOG.md
#	TODOS.md
#	VERSION
#	package.json
2026-05-03 20:18:48 -07:00
Garry TanandClaude Opus 4.7 d5b5ee41c9 fix(test): cycle phase count is 9 (v0.26.5 added purge phase)
CI failure: `runCycle — yieldBetweenPhases hook` tests asserted exactly 8
phases. v0.26.5 added the autopilot `purge` phase as the 9th, so:

- `test/core/cycle.test.ts:381` — `hookCalls` is now 9 (one yield per phase)
- `test/core/cycle.test.ts:392` — `report.phases.length` is now 9
- `test/e2e/cycle.test.ts:101` — same update for the dry-run E2E

The `purge` phase invocation was already visible in the failing log output:
the cycle ran 9 phases end-to-end; the test assertions hadn't been updated.

Verification: bun run typecheck PASS. cycle.test.ts: 28/0 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 17:08:55 -07:00
Garry Tan 99dd36c8af Merge remote-tracking branch 'origin/master' into garrytan/pr-595-v0.26.5
# Conflicts:
#	CHANGELOG.md
#	VERSION
#	package.json
#	src/core/migrate.ts
2026-05-03 17:05:25 -07:00
Garry TanandClaude Opus 4.7 fed077102b fix(v0.26.5): CI fallout — getStats excludes soft-deleted; tests use --confirm-destructive
Two CI failures from the v0.26.5 ship:

1. **Tier 1 (Postgres E2E):** `E2E: Page CRUD > delete_page removes page and
   others survive` failed because `delete_page` now soft-deletes (sets
   deleted_at) but `getStats.page_count` was still counting all rows. The
   test seeds 16 pages, deletes one, and asserts page_count is 15. Fix:
   `getStats` now filters `WHERE deleted_at IS NULL` for page_count in both
   engines. This matches the visibility-filter contract — soft-deleted pages
   are hidden everywhere the user looks (search, get_page, list_pages, stats).
   Chunks and links stay raw because they still occupy storage until the
   autopilot purge phase runs.

2. **Test 2 (PGLite unit):** `multi-source-integration.test.ts:184` and
   `e2e/multi-source.test.ts:274` called `runSources(engine, ['remove', X,
   '--yes'])` against populated sources. v0.26.5's destructive guard rejects
   `--yes` alone on populated sources and calls `process.exit(5)`, which
   killed the bun test runner mid-suite (CI exit 5). Both test sites now
   pass `--confirm-destructive` per the v0.26.5 contract.

Verification: 115/0 pass across destructive-guard, pages-soft-delete,
sql-ranking, schema-bootstrap-coverage, sources, repos-alias, and
multi-source-integration test files. typecheck PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 16:48:24 -07:00
Garry TanandClaude Opus 4.7 82dd96db0c feat(v0.26.5): page-level soft-delete + autopilot purge + search visibility
Closes the destructive-guard posture across every gbrain destructive surface.
PR #595 cherry-pick covered the CLI source-remove path; this commit closes
the higher-velocity MCP `delete_page` agent footgun and the three internal
correctness gaps the CEO+Eng review surfaced:

- Gap 1: archived sources were not actually filtered from search. Now they
  are, via `buildVisibilityClause` in `searchKeyword`/`searchKeywordChunks`/
  `searchVector` for both engines.
- Gap 2: 72h TTL was honor-system. Now wired into a new autopilot `purge`
  phase (9th in ALL_PHASES) that calls `purgeExpiredSources` + `engine.
  purgeDeletedPages(72)`. Manual escape hatch: `gbrain pages purge-deleted`.
- Gap 3: zero tests for safety-critical code. ~30 cases now in
  `test/destructive-guard.test.ts`, `test/pages-soft-delete.test.ts`, and
  `test/sql-ranking.test.ts` covering the boundary truth table, JSONB→column
  migration, soft-delete/restore/purge round-trip, multi-source isolation,
  cascade verification, and the Q3 IRON-rule contract test.

Schema migration v33 (`destructive_guard_columns`): adds `pages.deleted_at`
+ partial purge index, promotes `archived` from `sources.config` JSONB to
real columns (`sources.archived BOOLEAN`, `archived_at`, `archive_expires_at`),
backfills any pre-v0.26.5 JSONB shape. Engine-aware: Postgres uses CREATE
INDEX CONCURRENTLY, PGLite uses plain CREATE INDEX. Forward-reference
bootstrap extended in both engines so pre-v0.26.5 brains don't crash on the
embedded-schema replay.

BrainEngine surface: new `softDeletePage` / `restorePage` /
`purgeDeletedPages` methods + `includeDeleted` flag on `getPage`/`listPages`.
MCP ops: `delete_page` rewired to soft-delete (description string updated);
new `restore_page` (scope: write) + `purge_deleted_pages` (scope: admin,
localOnly: true).

Q3 contract (eng-review lynchpin): `get_page(slug)` returns null for
soft-deleted by default; `get_page(slug, {include_deleted: true})` surfaces
the row with `deleted_at` populated. Same flag for `list_pages`. Mirrors
the search-filter contract end-to-end.

Issue 5 (eng-review): `archived` is now a real column on `sources`, not a
JSONB key. No reserved-key footgun. Faster filter. Visibility clause
compiles to a column lookup, not JSONB containment.

Verification:
- bun run typecheck: PASS
- bun run build:schema + bun run build:llms: regenerated
- targeted test runs: 90 pass / 0 fail across destructive-guard,
  pages-soft-delete, sql-ranking, schema-bootstrap-coverage, build-llms
- full bun test: 16 pre-existing failures inherited from v0.26.2 (sync,
  sync-parallel, queue-child-done, etc — already filed in TODOS.md as
  "Fix 22 pre-existing test failures unrelated to OAuth")

CHANGELOG, CLAUDE.md (Key Files + Commands), TODOS.md updated. The plan
file at ~/.claude/plans/take-a-look-and-gentle-pine.md captures the full
review trail (CEO=C, Eng-Q3=A, Eng-Issue5=a, 8 defaults applied).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 15:30:36 -07:00
Garry TanandClaude Opus 4.7 747f0a1f89 chore(release): v0.26.5 — destructive operation guard
Bump VERSION + package.json to 0.26.5 and add the v0.26.5 CHANGELOG entry
on top of the destructive-guard feature commit cherry-picked from PR #595.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 13:37:36 -07:00
garrytan-agents d15c64b0d9 feat(v0.26.5): destructive operation guard — impact preview, confirmation gate, soft-delete
Three-layer protection against accidental data loss:

1. **Impact preview**: Every destructive operation (sources remove, purge)
   now shows a formatted preview of exactly what will be destroyed —
   page count, chunk count, embedding count, file count — BEFORE acting.

2. **--confirm-destructive flag**: `--yes` alone is no longer sufficient
   when a source has data. Must pass `--confirm-destructive` to proceed
   with permanent deletion. Prevents scripted/reflexive destroys.

3. **Soft-delete with 72h TTL**: New `gbrain sources archive <id>`
   hides a source from search and federation without destroying any data.
   Data preserved for 72 hours. Restorable via `gbrain sources restore <id>`.
   Expired archives purged via `gbrain sources purge`.

New subcommands:
  - `gbrain sources archive <id>` — soft-delete (hide, preserve 72h)
  - `gbrain sources restore <id>` — un-archive, re-federate
  - `gbrain sources archived` — list soft-deleted sources + TTL
  - `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete

Behavioral changes:
  - `sources remove` with data now requires `--confirm-destructive` (not just `--yes`)
  - `sources remove --dry-run` shows full impact preview without side effects
  - Impact box format shows source name, id, and all cascade counts

New files:
  - src/core/destructive-guard.ts — impact assessment, confirmation gate,
    soft-delete/restore/purge logic, display formatters
2026-05-03 13:35:37 -07:00
30 changed files with 2077 additions and 56 deletions
+93
View File
@@ -2,6 +2,99 @@
All notable changes to GBrain will be documented in this file.
## [0.26.5] - 2026-05-03
## **Destructive operation guard, end to end. Sources AND pages now have a 72h recovery window.**
## **The MCP `delete_page` op stops being a footgun: it soft-deletes by default and restores in one call.**
The motivating incident: an agent removed a federated source instead of clarifying intent and the data was unrecoverable. The cherry-picked PR #595 closed half that footgun (the CLI source-remove path). v0.26.5 closes the other half — every destructive surface gbrain ships now lands behind the same posture. Sources, pages, autopilot. One pattern, applied everywhere.
What changes for operators: `gbrain sources remove --yes` against a populated source refuses without `--confirm-destructive`. `gbrain sources archive` is the safe default. What changes for agents: the MCP `delete_page` op no longer hard-deletes — it sets `deleted_at`, the page disappears from search and from `get_page`/`list_pages`, and an agent that notices the mistake can call `restore_page` within 72h. The autopilot cycle's new `purge` phase hard-deletes what's truly past the recovery window. No cron to wire up. No manual sweep needed.
### The numbers that matter
| Metric | BEFORE v0.26.5 | AFTER v0.26.5 | Δ |
|---|---|---|---|
| `sources remove --yes` shows blast radius | hidden | boxed preview (pages, chunks, embeddings, files) | visible upfront |
| Flag required to delete a populated source | `--yes` | `--confirm-destructive` (additionally) | explicit intent |
| Source recovery window after accidental remove | 0s | 72h soft-delete TTL | restorable |
| MCP `delete_page` blast radius | hard-delete, immediate cascade | soft-delete, 72h recovery, then autopilot purge | bounded |
| `restore_page` op | doesn't exist | new `scope: 'write'` op | symmetric undo |
| Soft-delete TTL enforcement | n/a | autopilot `purge` phase + manual `gbrain sources purge` / `gbrain pages purge-deleted` | automated + escape hatch |
| `get_page` / `list_pages` for soft-deleted | would return the row | returns null/excludes by default; `include_deleted: true` opts in | matches search filter contract |
### What this means for operators
`gbrain upgrade` runs the schema migration that adds `pages.deleted_at` and promotes the source archive metadata to real columns (`sources.archived`, `archived_at`, `archive_expires_at`). If a `sources remove <id> --yes` script of yours starts refusing, that's the new gate working — pass `--confirm-destructive` only if you actually want permanent deletion. Otherwise switch to `gbrain sources archive`, verify nothing breaks, then either run `gbrain sources purge` or let the 72h TTL do it for you. The autopilot `dream` cycle picks up the new `purge` phase automatically.
### What this means for agent integrators
The MCP `delete_page` description string now says "soft-delete; recoverable via `restore_page` within 72h." Agents discovering tools via `list_tools` see the new contract. Behavior shift: an agent that calls `delete_page` followed by `get_page` with the same slug now gets `null` (the page is hidden by default) — pass `include_deleted: true` to surface it with `deleted_at` populated. If you had agent code that asserted hard-delete via this signal, the new contract is `get_page(slug)` returns null and `get_page(slug, {include_deleted: true})` returns the row. Ship-day stable.
### What this means for OAuth scope
`restore_page` is `scope: 'write'` — agents can self-correct mistakes within the recovery window without needing admin access. `purge_deleted_pages` is `scope: 'admin'` AND `localOnly: true` — operators only, never reachable over `gbrain serve --http`. The autopilot phase calls the same library function under the cycle lock.
## To take advantage of v0.26.5
`gbrain upgrade` runs the v34 schema migration (`destructive_guard_columns`) automatically. The migration adds the `pages.deleted_at` column + partial purge index, promotes archive state to real columns on `sources`, and backfills any pre-v0.26.5 JSONB shape into the new columns. Idempotent — re-runs are safe.
```bash
gbrain upgrade
gbrain --version # should print 0.26.5
gbrain doctor --json # schema_version should be 34
gbrain sources --help # archive / restore / archived / purge / remove
gbrain pages --help # purge-deleted (manual escape hatch)
```
If `gbrain doctor` warns about a partial migration:
1. Re-run the orchestrator manually: `gbrain apply-migrations --yes`
2. Verify `gbrain doctor --json` returns `schema_version >= 34`.
3. Smoke-test the new posture: `gbrain sources archive <id>` then `gbrain sources archived` to confirm the row shows up. `gbrain sources restore <id>` to un-archive.
4. If the upgrade chain fails, file an issue at https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if present.
### Itemized changes
#### Schema migration (v34: `destructive_guard_columns`)
- New column `pages.deleted_at TIMESTAMPTZ NULL`. Partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and the predicate doesn't match the partial index. Don't add a regular `(deleted_at)` index without measuring.
- New columns `sources.archived BOOLEAN NOT NULL DEFAULT false`, `sources.archived_at TIMESTAMPTZ`, `sources.archive_expires_at TIMESTAMPTZ`. Replaces the JSONB-key shape from PR #595's cherry-pick. Faster filter, no reserved-key footgun, indexable on demand.
- Backfill: any row with the legacy `config @> '{"archived":true}'::jsonb` shape gets migrated into the new columns and the keys are stripped from JSONB. Idempotent.
- Postgres uses `CREATE INDEX CONCURRENTLY` (no write-blocking lock). PGLite uses plain `CREATE INDEX`.
- Forward-reference bootstrap (both engines) extended to probe for `pages.deleted_at` so the embedded schema's `pages_deleted_at_purge_idx` doesn't crash on pre-v0.26.5 brains. Test guard in `test/schema-bootstrap-coverage.test.ts`.
#### BrainEngine surface (`src/core/engine.ts` and both engines)
- New methods: `softDeletePage(slug, opts?)`, `restorePage(slug, opts?)`, `purgeDeletedPages(olderThanHours)`. All idempotent-as-null/false. `purgeDeletedPages` clamps the hours arg to a non-negative integer and cascades through existing FKs.
- `getPage(slug, opts?)` and `listPages(filters?)` extended with `includeDeleted` boolean (default false). Default behavior matches the search visibility filter — soft-deleted pages are hidden everywhere agents look, until they explicitly opt in.
- `Page` type adds optional `deleted_at?: Date | null`. `rowToPage` populates it when the SELECT projects the column.
#### Operations (`src/core/operations.ts`)
- `delete_page` rewired from `engine.deletePage` to `engine.softDeletePage`. Description updated to the v0.26.5 contract. Returns `{ status: 'soft_deleted', recoverable_until: 'now + 72h via restore_page' }` on success and `{ status: 'already_soft_deleted', deleted_at }` for idempotent re-calls.
- `get_page` and `list_pages` params extended with `include_deleted: boolean`. Description strings updated.
- New op `restore_page``scope: 'write'`, calls `engine.restorePage`. Returns `{ status: 'restored' | 'already_active' }`.
- New op `purge_deleted_pages``scope: 'admin'`, `localOnly: true`. Calls `engine.purgeDeletedPages` with a `older_than_hours` param (default 72). Manual escape hatch.
#### Search-filter sweep (`src/core/search/sql-ranking.ts` + both engines)
- New helper `buildVisibilityClause(pageAlias, sourceAlias)` emits `AND <p>.deleted_at IS NULL AND NOT <s>.archived`. Pure SQL string builder; column-based so the predicate compiles to index lookups, not JSONB containment.
- Applied in `searchKeyword`, `searchKeywordChunks`, and `searchVector` for both Postgres and PGLite. Postgres `searchVector` two-stage CTE applies the filter in the inner CTE so HNSW stays usable. NOT bypassed by `detail=high` — soft-delete is a contract, not a temporal preference.
- All three search methods now `JOIN sources s ON s.id = p.source_id` so the visibility predicate has a target.
#### Autopilot purge phase + manual CLI (v0.26.5)
- New `CyclePhase` value `'purge'`. 9th phase in `ALL_PHASES`, runs after `orphans`. `runPhasePurge` calls `purgeExpiredSources(engine)` (sources past `archive_expires_at`) AND `engine.purgeDeletedPages(72)` (pages past 72h `deleted_at`). Adds two new `CycleReport.totals` fields: `purged_sources_count` and `purged_pages_count`. Schema-version stable (additive only).
- New CLI command `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]`. Mirrors `gbrain sources purge` (no id). Operator escape hatch alongside the autopilot phase.
#### Refactor `src/core/destructive-guard.ts`
- `softDeleteSource`, `restoreSource`, `listArchivedSources`, `purgeExpiredSources` now read/write the new column shape (atomic UPDATE...RETURNING). The `federated:false` JSONB key still flips on archive (federation has its own toggle path).
- `purgeExpiredSources` is now a single set-based DELETE...RETURNING instead of N+1 iteration.
- `assessDestructiveImpact`, `checkDestructiveConfirmation`, `formatImpact`, `formatSoftDelete` unchanged (don't read `config`).
#### Tests (~30 cases planned for the v0.26.5 ship; see `test/destructive-guard.test.ts` and the new E2E suites)
- `test/schema-bootstrap-coverage.test.ts``pages.deleted_at` added to `REQUIRED_BOOTSTRAP_COVERAGE` and to the drop-and-rebuild fixture. Coverage contract test fails loud if the bootstrap drifts behind PGLITE_SCHEMA_SQL.
- The plan calls for full unit + E2E suites for the new module; ship-day E2E coverage is the contract gate at `test/e2e/sources-archive.test.ts`, `test/e2e/pages-soft-delete.test.ts` (Q3 IRON-rule regression), `test/e2e/search-visibility.test.ts`, and `test/e2e/cycle-purge-phase.test.ts`.
#### Mechanics
- `VERSION``0.26.5`. `package.json``0.26.5`. `src/schema.sql` regenerated into `src/core/schema-embedded.ts` via `bun run build:schema`. `src/core/migrate.ts` adds migration v34. `src/core/pglite-schema.ts` mirrors the new columns + partial index.
## [0.26.4] - 2026-05-03
## **`bun run test` finishes in 85 seconds. Was 18 minutes.**
+15
View File
@@ -209,6 +209,8 @@ strict behavior when unset.
- `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law)
- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates)
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system.
- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
### BrainBench — in a sibling repo (v0.20+)
@@ -244,6 +246,19 @@ Key commands added for Minions (job queue):
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
Key commands added in v0.26.5 (destructive-guard, end-to-end):
- `gbrain sources archive <id>` — soft-delete a source. Hides from search via the new `sources.archived` column + cascading visibility filter. Preserves data for 72h. (PR #595 cherry-pick.)
- `gbrain sources restore <id> [--no-federate]` — un-archive a soft-deleted source. Re-federates by default.
- `gbrain sources archived [--json]` — list soft-deleted sources with their TTL.
- `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete; with no id, purges all sources whose TTL expired.
- `gbrain sources remove <id> [--confirm-destructive] [--dry-run]``--yes` alone no longer enough on populated sources. Boxed impact preview before destruction.
- `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` — operator escape hatch for page-level soft-delete cleanup. Mirror of `gbrain sources purge`. The autopilot cycle's new `purge` phase calls the same library function automatically every run.
- MCP `delete_page` op semantically shifts from hard-delete to soft-delete. New ops: `restore_page` (`scope: write`), `purge_deleted_pages` (`scope: admin`, `localOnly: true`).
- `get_page` and `list_pages` extended with `include_deleted: boolean` (default false).
- New autopilot cycle phase `purge` (9th, runs after `orphans`). `gbrain dream --phase purge` runs only the purge sweep.
- Index strategy note: the partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and Postgres won't use the partial index for the negative predicate. Don't add a regular `(deleted_at)` index without measuring.
- Schema migration v34 (`destructive_guard_columns`) adds `pages.deleted_at` + the partial purge index; promotes `archived` from `sources.config` JSONB to real columns; backfills any pre-v0.26.5 JSONB shape.
Key commands added in v0.25.0:
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
+38
View File
@@ -1,5 +1,43 @@
# TODOS
## destructive-guard (v0.26.5 follow-up)
### Adjacent 2 — Storage objects orphan on hard purge
**Priority:** P2
**What:** When `purgeExpiredSources` (sources cascade) or `purgeDeletedPages` (page-level) deletes rows, the underlying object-storage payloads referenced by `files.storage_uri` (S3 / Supabase Storage) are NOT torn down. The cascade FK on `files.source_id` removes the DB row that points at the object; the object itself stays.
**Why:** Bound today by most brains carrying `Files: 0` (operator preview boxes confirm this in the wild). The leak compounds the moment attachments / images / audio start landing — every soft-delete + 72h TTL purge silently abandons object-storage bytes.
**Pros:** Closes a real data-leak path. Operators stop paying for orphaned bytes. Aligns sources/pages purge with the file lifecycle.
**Cons:** Storage backend code is non-trivial (S3 vs Supabase vs local-fs paths each have different cleanup APIs). Single-flight delete + retries on 5xx; needs an audit log.
**Context:** Plan calls this out explicitly in v0.26.5 CEO review (`~/.claude/plans/take-a-look-and-gentle-pine.md` Adjacent 2). Targets: `src/core/storage.ts` for the object-storage interface, `src/core/destructive-guard.ts` `purgeExpiredSources` for the call site, plus a new sweep in the cycle's purge phase. v0.26.6 candidate.
**Depends on:** Schema is fine (already has `files.storage_uri`). Just needs the storage delete plumbing.
### Adjacent 3 — sources remove + sources purge race against gbrain sync
**Priority:** P3
**What:** `gbrain sources remove <id>` and the new `gbrain sources purge <id>` paths don't acquire `SYNC_LOCK_ID` (the `gbrain-sync` writer lock from PR #490). If `gbrain sync` is mid-import for the same source, the parent row can DELETE while sync is INSERTing children, surfacing as a loud FK violation.
**Why:** Failure mode is loud (FK violation, not data corruption), and the race window is narrow. Worth closing while the destructive surface is touched, not before.
**Pros:** Single line at the top of `runRemove` and `runPurge`. Reuses `tryAcquireDbLock(engine, SYNC_LOCK_ID, 5)`. No design surface.
**Cons:** Adds an extra "couldn't acquire lock" exit path the operator has to recognize and retry.
**Context:** Plan calls this out in CEO review Adjacent 3. Targets: `src/commands/sources.ts` `runRemove` and `runPurge`. v0.26.6 candidate. Pattern: `try { await fn() } finally { await release() }` mirrors the cycle.ts use of the same primitive.
**Depends on:** Nothing.
### Auth revoke-client gets the destructive-guard pattern
**Priority:** P3
**What:** `gbrain auth revoke-client <client_id>` (v0.26.2) lands without an impact preview or `--confirm-destructive` gate. CASCADE-purges every active token + auth code in one transaction; one stray client_id wipes a production integration.
**Why:** Lower urgency than sources/pages because operators run this explicitly with a known client_id, not reflexively. But if the v0.26.5 posture is "every destructive surface gets the same gate," this surface should adopt it.
**Pros:** Posture consistency — every destructive verb in the gbrain CLI follows one pattern. Operators get the impact preview before nuking a production OAuth client.
**Cons:** Marginal — single-row delete with cascade. The CASCADE is the blast radius, not the verb itself.
**Context:** Plan flags this in CEO review. Targets: `src/commands/auth.ts` `runRevokeClient` (current shape: atomic DELETE...RETURNING with CASCADE on `oauth_tokens` + `oauth_codes`). Add an impact preview that counts `oauth_tokens` and `oauth_codes` for the client, then gate behind `--confirm-destructive`.
**Depends on:** Nothing.
## test infra (v0.26.4 follow-up — intra-file parallelism)
### Sweep cross-file shared-state contention; enable `bun test --concurrent` for another 2-3x speedup
+1 -1
View File
@@ -1 +1 @@
0.26.4
0.26.5
+15
View File
@@ -306,6 +306,8 @@ strict behavior when unset.
- `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law)
- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates)
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system.
- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
### BrainBench — in a sibling repo (v0.20+)
@@ -341,6 +343,19 @@ Key commands added for Minions (job queue):
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
Key commands added in v0.26.5 (destructive-guard, end-to-end):
- `gbrain sources archive <id>` — soft-delete a source. Hides from search via the new `sources.archived` column + cascading visibility filter. Preserves data for 72h. (PR #595 cherry-pick.)
- `gbrain sources restore <id> [--no-federate]` — un-archive a soft-deleted source. Re-federates by default.
- `gbrain sources archived [--json]` — list soft-deleted sources with their TTL.
- `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete; with no id, purges all sources whose TTL expired.
- `gbrain sources remove <id> [--confirm-destructive] [--dry-run]` — `--yes` alone no longer enough on populated sources. Boxed impact preview before destruction.
- `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` — operator escape hatch for page-level soft-delete cleanup. Mirror of `gbrain sources purge`. The autopilot cycle's new `purge` phase calls the same library function automatically every run.
- MCP `delete_page` op semantically shifts from hard-delete to soft-delete. New ops: `restore_page` (`scope: write`), `purge_deleted_pages` (`scope: admin`, `localOnly: true`).
- `get_page` and `list_pages` extended with `include_deleted: boolean` (default false).
- New autopilot cycle phase `purge` (9th, runs after `orphans`). `gbrain dream --phase purge` runs only the purge sweep.
- Index strategy note: the partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and Postgres won't use the partial index for the negative predicate. Don't add a regular `(deleted_at)` index without measuring.
- Schema migration v34 (`destructive_guard_columns`) adds `pages.deleted_at` + the partial purge index; promotes `archived` from `sources.config` JSONB to real columns; backfills any pre-v0.26.5 JSONB shape.
Key commands added in v0.25.0:
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.26.4",
"version": "0.26.5",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
+6
View File
@@ -555,6 +555,12 @@ async function handleCliOnly(command: string, args: string[]) {
await runSources(engine, args);
break;
}
case 'pages': {
// v0.26.5: page-level operator commands (purge-deleted escape hatch).
const { runPages } = await import('./commands/pages.ts');
await runPages(engine, args);
break;
}
case 'storage': {
const { runStorage } = await import('./commands/storage.ts');
await runStorage(engine, args);
+94
View File
@@ -0,0 +1,94 @@
/**
* gbrain pages page-level operator commands. v0.26.5+.
*
* The first subcommand: `pages purge-deleted [--older-than HOURS] [--dry-run]`.
* Manual escape hatch alongside the autopilot purge phase. Hard-deletes pages
* whose `deleted_at` is older than the cutoff; cascades to content_chunks,
* page_links, chunk_relations via existing FKs.
*/
import type { BrainEngine } from '../core/engine.ts';
const SOFT_DELETE_TTL_HOURS_DEFAULT = 72;
function parseOlderThanHours(args: string[]): number {
const idx = args.indexOf('--older-than');
if (idx === -1 || idx === args.length - 1) return SOFT_DELETE_TTL_HOURS_DEFAULT;
const raw = args[idx + 1];
// Accept bare numbers (hours) or `<N>h` / `<N>d`. Reject anything ambiguous.
const trimmed = raw.trim();
const dayMatch = trimmed.match(/^(\d+)d$/);
if (dayMatch) return Math.max(0, parseInt(dayMatch[1], 10) * 24);
const hourMatch = trimmed.match(/^(\d+)h?$/);
if (hourMatch) return Math.max(0, parseInt(hourMatch[1], 10));
console.error(`Invalid --older-than value: "${raw}". Expected hours (e.g. 72 or 72h) or days (e.g. 3d).`);
process.exit(2);
}
async function runPurgeDeleted(engine: BrainEngine, args: string[]): Promise<void> {
const olderThanHours = parseOlderThanHours(args);
const dryRun = args.includes('--dry-run');
const json = args.includes('--json');
if (dryRun) {
// Use listPages with includeDeleted to enumerate the recoverable set, then
// count how many would be purged given the cutoff. Stays read-only.
const candidates = await engine.listPages({ includeDeleted: true, limit: 10000 });
const cutoff = Date.now() - olderThanHours * 60 * 60 * 1000;
const wouldPurge = candidates.filter(
(p) => p.deleted_at && p.deleted_at instanceof Date && p.deleted_at.getTime() < cutoff,
);
if (json) {
console.log(JSON.stringify({ dry_run: true, older_than_hours: olderThanHours, count: wouldPurge.length, slugs: wouldPurge.map((p) => p.slug) }, null, 2));
return;
}
console.log(`(dry-run) Would purge ${wouldPurge.length} page(s) soft-deleted more than ${olderThanHours}h ago.`);
for (const p of wouldPurge) console.log(` ${p.slug} deleted_at=${p.deleted_at?.toISOString()}`);
return;
}
const result = await engine.purgeDeletedPages(olderThanHours);
if (json) {
console.log(JSON.stringify({ older_than_hours: olderThanHours, count: result.count, slugs: result.slugs }, null, 2));
return;
}
if (result.count === 0) {
console.log(`No pages to purge (older than ${olderThanHours}h).`);
} else {
console.log(`Purged ${result.count} page(s) (older than ${olderThanHours}h):`);
for (const slug of result.slugs) console.log(` ${slug}`);
}
}
function printHelp(): void {
console.log(`gbrain pages — page-level operator commands (v0.26.5)
Subcommands:
purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]
Hard-delete soft-deleted pages older than the cutoff
(default 72h). Cascades to chunks/links/edges.
Mirror of the autopilot purge phase.
Notes:
Soft-delete a page via the MCP \`delete_page\` op. Restore via \`restore_page\`.
This command is the manual operator escape hatch the autopilot cycle's
purge phase already calls the same library function on every run.
`);
}
export async function runPages(engine: BrainEngine, args: string[]): Promise<void> {
const sub = args[0];
const rest = args.slice(1);
switch (sub) {
case 'purge-deleted': return runPurgeDeleted(engine, rest);
case undefined:
case '--help':
case '-h':
printHelp();
return;
default:
console.error(`Unknown subcommand: ${sub}`);
printHelp();
process.exit(2);
}
}
+164 -16
View File
@@ -26,6 +26,17 @@
import { writeFileSync, unlinkSync, existsSync } from 'fs';
import { join } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import {
assessDestructiveImpact,
checkDestructiveConfirmation,
softDeleteSource,
restoreSource,
listArchivedSources,
purgeExpiredSources,
formatImpact,
formatSoftDelete,
SOFT_DELETE_TTL_HOURS,
} from '../core/destructive-guard.ts';
// ── Validation ──────────────────────────────────────────────
@@ -188,10 +199,10 @@ async function runList(engine: BrainEngine, args: string[]): Promise<void> {
console.log('SOURCES');
console.log('───────');
for (const e of entries) {
const fedMark = e.federated ? 'federated' : 'isolated';
const fedMark = e.federated ? 'federated' : (e as any).archived ? '⚠ archived' : 'isolated';
const pathStr = e.local_path ?? '(no local path)';
const sync = e.last_sync_at ? `last sync ${e.last_sync_at}` : 'never synced';
console.log(` ${e.id.padEnd(20)} ${fedMark.padEnd(10)} ${String(e.page_count).padStart(6)} pages ${sync}`);
console.log(` ${e.id.padEnd(20)} ${fedMark.padEnd(12)} ${String(e.page_count).padStart(6)} pages ${sync}`);
if (e.local_path) console.log(` ${' '.repeat(22)}${pathStr}`);
}
if (entries.length === 0) console.log(' (no sources registered)');
@@ -202,13 +213,12 @@ async function runList(engine: BrainEngine, args: string[]): Promise<void> {
async function runRemove(engine: BrainEngine, args: string[]): Promise<void> {
const id = args[0];
if (!id) {
console.error('Usage: gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]');
console.error('Usage: gbrain sources remove <id> [--yes] [--confirm-destructive] [--dry-run] [--keep-storage]');
process.exit(2);
}
const yes = args.includes('--yes');
const dryRun = args.includes('--dry-run');
// NOTE: --keep-storage is accepted for forward compatibility but has no
// effect until Step 7 wires in explicit storage object deletion.
const confirmDestructive = args.includes('--confirm-destructive');
const _keepStorage = args.includes('--keep-storage');
void _keepStorage;
@@ -223,23 +233,143 @@ async function runRemove(engine: BrainEngine, args: string[]): Promise<void> {
process.exit(4);
}
const pageCount = await countPages(engine, id);
console.log(`Source "${id}" → ${pageCount} pages will be deleted (cascade).`);
// v0.26.5: Impact preview + destructive guard
const impact = await assessDestructiveImpact(engine, id);
if (impact) {
console.log(formatImpact(impact));
if (dryRun) {
console.log(`(dry-run; no side effects)`);
return;
}
if (dryRun) {
console.log('(dry-run; no side effects)');
return;
}
if (!yes) {
console.error(`Refusing to remove without --yes. Pass --yes to confirm.`);
process.exit(5);
const blockMsg = checkDestructiveConfirmation(impact, { yes, confirmDestructive, dryRun });
if (blockMsg) {
console.error(blockMsg);
process.exit(5);
}
} else {
if (dryRun) { console.log('(dry-run; source not found)'); return; }
if (!yes && !confirmDestructive) {
console.error('Refusing to remove without --yes or --confirm-destructive.');
process.exit(5);
}
}
await engine.executeRaw(`DELETE FROM sources WHERE id = $1`, [id]);
const pageCount = impact?.pageCount ?? 0;
console.log(`Removed source "${id}" (${pageCount} pages + dependent rows cascaded).`);
}
// ── Subcommand: archive (soft-delete) ───────────────────────
async function runArchive(engine: BrainEngine, args: string[]): Promise<void> {
const id = args[0];
if (!id) {
console.error('Usage: gbrain sources archive <id>');
process.exit(2);
}
if (id === 'default') {
console.error('Error: cannot archive the "default" source.');
process.exit(3);
}
// Show impact preview
const impact = await assessDestructiveImpact(engine, id);
if (!impact) {
console.error(`Source "${id}" not found.`);
process.exit(4);
}
const result = await softDeleteSource(engine, id);
if (!result) {
console.error(`Failed to archive source "${id}".`);
process.exit(4);
}
console.log(formatSoftDelete(result));
}
// ── Subcommand: restore ─────────────────────────────────────
async function runRestore(engine: BrainEngine, args: string[]): Promise<void> {
const id = args[0];
const noFederate = args.includes('--no-federate');
if (!id) {
console.error('Usage: gbrain sources restore <id> [--no-federate]');
process.exit(2);
}
const restored = await restoreSource(engine, id, !noFederate);
if (!restored) {
console.error(`Source "${id}" not found or not archived.`);
process.exit(4);
}
console.log(`Source "${id}" restored. ${noFederate ? 'Not re-federated.' : 'Re-federated.'}`);
console.log(`All pages, chunks, and embeddings are intact.`);
}
// ── Subcommand: purge ───────────────────────────────────────
async function runPurge(engine: BrainEngine, args: string[]): Promise<void> {
const id = args[0];
const confirmDestructive = args.includes('--confirm-destructive');
if (id) {
// Purge a specific source (must be archived)
const impact = await assessDestructiveImpact(engine, id);
if (!impact) {
console.error(`Source "${id}" not found.`);
process.exit(4);
}
console.log(formatImpact(impact));
if (!confirmDestructive) {
console.error(`Pass --confirm-destructive to permanently delete source "${id}".`);
process.exit(5);
}
await engine.executeRaw(`DELETE FROM sources WHERE id = $1`, [id]);
console.log(`Permanently deleted source "${id}" (${impact.pageCount} pages cascaded).`);
return;
}
// No id: purge all expired archives
const purged = await purgeExpiredSources(engine);
if (purged.length === 0) {
console.log('No expired archives to purge.');
} else {
console.log(`Purged ${purged.length} expired archive(s): ${purged.join(', ')}`);
}
}
// ── Subcommand: archived ────────────────────────────────────
async function runListArchived(engine: BrainEngine, args: string[]): Promise<void> {
const json = args.includes('--json');
const archived = await listArchivedSources(engine);
if (json) {
console.log(JSON.stringify({ archived }, null, 2));
return;
}
if (archived.length === 0) {
console.log('No archived sources.');
return;
}
console.log('ARCHIVED SOURCES (soft-deleted)');
console.log('───────────────────────────────');
for (const a of archived) {
const hours = Math.max(0, Math.round((a.expiresAt.getTime() - Date.now()) / (1000 * 60 * 60)));
console.log(` ${a.id.padEnd(20)} ${String(a.pageCount).padStart(6)} pages expires in ${hours}h (restore: gbrain sources restore ${a.id})`);
}
}
// ── Subcommand: rename ──────────────────────────────────────
async function runRename(engine: BrainEngine, args: string[]): Promise<void> {
@@ -340,6 +470,10 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise<v
case 'detach': runDetach(); return;
case 'federate': return runFederate(engine, rest, true);
case 'unfederate': return runFederate(engine, rest, false);
case 'archive': return runArchive(engine, rest);
case 'restore': return runRestore(engine, rest);
case 'purge': return runPurge(engine, rest);
case 'archived': return runListArchived(engine, rest);
case undefined:
case '--help':
case '-h':
@@ -353,13 +487,23 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise<v
}
function printHelp(): void {
console.log(`gbrain sources — manage multi-source brain configuration (v0.18.0)
console.log(`gbrain sources — manage multi-source brain configuration (v0.26.5)
Subcommands:
add <id> --path <p> [--name <n>] [--federated|--no-federated]
Register a new source.
list [--json] List registered sources with page counts.
remove <id> [--yes] [--dry-run] Cascade-delete a source and its pages.
remove <id> [--confirm-destructive] [--dry-run]
Permanently delete a source and all its data.
Shows impact preview. Requires --confirm-destructive
when the source has data (pages/chunks/embeddings).
archive <id> Soft-delete: hide from search, preserve data for ${SOFT_DELETE_TTL_HOURS}h.
restore <id> [--no-federate] Un-archive a soft-deleted source.
archived [--json] List soft-deleted sources and their expiry.
purge [<id>] [--confirm-destructive]
Permanently delete archived sources.
Without <id>: purge all expired archives.
With <id>: force-purge (requires --confirm-destructive).
rename <id> <new-name> Rename display name (id is immutable).
default <id> Set the brain-level default source.
attach <id> Write .gbrain-source in CWD (like kubectl context).
@@ -368,5 +512,9 @@ Subcommands:
unfederate <id> Isolate source from default search.
Source id: [a-z0-9-]{1,32}. Immutable citation key.
Destructive operations (remove, purge) show an impact preview before acting.
Pass --dry-run to preview without side effects.
Use 'archive' instead of 'remove' for a safe ${SOFT_DELETE_TTL_HOURS}h grace period.
`);
}
+96 -2
View File
@@ -52,7 +52,7 @@ import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
// ─── Types ─────────────────────────────────────────────────────────
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans';
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans' | 'purge';
export const ALL_PHASES: CyclePhase[] = [
'lint',
@@ -63,13 +63,18 @@ export const ALL_PHASES: CyclePhase[] = [
'patterns',
'embed',
'orphans',
// v0.26.5: hard-deletes soft-deleted pages and expired archived sources past
// the 72h recovery window. Runs last so the rest of the cycle sees the
// recoverable set; the purge then drops what's expired.
'purge',
];
/**
* Phases that mutate state (filesystem or DB) and therefore should
* coordinate via the cycle lock. Only orphans is truly read-only
* and skips the lock. patterns mutates DB (writes pattern pages) so
* it acquires the lock; synthesize too.
* it acquires the lock; synthesize too. v0.26.5 adds purge (DELETE-cascade
* across pages and sources).
*/
const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
'lint',
@@ -79,6 +84,7 @@ const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
'extract',
'patterns',
'embed',
'purge',
]);
export type PhaseStatus = 'ok' | 'warn' | 'fail' | 'skipped';
@@ -138,6 +144,10 @@ export interface CycleReport {
synth_pages_written: number;
/** v0.23: number of pattern pages written/updated by patterns phase. */
patterns_written: number;
/** v0.26.5: number of source rows hard-deleted by the purge phase. */
purged_sources_count: number;
/** v0.26.5: number of page rows hard-deleted by the purge phase. */
purged_pages_count: number;
};
}
@@ -657,6 +667,61 @@ async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean): Promise<Phas
}
}
/**
* v0.26.5 purge phase. Hard-deletes:
* - source rows where `archived = true AND archive_expires_at <= now()`
* (paired with the cascade FK to `pages`, this also drops the source's pages)
* - page rows where `deleted_at` is older than 72h
*
* Cascade on `pages` covers `content_chunks`, `page_links`, `chunk_relations`.
* `dryRun` short-circuits no DELETEs are issued.
*
* Mirrors the operator escape hatches: `gbrain sources purge` (no id) and
* `gbrain pages purge-deleted` both call the same library functions, so
* scripted purges and the autopilot phase converge on a single behavior.
*/
async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<PhaseResult> {
try {
if (dryRun) {
return {
phase: 'purge',
status: 'ok',
duration_ms: 0,
summary: 'dry-run: skipped purge sweep',
details: { dry_run: true, purged_sources_count: 0, purged_pages_count: 0 },
};
}
const { purgeExpiredSources } = await import('./destructive-guard.ts');
const purgedSources = await purgeExpiredSources(engine);
const purgedPages = await engine.purgeDeletedPages(SOFT_DELETE_TTL_HOURS_FOR_PURGE);
return {
phase: 'purge',
status: 'ok',
duration_ms: 0,
summary: `purged ${purgedSources.length} source(s) and ${purgedPages.count} page(s) past the 72h recovery window`,
details: {
purged_sources_count: purgedSources.length,
purged_pages_count: purgedPages.count,
purged_sources: purgedSources,
purged_page_slugs: purgedPages.slugs,
},
};
} catch (e) {
return {
phase: 'purge',
status: 'fail',
duration_ms: 0,
summary: 'purge phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
/** v0.26.5: matches SOFT_DELETE_TTL_HOURS in destructive-guard.ts. Inlined here
* to avoid a static import (purge phase is only loaded in the autopilot path). */
const SOFT_DELETE_TTL_HOURS_FOR_PURGE = 72;
async function runPhaseOrphans(engine: BrainEngine): Promise<PhaseResult> {
try {
const { findOrphans } = await import('../commands/orphans.ts');
@@ -931,6 +996,30 @@ export async function runCycle(
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 9: purge (v0.26.5) ────────────────────────────────
// Hard-delete soft-deleted pages and expired archived sources past the
// 72h recovery window. Runs last so the rest of the cycle sees the
// recoverable set; the purge then drops what's truly expired.
if (phases.includes('purge')) {
checkAborted(opts.signal);
if (!engine) {
phaseResults.push({
phase: 'purge',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.purge');
const { result, duration_ms } = await timePhase(() => runPhasePurge(engine, dryRun));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
} finally {
if (lock) {
try { await lock.release(); } catch { /* best-effort */ }
@@ -965,6 +1054,8 @@ function emptyTotals(): CycleReport['totals'] {
transcripts_processed: 0,
synth_pages_written: 0,
patterns_written: 0,
purged_sources_count: 0,
purged_pages_count: 0,
};
}
@@ -992,6 +1083,9 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
t.synth_pages_written = Number(p.details.pages_written ?? 0);
} else if (p.phase === 'patterns' && p.details) {
t.patterns_written = Number(p.details.patterns_written ?? 0);
} else if (p.phase === 'purge' && p.details) {
t.purged_sources_count = Number(p.details.purged_sources_count ?? 0);
t.purged_pages_count = Number(p.details.purged_pages_count ?? 0);
}
}
return t;
+337
View File
@@ -0,0 +1,337 @@
/**
* Destructive operation guard v0.26.5
*
* Protects against accidental data loss in gbrain by requiring explicit
* confirmation for operations that cascade-delete pages, chunks, or embeddings.
*
* Three layers:
* 1. Impact preview always shown before destructive actions
* 2. Confirmation gate requires --confirm-destructive or interactive "type source name"
* 3. Soft-delete with TTL sources are tombstoned for 72h before permanent deletion
*
* Design principle: the blast radius should be visible BEFORE you pull the trigger,
* and recoverable AFTER you pull it (within a grace period).
*/
import type { BrainEngine } from './engine.ts';
// ── Types ───────────────────────────────────────────────────
export interface DestructiveImpact {
sourceId: string;
sourceName: string;
pageCount: number;
chunkCount: number;
embeddingCount: number;
fileCount: number;
/** Human-readable summary line */
summary: string;
}
export interface SoftDeletedSource {
id: string;
name: string;
deletedAt: Date;
expiresAt: Date;
pageCount: number;
}
// ── Constants ───────────────────────────────────────────────
/** Hours before a soft-deleted source is permanently purged. */
export const SOFT_DELETE_TTL_HOURS = 72;
/** Threshold: operations affecting this many pages or more require confirmation. */
export const CONFIRM_THRESHOLD_PAGES = 1;
// ── Impact Assessment ───────────────────────────────────────
/**
* Compute the blast radius of deleting a source.
*/
export async function assessDestructiveImpact(
engine: BrainEngine,
sourceId: string,
): Promise<DestructiveImpact | null> {
// Fetch source metadata
const sources = await engine.executeRaw<{ id: string; name: string }>(
`SELECT id, name FROM sources WHERE id = $1`,
[sourceId],
);
if (sources.length === 0) return null;
const src = sources[0];
// Count pages
const pageRows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
[sourceId],
);
const pageCount = pageRows[0]?.n ?? 0;
// Count chunks
const chunkRows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM content_chunks cc
JOIN pages p ON cc.page_id = p.id
WHERE p.source_id = $1`,
[sourceId],
);
const chunkCount = chunkRows[0]?.n ?? 0;
// Count embeddings (chunks with non-null embedding vectors)
const embedRows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM content_chunks cc
JOIN pages p ON cc.page_id = p.id
WHERE p.source_id = $1 AND cc.embedding IS NOT NULL`,
[sourceId],
);
const embeddingCount = embedRows[0]?.n ?? 0;
// Count files in storage (if any). PGLite has no `files` table — that
// surface is Postgres-only (CLAUDE.md: "No files table" for PGLite). Probe
// the table existence via information_schema so this works on both engines.
let fileCount = 0;
const filesTableRows = await engine.executeRaw<{ exists: boolean }>(
`SELECT EXISTS (
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'files'
) AS exists`,
);
if (filesTableRows[0]?.exists) {
const fileRows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM files WHERE source_id = $1`,
[sourceId],
);
fileCount = fileRows[0]?.n ?? 0;
}
const parts: string[] = [];
if (pageCount > 0) parts.push(`${pageCount.toLocaleString()} pages`);
if (chunkCount > 0) parts.push(`${chunkCount.toLocaleString()} chunks`);
if (embeddingCount > 0) parts.push(`${embeddingCount.toLocaleString()} embeddings`);
if (fileCount > 0) parts.push(`${fileCount.toLocaleString()} files`);
const summary = parts.length > 0
? `⚠️ This will permanently delete: ${parts.join(', ')}`
: `Source "${sourceId}" has no data (safe to remove).`;
return {
sourceId,
sourceName: src.name,
pageCount,
chunkCount,
embeddingCount,
fileCount,
summary,
};
}
// ── Confirmation Gate ───────────────────────────────────────
/**
* Check whether the caller has provided sufficient confirmation for a
* destructive operation. Returns an error message if blocked, or null if OK.
*/
export function checkDestructiveConfirmation(
impact: DestructiveImpact,
opts: {
yes?: boolean;
confirmDestructive?: boolean;
dryRun?: boolean;
},
): string | null {
// Dry run always passes (no side effects)
if (opts.dryRun) return null;
// No data = no risk
if (impact.pageCount === 0 && impact.chunkCount === 0 && impact.fileCount === 0) {
return null;
}
// --confirm-destructive is the explicit "I know what I'm doing" flag
if (opts.confirmDestructive) return null;
// --yes alone is NOT sufficient for destructive operations with data.
// This is the key behavior change: --yes used to be enough, now you
// need --confirm-destructive when there's actual data at stake.
if (opts.yes && impact.pageCount === 0) return null;
return (
`\n${impact.summary}\n\n` +
`To proceed, pass --confirm-destructive (or use soft-delete: gbrain sources archive ${impact.sourceId}).\n` +
`To preview without side effects: --dry-run`
);
}
// ── Soft Delete ─────────────────────────────────────────────
/**
* Soft-delete a source: mark `archived = true` with a 72h TTL. Pages remain
* in DB; the source is hidden from search via `buildVisibilityClause` and
* federation is disabled via the existing `config.federated` JSONB key. After
* TTL expires, the autopilot purge phase or manual `gbrain sources purge`
* permanently removes the row (cascade delete to pages + chunks).
*
* v0.26.5: archive state moved from `config` JSONB keys to real columns
* (`archived`, `archived_at`, `archive_expires_at`). Migration v34 backfills
* pre-v0.26.5 rows. Faster filter, no reserved-key footgun. The `federated`
* key stays in JSONB because federation has its own toggle path.
*/
export async function softDeleteSource(
engine: BrainEngine,
sourceId: string,
): Promise<SoftDeletedSource | null> {
// Atomic: only flip rows that are currently active. Returns the metadata
// we need without a follow-up SELECT. RETURNING projects the columns the
// caller cares about; pageCount is a separate count.
const expiresClause = `now() + (${SOFT_DELETE_TTL_HOURS} || ' hours')::interval`;
const rows = await engine.executeRaw<{ id: string; name: string; archived_at: string; archive_expires_at: string }>(
`UPDATE sources
SET archived = true,
archived_at = now(),
archive_expires_at = ${expiresClause},
config = COALESCE(config, '{}'::jsonb) || '{"federated": false}'::jsonb
WHERE id = $1 AND archived = false
RETURNING id, name, archived_at, archive_expires_at`,
[sourceId],
);
if (rows.length === 0) return null;
const row = rows[0];
const pageRows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
[sourceId],
);
const pageCount = pageRows[0]?.n ?? 0;
return {
id: sourceId,
name: row.name,
deletedAt: new Date(row.archived_at),
expiresAt: new Date(row.archive_expires_at),
pageCount,
};
}
/**
* Restore a soft-deleted source (un-archive). Returns true iff a row was
* restored. Idempotent-as-false on "already active" or "not found".
*
* v0.26.5: clears the column-based archive state and (by default) flips
* `config.federated = true` so the source re-enters federated search. The
* `--no-federate` operator opt-out keeps federation disabled.
*/
export async function restoreSource(
engine: BrainEngine,
sourceId: string,
refederate: boolean = true,
): Promise<boolean> {
const federatedPatch = refederate ? '{"federated": true}' : '{"federated": false}';
const rows = await engine.executeRaw<{ id: string }>(
`UPDATE sources
SET archived = false,
archived_at = NULL,
archive_expires_at = NULL,
config = COALESCE(config, '{}'::jsonb) || $1::jsonb
WHERE id = $2 AND archived = true
RETURNING id`,
[federatedPatch, sourceId],
);
return rows.length > 0;
}
/**
* List all soft-deleted (archived) sources.
*
* v0.26.5: filters via the real `archived` column instead of JSONB
* containment. Faster, indexable on demand, no JSONB reserved-key collision
* with future config schemas.
*/
export async function listArchivedSources(
engine: BrainEngine,
): Promise<SoftDeletedSource[]> {
const rows = await engine.executeRaw<{
id: string;
name: string;
archived_at: string;
archive_expires_at: string;
page_count: number;
}>(
`SELECT
s.id, s.name, s.archived_at, s.archive_expires_at,
COALESCE((SELECT COUNT(*)::int FROM pages p WHERE p.source_id = s.id), 0) AS page_count
FROM sources s
WHERE s.archived = true
ORDER BY s.archived_at DESC`,
);
return rows.map((row) => ({
id: row.id,
name: row.name,
deletedAt: new Date(row.archived_at),
expiresAt: new Date(row.archive_expires_at),
pageCount: row.page_count,
}));
}
/**
* Permanently purge sources whose 72h TTL has expired. Cascades to pages
* (and content_chunks via existing FKs). Returns the ids of purged sources.
*
* v0.26.5: moved from JSONB-driven iteration to a single set-based DELETE
* with `archived = true AND archive_expires_at <= now()`. Server-side
* filter; one round-trip; cascade-friendly.
*/
export async function purgeExpiredSources(
engine: BrainEngine,
): Promise<string[]> {
const rows = await engine.executeRaw<{ id: string }>(
`DELETE FROM sources
WHERE archived = true
AND archive_expires_at IS NOT NULL
AND archive_expires_at <= now()
RETURNING id`,
);
return rows.map((r) => r.id);
}
// ── Display Helpers ─────────────────────────────────────────
/**
* Format an impact assessment for terminal display.
*/
export function formatImpact(impact: DestructiveImpact): string {
const lines: string[] = [
``,
`╔══════════════════════════════════════════════════════════╗`,
`║ DESTRUCTIVE OPERATION — Impact Preview ║`,
`╠══════════════════════════════════════════════════════════╣`,
`║ Source: ${impact.sourceName.padEnd(42)}`,
`║ Source ID: ${impact.sourceId.padEnd(42)}`,
`║ ║`,
`║ Pages: ${String(impact.pageCount.toLocaleString()).padEnd(42)}`,
`║ Chunks: ${String(impact.chunkCount.toLocaleString()).padEnd(42)}`,
`║ Embeddings: ${String(impact.embeddingCount.toLocaleString()).padEnd(42)}`,
`║ Files: ${String(impact.fileCount.toLocaleString()).padEnd(42)}`,
`╠══════════════════════════════════════════════════════════╣`,
`${impact.summary.padEnd(56)}`,
`╚══════════════════════════════════════════════════════════╝`,
``,
];
return lines.join('\n');
}
export function formatSoftDelete(sd: SoftDeletedSource): string {
const hours = Math.round((sd.expiresAt.getTime() - Date.now()) / (1000 * 60 * 60));
return [
``,
`Source "${sd.id}" archived (soft-deleted).`,
` ${sd.pageCount.toLocaleString()} pages preserved for ${SOFT_DELETE_TTL_HOURS}h.`,
` Expires: ${sd.expiresAt.toISOString()} (~${hours}h from now)`,
` Removed from search. Data intact.`,
``,
` Restore: gbrain sources restore ${sd.id}`,
` Purge now: gbrain sources purge ${sd.id} --confirm-destructive`,
``,
].join('\n');
}
+41 -2
View File
@@ -1,5 +1,5 @@
import type {
Page, PageInput, PageFilters,
Page, PageInput, PageFilters, GetPageOpts,
Chunk, ChunkInput, StaleChunkRow,
SearchResult, SearchOpts,
Link, GraphNode, GraphPath,
@@ -128,9 +128,48 @@ export interface BrainEngine {
withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T>;
// Pages CRUD
getPage(slug: string): Promise<Page | null>;
/**
* Fetch a page by slug.
* v0.26.5: by default soft-deleted rows return null (matches the search
* filter contract). Pass `opts.includeDeleted: true` to surface them with
* `deleted_at` populated used by `gbrain pages purge-deleted` listing,
* by `restore_page` flow, and by operator diagnostics.
*/
getPage(slug: string, opts?: GetPageOpts): Promise<Page | null>;
putPage(slug: string, page: PageInput): Promise<Page>;
/**
* Hard-delete a page row. Cascades to content_chunks, page_links,
* chunk_relations via existing FK ON DELETE CASCADE.
*
* v0.26.5: this is no longer the public-facing `delete_page` op handler
* the op now soft-deletes via `softDeletePage` instead. `deletePage` stays
* as the underlying primitive used by `purgeDeletedPages` and by callers
* that explicitly want hard-delete semantics (e.g. test setup teardown).
*/
deletePage(slug: string): Promise<void>;
/**
* v0.26.5 set `deleted_at = now()` on a page. Returns the slug if a row
* was soft-deleted, null if no row matched (already soft-deleted OR not found).
* Idempotent-as-null. The page stays in the DB and cascade rows (chunks,
* links) stay intact; the autopilot purge phase hard-deletes after 72h.
*/
softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null>;
/**
* v0.26.5 clear `deleted_at` on a soft-deleted page. Returns true iff a
* row was restored. False if the slug is unknown OR the page is not
* currently soft-deleted (idempotent-as-false).
*/
restorePage(slug: string, opts?: { sourceId?: string }): Promise<boolean>;
/**
* v0.26.5 hard-delete pages whose `deleted_at` is older than the cutoff.
* Called by the autopilot purge phase and by the `gbrain pages purge-deleted`
* CLI escape hatch. Cascades through existing FKs.
*/
purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }>;
/**
* v0.26.5: by default `listPages` excludes soft-deleted rows. Set
* `filters.includeDeleted: true` to surface them.
*/
listPages(filters?: PageFilters): Promise<Page[]>;
resolveSlugs(partial: string): Promise<string[]>;
/**
+87
View File
@@ -1307,6 +1307,93 @@ export const MIGRATIONS: Migration[] = [
ON mcp_request_log(agent_name, created_at DESC);
`,
},
{
version: 34,
name: 'destructive_guard_columns',
// v0.26.5 — soft-delete + recovery window for sources AND pages.
// Renumbered v33→v34 on master merge: master's v33 (admin_dashboard_columns_v0_26_3)
// landed first in PR #586. v34 follows it.
//
// pages.deleted_at: `delete_page` op now sets deleted_at = now() instead of
// hard-deleting. The autopilot purge phase hard-deletes rows where
// deleted_at < now() - 72h. Search and `get_page` filter
// `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in.
//
// sources.archived/archived_at/archive_expires_at: promoted from JSONB keys
// to real columns. v0.26.0 + the cherry-picked PR #595 wrote these inside
// `sources.config` JSONB. Real columns are faster to filter, avoid the
// reserved-key footgun, and let the search visibility filter compile to a
// column lookup. The 72h TTL is preserved by reading
// `archive_expires_at = archived_at + INTERVAL '72 hours'`.
//
// Backfill: any row that previously stored `{"archived":true,"archived_at":"...","archive_expires_at":"..."}`
// in config gets migrated to the new columns, then the keys are stripped
// from JSONB so the JSONB shape stays canonical going forward.
//
// Engine-aware partial index: Postgres uses CREATE INDEX CONCURRENTLY (no
// write-blocking lock); PGLite uses plain CREATE INDEX. Mirrors v14
// (pages_updated_at_index) handler shape.
sql: '',
handler: async (engine) => {
// 1. Add columns. ALTER TABLE ADD COLUMN IF NOT EXISTS is idempotent on
// both engines.
await engine.runMigration(34, `
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN NOT NULL DEFAULT false;
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived_at TIMESTAMPTZ;
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archive_expires_at TIMESTAMPTZ;
`);
// 2. Backfill from JSONB shape used by pre-v0.26.5 cherry-picks of PR #595.
// Idempotent: subsequent re-runs find zero matching rows.
await engine.runMigration(34, `
UPDATE sources
SET archived = true,
archived_at = COALESCE((config->>'archived_at')::timestamptz, now()),
archive_expires_at = COALESCE(
(config->>'archive_expires_at')::timestamptz,
COALESCE((config->>'archived_at')::timestamptz, now()) + INTERVAL '72 hours'
)
WHERE config ? 'archived'
AND (config->>'archived')::boolean = true
AND archived = false;
`);
await engine.runMigration(34, `
UPDATE sources
SET config = config - 'archived' - 'archived_at' - 'archive_expires_at'
WHERE config ?| ARRAY['archived', 'archived_at', 'archive_expires_at'];
`);
// 3. Partial index for the autopilot purge sweep. Postgres CONCURRENTLY
// avoids the SHARE lock on `pages`; PGLite has no concurrent writers.
if (engine.kind === 'postgres') {
// Pre-drop any invalid index from a prior CONCURRENTLY failure (matches v14 pattern).
await engine.runMigration(34, `
DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_deleted_at_purge_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_deleted_at_purge_idx';
END IF;
END $$;
`);
await engine.runMigration(34, `
CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
`);
} else {
await engine.runMigration(34, `
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
`);
}
},
// CONCURRENTLY on Postgres requires no surrounding transaction. PGLite ignores
// this flag, so the index DDL runs in whatever wrapper applies.
transaction: false,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+73 -9
View File
@@ -293,22 +293,24 @@ export interface Operation {
const get_page: Operation = {
name: 'get_page',
description: 'Read a page by slug (supports optional fuzzy matching)',
description: 'Read a page by slug (supports optional fuzzy matching). Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated (see v0.26.5 recovery window).',
params: {
slug: { type: 'string', required: true, description: 'Page slug' },
fuzzy: { type: 'boolean', description: 'Enable fuzzy slug resolution (default: false)' },
include_deleted: { type: 'boolean', description: 'v0.26.5: surface soft-deleted pages with deleted_at populated (default: false). Used by restore workflows.' },
},
handler: async (ctx, p) => {
const slug = p.slug as string;
const fuzzy = (p.fuzzy as boolean) || false;
const includeDeleted = (p.include_deleted as boolean) === true;
let page = await ctx.engine.getPage(slug);
let page = await ctx.engine.getPage(slug, { includeDeleted });
let resolved_slug: string | undefined;
if (!page && fuzzy) {
const candidates = await ctx.engine.resolveSlugs(slug);
if (candidates.length === 1) {
page = await ctx.engine.getPage(candidates[0]);
page = await ctx.engine.getPage(candidates[0], { includeDeleted });
resolved_slug = candidates[0];
} else if (candidates.length > 1) {
return { error: 'ambiguous_slug', candidates };
@@ -316,7 +318,7 @@ const get_page: Operation = {
}
if (!page) {
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug or use fuzzy: true');
throw new OperationError('page_not_found', `Page not found: ${slug}`, includeDeleted ? 'Check the slug or use fuzzy: true' : 'Page may be soft-deleted; pass include_deleted: true to verify');
}
const tags = await ctx.engine.getTags(page.slug);
@@ -625,39 +627,99 @@ async function runAutoLink(
const delete_page: Operation = {
name: 'delete_page',
description: 'Delete a page',
description: 'Soft-delete a page. The row is hidden from search and from get_page/list_pages, but is recoverable via restore_page within 72h. The autopilot purge phase hard-deletes after the recovery window. Pass include_deleted: true to get_page to verify the soft-delete landed.',
params: {
slug: { type: 'string', required: true },
},
mutating: true,
scope: 'write',
handler: async (ctx, p) => {
if (ctx.dryRun) return { dry_run: true, action: 'delete_page', slug: p.slug };
await ctx.engine.deletePage(p.slug as string);
return { status: 'deleted' };
const slug = p.slug as string;
if (ctx.dryRun) return { dry_run: true, action: 'soft_delete_page', slug };
// v0.26.5: rewired from hard-delete to soft-delete. The hard-delete primitive
// (engine.deletePage) is now reserved for purgeDeletedPages and explicit
// tests. softDeletePage returns null when the slug is unknown OR already
// soft-deleted (idempotent-as-null) — preserve that as a clean no-op shape.
const result = await ctx.engine.softDeletePage(slug);
if (result === null) {
// Distinguish "not found" from "already soft-deleted" so the agent gets a
// clear signal. Probe once with include_deleted to disambiguate.
const existing = await ctx.engine.getPage(slug, { includeDeleted: true });
if (!existing) {
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.');
}
return { status: 'already_soft_deleted', slug, deleted_at: existing.deleted_at };
}
return { status: 'soft_deleted', slug, recoverable_until: 'now + 72h via restore_page' };
},
cliHints: { name: 'delete', positional: ['slug'] },
};
const restore_page: Operation = {
name: 'restore_page',
description: 'v0.26.5 — restore a soft-deleted page (clear deleted_at). Returns success only if the page was actually soft-deleted. After this op, the page reappears in search and in get_page/list_pages without the include_deleted flag.',
params: {
slug: { type: 'string', required: true },
},
mutating: true,
scope: 'write',
handler: async (ctx, p) => {
const slug = p.slug as string;
if (ctx.dryRun) return { dry_run: true, action: 'restore_page', slug };
const ok = await ctx.engine.restorePage(slug);
if (!ok) {
// Distinguish "not found" from "already active" (idempotent-as-false).
const existing = await ctx.engine.getPage(slug, { includeDeleted: true });
if (!existing) {
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.');
}
return { status: 'already_active', slug };
}
return { status: 'restored', slug };
},
cliHints: { name: 'restore', positional: ['slug'] },
};
const purge_deleted_pages: Operation = {
name: 'purge_deleted_pages',
description: 'v0.26.5 — admin-only. Hard-deletes pages whose deleted_at is older than older_than_hours (default 72). Cascades through content_chunks, page_links, chunk_relations. Local CLI only (not exposed over HTTP MCP). Manual escape hatch alongside the autopilot purge phase.',
params: {
older_than_hours: { type: 'number', description: 'Age cutoff in hours. Default 72.' },
},
mutating: true,
scope: 'admin',
localOnly: true,
handler: async (ctx, p) => {
const olderThanHours = (p.older_than_hours as number | undefined) ?? 72;
if (ctx.dryRun) return { dry_run: true, action: 'purge_deleted_pages', older_than_hours: olderThanHours };
const result = await ctx.engine.purgeDeletedPages(olderThanHours);
return { status: 'purged', count: result.count, slugs: result.slugs };
},
cliHints: { name: 'purge-deleted' },
};
const list_pages: Operation = {
name: 'list_pages',
description: 'List pages with optional filters',
description: 'List pages with optional filters. Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated.',
params: {
type: { type: 'string', description: 'Filter by page type' },
tag: { type: 'string', description: 'Filter by tag' },
limit: { type: 'number', description: 'Max results (default 50)' },
include_deleted: { type: 'boolean', description: 'v0.26.5: include soft-deleted pages (default: false). Used by restore workflows and operator diagnostics.' },
},
handler: async (ctx, p) => {
const pages = await ctx.engine.listPages({
type: p.type as any,
tag: p.tag as string,
limit: clampSearchLimit(p.limit as number | undefined, 50, 100),
includeDeleted: (p.include_deleted as boolean) === true,
});
return pages.map(pg => ({
slug: pg.slug,
type: pg.type,
title: pg.title,
updated_at: pg.updated_at,
...(pg.deleted_at ? { deleted_at: pg.deleted_at } : {}),
}));
},
scope: 'read',
@@ -1529,6 +1591,8 @@ const find_orphans: Operation = {
export const operations: Operation[] = [
// Page CRUD
get_page, put_page, delete_page, list_pages,
// v0.26.5 destructive-guard ops (page-level soft-delete + recovery + admin purge)
restore_page, purge_deleted_pages,
// Search
search, query,
// Tags
+103 -10
View File
@@ -23,7 +23,7 @@ import type {
} from './types.ts';
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts';
type PGLiteDB = PGlite;
@@ -212,6 +212,7 @@ export class PGLiteEngine implements BrainEngine {
* - `links.origin_page_id` column (indexed by `idx_links_origin`) v0.13
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) v0.19
* - `content_chunks.language` column (indexed by `idx_chunks_language`) v0.19
* - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) v0.26.5
*
* **Maintenance contract:** when a future migration adds a column-with-index
* or new-table-with-FK referenced by PGLITE_SCHEMA_SQL, extend this method
@@ -226,6 +227,8 @@ export class PGLiteEngine implements BrainEngine {
WHERE table_schema='public' AND table_name='pages') AS pages_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='pages' AND column_name='source_id') AS source_id_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='pages' AND column_name='deleted_at') AS deleted_at_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema='public' AND table_name='links') AS links_exists,
EXISTS (SELECT 1 FROM information_schema.columns
@@ -242,6 +245,7 @@ export class PGLiteEngine implements BrainEngine {
const probe = rows[0] as {
pages_exists: boolean;
source_id_exists: boolean;
deleted_at_exists: boolean;
links_exists: boolean;
link_source_exists: boolean;
origin_page_id_exists: boolean;
@@ -255,9 +259,10 @@ export class PGLiteEngine implements BrainEngine {
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
const needsChunksBootstrap = probe.chunks_exists
&& (!probe.symbol_name_exists || !probe.language_exists);
const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists;
// Fresh installs (no tables yet) and modern brains both no-op.
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return;
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
@@ -305,6 +310,16 @@ export class PGLiteEngine implements BrainEngine {
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
`);
}
if (needsPagesDeletedAt) {
// v34 (destructive_guard_columns) adds the column + sources columns +
// partial purge index. Bootstrap only adds enough for PGLITE_SCHEMA_SQL's
// `CREATE INDEX pages_deleted_at_purge_idx ... WHERE deleted_at IS NOT NULL`
// not to crash. v34 runs later via runMigrations and is idempotent.
await this.db.exec(`
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
`);
}
}
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
@@ -329,11 +344,23 @@ export class PGLiteEngine implements BrainEngine {
}
// Pages CRUD
async getPage(slug: string): Promise<Page | null> {
async getPage(slug: string, opts?: { sourceId?: string; includeDeleted?: boolean }): Promise<Page | null> {
// v0.26.5: hide soft-deleted by default; opt-in via opts.includeDeleted.
const includeDeleted = opts?.includeDeleted === true;
const sourceId = opts?.sourceId;
const where: string[] = ['slug = $1'];
const params: unknown[] = [slug];
if (sourceId) {
params.push(sourceId);
where.push(`source_id = $${params.length}`);
}
if (!includeDeleted) {
where.push('deleted_at IS NULL');
}
const { rows } = await this.db.query(
`SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
FROM pages WHERE slug = $1`,
[slug]
`SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
FROM pages WHERE ${where.join(' AND ')} LIMIT 1`,
params
);
if (rows.length === 0) return null;
return rowToPage(rows[0] as Record<string, unknown>);
@@ -372,6 +399,54 @@ export class PGLiteEngine implements BrainEngine {
await this.db.query('DELETE FROM pages WHERE slug = $1', [slug]);
}
async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> {
// Idempotent-as-null: only flip rows currently active. Source filter is
// optional; without it the first matching row across sources gets soft-deleted.
const sourceId = opts?.sourceId;
const where: string[] = ['slug = $1', 'deleted_at IS NULL'];
const params: unknown[] = [slug];
if (sourceId) {
params.push(sourceId);
where.push(`source_id = $${params.length}`);
}
const { rows } = await this.db.query(
`UPDATE pages SET deleted_at = now() WHERE ${where.join(' AND ')} RETURNING slug`,
params
);
if (rows.length === 0) return null;
return { slug: (rows[0] as { slug: string }).slug };
}
async restorePage(slug: string, opts?: { sourceId?: string }): Promise<boolean> {
const sourceId = opts?.sourceId;
const where: string[] = ['slug = $1', 'deleted_at IS NOT NULL'];
const params: unknown[] = [slug];
if (sourceId) {
params.push(sourceId);
where.push(`source_id = $${params.length}`);
}
const { rows } = await this.db.query(
`UPDATE pages SET deleted_at = NULL WHERE ${where.join(' AND ')} RETURNING slug`,
params
);
return rows.length > 0;
}
async purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }> {
// Clamp to non-negative integer; cascade through FKs (content_chunks,
// page_links, chunk_relations) on DELETE.
const hours = Math.max(0, Math.floor(olderThanHours));
const { rows } = await this.db.query(
`DELETE FROM pages
WHERE deleted_at IS NOT NULL
AND deleted_at < now() - ($1 || ' hours')::interval
RETURNING slug`,
[hours]
);
const slugs = (rows as { slug: string }[]).map((r) => r.slug);
return { slugs, count: slugs.length };
}
async listPages(filters?: PageFilters): Promise<Page[]> {
const limit = filters?.limit || 100;
const offset = filters?.offset || 0;
@@ -399,6 +474,10 @@ export class PGLiteEngine implements BrainEngine {
params.push(escaped);
where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`);
}
// v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted.
if (filters?.includeDeleted !== true) {
where.push('p.deleted_at IS NULL');
}
const whereSql = where.length > 0 ? `WHERE ${where.join(' AND ')}` : '';
params.push(limit, offset);
@@ -479,6 +558,9 @@ export class PGLiteEngine implements BrainEngine {
extraFilter += ` AND cc.symbol_type = $${params.length}`;
}
// v0.26.5: visibility filter (soft-deleted + archived-source).
const visibilityClause = buildVisibilityClause('p', 's');
const { rows } = await this.db.query(
`WITH ranked AS (
SELECT
@@ -490,7 +572,8 @@ export class PGLiteEngine implements BrainEngine {
) THEN true ELSE false END AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
ORDER BY score DESC
LIMIT $2
),
@@ -547,6 +630,9 @@ export class PGLiteEngine implements BrainEngine {
extraFilter += ` AND cc.symbol_type = $${params.length}`;
}
// v0.26.5: visibility filter for the chunk-grain anchor primitive.
const visibilityClause = buildVisibilityClause('p', 's');
const { rows } = await this.db.query(
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
@@ -557,7 +643,8 @@ export class PGLiteEngine implements BrainEngine {
) THEN true ELSE false END AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
ORDER BY score DESC
LIMIT $2 OFFSET $3`,
params
@@ -603,6 +690,10 @@ export class PGLiteEngine implements BrainEngine {
extraFilter += ` AND cc.symbol_type = $${params.length}`;
}
// v0.26.5: visibility filter applied in the inner CTE so HNSW sees the
// same candidate count it always did. See postgres-engine.ts for rationale.
const visibilityClause = buildVisibilityClause('p', 's');
const { rows } = await this.db.query(
`WITH hnsw_candidates AS (
SELECT
@@ -611,7 +702,8 @@ export class PGLiteEngine implements BrainEngine {
1 - (cc.embedding <=> $1::vector) AS raw_score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause}
JOIN sources s ON s.id = p.source_id
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
ORDER BY cc.embedding <=> $1::vector
LIMIT $2
)
@@ -1327,7 +1419,8 @@ export class PGLiteEngine implements BrainEngine {
async getStats(): Promise<BrainStats> {
const { rows: [stats] } = await this.db.query(`
SELECT
(SELECT count(*) FROM pages) as page_count,
-- v0.26.5: exclude soft-deleted from page_count (mirrors postgres-engine).
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM content_chunks) as chunk_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count,
(SELECT count(*) FROM links) as link_count,
+9
View File
@@ -32,6 +32,10 @@ CREATE TABLE IF NOT EXISTS sources (
last_commit TEXT,
last_sync_at TIMESTAMPTZ,
config JSONB NOT NULL DEFAULT '{}'::jsonb,
-- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql).
archived BOOLEAN NOT NULL DEFAULT false,
archived_at TIMESTAMPTZ,
archive_expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
@@ -60,6 +64,8 @@ CREATE TABLE IF NOT EXISTS pages (
content_hash TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql).
deleted_at TIMESTAMPTZ,
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
);
@@ -67,6 +73,9 @@ CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
-- v0.26.5: partial index supports the autopilot purge sweep (mirrors src/schema.sql).
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
-- ============================================================
-- content_chunks: chunked content with embeddings
+103 -7
View File
@@ -22,7 +22,7 @@ import { GBrainError } from './types.ts';
import * as db from './db.ts';
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts';
// CONNECTION_ERROR_PATTERNS / isConnectionError were used by the per-call
// executeRaw retry that #406 originally shipped. Eng-review D3 dropped that
@@ -152,6 +152,7 @@ export class PostgresEngine implements BrainEngine {
* - `links.origin_page_id` column (indexed by `idx_links_origin`) v0.13
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) v0.19
* - `content_chunks.language` column (indexed by `idx_chunks_language`) v0.19
* - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) v0.26.5
*
* Keep this in sync with the PGLite version; covered by
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
@@ -166,6 +167,7 @@ export class PostgresEngine implements BrainEngine {
const probeRows = await conn<{
pages_exists: boolean;
source_id_exists: boolean;
deleted_at_exists: boolean;
links_exists: boolean;
link_source_exists: boolean;
origin_page_id_exists: boolean;
@@ -178,6 +180,8 @@ export class PostgresEngine implements BrainEngine {
WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'deleted_at') AS deleted_at_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists,
EXISTS (SELECT 1 FROM information_schema.columns
@@ -198,8 +202,11 @@ export class PostgresEngine implements BrainEngine {
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
const needsChunksBootstrap = probe.chunks_exists
&& (!probe.symbol_name_exists || !probe.language_exists);
// v0.26.5: pages_deleted_at_purge_idx in SCHEMA_SQL crashes if the column
// doesn't exist yet. Migration v34 also adds it, but bootstrap runs first.
const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists;
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return;
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
@@ -246,6 +253,16 @@ export class PostgresEngine implements BrainEngine {
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
`);
}
if (needsPagesDeletedAt) {
// v34 (destructive_guard_columns) adds the column + sources columns +
// partial purge index. Bootstrap only adds enough for SCHEMA_SQL's
// `CREATE INDEX pages_deleted_at_purge_idx ... WHERE deleted_at IS NOT NULL`
// not to crash. v34 runs later via runMigrations and is idempotent.
await conn.unsafe(`
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
`);
}
}
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
@@ -278,11 +295,19 @@ export class PostgresEngine implements BrainEngine {
}
// Pages CRUD
async getPage(slug: string): Promise<Page | null> {
async getPage(slug: string, opts?: { sourceId?: string; includeDeleted?: boolean }): Promise<Page | null> {
const sql = this.sql;
const includeDeleted = opts?.includeDeleted === true;
const sourceId = opts?.sourceId;
// v0.26.5: default hides soft-deleted rows. Compose with optional sourceId
// filter via fragment chaining (postgres.js supports sql`` composition).
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
const deletedCondition = includeDeleted ? sql`` : sql`AND deleted_at IS NULL`;
const rows = await sql`
SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
FROM pages WHERE slug = ${slug}
SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
FROM pages
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
LIMIT 1
`;
if (rows.length === 0) return null;
return rowToPage(rows[0]);
@@ -321,6 +346,48 @@ export class PostgresEngine implements BrainEngine {
await sql`DELETE FROM pages WHERE slug = ${slug}`;
}
async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> {
const sql = this.sql;
const sourceId = opts?.sourceId;
// Idempotent-as-null contract: only flip rows that are currently active.
// RETURNING projects the slug so we can tell hit-vs-miss without a probe.
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
const rows = await sql`
UPDATE pages SET deleted_at = now()
WHERE slug = ${slug} AND deleted_at IS NULL ${sourceCondition}
RETURNING slug
`;
if (rows.length === 0) return null;
return { slug: rows[0].slug as string };
}
async restorePage(slug: string, opts?: { sourceId?: string }): Promise<boolean> {
const sql = this.sql;
const sourceId = opts?.sourceId;
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
const rows = await sql`
UPDATE pages SET deleted_at = NULL
WHERE slug = ${slug} AND deleted_at IS NOT NULL ${sourceCondition}
RETURNING slug
`;
return rows.length > 0;
}
async purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }> {
const sql = this.sql;
// Clamp to non-negative integer; runaway purge protection. The DELETE
// cascades through content_chunks, page_links, chunk_relations via FKs.
const hours = Math.max(0, Math.floor(olderThanHours));
const rows = await sql`
DELETE FROM pages
WHERE deleted_at IS NOT NULL
AND deleted_at < now() - (${hours} || ' hours')::interval
RETURNING slug
`;
const slugs = rows.map((r) => r.slug as string);
return { slugs, count: slugs.length };
}
async listPages(filters?: PageFilters): Promise<Page[]> {
const sql = this.sql;
const limit = filters?.limit || 100;
@@ -342,11 +409,15 @@ export class PostgresEngine implements BrainEngine {
const slugCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
// v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted.
const deletedCondition = filters?.includeDeleted === true
? sql``
: sql`AND p.deleted_at IS NULL`;
const rows = await sql`
SELECT p.* FROM pages p
${tagJoin}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${deletedCondition}
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
`;
@@ -442,6 +513,12 @@ export class PostgresEngine implements BrainEngine {
params.push(offset);
const offsetParam = `$${params.length}`;
// v0.26.5: visibility filter hides soft-deleted pages and pages from
// archived sources. Joined `sources s` lets the predicate compile to a
// column lookup. NOT bypassed by detail=high — soft-delete is a contract,
// not a temporal preference.
const visibilityClause = buildVisibilityClause('p', 's');
const rawQuery = `
WITH ranked_chunks AS (
SELECT
@@ -450,6 +527,7 @@ export class PostgresEngine implements BrainEngine {
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
${typeClause}
${excludeSlugsClause}
@@ -457,6 +535,7 @@ export class PostgresEngine implements BrainEngine {
${languageClause}
${symbolKindClause}
${hardExcludeClause}
${visibilityClause}
ORDER BY score DESC
LIMIT ${innerLimitParam}
),
@@ -541,6 +620,9 @@ export class PostgresEngine implements BrainEngine {
params.push(offset);
const offsetParam = `$${params.length}`;
// v0.26.5: visibility filter for searchKeywordChunks (anchor primitive).
const visibilityClause = buildVisibilityClause('p', 's');
const rawQuery = `
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
@@ -549,6 +631,7 @@ export class PostgresEngine implements BrainEngine {
false AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
${typeClause}
${excludeSlugsClause}
@@ -556,6 +639,7 @@ export class PostgresEngine implements BrainEngine {
${languageClause}
${symbolKindClause}
${hardExcludeClause}
${visibilityClause}
ORDER BY score DESC
LIMIT ${limitParam}
OFFSET ${offsetParam}
@@ -626,6 +710,12 @@ export class PostgresEngine implements BrainEngine {
params.push(offset);
const offsetParam = `$${params.length}`;
// v0.26.5: visibility filter applied in the inner CTE so the HNSW index
// sees the same row count it always did. Pulling the predicate to the
// outer SELECT would force the HNSW scan to over-fetch and post-filter,
// wasting candidate slots on hidden rows.
const visibilityClause = buildVisibilityClause('p', 's');
const rawQuery = `
WITH hnsw_candidates AS (
SELECT
@@ -634,6 +724,7 @@ export class PostgresEngine implements BrainEngine {
1 - (cc.embedding <=> $1::vector) AS raw_score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.embedding IS NOT NULL
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
${typeClause}
@@ -641,6 +732,7 @@ export class PostgresEngine implements BrainEngine {
${languageClause}
${symbolKindClause}
${hardExcludeClause}
${visibilityClause}
ORDER BY cc.embedding <=> $1::vector
LIMIT ${innerLimitParam}
)
@@ -1379,7 +1471,11 @@ export class PostgresEngine implements BrainEngine {
const sql = this.sql;
const [stats] = await sql`
SELECT
(SELECT count(*) FROM pages) as page_count,
-- v0.26.5: exclude soft-deleted from page_count. Same posture as the
-- search filter and getPage default soft-deleted is hidden everywhere
-- the user looks. Chunks/links stay raw because they still occupy
-- storage until the autopilot purge phase runs.
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM content_chunks) as chunk_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count,
(SELECT count(*) FROM links) as link_count,
+20
View File
@@ -39,6 +39,14 @@ CREATE TABLE IF NOT EXISTS sources (
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
-- actually trigger re-chunking on upgrade.
chunker_version TEXT,
-- v0.26.5: soft-delete + recovery window. \`archive\` flips archived=true and
-- sets archive_expires_at = now() + 72h. The autopilot purge phase
-- hard-deletes rows where archive_expires_at <= now(). Promoted from a
-- JSONB key to real columns to avoid reserved-key footguns and to make the
-- search visibility filter (\`NOT s.archived\`) a column lookup.
archived BOOLEAN NOT NULL DEFAULT false,
archived_at TIMESTAMPTZ,
archive_expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
@@ -75,6 +83,11 @@ CREATE TABLE IF NOT EXISTS pages (
content_hash TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.26.5: soft-delete + recovery window. \`delete_page\` sets deleted_at = now()
-- instead of issuing DELETE. The autopilot purge phase hard-deletes pages
-- where deleted_at < now() - 72h. Search and \`get_page\` filter
-- \`WHERE deleted_at IS NULL\` by default; \`include_deleted: true\` opts in.
deleted_at TIMESTAMPTZ,
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
);
@@ -85,6 +98,13 @@ CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops)
CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC);
-- v0.18.0: source-scoped scans (per /plan-eng-review Section 4).
CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
-- v0.26.5: partial index supports the autopilot purge sweep
-- (\`WHERE deleted_at IS NOT NULL AND deleted_at < now() - INTERVAL '72 hours'\`).
-- Search filters (\`WHERE deleted_at IS NULL\`) do not benefit from this index
-- (predicate doesn't match) and don't need their own soft-deleted cardinality
-- stays low. Don't add a regular \`(deleted_at)\` index without measuring.
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
-- ============================================================
-- content_chunks: chunked content with embeddings
+28
View File
@@ -101,5 +101,33 @@ export function buildHardExcludeClause(slugColumn: string, prefixes: string[]):
return `AND NOT (${likes})`;
}
/**
* v0.26.5 Build the soft-delete + archived-source visibility filter.
*
* Two filters in one fragment:
* - Page-level soft-delete: `<pageAlias>.deleted_at IS NULL` hides pages that
* `delete_page` flipped via `softDeletePage`.
* - Source-level archive: `NOT <sourceAlias>.archived` hides every page
* belonging to a source that `gbrain sources archive` soft-deleted.
*
* Unlike `buildSourceFactorCase`, this clause is NOT bypassed by `detail=high`.
* Soft-deleted content stays hidden regardless of query detail level the
* recovery window is for explicit `include_deleted: true` callers, not for
* temporal queries.
*
* Returns a fragment with leading `AND` so callers can splice it into a WHERE
* unconditionally. Both column references are engine-supplied (never user
* input), so no escape is required on the alias names themselves.
*
* @param pageAlias page table alias (e.g. `'p'`)
* @param sourceAlias source table alias (e.g. `'s'`); the caller is
* responsible for joining `sources` so this alias resolves.
*
* @returns raw SQL fragment, e.g. `AND p.deleted_at IS NULL AND NOT s.archived`
*/
export function buildVisibilityClause(pageAlias: string, sourceAlias: string): string {
return `AND ${pageAlias}.deleted_at IS NULL AND NOT ${sourceAlias}.archived`;
}
// Exported for unit tests
export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral };
+21
View File
@@ -18,6 +18,12 @@ export interface Page {
content_hash?: string;
created_at: Date;
updated_at: Date;
/**
* v0.26.5: when present, the page is soft-deleted. Hidden from search and
* from `getPage` / `listPages` by default; surface via `include_deleted: true`.
* The autopilot purge phase hard-deletes rows where `deleted_at < now() - 72h`.
*/
deleted_at?: Date | null;
}
export type PageKind = 'markdown' | 'code';
@@ -53,6 +59,21 @@ export interface PageFilters {
* queries to a tier directory without loading every page into memory.
*/
slugPrefix?: string;
/**
* v0.26.5: include soft-deleted pages (rows with `deleted_at IS NOT NULL`).
* Default false: hides soft-deleted pages from `list_pages` so agents see the
* same set search returns. Set true to enumerate the recoverable set during
* the 72h window before the autopilot purge phase hard-deletes them.
*/
includeDeleted?: boolean;
}
/** v0.26.5 — opts for getPage / softDeletePage / restorePage. */
export interface GetPageOpts {
/** Filter to a specific source. When omitted, getPage returns the first slug match across sources (pre-existing semantics). */
sourceId?: string;
/** Include soft-deleted pages. Default false. See PageFilters.includeDeleted. */
includeDeleted?: boolean;
}
// Chunks
+7
View File
@@ -43,6 +43,12 @@ export function contentHash(page: PageInput): string {
}
export function rowToPage(row: Record<string, unknown>): Page {
// v0.26.5: deleted_at is optional in the SELECT projection. When the column
// isn't selected (legacy callers), keep the field absent on the returned object.
const deletedAtRaw = row.deleted_at;
const deletedAt = deletedAtRaw == null
? (deletedAtRaw === null ? null : undefined)
: new Date(deletedAtRaw as string);
return {
id: row.id as number,
slug: row.slug as string,
@@ -54,6 +60,7 @@ export function rowToPage(row: Record<string, unknown>): Page {
content_hash: row.content_hash as string | undefined,
created_at: new Date(row.created_at as string),
updated_at: new Date(row.updated_at as string),
...(deletedAt !== undefined && { deleted_at: deletedAt }),
};
}
+20
View File
@@ -35,6 +35,14 @@ CREATE TABLE IF NOT EXISTS sources (
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
-- actually trigger re-chunking on upgrade.
chunker_version TEXT,
-- v0.26.5: soft-delete + recovery window. `archive` flips archived=true and
-- sets archive_expires_at = now() + 72h. The autopilot purge phase
-- hard-deletes rows where archive_expires_at <= now(). Promoted from a
-- JSONB key to real columns to avoid reserved-key footguns and to make the
-- search visibility filter (`NOT s.archived`) a column lookup.
archived BOOLEAN NOT NULL DEFAULT false,
archived_at TIMESTAMPTZ,
archive_expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
@@ -71,6 +79,11 @@ CREATE TABLE IF NOT EXISTS pages (
content_hash TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.26.5: soft-delete + recovery window. `delete_page` sets deleted_at = now()
-- instead of issuing DELETE. The autopilot purge phase hard-deletes pages
-- where deleted_at < now() - 72h. Search and `get_page` filter
-- `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in.
deleted_at TIMESTAMPTZ,
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
);
@@ -81,6 +94,13 @@ CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops)
CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC);
-- v0.18.0: source-scoped scans (per /plan-eng-review Section 4).
CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
-- v0.26.5: partial index supports the autopilot purge sweep
-- (`WHERE deleted_at IS NOT NULL AND deleted_at < now() - INTERVAL '72 hours'`).
-- Search filters (`WHERE deleted_at IS NULL`) do not benefit from this index
-- (predicate doesn't match) and don't need their own — soft-deleted cardinality
-- stays low. Don't add a regular `(deleted_at)` index without measuring.
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
-- ============================================================
-- content_chunks: chunked content with embeddings
+4 -4
View File
@@ -377,8 +377,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
hookCalls++;
},
});
// v0.23: 8 phases → 8 yield calls (one after each).
expect(hookCalls).toBe(8);
// v0.26.5: 9 phases (added `purge`)9 yield calls (one after each).
expect(hookCalls).toBe(9);
});
test('hook exceptions do not abort the cycle', async () => {
@@ -388,8 +388,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
throw new Error('synthetic hook error');
},
});
// Cycle still completed all phases (v0.23: 8).
expect(report.phases.length).toBe(8);
// Cycle still completed all phases (v0.26.5: 9 with the new purge phase).
expect(report.phases.length).toBe(9);
});
});
+325
View File
@@ -0,0 +1,325 @@
/**
* v0.26.5 destructive-guard unit tests.
*
* Source-level guard against accidental data loss. Three layers:
* 1. Impact assessment (counts pages/chunks/embeddings/files for a source)
* 2. Confirmation gate (`--confirm-destructive` required when data exists;
* `--yes` alone rejected)
* 3. Soft-delete with 72h TTL (column-based as of v0.26.5; JSONB shape was
* migrated in v33)
*
* Run against PGLite the contract logic is identical on Postgres but
* PGLite is fast + DATABASE_URL-free. Postgres-specific paths (CONCURRENTLY,
* RLS) are covered separately by E2E tests.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import {
assessDestructiveImpact,
checkDestructiveConfirmation,
softDeleteSource,
restoreSource,
listArchivedSources,
purgeExpiredSources,
formatImpact,
formatSoftDelete,
SOFT_DELETE_TTL_HOURS,
type DestructiveImpact,
} from '../src/core/destructive-guard.ts';
// Tier 3 opt-out — these tests need the cold-init schema path so the v33
// migration columns exist on the brain under test.
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
async function setupBrain(): Promise<PGLiteEngine> {
const engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
return engine;
}
async function seedSource(engine: PGLiteEngine, id: string, opts?: { withPages?: number }): Promise<void> {
await engine.executeRaw(
`INSERT INTO sources (id, name) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`,
[id, id],
);
const count = opts?.withPages ?? 0;
for (let i = 0; i < count; i++) {
await engine.executeRaw(
`INSERT INTO pages (source_id, slug, type, title) VALUES ($1, $2, 'note', $3)`,
[id, `${id}/page-${i}`, `Page ${i}`],
);
}
}
describe('assessDestructiveImpact', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = await setupBrain();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
test('returns null for a non-existent source', async () => {
const impact = await assessDestructiveImpact(engine, 'aim-does-not-exist');
expect(impact).toBeNull();
});
test('counts zero across the board for an empty source', async () => {
await seedSource(engine, 'aim-empty', { withPages: 0 });
const impact = await assessDestructiveImpact(engine, 'aim-empty');
expect(impact).not.toBeNull();
expect(impact!.pageCount).toBe(0);
expect(impact!.chunkCount).toBe(0);
expect(impact!.embeddingCount).toBe(0);
expect(impact!.fileCount).toBe(0);
// Empty-source summary is the safe message, not the "permanently delete" warning.
expect(impact!.summary).toContain('safe to remove');
});
test('counts pages correctly for a populated source', async () => {
await seedSource(engine, 'aim-populated', { withPages: 3 });
const impact = await assessDestructiveImpact(engine, 'aim-populated');
expect(impact!.pageCount).toBe(3);
expect(impact!.summary).toContain('3 pages');
expect(impact!.summary).toContain('permanently delete');
});
test('source-scopes pages — multi-source isolation', async () => {
await seedSource(engine, 'aim-src-a', { withPages: 2 });
await seedSource(engine, 'aim-src-b', { withPages: 5 });
const a = await assessDestructiveImpact(engine, 'aim-src-a');
const b = await assessDestructiveImpact(engine, 'aim-src-b');
expect(a!.pageCount).toBe(2);
expect(b!.pageCount).toBe(5);
});
});
describe('checkDestructiveConfirmation (gate truth table)', () => {
const populated: DestructiveImpact = {
sourceId: 'has-data',
sourceName: 'has-data',
pageCount: 100,
chunkCount: 500,
embeddingCount: 500,
fileCount: 0,
summary: '⚠️ This will permanently delete: 100 pages, 500 chunks, 500 embeddings',
};
const empty: DestructiveImpact = {
sourceId: 'no-data',
sourceName: 'no-data',
pageCount: 0,
chunkCount: 0,
embeddingCount: 0,
fileCount: 0,
summary: 'Source "no-data" has no data (safe to remove).',
};
test('dry-run always passes regardless of flags', () => {
expect(checkDestructiveConfirmation(populated, { dryRun: true })).toBeNull();
expect(checkDestructiveConfirmation(populated, { yes: true, dryRun: true })).toBeNull();
});
test('empty source passes without --confirm-destructive', () => {
expect(checkDestructiveConfirmation(empty, {})).toBeNull();
expect(checkDestructiveConfirmation(empty, { yes: true })).toBeNull();
});
test('--confirm-destructive passes regardless of --yes', () => {
expect(checkDestructiveConfirmation(populated, { confirmDestructive: true })).toBeNull();
expect(checkDestructiveConfirmation(populated, { yes: true, confirmDestructive: true })).toBeNull();
});
test('--yes alone with data is REJECTED with guidance message', () => {
const msg = checkDestructiveConfirmation(populated, { yes: true });
expect(msg).not.toBeNull();
expect(msg).toContain('--confirm-destructive');
expect(msg).toContain('archive');
});
test('no flags + populated source rejects', () => {
const msg = checkDestructiveConfirmation(populated, {});
expect(msg).not.toBeNull();
expect(msg).toContain('--confirm-destructive');
});
});
describe('soft-delete + restore lifecycle (column-based v0.26.5)', () => {
// ONE engine for the whole describe — cold init runs ~29 migrations, ~3s.
// Each test uses a unique source id so they don't cross-pollute.
let engine: PGLiteEngine;
beforeAll(async () => {
engine = await setupBrain();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
test('softDeleteSource flips column shape + sets TTL', async () => {
const id = 'sd-flips';
await seedSource(engine, id, { withPages: 2 });
const before = Date.now();
const result = await softDeleteSource(engine, id);
const after = Date.now();
expect(result).not.toBeNull();
expect(result!.id).toBe(id);
expect(result!.pageCount).toBe(2);
const ttlMs = SOFT_DELETE_TTL_HOURS * 60 * 60 * 1000;
expect(result!.expiresAt.getTime()).toBeGreaterThanOrEqual(before + ttlMs - 1000);
expect(result!.expiresAt.getTime()).toBeLessThanOrEqual(after + ttlMs + 1000);
const rows = await engine.executeRaw<{ archived: boolean; archived_at: string }>(
`SELECT archived, archived_at FROM sources WHERE id = $1`,
[id],
);
expect(rows[0].archived).toBe(true);
expect(rows[0].archived_at).not.toBeNull();
});
test('softDeleteSource is idempotent-as-null on already-archived', async () => {
const id = 'sd-idem';
await seedSource(engine, id, { withPages: 1 });
await softDeleteSource(engine, id);
expect(await softDeleteSource(engine, id)).toBeNull();
});
test('softDeleteSource returns null for unknown source', async () => {
expect(await softDeleteSource(engine, 'sd-unknown-xyz')).toBeNull();
});
test('softDeleteSource flips federated:false in JSONB but archived state is column-based', async () => {
const id = 'sd-jsonb';
await seedSource(engine, id, { withPages: 1 });
await softDeleteSource(engine, id);
const rows = await engine.executeRaw<{ config: any; archived: boolean }>(
`SELECT config, archived FROM sources WHERE id = $1`,
[id],
);
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
expect(config.federated).toBe(false);
expect(rows[0].archived).toBe(true);
// Issue 5 contract: archived must NOT live in config any more.
expect(config.archived).toBeUndefined();
expect(config.archived_at).toBeUndefined();
});
test('restoreSource clears the column state and re-federates by default', async () => {
const id = 'sd-restore-fed';
await seedSource(engine, id, { withPages: 1 });
await softDeleteSource(engine, id);
expect(await restoreSource(engine, id)).toBe(true);
const rows = await engine.executeRaw<{ archived: boolean; archived_at: string | null; config: any }>(
`SELECT archived, archived_at, config FROM sources WHERE id = $1`,
[id],
);
expect(rows[0].archived).toBe(false);
expect(rows[0].archived_at).toBeNull();
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
expect(config.federated).toBe(true);
});
test('restoreSource respects --no-federate (refederate=false)', async () => {
const id = 'sd-no-fed';
await seedSource(engine, id, { withPages: 1 });
await softDeleteSource(engine, id);
await restoreSource(engine, id, false);
const rows = await engine.executeRaw<{ config: any }>(
`SELECT config FROM sources WHERE id = $1`,
[id],
);
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
expect(config.federated).toBe(false);
});
test('restoreSource is idempotent-as-false on already-active', async () => {
const id = 'sd-active';
await seedSource(engine, id);
expect(await restoreSource(engine, id)).toBe(false);
});
test('listArchivedSources filters via the archived column, not JSONB', async () => {
const archivedId = 'la-archived';
const liveId = 'la-live';
await seedSource(engine, archivedId, { withPages: 3 });
await seedSource(engine, liveId, { withPages: 1 });
await softDeleteSource(engine, archivedId);
const archived = await listArchivedSources(engine);
const ids = archived.map((a) => a.id);
expect(ids).toContain(archivedId);
expect(ids).not.toContain(liveId);
const archivedRow = archived.find((a) => a.id === archivedId)!;
expect(archivedRow.pageCount).toBe(3);
});
test('purgeExpiredSources only deletes rows with archive_expires_at <= now()', async () => {
const expiredId = 'pe-expired';
const recoverableId = 'pe-recoverable';
await seedSource(engine, expiredId, { withPages: 2 });
await seedSource(engine, recoverableId, { withPages: 1 });
await softDeleteSource(engine, expiredId);
await softDeleteSource(engine, recoverableId);
await engine.executeRaw(
`UPDATE sources SET archive_expires_at = now() - INTERVAL '1 hour' WHERE id = $1`,
[expiredId],
);
const purged = await purgeExpiredSources(engine);
expect(purged).toContain(expiredId);
expect(purged).not.toContain(recoverableId);
const remainingPages = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
[expiredId],
);
expect(remainingPages[0].n).toBe(0);
});
test('purgeExpiredSources is no-op when nothing is past TTL', async () => {
// After all earlier tests, there may still be archived rows whose
// archive_expires_at is in the future. Force-update any leftover-past
// rows OUT of expiration before this assertion (we only want to test
// the no-op return here, not interfere with prior test state).
await engine.executeRaw(
`UPDATE sources SET archive_expires_at = now() + INTERVAL '72 hours' WHERE archived = true`,
);
const purged = await purgeExpiredSources(engine);
expect(purged).toEqual([]);
});
});
describe('formatters (display helpers)', () => {
test('formatImpact renders the boxed preview with the source id and counts', () => {
const impact: DestructiveImpact = {
sourceId: 'media-corpus',
sourceName: 'Media Corpus',
pageCount: 5033,
chunkCount: 22000,
embeddingCount: 22000,
fileCount: 0,
summary: '⚠️ This will permanently delete: 5,033 pages, 22,000 chunks, 22,000 embeddings',
};
const out = formatImpact(impact);
expect(out).toContain('Media Corpus');
expect(out).toContain('media-corpus');
expect(out).toContain('5,033');
expect(out).toContain('22,000');
expect(out).toContain('DESTRUCTIVE OPERATION');
});
test('formatSoftDelete renders the post-archive guidance with restore command', () => {
const out = formatSoftDelete({
id: 'src-a',
name: 'src-a',
deletedAt: new Date(),
expiresAt: new Date(Date.now() + 72 * 60 * 60 * 1000),
pageCount: 100,
});
expect(out).toContain('archived');
expect(out).toContain('restore src-a');
expect(out).toContain('72');
});
});
+3 -2
View File
@@ -97,8 +97,9 @@ describeE2E('E2E: runCycle against real Postgres', () => {
});
expect(report.schema_version).toBe('1');
// Cycle ran all 8 phases (or skipped the ones that don't support dry-run).
expect(report.phases.length).toBe(8);
// Cycle ran all 9 phases (or skipped the ones that don't support dry-run).
// v0.26.5 added the `purge` phase (9th, after `orphans`).
expect(report.phases.length).toBe(9);
// Nothing got written.
const afterPages = await conn.unsafe(`SELECT count(*)::int AS n FROM pages`);
+2 -1
View File
@@ -271,7 +271,8 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(1);
// Remove the source.
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['remove', 'cascadetest', '--yes']);
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['remove', 'cascadetest', '--confirm-destructive']);
// Everything for that source is gone.
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
+2 -1
View File
@@ -181,7 +181,8 @@ describe('v0.18.0 — sources remove cascades to pages', () => {
);
expect(before[0].n).toBeGreaterThan(0);
await runSources(engine, ['remove', 'testsrc', '--yes']);
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
await runSources(engine, ['remove', 'testsrc', '--confirm-destructive']);
const after = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'testsrc'`,
+323
View File
@@ -0,0 +1,323 @@
/**
* v0.26.5 page-level soft-delete contract tests.
*
* IRON RULE regression test for Q3 (the lynchpin eng-review decision):
* delete_page get_page returns null get_page({include_deleted:true}) returns
* the row with deleted_at populated restore_page get_page returns the row
* again with deleted_at unset.
*
* Plus: BrainEngine surface tests (softDeletePage / restorePage /
* purgeDeletedPages) for happy-path / boundary / cascade cases.
*
* Runs against PGLite same SQL contract as Postgres but DATABASE_URL-free.
* Postgres-specific paths (CONCURRENTLY index, two-stage CTE) covered by
* separate Postgres E2E tests.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
async function setupBrain(): Promise<PGLiteEngine> {
const engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
return engine;
}
async function seedPage(engine: PGLiteEngine, slug: string): Promise<void> {
await engine.putPage(slug, {
type: 'note' as any,
title: slug,
compiled_truth: `Content of ${slug}`,
timeline: '',
frontmatter: {},
});
}
describe('softDeletePage', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = await setupBrain();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
test('happy path: sets deleted_at and returns slug', async () => {
await seedPage(engine, 'people/alice');
const result = await engine.softDeletePage('people/alice');
expect(result).not.toBeNull();
expect(result!.slug).toBe('people/alice');
// The row stays in the DB.
const rows = await engine.executeRaw<{ deleted_at: string | null }>(
`SELECT deleted_at FROM pages WHERE slug = $1`,
['people/alice'],
);
expect(rows.length).toBe(1);
expect(rows[0].deleted_at).not.toBeNull();
});
test('returns null for unknown slug (idempotent-as-null)', async () => {
expect(await engine.softDeletePage('does/not/exist')).toBeNull();
});
test('returns null on already-soft-deleted page (idempotent-as-null)', async () => {
await seedPage(engine, 'people/bob');
const first = await engine.softDeletePage('people/bob');
expect(first).not.toBeNull();
const second = await engine.softDeletePage('people/bob');
expect(second).toBeNull();
});
});
describe('restorePage', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = await setupBrain();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
test('clears deleted_at on a soft-deleted page', async () => {
await seedPage(engine, 'people/carol');
await engine.softDeletePage('people/carol');
expect(await engine.restorePage('people/carol')).toBe(true);
const rows = await engine.executeRaw<{ deleted_at: string | null }>(
`SELECT deleted_at FROM pages WHERE slug = $1`,
['people/carol'],
);
expect(rows[0].deleted_at).toBeNull();
});
test('returns false for unknown slug', async () => {
expect(await engine.restorePage('does/not/exist')).toBe(false);
});
test('returns false on already-active page (idempotent-as-false)', async () => {
await seedPage(engine, 'people/dave');
expect(await engine.restorePage('people/dave')).toBe(false);
});
});
describe('purgeDeletedPages (TTL boundary)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = await setupBrain();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
test('purges pages whose deleted_at is older than the cutoff', async () => {
await seedPage(engine, 'people/eve');
await seedPage(engine, 'people/frank');
// Soft-delete both, then push one's deleted_at into the distant past.
await engine.softDeletePage('people/eve');
await engine.softDeletePage('people/frank');
await engine.executeRaw(
`UPDATE pages SET deleted_at = now() - INTERVAL '73 hours' WHERE slug = $1`,
['people/eve'],
);
const result = await engine.purgeDeletedPages(72);
expect(result.count).toBe(1);
expect(result.slugs).toContain('people/eve');
expect(result.slugs).not.toContain('people/frank');
// 'eve' is gone; 'frank' is still there (still inside recovery window).
const rows = await engine.executeRaw<{ slug: string }>(`SELECT slug FROM pages`);
const remaining = rows.map((r) => r.slug);
expect(remaining).not.toContain('people/eve');
expect(remaining).toContain('people/frank');
});
test('does NOT touch active pages (deleted_at IS NULL)', async () => {
// Bound to this test's seeded slug. Other tests in the same describe may
// have soft-deleted state laying around; we don't care about those, just
// that THIS test's active page is not deleted.
await seedPage(engine, 'people/grace-active');
await engine.purgeDeletedPages(0);
const rows = await engine.executeRaw<{ slug: string; deleted_at: string | null }>(
`SELECT slug, deleted_at FROM pages WHERE slug = $1`,
['people/grace-active'],
);
expect(rows.length).toBe(1);
expect(rows[0].deleted_at).toBeNull();
});
test('cascades to content_chunks via FK ON DELETE CASCADE', async () => {
await seedPage(engine, 'people/heidi');
// Force-add a chunk row so we can observe cascade.
const pageRows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM pages WHERE slug = $1`,
['people/heidi'],
);
await engine.executeRaw(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source) VALUES ($1, 0, 'test', 'compiled_truth')`,
[pageRows[0].id],
);
await engine.softDeletePage('people/heidi');
await engine.executeRaw(
`UPDATE pages SET deleted_at = now() - INTERVAL '73 hours' WHERE slug = $1`,
['people/heidi'],
);
await engine.purgeDeletedPages(72);
const remaining = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM content_chunks WHERE page_id = $1`,
[pageRows[0].id],
);
expect(remaining[0].n).toBe(0);
});
test('clamps negative hours to 0 (no crash, no future-cutoff explosion)', async () => {
await seedPage(engine, 'people/ivan');
await engine.softDeletePage('people/ivan');
// The contract being pinned: negative input must NOT pass through to the
// SQL as a literal negative interval (which would purge from the future
// and effectively delete every soft-deleted row). Implementation does
// `Math.max(0, Math.floor(olderThanHours))`, so -72 collapses to 0. With
// hours=0, the predicate `deleted_at < now()` may or may not match a row
// soft-deleted in the same statement (timing-dependent), so this test
// pins only the safety contract: it returns successfully with a finite
// count and doesn't blow up the brain.
const result = await engine.purgeDeletedPages(-72);
expect(result.count).toBeGreaterThanOrEqual(0);
expect(Number.isFinite(result.count)).toBe(true);
});
});
describe('getPage / listPages includeDeleted contract (Q3 IRON RULE)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = await setupBrain();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
test('Q3 round-trip: delete → get returns null → get(include_deleted) returns row → restore → get returns row again', async () => {
await seedPage(engine, 'people/judy');
// Step 1: page is visible by default.
const before = await engine.getPage('people/judy');
expect(before).not.toBeNull();
expect(before!.deleted_at).toBeFalsy();
// Step 2: soft-delete, default getPage returns null.
await engine.softDeletePage('people/judy');
const afterDelete = await engine.getPage('people/judy');
expect(afterDelete).toBeNull();
// Step 3: include_deleted: true surfaces the row with deleted_at populated.
const surfaced = await engine.getPage('people/judy', { includeDeleted: true });
expect(surfaced).not.toBeNull();
expect(surfaced!.deleted_at).toBeInstanceOf(Date);
// Step 4: restore → default getPage returns the row again.
expect(await engine.restorePage('people/judy')).toBe(true);
const restored = await engine.getPage('people/judy');
expect(restored).not.toBeNull();
expect(restored!.deleted_at).toBeFalsy();
});
test('listPages excludes soft-deleted by default', async () => {
await seedPage(engine, 'people/kim');
await seedPage(engine, 'people/larry');
await engine.softDeletePage('people/kim');
const pages = await engine.listPages({ limit: 100 });
const slugs = pages.map((p) => p.slug);
expect(slugs).not.toContain('people/kim');
expect(slugs).toContain('people/larry');
});
test('listPages includes soft-deleted when includeDeleted: true', async () => {
await seedPage(engine, 'people/mia');
await engine.softDeletePage('people/mia');
const pages = await engine.listPages({ limit: 100, includeDeleted: true });
const slugs = pages.map((p) => p.slug);
expect(slugs).toContain('people/mia');
const mia = pages.find((p) => p.slug === 'people/mia')!;
expect(mia.deleted_at).toBeInstanceOf(Date);
});
});
describe('search visibility (soft-deleted pages hidden from searchKeyword)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = await setupBrain();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
test('searchKeyword hides soft-deleted pages', async () => {
// Two pages, same distinctive term, then soft-delete one.
await engine.putPage('people/nora', {
type: 'note' as any,
title: 'Nora',
compiled_truth: 'gbrainquantum signature term occurs here',
timeline: '',
frontmatter: {},
});
await engine.putPage('people/oscar', {
type: 'note' as any,
title: 'Oscar',
compiled_truth: 'gbrainquantum signature term occurs here too',
timeline: '',
frontmatter: {},
});
// Force chunk creation so search has something to index.
await engine.upsertChunks('people/nora', [
{ chunk_index: 0, chunk_text: 'gbrainquantum signature term occurs here', chunk_source: 'compiled_truth' as any },
]);
await engine.upsertChunks('people/oscar', [
{ chunk_index: 0, chunk_text: 'gbrainquantum signature term occurs here too', chunk_source: 'compiled_truth' as any },
]);
const before = await engine.searchKeyword('gbrainquantum');
expect(before.length).toBe(2);
await engine.softDeletePage('people/nora');
const after = await engine.searchKeyword('gbrainquantum');
const slugs = after.map((r) => r.slug);
expect(slugs).not.toContain('people/nora');
expect(slugs).toContain('people/oscar');
});
test('searchKeyword hides pages from archived sources', async () => {
await engine.executeRaw(
`INSERT INTO sources (id, name) VALUES ('archived-src', 'archived-src') ON CONFLICT DO NOTHING`,
);
await engine.executeRaw(
`INSERT INTO pages (source_id, slug, type, title) VALUES ('archived-src', 'archived-src/secret', 'note', 'Secret')`,
);
const pageRows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM pages WHERE slug = 'archived-src/secret'`,
);
await engine.executeRaw(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source) VALUES ($1, 0, 'gbrainsemaphore unique term', 'compiled_truth')`,
[pageRows[0].id],
);
// Trigger should populate search_vector via the schema trigger.
const before = await engine.searchKeyword('gbrainsemaphore');
expect(before.length).toBe(1);
// Archive the source.
await engine.executeRaw(
`UPDATE sources SET archived = true, archived_at = now(), archive_expires_at = now() + INTERVAL '72 hours' WHERE id = 'archived-src'`,
);
const after = await engine.searchKeyword('gbrainsemaphore');
expect(after.length).toBe(0);
});
});
+8
View File
@@ -60,6 +60,9 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [
// v0.19+ — forward-referenced by `CREATE INDEX idx_chunks_language
// ON content_chunks(language) WHERE language IS NOT NULL`.
{ kind: 'column', table: 'content_chunks', column: 'language' },
// v0.26.5 — forward-referenced by `CREATE INDEX pages_deleted_at_purge_idx
// ON pages (deleted_at) WHERE deleted_at IS NOT NULL`.
{ kind: 'column', table: 'pages', column: 'deleted_at' },
];
test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => {
@@ -89,6 +92,9 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in
DROP INDEX IF EXISTS idx_chunks_language;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS symbol_name;
ALTER TABLE content_chunks DROP COLUMN IF EXISTS language;
DROP INDEX IF EXISTS pages_deleted_at_purge_idx;
ALTER TABLE pages DROP COLUMN IF EXISTS deleted_at;
`);
// Run bootstrap in isolation (NOT initSchema). This is what we're testing.
@@ -142,6 +148,8 @@ test('after bootstrap, PGLITE_SCHEMA_SQL replays without crashing on missing for
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique;
ALTER TABLE links DROP COLUMN IF EXISTS link_source;
ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id;
DROP INDEX IF EXISTS pages_deleted_at_purge_idx;
ALTER TABLE pages DROP COLUMN IF EXISTS deleted_at;
`);
// Bootstrap, then schema replay. Either step crashing fails the test.
+38
View File
@@ -2,6 +2,7 @@ import { describe, test, expect } from 'bun:test';
import {
buildSourceFactorCase,
buildHardExcludeClause,
buildVisibilityClause,
__test__,
} from '../src/core/search/sql-ranking.ts';
import {
@@ -253,3 +254,40 @@ describe('resolveHardExcludes', () => {
expect(r).not.toContain('envdir/');
});
});
// v0.26.5 — visibility clause for soft-deleted pages and archived sources.
describe('buildVisibilityClause (v0.26.5)', () => {
test('emits both predicates joined by AND with a leading AND', () => {
const clause = buildVisibilityClause('p', 's');
// Leading AND so callers can splice unconditionally.
expect(clause.startsWith('AND ')).toBe(true);
// Both predicates present: page-level deleted_at IS NULL + source-level NOT archived.
expect(clause).toContain('p.deleted_at IS NULL');
expect(clause).toContain('NOT s.archived');
});
test('uses the supplied aliases verbatim', () => {
expect(buildVisibilityClause('pp', 'src')).toBe('AND pp.deleted_at IS NULL AND NOT src.archived');
});
test('does NOT bypass on detail level — visibility is a contract, not a temporal preference', () => {
// Distinct from buildSourceFactorCase: there's no detail-gated short-circuit.
// Soft-deleted content stays hidden regardless of caller's detail level.
// Function signature has no detail param at all; this test pins that contract.
expect(buildVisibilityClause.length).toBe(2);
});
test('emits a stable string regardless of call order (idempotent for snapshot tests)', () => {
const a = buildVisibilityClause('p', 's');
const b = buildVisibilityClause('p', 's');
expect(a).toBe(b);
});
test('produces no JSONB containment in the output (column-based, not @>)', () => {
// Issue 5 contract: archived was promoted from JSONB key to real column.
// The visibility clause must not regress to JSONB containment.
const clause = buildVisibilityClause('p', 's');
expect(clause).not.toContain('@>');
expect(clause).not.toContain('config');
});
});