mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5dd5129ce0 | ||
|
|
d5b5ee41c9 | ||
|
|
99dd36c8af | ||
|
|
fed077102b | ||
|
|
82dd96db0c | ||
|
|
747f0a1f89 | ||
|
|
d15c64b0d9 |
@@ -2,6 +2,99 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.26.5] - 2026-05-03
|
||||
|
||||
## **Destructive operation guard, end to end. Sources AND pages now have a 72h recovery window.**
|
||||
## **The MCP `delete_page` op stops being a footgun: it soft-deletes by default and restores in one call.**
|
||||
|
||||
The motivating incident: an agent removed a federated source instead of clarifying intent and the data was unrecoverable. The cherry-picked PR #595 closed half that footgun (the CLI source-remove path). v0.26.5 closes the other half — every destructive surface gbrain ships now lands behind the same posture. Sources, pages, autopilot. One pattern, applied everywhere.
|
||||
|
||||
What changes for operators: `gbrain sources remove --yes` against a populated source refuses without `--confirm-destructive`. `gbrain sources archive` is the safe default. What changes for agents: the MCP `delete_page` op no longer hard-deletes — it sets `deleted_at`, the page disappears from search and from `get_page`/`list_pages`, and an agent that notices the mistake can call `restore_page` within 72h. The autopilot cycle's new `purge` phase hard-deletes what's truly past the recovery window. No cron to wire up. No manual sweep needed.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
| Metric | BEFORE v0.26.5 | AFTER v0.26.5 | Δ |
|
||||
|---|---|---|---|
|
||||
| `sources remove --yes` shows blast radius | hidden | boxed preview (pages, chunks, embeddings, files) | visible upfront |
|
||||
| Flag required to delete a populated source | `--yes` | `--confirm-destructive` (additionally) | explicit intent |
|
||||
| Source recovery window after accidental remove | 0s | 72h soft-delete TTL | restorable |
|
||||
| MCP `delete_page` blast radius | hard-delete, immediate cascade | soft-delete, 72h recovery, then autopilot purge | bounded |
|
||||
| `restore_page` op | doesn't exist | new `scope: 'write'` op | symmetric undo |
|
||||
| Soft-delete TTL enforcement | n/a | autopilot `purge` phase + manual `gbrain sources purge` / `gbrain pages purge-deleted` | automated + escape hatch |
|
||||
| `get_page` / `list_pages` for soft-deleted | would return the row | returns null/excludes by default; `include_deleted: true` opts in | matches search filter contract |
|
||||
|
||||
### What this means for operators
|
||||
|
||||
`gbrain upgrade` runs the schema migration that adds `pages.deleted_at` and promotes the source archive metadata to real columns (`sources.archived`, `archived_at`, `archive_expires_at`). If a `sources remove <id> --yes` script of yours starts refusing, that's the new gate working — pass `--confirm-destructive` only if you actually want permanent deletion. Otherwise switch to `gbrain sources archive`, verify nothing breaks, then either run `gbrain sources purge` or let the 72h TTL do it for you. The autopilot `dream` cycle picks up the new `purge` phase automatically.
|
||||
|
||||
### What this means for agent integrators
|
||||
|
||||
The MCP `delete_page` description string now says "soft-delete; recoverable via `restore_page` within 72h." Agents discovering tools via `list_tools` see the new contract. Behavior shift: an agent that calls `delete_page` followed by `get_page` with the same slug now gets `null` (the page is hidden by default) — pass `include_deleted: true` to surface it with `deleted_at` populated. If you had agent code that asserted hard-delete via this signal, the new contract is `get_page(slug)` returns null and `get_page(slug, {include_deleted: true})` returns the row. Ship-day stable.
|
||||
|
||||
### What this means for OAuth scope
|
||||
|
||||
`restore_page` is `scope: 'write'` — agents can self-correct mistakes within the recovery window without needing admin access. `purge_deleted_pages` is `scope: 'admin'` AND `localOnly: true` — operators only, never reachable over `gbrain serve --http`. The autopilot phase calls the same library function under the cycle lock.
|
||||
|
||||
## To take advantage of v0.26.5
|
||||
|
||||
`gbrain upgrade` runs the v34 schema migration (`destructive_guard_columns`) automatically. The migration adds the `pages.deleted_at` column + partial purge index, promotes archive state to real columns on `sources`, and backfills any pre-v0.26.5 JSONB shape into the new columns. Idempotent — re-runs are safe.
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
gbrain --version # should print 0.26.5
|
||||
gbrain doctor --json # schema_version should be 34
|
||||
gbrain sources --help # archive / restore / archived / purge / remove
|
||||
gbrain pages --help # purge-deleted (manual escape hatch)
|
||||
```
|
||||
|
||||
If `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. Re-run the orchestrator manually: `gbrain apply-migrations --yes`
|
||||
2. Verify `gbrain doctor --json` returns `schema_version >= 34`.
|
||||
3. Smoke-test the new posture: `gbrain sources archive <id>` then `gbrain sources archived` to confirm the row shows up. `gbrain sources restore <id>` to un-archive.
|
||||
4. If the upgrade chain fails, file an issue at https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if present.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Schema migration (v34: `destructive_guard_columns`)
|
||||
- New column `pages.deleted_at TIMESTAMPTZ NULL`. Partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and the predicate doesn't match the partial index. Don't add a regular `(deleted_at)` index without measuring.
|
||||
- New columns `sources.archived BOOLEAN NOT NULL DEFAULT false`, `sources.archived_at TIMESTAMPTZ`, `sources.archive_expires_at TIMESTAMPTZ`. Replaces the JSONB-key shape from PR #595's cherry-pick. Faster filter, no reserved-key footgun, indexable on demand.
|
||||
- Backfill: any row with the legacy `config @> '{"archived":true}'::jsonb` shape gets migrated into the new columns and the keys are stripped from JSONB. Idempotent.
|
||||
- Postgres uses `CREATE INDEX CONCURRENTLY` (no write-blocking lock). PGLite uses plain `CREATE INDEX`.
|
||||
- Forward-reference bootstrap (both engines) extended to probe for `pages.deleted_at` so the embedded schema's `pages_deleted_at_purge_idx` doesn't crash on pre-v0.26.5 brains. Test guard in `test/schema-bootstrap-coverage.test.ts`.
|
||||
|
||||
#### BrainEngine surface (`src/core/engine.ts` and both engines)
|
||||
- New methods: `softDeletePage(slug, opts?)`, `restorePage(slug, opts?)`, `purgeDeletedPages(olderThanHours)`. All idempotent-as-null/false. `purgeDeletedPages` clamps the hours arg to a non-negative integer and cascades through existing FKs.
|
||||
- `getPage(slug, opts?)` and `listPages(filters?)` extended with `includeDeleted` boolean (default false). Default behavior matches the search visibility filter — soft-deleted pages are hidden everywhere agents look, until they explicitly opt in.
|
||||
- `Page` type adds optional `deleted_at?: Date | null`. `rowToPage` populates it when the SELECT projects the column.
|
||||
|
||||
#### Operations (`src/core/operations.ts`)
|
||||
- `delete_page` rewired from `engine.deletePage` to `engine.softDeletePage`. Description updated to the v0.26.5 contract. Returns `{ status: 'soft_deleted', recoverable_until: 'now + 72h via restore_page' }` on success and `{ status: 'already_soft_deleted', deleted_at }` for idempotent re-calls.
|
||||
- `get_page` and `list_pages` params extended with `include_deleted: boolean`. Description strings updated.
|
||||
- New op `restore_page` — `scope: 'write'`, calls `engine.restorePage`. Returns `{ status: 'restored' | 'already_active' }`.
|
||||
- New op `purge_deleted_pages` — `scope: 'admin'`, `localOnly: true`. Calls `engine.purgeDeletedPages` with a `older_than_hours` param (default 72). Manual escape hatch.
|
||||
|
||||
#### Search-filter sweep (`src/core/search/sql-ranking.ts` + both engines)
|
||||
- New helper `buildVisibilityClause(pageAlias, sourceAlias)` emits `AND <p>.deleted_at IS NULL AND NOT <s>.archived`. Pure SQL string builder; column-based so the predicate compiles to index lookups, not JSONB containment.
|
||||
- Applied in `searchKeyword`, `searchKeywordChunks`, and `searchVector` for both Postgres and PGLite. Postgres `searchVector` two-stage CTE applies the filter in the inner CTE so HNSW stays usable. NOT bypassed by `detail=high` — soft-delete is a contract, not a temporal preference.
|
||||
- All three search methods now `JOIN sources s ON s.id = p.source_id` so the visibility predicate has a target.
|
||||
|
||||
#### Autopilot purge phase + manual CLI (v0.26.5)
|
||||
- New `CyclePhase` value `'purge'`. 9th phase in `ALL_PHASES`, runs after `orphans`. `runPhasePurge` calls `purgeExpiredSources(engine)` (sources past `archive_expires_at`) AND `engine.purgeDeletedPages(72)` (pages past 72h `deleted_at`). Adds two new `CycleReport.totals` fields: `purged_sources_count` and `purged_pages_count`. Schema-version stable (additive only).
|
||||
- New CLI command `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]`. Mirrors `gbrain sources purge` (no id). Operator escape hatch alongside the autopilot phase.
|
||||
|
||||
#### Refactor `src/core/destructive-guard.ts`
|
||||
- `softDeleteSource`, `restoreSource`, `listArchivedSources`, `purgeExpiredSources` now read/write the new column shape (atomic UPDATE...RETURNING). The `federated:false` JSONB key still flips on archive (federation has its own toggle path).
|
||||
- `purgeExpiredSources` is now a single set-based DELETE...RETURNING instead of N+1 iteration.
|
||||
- `assessDestructiveImpact`, `checkDestructiveConfirmation`, `formatImpact`, `formatSoftDelete` unchanged (don't read `config`).
|
||||
|
||||
#### Tests (~30 cases planned for the v0.26.5 ship; see `test/destructive-guard.test.ts` and the new E2E suites)
|
||||
- `test/schema-bootstrap-coverage.test.ts` — `pages.deleted_at` added to `REQUIRED_BOOTSTRAP_COVERAGE` and to the drop-and-rebuild fixture. Coverage contract test fails loud if the bootstrap drifts behind PGLITE_SCHEMA_SQL.
|
||||
- The plan calls for full unit + E2E suites for the new module; ship-day E2E coverage is the contract gate at `test/e2e/sources-archive.test.ts`, `test/e2e/pages-soft-delete.test.ts` (Q3 IRON-rule regression), `test/e2e/search-visibility.test.ts`, and `test/e2e/cycle-purge-phase.test.ts`.
|
||||
|
||||
#### Mechanics
|
||||
- `VERSION` → `0.26.5`. `package.json` → `0.26.5`. `src/schema.sql` regenerated into `src/core/schema-embedded.ts` via `bun run build:schema`. `src/core/migrate.ts` adds migration v34. `src/core/pglite-schema.ts` mirrors the new columns + partial index.
|
||||
|
||||
## [0.26.4] - 2026-05-03
|
||||
|
||||
## **`bun run test` finishes in 85 seconds. Was 18 minutes.**
|
||||
|
||||
@@ -209,6 +209,8 @@ strict behavior when unset.
|
||||
- `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law)
|
||||
- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates)
|
||||
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
|
||||
- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system.
|
||||
- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
|
||||
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
|
||||
|
||||
### BrainBench — in a sibling repo (v0.20+)
|
||||
@@ -244,6 +246,19 @@ Key commands added for Minions (job queue):
|
||||
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
Key commands added in v0.26.5 (destructive-guard, end-to-end):
|
||||
- `gbrain sources archive <id>` — soft-delete a source. Hides from search via the new `sources.archived` column + cascading visibility filter. Preserves data for 72h. (PR #595 cherry-pick.)
|
||||
- `gbrain sources restore <id> [--no-federate]` — un-archive a soft-deleted source. Re-federates by default.
|
||||
- `gbrain sources archived [--json]` — list soft-deleted sources with their TTL.
|
||||
- `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete; with no id, purges all sources whose TTL expired.
|
||||
- `gbrain sources remove <id> [--confirm-destructive] [--dry-run]` — `--yes` alone no longer enough on populated sources. Boxed impact preview before destruction.
|
||||
- `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` — operator escape hatch for page-level soft-delete cleanup. Mirror of `gbrain sources purge`. The autopilot cycle's new `purge` phase calls the same library function automatically every run.
|
||||
- MCP `delete_page` op semantically shifts from hard-delete to soft-delete. New ops: `restore_page` (`scope: write`), `purge_deleted_pages` (`scope: admin`, `localOnly: true`).
|
||||
- `get_page` and `list_pages` extended with `include_deleted: boolean` (default false).
|
||||
- New autopilot cycle phase `purge` (9th, runs after `orphans`). `gbrain dream --phase purge` runs only the purge sweep.
|
||||
- Index strategy note: the partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and Postgres won't use the partial index for the negative predicate. Don't add a regular `(deleted_at)` index without measuring.
|
||||
- Schema migration v34 (`destructive_guard_columns`) adds `pages.deleted_at` + the partial purge index; promotes `archived` from `sources.config` JSONB to real columns; backfills any pre-v0.26.5 JSONB shape.
|
||||
|
||||
Key commands added in v0.25.0:
|
||||
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
|
||||
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
|
||||
|
||||
@@ -1,5 +1,43 @@
|
||||
# TODOS
|
||||
|
||||
## destructive-guard (v0.26.5 follow-up)
|
||||
|
||||
### Adjacent 2 — Storage objects orphan on hard purge
|
||||
**Priority:** P2
|
||||
|
||||
**What:** When `purgeExpiredSources` (sources cascade) or `purgeDeletedPages` (page-level) deletes rows, the underlying object-storage payloads referenced by `files.storage_uri` (S3 / Supabase Storage) are NOT torn down. The cascade FK on `files.source_id` removes the DB row that points at the object; the object itself stays.
|
||||
|
||||
**Why:** Bound today by most brains carrying `Files: 0` (operator preview boxes confirm this in the wild). The leak compounds the moment attachments / images / audio start landing — every soft-delete + 72h TTL purge silently abandons object-storage bytes.
|
||||
|
||||
**Pros:** Closes a real data-leak path. Operators stop paying for orphaned bytes. Aligns sources/pages purge with the file lifecycle.
|
||||
**Cons:** Storage backend code is non-trivial (S3 vs Supabase vs local-fs paths each have different cleanup APIs). Single-flight delete + retries on 5xx; needs an audit log.
|
||||
**Context:** Plan calls this out explicitly in v0.26.5 CEO review (`~/.claude/plans/take-a-look-and-gentle-pine.md` Adjacent 2). Targets: `src/core/storage.ts` for the object-storage interface, `src/core/destructive-guard.ts` `purgeExpiredSources` for the call site, plus a new sweep in the cycle's purge phase. v0.26.6 candidate.
|
||||
**Depends on:** Schema is fine (already has `files.storage_uri`). Just needs the storage delete plumbing.
|
||||
|
||||
### Adjacent 3 — sources remove + sources purge race against gbrain sync
|
||||
**Priority:** P3
|
||||
|
||||
**What:** `gbrain sources remove <id>` and the new `gbrain sources purge <id>` paths don't acquire `SYNC_LOCK_ID` (the `gbrain-sync` writer lock from PR #490). If `gbrain sync` is mid-import for the same source, the parent row can DELETE while sync is INSERTing children, surfacing as a loud FK violation.
|
||||
|
||||
**Why:** Failure mode is loud (FK violation, not data corruption), and the race window is narrow. Worth closing while the destructive surface is touched, not before.
|
||||
|
||||
**Pros:** Single line at the top of `runRemove` and `runPurge`. Reuses `tryAcquireDbLock(engine, SYNC_LOCK_ID, 5)`. No design surface.
|
||||
**Cons:** Adds an extra "couldn't acquire lock" exit path the operator has to recognize and retry.
|
||||
**Context:** Plan calls this out in CEO review Adjacent 3. Targets: `src/commands/sources.ts` `runRemove` and `runPurge`. v0.26.6 candidate. Pattern: `try { await fn() } finally { await release() }` mirrors the cycle.ts use of the same primitive.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### Auth revoke-client gets the destructive-guard pattern
|
||||
**Priority:** P3
|
||||
|
||||
**What:** `gbrain auth revoke-client <client_id>` (v0.26.2) lands without an impact preview or `--confirm-destructive` gate. CASCADE-purges every active token + auth code in one transaction; one stray client_id wipes a production integration.
|
||||
|
||||
**Why:** Lower urgency than sources/pages because operators run this explicitly with a known client_id, not reflexively. But if the v0.26.5 posture is "every destructive surface gets the same gate," this surface should adopt it.
|
||||
|
||||
**Pros:** Posture consistency — every destructive verb in the gbrain CLI follows one pattern. Operators get the impact preview before nuking a production OAuth client.
|
||||
**Cons:** Marginal — single-row delete with cascade. The CASCADE is the blast radius, not the verb itself.
|
||||
**Context:** Plan flags this in CEO review. Targets: `src/commands/auth.ts` `runRevokeClient` (current shape: atomic DELETE...RETURNING with CASCADE on `oauth_tokens` + `oauth_codes`). Add an impact preview that counts `oauth_tokens` and `oauth_codes` for the client, then gate behind `--confirm-destructive`.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
## test infra (v0.26.4 follow-up — intra-file parallelism)
|
||||
|
||||
### Sweep cross-file shared-state contention; enable `bun test --concurrent` for another 2-3x speedup
|
||||
|
||||
@@ -306,6 +306,8 @@ strict behavior when unset.
|
||||
- `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law)
|
||||
- `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates)
|
||||
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
|
||||
- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system.
|
||||
- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
|
||||
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
|
||||
|
||||
### BrainBench — in a sibling repo (v0.20+)
|
||||
@@ -341,6 +343,19 @@ Key commands added for Minions (job queue):
|
||||
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
Key commands added in v0.26.5 (destructive-guard, end-to-end):
|
||||
- `gbrain sources archive <id>` — soft-delete a source. Hides from search via the new `sources.archived` column + cascading visibility filter. Preserves data for 72h. (PR #595 cherry-pick.)
|
||||
- `gbrain sources restore <id> [--no-federate]` — un-archive a soft-deleted source. Re-federates by default.
|
||||
- `gbrain sources archived [--json]` — list soft-deleted sources with their TTL.
|
||||
- `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete; with no id, purges all sources whose TTL expired.
|
||||
- `gbrain sources remove <id> [--confirm-destructive] [--dry-run]` — `--yes` alone no longer enough on populated sources. Boxed impact preview before destruction.
|
||||
- `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` — operator escape hatch for page-level soft-delete cleanup. Mirror of `gbrain sources purge`. The autopilot cycle's new `purge` phase calls the same library function automatically every run.
|
||||
- MCP `delete_page` op semantically shifts from hard-delete to soft-delete. New ops: `restore_page` (`scope: write`), `purge_deleted_pages` (`scope: admin`, `localOnly: true`).
|
||||
- `get_page` and `list_pages` extended with `include_deleted: boolean` (default false).
|
||||
- New autopilot cycle phase `purge` (9th, runs after `orphans`). `gbrain dream --phase purge` runs only the purge sweep.
|
||||
- Index strategy note: the partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and Postgres won't use the partial index for the negative predicate. Don't add a regular `(deleted_at)` index without measuring.
|
||||
- Schema migration v34 (`destructive_guard_columns`) adds `pages.deleted_at` + the partial purge index; promotes `archived` from `sources.config` JSONB to real columns; backfills any pre-v0.26.5 JSONB shape.
|
||||
|
||||
Key commands added in v0.25.0:
|
||||
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
|
||||
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.26.4",
|
||||
"version": "0.26.5",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -555,6 +555,12 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runSources(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'pages': {
|
||||
// v0.26.5: page-level operator commands (purge-deleted escape hatch).
|
||||
const { runPages } = await import('./commands/pages.ts');
|
||||
await runPages(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'storage': {
|
||||
const { runStorage } = await import('./commands/storage.ts');
|
||||
await runStorage(engine, args);
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
/**
|
||||
* gbrain pages — page-level operator commands. v0.26.5+.
|
||||
*
|
||||
* The first subcommand: `pages purge-deleted [--older-than HOURS] [--dry-run]`.
|
||||
* Manual escape hatch alongside the autopilot purge phase. Hard-deletes pages
|
||||
* whose `deleted_at` is older than the cutoff; cascades to content_chunks,
|
||||
* page_links, chunk_relations via existing FKs.
|
||||
*/
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
|
||||
const SOFT_DELETE_TTL_HOURS_DEFAULT = 72;
|
||||
|
||||
function parseOlderThanHours(args: string[]): number {
|
||||
const idx = args.indexOf('--older-than');
|
||||
if (idx === -1 || idx === args.length - 1) return SOFT_DELETE_TTL_HOURS_DEFAULT;
|
||||
const raw = args[idx + 1];
|
||||
// Accept bare numbers (hours) or `<N>h` / `<N>d`. Reject anything ambiguous.
|
||||
const trimmed = raw.trim();
|
||||
const dayMatch = trimmed.match(/^(\d+)d$/);
|
||||
if (dayMatch) return Math.max(0, parseInt(dayMatch[1], 10) * 24);
|
||||
const hourMatch = trimmed.match(/^(\d+)h?$/);
|
||||
if (hourMatch) return Math.max(0, parseInt(hourMatch[1], 10));
|
||||
console.error(`Invalid --older-than value: "${raw}". Expected hours (e.g. 72 or 72h) or days (e.g. 3d).`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
async function runPurgeDeleted(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const olderThanHours = parseOlderThanHours(args);
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const json = args.includes('--json');
|
||||
|
||||
if (dryRun) {
|
||||
// Use listPages with includeDeleted to enumerate the recoverable set, then
|
||||
// count how many would be purged given the cutoff. Stays read-only.
|
||||
const candidates = await engine.listPages({ includeDeleted: true, limit: 10000 });
|
||||
const cutoff = Date.now() - olderThanHours * 60 * 60 * 1000;
|
||||
const wouldPurge = candidates.filter(
|
||||
(p) => p.deleted_at && p.deleted_at instanceof Date && p.deleted_at.getTime() < cutoff,
|
||||
);
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ dry_run: true, older_than_hours: olderThanHours, count: wouldPurge.length, slugs: wouldPurge.map((p) => p.slug) }, null, 2));
|
||||
return;
|
||||
}
|
||||
console.log(`(dry-run) Would purge ${wouldPurge.length} page(s) soft-deleted more than ${olderThanHours}h ago.`);
|
||||
for (const p of wouldPurge) console.log(` ${p.slug} deleted_at=${p.deleted_at?.toISOString()}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const result = await engine.purgeDeletedPages(olderThanHours);
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ older_than_hours: olderThanHours, count: result.count, slugs: result.slugs }, null, 2));
|
||||
return;
|
||||
}
|
||||
if (result.count === 0) {
|
||||
console.log(`No pages to purge (older than ${olderThanHours}h).`);
|
||||
} else {
|
||||
console.log(`Purged ${result.count} page(s) (older than ${olderThanHours}h):`);
|
||||
for (const slug of result.slugs) console.log(` ${slug}`);
|
||||
}
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.log(`gbrain pages — page-level operator commands (v0.26.5)
|
||||
|
||||
Subcommands:
|
||||
purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]
|
||||
Hard-delete soft-deleted pages older than the cutoff
|
||||
(default 72h). Cascades to chunks/links/edges.
|
||||
Mirror of the autopilot purge phase.
|
||||
|
||||
Notes:
|
||||
Soft-delete a page via the MCP \`delete_page\` op. Restore via \`restore_page\`.
|
||||
This command is the manual operator escape hatch — the autopilot cycle's
|
||||
purge phase already calls the same library function on every run.
|
||||
`);
|
||||
}
|
||||
|
||||
export async function runPages(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
const rest = args.slice(1);
|
||||
|
||||
switch (sub) {
|
||||
case 'purge-deleted': return runPurgeDeleted(engine, rest);
|
||||
case undefined:
|
||||
case '--help':
|
||||
case '-h':
|
||||
printHelp();
|
||||
return;
|
||||
default:
|
||||
console.error(`Unknown subcommand: ${sub}`);
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
+164
-16
@@ -26,6 +26,17 @@
|
||||
import { writeFileSync, unlinkSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import {
|
||||
assessDestructiveImpact,
|
||||
checkDestructiveConfirmation,
|
||||
softDeleteSource,
|
||||
restoreSource,
|
||||
listArchivedSources,
|
||||
purgeExpiredSources,
|
||||
formatImpact,
|
||||
formatSoftDelete,
|
||||
SOFT_DELETE_TTL_HOURS,
|
||||
} from '../core/destructive-guard.ts';
|
||||
|
||||
// ── Validation ──────────────────────────────────────────────
|
||||
|
||||
@@ -188,10 +199,10 @@ async function runList(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
console.log('SOURCES');
|
||||
console.log('───────');
|
||||
for (const e of entries) {
|
||||
const fedMark = e.federated ? 'federated' : 'isolated';
|
||||
const fedMark = e.federated ? 'federated' : (e as any).archived ? '⚠ archived' : 'isolated';
|
||||
const pathStr = e.local_path ?? '(no local path)';
|
||||
const sync = e.last_sync_at ? `last sync ${e.last_sync_at}` : 'never synced';
|
||||
console.log(` ${e.id.padEnd(20)} ${fedMark.padEnd(10)} ${String(e.page_count).padStart(6)} pages ${sync}`);
|
||||
console.log(` ${e.id.padEnd(20)} ${fedMark.padEnd(12)} ${String(e.page_count).padStart(6)} pages ${sync}`);
|
||||
if (e.local_path) console.log(` ${' '.repeat(22)}${pathStr}`);
|
||||
}
|
||||
if (entries.length === 0) console.log(' (no sources registered)');
|
||||
@@ -202,13 +213,12 @@ async function runList(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
async function runRemove(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]');
|
||||
console.error('Usage: gbrain sources remove <id> [--yes] [--confirm-destructive] [--dry-run] [--keep-storage]');
|
||||
process.exit(2);
|
||||
}
|
||||
const yes = args.includes('--yes');
|
||||
const dryRun = args.includes('--dry-run');
|
||||
// NOTE: --keep-storage is accepted for forward compatibility but has no
|
||||
// effect until Step 7 wires in explicit storage object deletion.
|
||||
const confirmDestructive = args.includes('--confirm-destructive');
|
||||
const _keepStorage = args.includes('--keep-storage');
|
||||
void _keepStorage;
|
||||
|
||||
@@ -223,23 +233,143 @@ async function runRemove(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
process.exit(4);
|
||||
}
|
||||
|
||||
const pageCount = await countPages(engine, id);
|
||||
console.log(`Source "${id}" → ${pageCount} pages will be deleted (cascade).`);
|
||||
// v0.26.5: Impact preview + destructive guard
|
||||
const impact = await assessDestructiveImpact(engine, id);
|
||||
if (impact) {
|
||||
console.log(formatImpact(impact));
|
||||
|
||||
if (dryRun) {
|
||||
console.log(`(dry-run; no side effects)`);
|
||||
return;
|
||||
}
|
||||
if (dryRun) {
|
||||
console.log('(dry-run; no side effects)');
|
||||
return;
|
||||
}
|
||||
|
||||
if (!yes) {
|
||||
console.error(`Refusing to remove without --yes. Pass --yes to confirm.`);
|
||||
process.exit(5);
|
||||
const blockMsg = checkDestructiveConfirmation(impact, { yes, confirmDestructive, dryRun });
|
||||
if (blockMsg) {
|
||||
console.error(blockMsg);
|
||||
process.exit(5);
|
||||
}
|
||||
} else {
|
||||
if (dryRun) { console.log('(dry-run; source not found)'); return; }
|
||||
if (!yes && !confirmDestructive) {
|
||||
console.error('Refusing to remove without --yes or --confirm-destructive.');
|
||||
process.exit(5);
|
||||
}
|
||||
}
|
||||
|
||||
await engine.executeRaw(`DELETE FROM sources WHERE id = $1`, [id]);
|
||||
const pageCount = impact?.pageCount ?? 0;
|
||||
console.log(`Removed source "${id}" (${pageCount} pages + dependent rows cascaded).`);
|
||||
}
|
||||
|
||||
// ── Subcommand: archive (soft-delete) ───────────────────────
|
||||
|
||||
async function runArchive(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources archive <id>');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
if (id === 'default') {
|
||||
console.error('Error: cannot archive the "default" source.');
|
||||
process.exit(3);
|
||||
}
|
||||
|
||||
// Show impact preview
|
||||
const impact = await assessDestructiveImpact(engine, id);
|
||||
if (!impact) {
|
||||
console.error(`Source "${id}" not found.`);
|
||||
process.exit(4);
|
||||
}
|
||||
|
||||
const result = await softDeleteSource(engine, id);
|
||||
if (!result) {
|
||||
console.error(`Failed to archive source "${id}".`);
|
||||
process.exit(4);
|
||||
}
|
||||
|
||||
console.log(formatSoftDelete(result));
|
||||
}
|
||||
|
||||
// ── Subcommand: restore ─────────────────────────────────────
|
||||
|
||||
async function runRestore(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
const noFederate = args.includes('--no-federate');
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources restore <id> [--no-federate]');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const restored = await restoreSource(engine, id, !noFederate);
|
||||
if (!restored) {
|
||||
console.error(`Source "${id}" not found or not archived.`);
|
||||
process.exit(4);
|
||||
}
|
||||
|
||||
console.log(`Source "${id}" restored. ${noFederate ? 'Not re-federated.' : 'Re-federated.'}`);
|
||||
console.log(`All pages, chunks, and embeddings are intact.`);
|
||||
}
|
||||
|
||||
// ── Subcommand: purge ───────────────────────────────────────
|
||||
|
||||
async function runPurge(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
const confirmDestructive = args.includes('--confirm-destructive');
|
||||
|
||||
if (id) {
|
||||
// Purge a specific source (must be archived)
|
||||
const impact = await assessDestructiveImpact(engine, id);
|
||||
if (!impact) {
|
||||
console.error(`Source "${id}" not found.`);
|
||||
process.exit(4);
|
||||
}
|
||||
|
||||
console.log(formatImpact(impact));
|
||||
|
||||
if (!confirmDestructive) {
|
||||
console.error(`Pass --confirm-destructive to permanently delete source "${id}".`);
|
||||
process.exit(5);
|
||||
}
|
||||
|
||||
await engine.executeRaw(`DELETE FROM sources WHERE id = $1`, [id]);
|
||||
console.log(`Permanently deleted source "${id}" (${impact.pageCount} pages cascaded).`);
|
||||
return;
|
||||
}
|
||||
|
||||
// No id: purge all expired archives
|
||||
const purged = await purgeExpiredSources(engine);
|
||||
if (purged.length === 0) {
|
||||
console.log('No expired archives to purge.');
|
||||
} else {
|
||||
console.log(`Purged ${purged.length} expired archive(s): ${purged.join(', ')}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Subcommand: archived ────────────────────────────────────
|
||||
|
||||
async function runListArchived(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const archived = await listArchivedSources(engine);
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ archived }, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (archived.length === 0) {
|
||||
console.log('No archived sources.');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log('ARCHIVED SOURCES (soft-deleted)');
|
||||
console.log('───────────────────────────────');
|
||||
for (const a of archived) {
|
||||
const hours = Math.max(0, Math.round((a.expiresAt.getTime() - Date.now()) / (1000 * 60 * 60)));
|
||||
console.log(` ${a.id.padEnd(20)} ${String(a.pageCount).padStart(6)} pages expires in ${hours}h (restore: gbrain sources restore ${a.id})`);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Subcommand: rename ──────────────────────────────────────
|
||||
|
||||
async function runRename(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
@@ -340,6 +470,10 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise<v
|
||||
case 'detach': runDetach(); return;
|
||||
case 'federate': return runFederate(engine, rest, true);
|
||||
case 'unfederate': return runFederate(engine, rest, false);
|
||||
case 'archive': return runArchive(engine, rest);
|
||||
case 'restore': return runRestore(engine, rest);
|
||||
case 'purge': return runPurge(engine, rest);
|
||||
case 'archived': return runListArchived(engine, rest);
|
||||
case undefined:
|
||||
case '--help':
|
||||
case '-h':
|
||||
@@ -353,13 +487,23 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise<v
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.log(`gbrain sources — manage multi-source brain configuration (v0.18.0)
|
||||
console.log(`gbrain sources — manage multi-source brain configuration (v0.26.5)
|
||||
|
||||
Subcommands:
|
||||
add <id> --path <p> [--name <n>] [--federated|--no-federated]
|
||||
Register a new source.
|
||||
list [--json] List registered sources with page counts.
|
||||
remove <id> [--yes] [--dry-run] Cascade-delete a source and its pages.
|
||||
remove <id> [--confirm-destructive] [--dry-run]
|
||||
Permanently delete a source and all its data.
|
||||
Shows impact preview. Requires --confirm-destructive
|
||||
when the source has data (pages/chunks/embeddings).
|
||||
archive <id> Soft-delete: hide from search, preserve data for ${SOFT_DELETE_TTL_HOURS}h.
|
||||
restore <id> [--no-federate] Un-archive a soft-deleted source.
|
||||
archived [--json] List soft-deleted sources and their expiry.
|
||||
purge [<id>] [--confirm-destructive]
|
||||
Permanently delete archived sources.
|
||||
Without <id>: purge all expired archives.
|
||||
With <id>: force-purge (requires --confirm-destructive).
|
||||
rename <id> <new-name> Rename display name (id is immutable).
|
||||
default <id> Set the brain-level default source.
|
||||
attach <id> Write .gbrain-source in CWD (like kubectl context).
|
||||
@@ -368,5 +512,9 @@ Subcommands:
|
||||
unfederate <id> Isolate source from default search.
|
||||
|
||||
Source id: [a-z0-9-]{1,32}. Immutable citation key.
|
||||
|
||||
Destructive operations (remove, purge) show an impact preview before acting.
|
||||
Pass --dry-run to preview without side effects.
|
||||
Use 'archive' instead of 'remove' for a safe ${SOFT_DELETE_TTL_HOURS}h grace period.
|
||||
`);
|
||||
}
|
||||
|
||||
+96
-2
@@ -52,7 +52,7 @@ import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
|
||||
|
||||
// ─── Types ─────────────────────────────────────────────────────────
|
||||
|
||||
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans';
|
||||
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans' | 'purge';
|
||||
|
||||
export const ALL_PHASES: CyclePhase[] = [
|
||||
'lint',
|
||||
@@ -63,13 +63,18 @@ export const ALL_PHASES: CyclePhase[] = [
|
||||
'patterns',
|
||||
'embed',
|
||||
'orphans',
|
||||
// v0.26.5: hard-deletes soft-deleted pages and expired archived sources past
|
||||
// the 72h recovery window. Runs last so the rest of the cycle sees the
|
||||
// recoverable set; the purge then drops what's expired.
|
||||
'purge',
|
||||
];
|
||||
|
||||
/**
|
||||
* Phases that mutate state (filesystem or DB) and therefore should
|
||||
* coordinate via the cycle lock. Only orphans is truly read-only
|
||||
* and skips the lock. patterns mutates DB (writes pattern pages) so
|
||||
* it acquires the lock; synthesize too.
|
||||
* it acquires the lock; synthesize too. v0.26.5 adds purge (DELETE-cascade
|
||||
* across pages and sources).
|
||||
*/
|
||||
const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
|
||||
'lint',
|
||||
@@ -79,6 +84,7 @@ const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
|
||||
'extract',
|
||||
'patterns',
|
||||
'embed',
|
||||
'purge',
|
||||
]);
|
||||
|
||||
export type PhaseStatus = 'ok' | 'warn' | 'fail' | 'skipped';
|
||||
@@ -138,6 +144,10 @@ export interface CycleReport {
|
||||
synth_pages_written: number;
|
||||
/** v0.23: number of pattern pages written/updated by patterns phase. */
|
||||
patterns_written: number;
|
||||
/** v0.26.5: number of source rows hard-deleted by the purge phase. */
|
||||
purged_sources_count: number;
|
||||
/** v0.26.5: number of page rows hard-deleted by the purge phase. */
|
||||
purged_pages_count: number;
|
||||
};
|
||||
}
|
||||
|
||||
@@ -657,6 +667,61 @@ async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean): Promise<Phas
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.26.5 — purge phase. Hard-deletes:
|
||||
* - source rows where `archived = true AND archive_expires_at <= now()`
|
||||
* (paired with the cascade FK to `pages`, this also drops the source's pages)
|
||||
* - page rows where `deleted_at` is older than 72h
|
||||
*
|
||||
* Cascade on `pages` covers `content_chunks`, `page_links`, `chunk_relations`.
|
||||
* `dryRun` short-circuits — no DELETEs are issued.
|
||||
*
|
||||
* Mirrors the operator escape hatches: `gbrain sources purge` (no id) and
|
||||
* `gbrain pages purge-deleted` both call the same library functions, so
|
||||
* scripted purges and the autopilot phase converge on a single behavior.
|
||||
*/
|
||||
async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<PhaseResult> {
|
||||
try {
|
||||
if (dryRun) {
|
||||
return {
|
||||
phase: 'purge',
|
||||
status: 'ok',
|
||||
duration_ms: 0,
|
||||
summary: 'dry-run: skipped purge sweep',
|
||||
details: { dry_run: true, purged_sources_count: 0, purged_pages_count: 0 },
|
||||
};
|
||||
}
|
||||
const { purgeExpiredSources } = await import('./destructive-guard.ts');
|
||||
const purgedSources = await purgeExpiredSources(engine);
|
||||
const purgedPages = await engine.purgeDeletedPages(SOFT_DELETE_TTL_HOURS_FOR_PURGE);
|
||||
return {
|
||||
phase: 'purge',
|
||||
status: 'ok',
|
||||
duration_ms: 0,
|
||||
summary: `purged ${purgedSources.length} source(s) and ${purgedPages.count} page(s) past the 72h recovery window`,
|
||||
details: {
|
||||
purged_sources_count: purgedSources.length,
|
||||
purged_pages_count: purgedPages.count,
|
||||
purged_sources: purgedSources,
|
||||
purged_page_slugs: purgedPages.slugs,
|
||||
},
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
phase: 'purge',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'purge phase failed',
|
||||
details: {},
|
||||
error: makeErrorFromException(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/** v0.26.5: matches SOFT_DELETE_TTL_HOURS in destructive-guard.ts. Inlined here
|
||||
* to avoid a static import (purge phase is only loaded in the autopilot path). */
|
||||
const SOFT_DELETE_TTL_HOURS_FOR_PURGE = 72;
|
||||
|
||||
async function runPhaseOrphans(engine: BrainEngine): Promise<PhaseResult> {
|
||||
try {
|
||||
const { findOrphans } = await import('../commands/orphans.ts');
|
||||
@@ -931,6 +996,30 @@ export async function runCycle(
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 9: purge (v0.26.5) ────────────────────────────────
|
||||
// Hard-delete soft-deleted pages and expired archived sources past the
|
||||
// 72h recovery window. Runs last so the rest of the cycle sees the
|
||||
// recoverable set; the purge then drops what's truly expired.
|
||||
if (phases.includes('purge')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'purge',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.purge');
|
||||
const { result, duration_ms } = await timePhase(() => runPhasePurge(engine, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
} finally {
|
||||
if (lock) {
|
||||
try { await lock.release(); } catch { /* best-effort */ }
|
||||
@@ -965,6 +1054,8 @@ function emptyTotals(): CycleReport['totals'] {
|
||||
transcripts_processed: 0,
|
||||
synth_pages_written: 0,
|
||||
patterns_written: 0,
|
||||
purged_sources_count: 0,
|
||||
purged_pages_count: 0,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -992,6 +1083,9 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
|
||||
t.synth_pages_written = Number(p.details.pages_written ?? 0);
|
||||
} else if (p.phase === 'patterns' && p.details) {
|
||||
t.patterns_written = Number(p.details.patterns_written ?? 0);
|
||||
} else if (p.phase === 'purge' && p.details) {
|
||||
t.purged_sources_count = Number(p.details.purged_sources_count ?? 0);
|
||||
t.purged_pages_count = Number(p.details.purged_pages_count ?? 0);
|
||||
}
|
||||
}
|
||||
return t;
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
/**
|
||||
* Destructive operation guard — v0.26.5
|
||||
*
|
||||
* Protects against accidental data loss in gbrain by requiring explicit
|
||||
* confirmation for operations that cascade-delete pages, chunks, or embeddings.
|
||||
*
|
||||
* Three layers:
|
||||
* 1. Impact preview — always shown before destructive actions
|
||||
* 2. Confirmation gate — requires --confirm-destructive or interactive "type source name"
|
||||
* 3. Soft-delete with TTL — sources are tombstoned for 72h before permanent deletion
|
||||
*
|
||||
* Design principle: the blast radius should be visible BEFORE you pull the trigger,
|
||||
* and recoverable AFTER you pull it (within a grace period).
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
// ── Types ───────────────────────────────────────────────────
|
||||
|
||||
export interface DestructiveImpact {
|
||||
sourceId: string;
|
||||
sourceName: string;
|
||||
pageCount: number;
|
||||
chunkCount: number;
|
||||
embeddingCount: number;
|
||||
fileCount: number;
|
||||
/** Human-readable summary line */
|
||||
summary: string;
|
||||
}
|
||||
|
||||
export interface SoftDeletedSource {
|
||||
id: string;
|
||||
name: string;
|
||||
deletedAt: Date;
|
||||
expiresAt: Date;
|
||||
pageCount: number;
|
||||
}
|
||||
|
||||
// ── Constants ───────────────────────────────────────────────
|
||||
|
||||
/** Hours before a soft-deleted source is permanently purged. */
|
||||
export const SOFT_DELETE_TTL_HOURS = 72;
|
||||
|
||||
/** Threshold: operations affecting this many pages or more require confirmation. */
|
||||
export const CONFIRM_THRESHOLD_PAGES = 1;
|
||||
|
||||
// ── Impact Assessment ───────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Compute the blast radius of deleting a source.
|
||||
*/
|
||||
export async function assessDestructiveImpact(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
): Promise<DestructiveImpact | null> {
|
||||
// Fetch source metadata
|
||||
const sources = await engine.executeRaw<{ id: string; name: string }>(
|
||||
`SELECT id, name FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
if (sources.length === 0) return null;
|
||||
|
||||
const src = sources[0];
|
||||
|
||||
// Count pages
|
||||
const pageRows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
const pageCount = pageRows[0]?.n ?? 0;
|
||||
|
||||
// Count chunks
|
||||
const chunkRows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM content_chunks cc
|
||||
JOIN pages p ON cc.page_id = p.id
|
||||
WHERE p.source_id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
const chunkCount = chunkRows[0]?.n ?? 0;
|
||||
|
||||
// Count embeddings (chunks with non-null embedding vectors)
|
||||
const embedRows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM content_chunks cc
|
||||
JOIN pages p ON cc.page_id = p.id
|
||||
WHERE p.source_id = $1 AND cc.embedding IS NOT NULL`,
|
||||
[sourceId],
|
||||
);
|
||||
const embeddingCount = embedRows[0]?.n ?? 0;
|
||||
|
||||
// Count files in storage (if any). PGLite has no `files` table — that
|
||||
// surface is Postgres-only (CLAUDE.md: "No files table" for PGLite). Probe
|
||||
// the table existence via information_schema so this works on both engines.
|
||||
let fileCount = 0;
|
||||
const filesTableRows = await engine.executeRaw<{ exists: boolean }>(
|
||||
`SELECT EXISTS (
|
||||
SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = 'public' AND table_name = 'files'
|
||||
) AS exists`,
|
||||
);
|
||||
if (filesTableRows[0]?.exists) {
|
||||
const fileRows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM files WHERE source_id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
fileCount = fileRows[0]?.n ?? 0;
|
||||
}
|
||||
|
||||
const parts: string[] = [];
|
||||
if (pageCount > 0) parts.push(`${pageCount.toLocaleString()} pages`);
|
||||
if (chunkCount > 0) parts.push(`${chunkCount.toLocaleString()} chunks`);
|
||||
if (embeddingCount > 0) parts.push(`${embeddingCount.toLocaleString()} embeddings`);
|
||||
if (fileCount > 0) parts.push(`${fileCount.toLocaleString()} files`);
|
||||
|
||||
const summary = parts.length > 0
|
||||
? `⚠️ This will permanently delete: ${parts.join(', ')}`
|
||||
: `Source "${sourceId}" has no data (safe to remove).`;
|
||||
|
||||
return {
|
||||
sourceId,
|
||||
sourceName: src.name,
|
||||
pageCount,
|
||||
chunkCount,
|
||||
embeddingCount,
|
||||
fileCount,
|
||||
summary,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Confirmation Gate ───────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Check whether the caller has provided sufficient confirmation for a
|
||||
* destructive operation. Returns an error message if blocked, or null if OK.
|
||||
*/
|
||||
export function checkDestructiveConfirmation(
|
||||
impact: DestructiveImpact,
|
||||
opts: {
|
||||
yes?: boolean;
|
||||
confirmDestructive?: boolean;
|
||||
dryRun?: boolean;
|
||||
},
|
||||
): string | null {
|
||||
// Dry run always passes (no side effects)
|
||||
if (opts.dryRun) return null;
|
||||
|
||||
// No data = no risk
|
||||
if (impact.pageCount === 0 && impact.chunkCount === 0 && impact.fileCount === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// --confirm-destructive is the explicit "I know what I'm doing" flag
|
||||
if (opts.confirmDestructive) return null;
|
||||
|
||||
// --yes alone is NOT sufficient for destructive operations with data.
|
||||
// This is the key behavior change: --yes used to be enough, now you
|
||||
// need --confirm-destructive when there's actual data at stake.
|
||||
if (opts.yes && impact.pageCount === 0) return null;
|
||||
|
||||
return (
|
||||
`\n${impact.summary}\n\n` +
|
||||
`To proceed, pass --confirm-destructive (or use soft-delete: gbrain sources archive ${impact.sourceId}).\n` +
|
||||
`To preview without side effects: --dry-run`
|
||||
);
|
||||
}
|
||||
|
||||
// ── Soft Delete ─────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Soft-delete a source: mark `archived = true` with a 72h TTL. Pages remain
|
||||
* in DB; the source is hidden from search via `buildVisibilityClause` and
|
||||
* federation is disabled via the existing `config.federated` JSONB key. After
|
||||
* TTL expires, the autopilot purge phase or manual `gbrain sources purge`
|
||||
* permanently removes the row (cascade delete to pages + chunks).
|
||||
*
|
||||
* v0.26.5: archive state moved from `config` JSONB keys to real columns
|
||||
* (`archived`, `archived_at`, `archive_expires_at`). Migration v34 backfills
|
||||
* pre-v0.26.5 rows. Faster filter, no reserved-key footgun. The `federated`
|
||||
* key stays in JSONB because federation has its own toggle path.
|
||||
*/
|
||||
export async function softDeleteSource(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
): Promise<SoftDeletedSource | null> {
|
||||
// Atomic: only flip rows that are currently active. Returns the metadata
|
||||
// we need without a follow-up SELECT. RETURNING projects the columns the
|
||||
// caller cares about; pageCount is a separate count.
|
||||
const expiresClause = `now() + (${SOFT_DELETE_TTL_HOURS} || ' hours')::interval`;
|
||||
const rows = await engine.executeRaw<{ id: string; name: string; archived_at: string; archive_expires_at: string }>(
|
||||
`UPDATE sources
|
||||
SET archived = true,
|
||||
archived_at = now(),
|
||||
archive_expires_at = ${expiresClause},
|
||||
config = COALESCE(config, '{}'::jsonb) || '{"federated": false}'::jsonb
|
||||
WHERE id = $1 AND archived = false
|
||||
RETURNING id, name, archived_at, archive_expires_at`,
|
||||
[sourceId],
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
const row = rows[0];
|
||||
|
||||
const pageRows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
const pageCount = pageRows[0]?.n ?? 0;
|
||||
|
||||
return {
|
||||
id: sourceId,
|
||||
name: row.name,
|
||||
deletedAt: new Date(row.archived_at),
|
||||
expiresAt: new Date(row.archive_expires_at),
|
||||
pageCount,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore a soft-deleted source (un-archive). Returns true iff a row was
|
||||
* restored. Idempotent-as-false on "already active" or "not found".
|
||||
*
|
||||
* v0.26.5: clears the column-based archive state and (by default) flips
|
||||
* `config.federated = true` so the source re-enters federated search. The
|
||||
* `--no-federate` operator opt-out keeps federation disabled.
|
||||
*/
|
||||
export async function restoreSource(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
refederate: boolean = true,
|
||||
): Promise<boolean> {
|
||||
const federatedPatch = refederate ? '{"federated": true}' : '{"federated": false}';
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`UPDATE sources
|
||||
SET archived = false,
|
||||
archived_at = NULL,
|
||||
archive_expires_at = NULL,
|
||||
config = COALESCE(config, '{}'::jsonb) || $1::jsonb
|
||||
WHERE id = $2 AND archived = true
|
||||
RETURNING id`,
|
||||
[federatedPatch, sourceId],
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* List all soft-deleted (archived) sources.
|
||||
*
|
||||
* v0.26.5: filters via the real `archived` column instead of JSONB
|
||||
* containment. Faster, indexable on demand, no JSONB reserved-key collision
|
||||
* with future config schemas.
|
||||
*/
|
||||
export async function listArchivedSources(
|
||||
engine: BrainEngine,
|
||||
): Promise<SoftDeletedSource[]> {
|
||||
const rows = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
archived_at: string;
|
||||
archive_expires_at: string;
|
||||
page_count: number;
|
||||
}>(
|
||||
`SELECT
|
||||
s.id, s.name, s.archived_at, s.archive_expires_at,
|
||||
COALESCE((SELECT COUNT(*)::int FROM pages p WHERE p.source_id = s.id), 0) AS page_count
|
||||
FROM sources s
|
||||
WHERE s.archived = true
|
||||
ORDER BY s.archived_at DESC`,
|
||||
);
|
||||
|
||||
return rows.map((row) => ({
|
||||
id: row.id,
|
||||
name: row.name,
|
||||
deletedAt: new Date(row.archived_at),
|
||||
expiresAt: new Date(row.archive_expires_at),
|
||||
pageCount: row.page_count,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Permanently purge sources whose 72h TTL has expired. Cascades to pages
|
||||
* (and content_chunks via existing FKs). Returns the ids of purged sources.
|
||||
*
|
||||
* v0.26.5: moved from JSONB-driven iteration to a single set-based DELETE
|
||||
* with `archived = true AND archive_expires_at <= now()`. Server-side
|
||||
* filter; one round-trip; cascade-friendly.
|
||||
*/
|
||||
export async function purgeExpiredSources(
|
||||
engine: BrainEngine,
|
||||
): Promise<string[]> {
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`DELETE FROM sources
|
||||
WHERE archived = true
|
||||
AND archive_expires_at IS NOT NULL
|
||||
AND archive_expires_at <= now()
|
||||
RETURNING id`,
|
||||
);
|
||||
return rows.map((r) => r.id);
|
||||
}
|
||||
|
||||
// ── Display Helpers ─────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Format an impact assessment for terminal display.
|
||||
*/
|
||||
export function formatImpact(impact: DestructiveImpact): string {
|
||||
const lines: string[] = [
|
||||
``,
|
||||
`╔══════════════════════════════════════════════════════════╗`,
|
||||
`║ DESTRUCTIVE OPERATION — Impact Preview ║`,
|
||||
`╠══════════════════════════════════════════════════════════╣`,
|
||||
`║ Source: ${impact.sourceName.padEnd(42)}║`,
|
||||
`║ Source ID: ${impact.sourceId.padEnd(42)}║`,
|
||||
`║ ║`,
|
||||
`║ Pages: ${String(impact.pageCount.toLocaleString()).padEnd(42)}║`,
|
||||
`║ Chunks: ${String(impact.chunkCount.toLocaleString()).padEnd(42)}║`,
|
||||
`║ Embeddings: ${String(impact.embeddingCount.toLocaleString()).padEnd(42)}║`,
|
||||
`║ Files: ${String(impact.fileCount.toLocaleString()).padEnd(42)}║`,
|
||||
`╠══════════════════════════════════════════════════════════╣`,
|
||||
`║ ${impact.summary.padEnd(56)}║`,
|
||||
`╚══════════════════════════════════════════════════════════╝`,
|
||||
``,
|
||||
];
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export function formatSoftDelete(sd: SoftDeletedSource): string {
|
||||
const hours = Math.round((sd.expiresAt.getTime() - Date.now()) / (1000 * 60 * 60));
|
||||
return [
|
||||
``,
|
||||
`Source "${sd.id}" archived (soft-deleted).`,
|
||||
` ${sd.pageCount.toLocaleString()} pages preserved for ${SOFT_DELETE_TTL_HOURS}h.`,
|
||||
` Expires: ${sd.expiresAt.toISOString()} (~${hours}h from now)`,
|
||||
` Removed from search. Data intact.`,
|
||||
``,
|
||||
` Restore: gbrain sources restore ${sd.id}`,
|
||||
` Purge now: gbrain sources purge ${sd.id} --confirm-destructive`,
|
||||
``,
|
||||
].join('\n');
|
||||
}
|
||||
+41
-2
@@ -1,5 +1,5 @@
|
||||
import type {
|
||||
Page, PageInput, PageFilters,
|
||||
Page, PageInput, PageFilters, GetPageOpts,
|
||||
Chunk, ChunkInput, StaleChunkRow,
|
||||
SearchResult, SearchOpts,
|
||||
Link, GraphNode, GraphPath,
|
||||
@@ -128,9 +128,48 @@ export interface BrainEngine {
|
||||
withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T>;
|
||||
|
||||
// Pages CRUD
|
||||
getPage(slug: string): Promise<Page | null>;
|
||||
/**
|
||||
* Fetch a page by slug.
|
||||
* v0.26.5: by default soft-deleted rows return null (matches the search
|
||||
* filter contract). Pass `opts.includeDeleted: true` to surface them with
|
||||
* `deleted_at` populated — used by `gbrain pages purge-deleted` listing,
|
||||
* by `restore_page` flow, and by operator diagnostics.
|
||||
*/
|
||||
getPage(slug: string, opts?: GetPageOpts): Promise<Page | null>;
|
||||
putPage(slug: string, page: PageInput): Promise<Page>;
|
||||
/**
|
||||
* Hard-delete a page row. Cascades to content_chunks, page_links,
|
||||
* chunk_relations via existing FK ON DELETE CASCADE.
|
||||
*
|
||||
* v0.26.5: this is no longer the public-facing `delete_page` op handler —
|
||||
* the op now soft-deletes via `softDeletePage` instead. `deletePage` stays
|
||||
* as the underlying primitive used by `purgeDeletedPages` and by callers
|
||||
* that explicitly want hard-delete semantics (e.g. test setup teardown).
|
||||
*/
|
||||
deletePage(slug: string): Promise<void>;
|
||||
/**
|
||||
* v0.26.5 — set `deleted_at = now()` on a page. Returns the slug if a row
|
||||
* was soft-deleted, null if no row matched (already soft-deleted OR not found).
|
||||
* Idempotent-as-null. The page stays in the DB and cascade rows (chunks,
|
||||
* links) stay intact; the autopilot purge phase hard-deletes after 72h.
|
||||
*/
|
||||
softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null>;
|
||||
/**
|
||||
* v0.26.5 — clear `deleted_at` on a soft-deleted page. Returns true iff a
|
||||
* row was restored. False if the slug is unknown OR the page is not
|
||||
* currently soft-deleted (idempotent-as-false).
|
||||
*/
|
||||
restorePage(slug: string, opts?: { sourceId?: string }): Promise<boolean>;
|
||||
/**
|
||||
* v0.26.5 — hard-delete pages whose `deleted_at` is older than the cutoff.
|
||||
* Called by the autopilot purge phase and by the `gbrain pages purge-deleted`
|
||||
* CLI escape hatch. Cascades through existing FKs.
|
||||
*/
|
||||
purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }>;
|
||||
/**
|
||||
* v0.26.5: by default `listPages` excludes soft-deleted rows. Set
|
||||
* `filters.includeDeleted: true` to surface them.
|
||||
*/
|
||||
listPages(filters?: PageFilters): Promise<Page[]>;
|
||||
resolveSlugs(partial: string): Promise<string[]>;
|
||||
/**
|
||||
|
||||
@@ -1307,6 +1307,93 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON mcp_request_log(agent_name, created_at DESC);
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 34,
|
||||
name: 'destructive_guard_columns',
|
||||
// v0.26.5 — soft-delete + recovery window for sources AND pages.
|
||||
// Renumbered v33→v34 on master merge: master's v33 (admin_dashboard_columns_v0_26_3)
|
||||
// landed first in PR #586. v34 follows it.
|
||||
//
|
||||
// pages.deleted_at: `delete_page` op now sets deleted_at = now() instead of
|
||||
// hard-deleting. The autopilot purge phase hard-deletes rows where
|
||||
// deleted_at < now() - 72h. Search and `get_page` filter
|
||||
// `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in.
|
||||
//
|
||||
// sources.archived/archived_at/archive_expires_at: promoted from JSONB keys
|
||||
// to real columns. v0.26.0 + the cherry-picked PR #595 wrote these inside
|
||||
// `sources.config` JSONB. Real columns are faster to filter, avoid the
|
||||
// reserved-key footgun, and let the search visibility filter compile to a
|
||||
// column lookup. The 72h TTL is preserved by reading
|
||||
// `archive_expires_at = archived_at + INTERVAL '72 hours'`.
|
||||
//
|
||||
// Backfill: any row that previously stored `{"archived":true,"archived_at":"...","archive_expires_at":"..."}`
|
||||
// in config gets migrated to the new columns, then the keys are stripped
|
||||
// from JSONB so the JSONB shape stays canonical going forward.
|
||||
//
|
||||
// Engine-aware partial index: Postgres uses CREATE INDEX CONCURRENTLY (no
|
||||
// write-blocking lock); PGLite uses plain CREATE INDEX. Mirrors v14
|
||||
// (pages_updated_at_index) handler shape.
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
// 1. Add columns. ALTER TABLE ADD COLUMN IF NOT EXISTS is idempotent on
|
||||
// both engines.
|
||||
await engine.runMigration(34, `
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN NOT NULL DEFAULT false;
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived_at TIMESTAMPTZ;
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archive_expires_at TIMESTAMPTZ;
|
||||
`);
|
||||
|
||||
// 2. Backfill from JSONB shape used by pre-v0.26.5 cherry-picks of PR #595.
|
||||
// Idempotent: subsequent re-runs find zero matching rows.
|
||||
await engine.runMigration(34, `
|
||||
UPDATE sources
|
||||
SET archived = true,
|
||||
archived_at = COALESCE((config->>'archived_at')::timestamptz, now()),
|
||||
archive_expires_at = COALESCE(
|
||||
(config->>'archive_expires_at')::timestamptz,
|
||||
COALESCE((config->>'archived_at')::timestamptz, now()) + INTERVAL '72 hours'
|
||||
)
|
||||
WHERE config ? 'archived'
|
||||
AND (config->>'archived')::boolean = true
|
||||
AND archived = false;
|
||||
`);
|
||||
await engine.runMigration(34, `
|
||||
UPDATE sources
|
||||
SET config = config - 'archived' - 'archived_at' - 'archive_expires_at'
|
||||
WHERE config ?| ARRAY['archived', 'archived_at', 'archive_expires_at'];
|
||||
`);
|
||||
|
||||
// 3. Partial index for the autopilot purge sweep. Postgres CONCURRENTLY
|
||||
// avoids the SHARE lock on `pages`; PGLite has no concurrent writers.
|
||||
if (engine.kind === 'postgres') {
|
||||
// Pre-drop any invalid index from a prior CONCURRENTLY failure (matches v14 pattern).
|
||||
await engine.runMigration(34, `
|
||||
DO $$ BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM pg_index i
|
||||
JOIN pg_class c ON c.oid = i.indexrelid
|
||||
WHERE c.relname = 'pages_deleted_at_purge_idx' AND NOT i.indisvalid
|
||||
) THEN
|
||||
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_deleted_at_purge_idx';
|
||||
END IF;
|
||||
END $$;
|
||||
`);
|
||||
await engine.runMigration(34, `
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_deleted_at_purge_idx
|
||||
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
|
||||
`);
|
||||
} else {
|
||||
await engine.runMigration(34, `
|
||||
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
|
||||
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
|
||||
`);
|
||||
}
|
||||
},
|
||||
// CONCURRENTLY on Postgres requires no surrounding transaction. PGLite ignores
|
||||
// this flag, so the index DDL runs in whatever wrapper applies.
|
||||
transaction: false,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
+73
-9
@@ -293,22 +293,24 @@ export interface Operation {
|
||||
|
||||
const get_page: Operation = {
|
||||
name: 'get_page',
|
||||
description: 'Read a page by slug (supports optional fuzzy matching)',
|
||||
description: 'Read a page by slug (supports optional fuzzy matching). Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated (see v0.26.5 recovery window).',
|
||||
params: {
|
||||
slug: { type: 'string', required: true, description: 'Page slug' },
|
||||
fuzzy: { type: 'boolean', description: 'Enable fuzzy slug resolution (default: false)' },
|
||||
include_deleted: { type: 'boolean', description: 'v0.26.5: surface soft-deleted pages with deleted_at populated (default: false). Used by restore workflows.' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const slug = p.slug as string;
|
||||
const fuzzy = (p.fuzzy as boolean) || false;
|
||||
const includeDeleted = (p.include_deleted as boolean) === true;
|
||||
|
||||
let page = await ctx.engine.getPage(slug);
|
||||
let page = await ctx.engine.getPage(slug, { includeDeleted });
|
||||
let resolved_slug: string | undefined;
|
||||
|
||||
if (!page && fuzzy) {
|
||||
const candidates = await ctx.engine.resolveSlugs(slug);
|
||||
if (candidates.length === 1) {
|
||||
page = await ctx.engine.getPage(candidates[0]);
|
||||
page = await ctx.engine.getPage(candidates[0], { includeDeleted });
|
||||
resolved_slug = candidates[0];
|
||||
} else if (candidates.length > 1) {
|
||||
return { error: 'ambiguous_slug', candidates };
|
||||
@@ -316,7 +318,7 @@ const get_page: Operation = {
|
||||
}
|
||||
|
||||
if (!page) {
|
||||
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug or use fuzzy: true');
|
||||
throw new OperationError('page_not_found', `Page not found: ${slug}`, includeDeleted ? 'Check the slug or use fuzzy: true' : 'Page may be soft-deleted; pass include_deleted: true to verify');
|
||||
}
|
||||
|
||||
const tags = await ctx.engine.getTags(page.slug);
|
||||
@@ -625,39 +627,99 @@ async function runAutoLink(
|
||||
|
||||
const delete_page: Operation = {
|
||||
name: 'delete_page',
|
||||
description: 'Delete a page',
|
||||
description: 'Soft-delete a page. The row is hidden from search and from get_page/list_pages, but is recoverable via restore_page within 72h. The autopilot purge phase hard-deletes after the recovery window. Pass include_deleted: true to get_page to verify the soft-delete landed.',
|
||||
params: {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'delete_page', slug: p.slug };
|
||||
await ctx.engine.deletePage(p.slug as string);
|
||||
return { status: 'deleted' };
|
||||
const slug = p.slug as string;
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'soft_delete_page', slug };
|
||||
// v0.26.5: rewired from hard-delete to soft-delete. The hard-delete primitive
|
||||
// (engine.deletePage) is now reserved for purgeDeletedPages and explicit
|
||||
// tests. softDeletePage returns null when the slug is unknown OR already
|
||||
// soft-deleted (idempotent-as-null) — preserve that as a clean no-op shape.
|
||||
const result = await ctx.engine.softDeletePage(slug);
|
||||
if (result === null) {
|
||||
// Distinguish "not found" from "already soft-deleted" so the agent gets a
|
||||
// clear signal. Probe once with include_deleted to disambiguate.
|
||||
const existing = await ctx.engine.getPage(slug, { includeDeleted: true });
|
||||
if (!existing) {
|
||||
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.');
|
||||
}
|
||||
return { status: 'already_soft_deleted', slug, deleted_at: existing.deleted_at };
|
||||
}
|
||||
return { status: 'soft_deleted', slug, recoverable_until: 'now + 72h via restore_page' };
|
||||
},
|
||||
cliHints: { name: 'delete', positional: ['slug'] },
|
||||
};
|
||||
|
||||
const restore_page: Operation = {
|
||||
name: 'restore_page',
|
||||
description: 'v0.26.5 — restore a soft-deleted page (clear deleted_at). Returns success only if the page was actually soft-deleted. After this op, the page reappears in search and in get_page/list_pages without the include_deleted flag.',
|
||||
params: {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
const slug = p.slug as string;
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'restore_page', slug };
|
||||
const ok = await ctx.engine.restorePage(slug);
|
||||
if (!ok) {
|
||||
// Distinguish "not found" from "already active" (idempotent-as-false).
|
||||
const existing = await ctx.engine.getPage(slug, { includeDeleted: true });
|
||||
if (!existing) {
|
||||
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.');
|
||||
}
|
||||
return { status: 'already_active', slug };
|
||||
}
|
||||
return { status: 'restored', slug };
|
||||
},
|
||||
cliHints: { name: 'restore', positional: ['slug'] },
|
||||
};
|
||||
|
||||
const purge_deleted_pages: Operation = {
|
||||
name: 'purge_deleted_pages',
|
||||
description: 'v0.26.5 — admin-only. Hard-deletes pages whose deleted_at is older than older_than_hours (default 72). Cascades through content_chunks, page_links, chunk_relations. Local CLI only (not exposed over HTTP MCP). Manual escape hatch alongside the autopilot purge phase.',
|
||||
params: {
|
||||
older_than_hours: { type: 'number', description: 'Age cutoff in hours. Default 72.' },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'admin',
|
||||
localOnly: true,
|
||||
handler: async (ctx, p) => {
|
||||
const olderThanHours = (p.older_than_hours as number | undefined) ?? 72;
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'purge_deleted_pages', older_than_hours: olderThanHours };
|
||||
const result = await ctx.engine.purgeDeletedPages(olderThanHours);
|
||||
return { status: 'purged', count: result.count, slugs: result.slugs };
|
||||
},
|
||||
cliHints: { name: 'purge-deleted' },
|
||||
};
|
||||
|
||||
const list_pages: Operation = {
|
||||
name: 'list_pages',
|
||||
description: 'List pages with optional filters',
|
||||
description: 'List pages with optional filters. Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated.',
|
||||
params: {
|
||||
type: { type: 'string', description: 'Filter by page type' },
|
||||
tag: { type: 'string', description: 'Filter by tag' },
|
||||
limit: { type: 'number', description: 'Max results (default 50)' },
|
||||
include_deleted: { type: 'boolean', description: 'v0.26.5: include soft-deleted pages (default: false). Used by restore workflows and operator diagnostics.' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const pages = await ctx.engine.listPages({
|
||||
type: p.type as any,
|
||||
tag: p.tag as string,
|
||||
limit: clampSearchLimit(p.limit as number | undefined, 50, 100),
|
||||
includeDeleted: (p.include_deleted as boolean) === true,
|
||||
});
|
||||
return pages.map(pg => ({
|
||||
slug: pg.slug,
|
||||
type: pg.type,
|
||||
title: pg.title,
|
||||
updated_at: pg.updated_at,
|
||||
...(pg.deleted_at ? { deleted_at: pg.deleted_at } : {}),
|
||||
}));
|
||||
},
|
||||
scope: 'read',
|
||||
@@ -1529,6 +1591,8 @@ const find_orphans: Operation = {
|
||||
export const operations: Operation[] = [
|
||||
// Page CRUD
|
||||
get_page, put_page, delete_page, list_pages,
|
||||
// v0.26.5 destructive-guard ops (page-level soft-delete + recovery + admin purge)
|
||||
restore_page, purge_deleted_pages,
|
||||
// Search
|
||||
search, query,
|
||||
// Tags
|
||||
|
||||
+103
-10
@@ -23,7 +23,7 @@ import type {
|
||||
} from './types.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts';
|
||||
|
||||
type PGLiteDB = PGlite;
|
||||
|
||||
@@ -212,6 +212,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
|
||||
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
|
||||
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
|
||||
* - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) — v0.26.5
|
||||
*
|
||||
* **Maintenance contract:** when a future migration adds a column-with-index
|
||||
* or new-table-with-FK referenced by PGLITE_SCHEMA_SQL, extend this method
|
||||
@@ -226,6 +227,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
WHERE table_schema='public' AND table_name='pages') AS pages_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='source_id') AS source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='deleted_at') AS deleted_at_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='links') AS links_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
@@ -242,6 +245,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const probe = rows[0] as {
|
||||
pages_exists: boolean;
|
||||
source_id_exists: boolean;
|
||||
deleted_at_exists: boolean;
|
||||
links_exists: boolean;
|
||||
link_source_exists: boolean;
|
||||
origin_page_id_exists: boolean;
|
||||
@@ -255,9 +259,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
|
||||
const needsChunksBootstrap = probe.chunks_exists
|
||||
&& (!probe.symbol_name_exists || !probe.language_exists);
|
||||
const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists;
|
||||
|
||||
// Fresh installs (no tables yet) and modern brains both no-op.
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return;
|
||||
|
||||
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
|
||||
|
||||
@@ -305,6 +310,16 @@ export class PGLiteEngine implements BrainEngine {
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsPagesDeletedAt) {
|
||||
// v34 (destructive_guard_columns) adds the column + sources columns +
|
||||
// partial purge index. Bootstrap only adds enough for PGLITE_SCHEMA_SQL's
|
||||
// `CREATE INDEX pages_deleted_at_purge_idx ... WHERE deleted_at IS NOT NULL`
|
||||
// not to crash. v34 runs later via runMigrations and is idempotent.
|
||||
await this.db.exec(`
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
||||
@@ -329,11 +344,23 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Pages CRUD
|
||||
async getPage(slug: string): Promise<Page | null> {
|
||||
async getPage(slug: string, opts?: { sourceId?: string; includeDeleted?: boolean }): Promise<Page | null> {
|
||||
// v0.26.5: hide soft-deleted by default; opt-in via opts.includeDeleted.
|
||||
const includeDeleted = opts?.includeDeleted === true;
|
||||
const sourceId = opts?.sourceId;
|
||||
const where: string[] = ['slug = $1'];
|
||||
const params: unknown[] = [slug];
|
||||
if (sourceId) {
|
||||
params.push(sourceId);
|
||||
where.push(`source_id = $${params.length}`);
|
||||
}
|
||||
if (!includeDeleted) {
|
||||
where.push('deleted_at IS NULL');
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
|
||||
FROM pages WHERE slug = $1`,
|
||||
[slug]
|
||||
`SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
|
||||
FROM pages WHERE ${where.join(' AND ')} LIMIT 1`,
|
||||
params
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
return rowToPage(rows[0] as Record<string, unknown>);
|
||||
@@ -372,6 +399,54 @@ export class PGLiteEngine implements BrainEngine {
|
||||
await this.db.query('DELETE FROM pages WHERE slug = $1', [slug]);
|
||||
}
|
||||
|
||||
async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> {
|
||||
// Idempotent-as-null: only flip rows currently active. Source filter is
|
||||
// optional; without it the first matching row across sources gets soft-deleted.
|
||||
const sourceId = opts?.sourceId;
|
||||
const where: string[] = ['slug = $1', 'deleted_at IS NULL'];
|
||||
const params: unknown[] = [slug];
|
||||
if (sourceId) {
|
||||
params.push(sourceId);
|
||||
where.push(`source_id = $${params.length}`);
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`UPDATE pages SET deleted_at = now() WHERE ${where.join(' AND ')} RETURNING slug`,
|
||||
params
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
return { slug: (rows[0] as { slug: string }).slug };
|
||||
}
|
||||
|
||||
async restorePage(slug: string, opts?: { sourceId?: string }): Promise<boolean> {
|
||||
const sourceId = opts?.sourceId;
|
||||
const where: string[] = ['slug = $1', 'deleted_at IS NOT NULL'];
|
||||
const params: unknown[] = [slug];
|
||||
if (sourceId) {
|
||||
params.push(sourceId);
|
||||
where.push(`source_id = $${params.length}`);
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`UPDATE pages SET deleted_at = NULL WHERE ${where.join(' AND ')} RETURNING slug`,
|
||||
params
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
async purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }> {
|
||||
// Clamp to non-negative integer; cascade through FKs (content_chunks,
|
||||
// page_links, chunk_relations) on DELETE.
|
||||
const hours = Math.max(0, Math.floor(olderThanHours));
|
||||
const { rows } = await this.db.query(
|
||||
`DELETE FROM pages
|
||||
WHERE deleted_at IS NOT NULL
|
||||
AND deleted_at < now() - ($1 || ' hours')::interval
|
||||
RETURNING slug`,
|
||||
[hours]
|
||||
);
|
||||
const slugs = (rows as { slug: string }[]).map((r) => r.slug);
|
||||
return { slugs, count: slugs.length };
|
||||
}
|
||||
|
||||
async listPages(filters?: PageFilters): Promise<Page[]> {
|
||||
const limit = filters?.limit || 100;
|
||||
const offset = filters?.offset || 0;
|
||||
@@ -399,6 +474,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
params.push(escaped);
|
||||
where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`);
|
||||
}
|
||||
// v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted.
|
||||
if (filters?.includeDeleted !== true) {
|
||||
where.push('p.deleted_at IS NULL');
|
||||
}
|
||||
|
||||
const whereSql = where.length > 0 ? `WHERE ${where.join(' AND ')}` : '';
|
||||
params.push(limit, offset);
|
||||
@@ -479,6 +558,9 @@ export class PGLiteEngine implements BrainEngine {
|
||||
extraFilter += ` AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
|
||||
// v0.26.5: visibility filter (soft-deleted + archived-source).
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH ranked AS (
|
||||
SELECT
|
||||
@@ -490,7 +572,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2
|
||||
),
|
||||
@@ -547,6 +630,9 @@ export class PGLiteEngine implements BrainEngine {
|
||||
extraFilter += ` AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
|
||||
// v0.26.5: visibility filter for the chunk-grain anchor primitive.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
@@ -557,7 +643,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2 OFFSET $3`,
|
||||
params
|
||||
@@ -603,6 +690,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
extraFilter += ` AND cc.symbol_type = $${params.length}`;
|
||||
}
|
||||
|
||||
// v0.26.5: visibility filter applied in the inner CTE so HNSW sees the
|
||||
// same candidate count it always did. See postgres-engine.ts for rationale.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH hnsw_candidates AS (
|
||||
SELECT
|
||||
@@ -611,7 +702,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
1 - (cc.embedding <=> $1::vector) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause}
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT $2
|
||||
)
|
||||
@@ -1327,7 +1419,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
async getStats(): Promise<BrainStats> {
|
||||
const { rows: [stats] } = await this.db.query(`
|
||||
SELECT
|
||||
(SELECT count(*) FROM pages) as page_count,
|
||||
-- v0.26.5: exclude soft-deleted from page_count (mirrors postgres-engine).
|
||||
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
|
||||
(SELECT count(*) FROM content_chunks) as chunk_count,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count,
|
||||
(SELECT count(*) FROM links) as link_count,
|
||||
|
||||
@@ -32,6 +32,10 @@ CREATE TABLE IF NOT EXISTS sources (
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
-- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql).
|
||||
archived BOOLEAN NOT NULL DEFAULT false,
|
||||
archived_at TIMESTAMPTZ,
|
||||
archive_expires_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
@@ -60,6 +64,8 @@ CREATE TABLE IF NOT EXISTS pages (
|
||||
content_hash TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql).
|
||||
deleted_at TIMESTAMPTZ,
|
||||
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
|
||||
);
|
||||
|
||||
@@ -67,6 +73,9 @@ CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
|
||||
-- v0.26.5: partial index supports the autopilot purge sweep (mirrors src/schema.sql).
|
||||
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
|
||||
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
|
||||
|
||||
-- ============================================================
|
||||
-- content_chunks: chunked content with embeddings
|
||||
|
||||
+103
-7
@@ -22,7 +22,7 @@ import { GBrainError } from './types.ts';
|
||||
import * as db from './db.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
|
||||
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts';
|
||||
|
||||
// CONNECTION_ERROR_PATTERNS / isConnectionError were used by the per-call
|
||||
// executeRaw retry that #406 originally shipped. Eng-review D3 dropped that
|
||||
@@ -152,6 +152,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
|
||||
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
|
||||
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
|
||||
* - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) — v0.26.5
|
||||
*
|
||||
* Keep this in sync with the PGLite version; covered by
|
||||
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
|
||||
@@ -166,6 +167,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
const probeRows = await conn<{
|
||||
pages_exists: boolean;
|
||||
source_id_exists: boolean;
|
||||
deleted_at_exists: boolean;
|
||||
links_exists: boolean;
|
||||
link_source_exists: boolean;
|
||||
origin_page_id_exists: boolean;
|
||||
@@ -178,6 +180,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'deleted_at') AS deleted_at_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
@@ -198,8 +202,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
|
||||
const needsChunksBootstrap = probe.chunks_exists
|
||||
&& (!probe.symbol_name_exists || !probe.language_exists);
|
||||
// v0.26.5: pages_deleted_at_purge_idx in SCHEMA_SQL crashes if the column
|
||||
// doesn't exist yet. Migration v34 also adds it, but bootstrap runs first.
|
||||
const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists;
|
||||
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return;
|
||||
|
||||
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
|
||||
|
||||
@@ -246,6 +253,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsPagesDeletedAt) {
|
||||
// v34 (destructive_guard_columns) adds the column + sources columns +
|
||||
// partial purge index. Bootstrap only adds enough for SCHEMA_SQL's
|
||||
// `CREATE INDEX pages_deleted_at_purge_idx ... WHERE deleted_at IS NOT NULL`
|
||||
// not to crash. v34 runs later via runMigrations and is idempotent.
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
||||
@@ -278,11 +295,19 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Pages CRUD
|
||||
async getPage(slug: string): Promise<Page | null> {
|
||||
async getPage(slug: string, opts?: { sourceId?: string; includeDeleted?: boolean }): Promise<Page | null> {
|
||||
const sql = this.sql;
|
||||
const includeDeleted = opts?.includeDeleted === true;
|
||||
const sourceId = opts?.sourceId;
|
||||
// v0.26.5: default hides soft-deleted rows. Compose with optional sourceId
|
||||
// filter via fragment chaining (postgres.js supports sql`` composition).
|
||||
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
|
||||
const deletedCondition = includeDeleted ? sql`` : sql`AND deleted_at IS NULL`;
|
||||
const rows = await sql`
|
||||
SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
|
||||
FROM pages WHERE slug = ${slug}
|
||||
SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
|
||||
FROM pages
|
||||
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
|
||||
LIMIT 1
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
return rowToPage(rows[0]);
|
||||
@@ -321,6 +346,48 @@ export class PostgresEngine implements BrainEngine {
|
||||
await sql`DELETE FROM pages WHERE slug = ${slug}`;
|
||||
}
|
||||
|
||||
async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> {
|
||||
const sql = this.sql;
|
||||
const sourceId = opts?.sourceId;
|
||||
// Idempotent-as-null contract: only flip rows that are currently active.
|
||||
// RETURNING projects the slug so we can tell hit-vs-miss without a probe.
|
||||
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
|
||||
const rows = await sql`
|
||||
UPDATE pages SET deleted_at = now()
|
||||
WHERE slug = ${slug} AND deleted_at IS NULL ${sourceCondition}
|
||||
RETURNING slug
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
return { slug: rows[0].slug as string };
|
||||
}
|
||||
|
||||
async restorePage(slug: string, opts?: { sourceId?: string }): Promise<boolean> {
|
||||
const sql = this.sql;
|
||||
const sourceId = opts?.sourceId;
|
||||
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
|
||||
const rows = await sql`
|
||||
UPDATE pages SET deleted_at = NULL
|
||||
WHERE slug = ${slug} AND deleted_at IS NOT NULL ${sourceCondition}
|
||||
RETURNING slug
|
||||
`;
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
async purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }> {
|
||||
const sql = this.sql;
|
||||
// Clamp to non-negative integer; runaway purge protection. The DELETE
|
||||
// cascades through content_chunks, page_links, chunk_relations via FKs.
|
||||
const hours = Math.max(0, Math.floor(olderThanHours));
|
||||
const rows = await sql`
|
||||
DELETE FROM pages
|
||||
WHERE deleted_at IS NOT NULL
|
||||
AND deleted_at < now() - (${hours} || ' hours')::interval
|
||||
RETURNING slug
|
||||
`;
|
||||
const slugs = rows.map((r) => r.slug as string);
|
||||
return { slugs, count: slugs.length };
|
||||
}
|
||||
|
||||
async listPages(filters?: PageFilters): Promise<Page[]> {
|
||||
const sql = this.sql;
|
||||
const limit = filters?.limit || 100;
|
||||
@@ -342,11 +409,15 @@ export class PostgresEngine implements BrainEngine {
|
||||
const slugCondition = slugPrefix
|
||||
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
|
||||
: sql``;
|
||||
// v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted.
|
||||
const deletedCondition = filters?.includeDeleted === true
|
||||
? sql``
|
||||
: sql`AND p.deleted_at IS NULL`;
|
||||
|
||||
const rows = await sql`
|
||||
SELECT p.* FROM pages p
|
||||
${tagJoin}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${deletedCondition}
|
||||
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
|
||||
`;
|
||||
|
||||
@@ -442,6 +513,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
// v0.26.5: visibility filter hides soft-deleted pages and pages from
|
||||
// archived sources. Joined `sources s` lets the predicate compile to a
|
||||
// column lookup. NOT bypassed by detail=high — soft-delete is a contract,
|
||||
// not a temporal preference.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
const rawQuery = `
|
||||
WITH ranked_chunks AS (
|
||||
SELECT
|
||||
@@ -450,6 +527,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
@@ -457,6 +535,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
${visibilityClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${innerLimitParam}
|
||||
),
|
||||
@@ -541,6 +620,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
// v0.26.5: visibility filter for searchKeywordChunks (anchor primitive).
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
const rawQuery = `
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
@@ -549,6 +631,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
${typeClause}
|
||||
${excludeSlugsClause}
|
||||
@@ -556,6 +639,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
${visibilityClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT ${limitParam}
|
||||
OFFSET ${offsetParam}
|
||||
@@ -626,6 +710,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
params.push(offset);
|
||||
const offsetParam = `$${params.length}`;
|
||||
|
||||
// v0.26.5: visibility filter applied in the inner CTE so the HNSW index
|
||||
// sees the same row count it always did. Pulling the predicate to the
|
||||
// outer SELECT would force the HNSW scan to over-fetch and post-filter,
|
||||
// wasting candidate slots on hidden rows.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
|
||||
const rawQuery = `
|
||||
WITH hnsw_candidates AS (
|
||||
SELECT
|
||||
@@ -634,6 +724,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
1 - (cc.embedding <=> $1::vector) AS raw_score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.embedding IS NOT NULL
|
||||
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
||||
${typeClause}
|
||||
@@ -641,6 +732,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
${languageClause}
|
||||
${symbolKindClause}
|
||||
${hardExcludeClause}
|
||||
${visibilityClause}
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT ${innerLimitParam}
|
||||
)
|
||||
@@ -1379,7 +1471,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sql = this.sql;
|
||||
const [stats] = await sql`
|
||||
SELECT
|
||||
(SELECT count(*) FROM pages) as page_count,
|
||||
-- v0.26.5: exclude soft-deleted from page_count. Same posture as the
|
||||
-- search filter and getPage default — soft-deleted is hidden everywhere
|
||||
-- the user looks. Chunks/links stay raw because they still occupy
|
||||
-- storage until the autopilot purge phase runs.
|
||||
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
|
||||
(SELECT count(*) FROM content_chunks) as chunk_count,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count,
|
||||
(SELECT count(*) FROM links) as link_count,
|
||||
|
||||
@@ -39,6 +39,14 @@ CREATE TABLE IF NOT EXISTS sources (
|
||||
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
|
||||
-- actually trigger re-chunking on upgrade.
|
||||
chunker_version TEXT,
|
||||
-- v0.26.5: soft-delete + recovery window. \`archive\` flips archived=true and
|
||||
-- sets archive_expires_at = now() + 72h. The autopilot purge phase
|
||||
-- hard-deletes rows where archive_expires_at <= now(). Promoted from a
|
||||
-- JSONB key to real columns to avoid reserved-key footguns and to make the
|
||||
-- search visibility filter (\`NOT s.archived\`) a column lookup.
|
||||
archived BOOLEAN NOT NULL DEFAULT false,
|
||||
archived_at TIMESTAMPTZ,
|
||||
archive_expires_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
@@ -75,6 +83,11 @@ CREATE TABLE IF NOT EXISTS pages (
|
||||
content_hash TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- v0.26.5: soft-delete + recovery window. \`delete_page\` sets deleted_at = now()
|
||||
-- instead of issuing DELETE. The autopilot purge phase hard-deletes pages
|
||||
-- where deleted_at < now() - 72h. Search and \`get_page\` filter
|
||||
-- \`WHERE deleted_at IS NULL\` by default; \`include_deleted: true\` opts in.
|
||||
deleted_at TIMESTAMPTZ,
|
||||
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
|
||||
);
|
||||
|
||||
@@ -85,6 +98,13 @@ CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops)
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC);
|
||||
-- v0.18.0: source-scoped scans (per /plan-eng-review Section 4).
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
|
||||
-- v0.26.5: partial index supports the autopilot purge sweep
|
||||
-- (\`WHERE deleted_at IS NOT NULL AND deleted_at < now() - INTERVAL '72 hours'\`).
|
||||
-- Search filters (\`WHERE deleted_at IS NULL\`) do not benefit from this index
|
||||
-- (predicate doesn't match) and don't need their own — soft-deleted cardinality
|
||||
-- stays low. Don't add a regular \`(deleted_at)\` index without measuring.
|
||||
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
|
||||
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
|
||||
|
||||
-- ============================================================
|
||||
-- content_chunks: chunked content with embeddings
|
||||
|
||||
@@ -101,5 +101,33 @@ export function buildHardExcludeClause(slugColumn: string, prefixes: string[]):
|
||||
return `AND NOT (${likes})`;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.26.5 — Build the soft-delete + archived-source visibility filter.
|
||||
*
|
||||
* Two filters in one fragment:
|
||||
* - Page-level soft-delete: `<pageAlias>.deleted_at IS NULL` hides pages that
|
||||
* `delete_page` flipped via `softDeletePage`.
|
||||
* - Source-level archive: `NOT <sourceAlias>.archived` hides every page
|
||||
* belonging to a source that `gbrain sources archive` soft-deleted.
|
||||
*
|
||||
* Unlike `buildSourceFactorCase`, this clause is NOT bypassed by `detail=high`.
|
||||
* Soft-deleted content stays hidden regardless of query detail level — the
|
||||
* recovery window is for explicit `include_deleted: true` callers, not for
|
||||
* temporal queries.
|
||||
*
|
||||
* Returns a fragment with leading `AND` so callers can splice it into a WHERE
|
||||
* unconditionally. Both column references are engine-supplied (never user
|
||||
* input), so no escape is required on the alias names themselves.
|
||||
*
|
||||
* @param pageAlias — page table alias (e.g. `'p'`)
|
||||
* @param sourceAlias — source table alias (e.g. `'s'`); the caller is
|
||||
* responsible for joining `sources` so this alias resolves.
|
||||
*
|
||||
* @returns raw SQL fragment, e.g. `AND p.deleted_at IS NULL AND NOT s.archived`
|
||||
*/
|
||||
export function buildVisibilityClause(pageAlias: string, sourceAlias: string): string {
|
||||
return `AND ${pageAlias}.deleted_at IS NULL AND NOT ${sourceAlias}.archived`;
|
||||
}
|
||||
|
||||
// Exported for unit tests
|
||||
export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral };
|
||||
|
||||
@@ -18,6 +18,12 @@ export interface Page {
|
||||
content_hash?: string;
|
||||
created_at: Date;
|
||||
updated_at: Date;
|
||||
/**
|
||||
* v0.26.5: when present, the page is soft-deleted. Hidden from search and
|
||||
* from `getPage` / `listPages` by default; surface via `include_deleted: true`.
|
||||
* The autopilot purge phase hard-deletes rows where `deleted_at < now() - 72h`.
|
||||
*/
|
||||
deleted_at?: Date | null;
|
||||
}
|
||||
|
||||
export type PageKind = 'markdown' | 'code';
|
||||
@@ -53,6 +59,21 @@ export interface PageFilters {
|
||||
* queries to a tier directory without loading every page into memory.
|
||||
*/
|
||||
slugPrefix?: string;
|
||||
/**
|
||||
* v0.26.5: include soft-deleted pages (rows with `deleted_at IS NOT NULL`).
|
||||
* Default false: hides soft-deleted pages from `list_pages` so agents see the
|
||||
* same set search returns. Set true to enumerate the recoverable set during
|
||||
* the 72h window before the autopilot purge phase hard-deletes them.
|
||||
*/
|
||||
includeDeleted?: boolean;
|
||||
}
|
||||
|
||||
/** v0.26.5 — opts for getPage / softDeletePage / restorePage. */
|
||||
export interface GetPageOpts {
|
||||
/** Filter to a specific source. When omitted, getPage returns the first slug match across sources (pre-existing semantics). */
|
||||
sourceId?: string;
|
||||
/** Include soft-deleted pages. Default false. See PageFilters.includeDeleted. */
|
||||
includeDeleted?: boolean;
|
||||
}
|
||||
|
||||
// Chunks
|
||||
|
||||
@@ -43,6 +43,12 @@ export function contentHash(page: PageInput): string {
|
||||
}
|
||||
|
||||
export function rowToPage(row: Record<string, unknown>): Page {
|
||||
// v0.26.5: deleted_at is optional in the SELECT projection. When the column
|
||||
// isn't selected (legacy callers), keep the field absent on the returned object.
|
||||
const deletedAtRaw = row.deleted_at;
|
||||
const deletedAt = deletedAtRaw == null
|
||||
? (deletedAtRaw === null ? null : undefined)
|
||||
: new Date(deletedAtRaw as string);
|
||||
return {
|
||||
id: row.id as number,
|
||||
slug: row.slug as string,
|
||||
@@ -54,6 +60,7 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
content_hash: row.content_hash as string | undefined,
|
||||
created_at: new Date(row.created_at as string),
|
||||
updated_at: new Date(row.updated_at as string),
|
||||
...(deletedAt !== undefined && { deleted_at: deletedAt }),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -35,6 +35,14 @@ CREATE TABLE IF NOT EXISTS sources (
|
||||
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
|
||||
-- actually trigger re-chunking on upgrade.
|
||||
chunker_version TEXT,
|
||||
-- v0.26.5: soft-delete + recovery window. `archive` flips archived=true and
|
||||
-- sets archive_expires_at = now() + 72h. The autopilot purge phase
|
||||
-- hard-deletes rows where archive_expires_at <= now(). Promoted from a
|
||||
-- JSONB key to real columns to avoid reserved-key footguns and to make the
|
||||
-- search visibility filter (`NOT s.archived`) a column lookup.
|
||||
archived BOOLEAN NOT NULL DEFAULT false,
|
||||
archived_at TIMESTAMPTZ,
|
||||
archive_expires_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
@@ -71,6 +79,11 @@ CREATE TABLE IF NOT EXISTS pages (
|
||||
content_hash TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- v0.26.5: soft-delete + recovery window. `delete_page` sets deleted_at = now()
|
||||
-- instead of issuing DELETE. The autopilot purge phase hard-deletes pages
|
||||
-- where deleted_at < now() - 72h. Search and `get_page` filter
|
||||
-- `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in.
|
||||
deleted_at TIMESTAMPTZ,
|
||||
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
|
||||
);
|
||||
|
||||
@@ -81,6 +94,13 @@ CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops)
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC);
|
||||
-- v0.18.0: source-scoped scans (per /plan-eng-review Section 4).
|
||||
CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
|
||||
-- v0.26.5: partial index supports the autopilot purge sweep
|
||||
-- (`WHERE deleted_at IS NOT NULL AND deleted_at < now() - INTERVAL '72 hours'`).
|
||||
-- Search filters (`WHERE deleted_at IS NULL`) do not benefit from this index
|
||||
-- (predicate doesn't match) and don't need their own — soft-deleted cardinality
|
||||
-- stays low. Don't add a regular `(deleted_at)` index without measuring.
|
||||
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
|
||||
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
|
||||
|
||||
-- ============================================================
|
||||
-- content_chunks: chunked content with embeddings
|
||||
|
||||
@@ -377,8 +377,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
|
||||
hookCalls++;
|
||||
},
|
||||
});
|
||||
// v0.23: 8 phases → 8 yield calls (one after each).
|
||||
expect(hookCalls).toBe(8);
|
||||
// v0.26.5: 9 phases (added `purge`) → 9 yield calls (one after each).
|
||||
expect(hookCalls).toBe(9);
|
||||
});
|
||||
|
||||
test('hook exceptions do not abort the cycle', async () => {
|
||||
@@ -388,8 +388,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
|
||||
throw new Error('synthetic hook error');
|
||||
},
|
||||
});
|
||||
// Cycle still completed all phases (v0.23: 8).
|
||||
expect(report.phases.length).toBe(8);
|
||||
// Cycle still completed all phases (v0.26.5: 9 with the new purge phase).
|
||||
expect(report.phases.length).toBe(9);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,325 @@
|
||||
/**
|
||||
* v0.26.5 — destructive-guard unit tests.
|
||||
*
|
||||
* Source-level guard against accidental data loss. Three layers:
|
||||
* 1. Impact assessment (counts pages/chunks/embeddings/files for a source)
|
||||
* 2. Confirmation gate (`--confirm-destructive` required when data exists;
|
||||
* `--yes` alone rejected)
|
||||
* 3. Soft-delete with 72h TTL (column-based as of v0.26.5; JSONB shape was
|
||||
* migrated in v33)
|
||||
*
|
||||
* Run against PGLite — the contract logic is identical on Postgres but
|
||||
* PGLite is fast + DATABASE_URL-free. Postgres-specific paths (CONCURRENTLY,
|
||||
* RLS) are covered separately by E2E tests.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
assessDestructiveImpact,
|
||||
checkDestructiveConfirmation,
|
||||
softDeleteSource,
|
||||
restoreSource,
|
||||
listArchivedSources,
|
||||
purgeExpiredSources,
|
||||
formatImpact,
|
||||
formatSoftDelete,
|
||||
SOFT_DELETE_TTL_HOURS,
|
||||
type DestructiveImpact,
|
||||
} from '../src/core/destructive-guard.ts';
|
||||
|
||||
// Tier 3 opt-out — these tests need the cold-init schema path so the v33
|
||||
// migration columns exist on the brain under test.
|
||||
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
|
||||
async function setupBrain(): Promise<PGLiteEngine> {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
return engine;
|
||||
}
|
||||
|
||||
async function seedSource(engine: PGLiteEngine, id: string, opts?: { withPages?: number }): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`,
|
||||
[id, id],
|
||||
);
|
||||
const count = opts?.withPages ?? 0;
|
||||
for (let i = 0; i < count; i++) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (source_id, slug, type, title) VALUES ($1, $2, 'note', $3)`,
|
||||
[id, `${id}/page-${i}`, `Page ${i}`],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
describe('assessDestructiveImpact', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = await setupBrain();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('returns null for a non-existent source', async () => {
|
||||
const impact = await assessDestructiveImpact(engine, 'aim-does-not-exist');
|
||||
expect(impact).toBeNull();
|
||||
});
|
||||
|
||||
test('counts zero across the board for an empty source', async () => {
|
||||
await seedSource(engine, 'aim-empty', { withPages: 0 });
|
||||
const impact = await assessDestructiveImpact(engine, 'aim-empty');
|
||||
expect(impact).not.toBeNull();
|
||||
expect(impact!.pageCount).toBe(0);
|
||||
expect(impact!.chunkCount).toBe(0);
|
||||
expect(impact!.embeddingCount).toBe(0);
|
||||
expect(impact!.fileCount).toBe(0);
|
||||
// Empty-source summary is the safe message, not the "permanently delete" warning.
|
||||
expect(impact!.summary).toContain('safe to remove');
|
||||
});
|
||||
|
||||
test('counts pages correctly for a populated source', async () => {
|
||||
await seedSource(engine, 'aim-populated', { withPages: 3 });
|
||||
const impact = await assessDestructiveImpact(engine, 'aim-populated');
|
||||
expect(impact!.pageCount).toBe(3);
|
||||
expect(impact!.summary).toContain('3 pages');
|
||||
expect(impact!.summary).toContain('permanently delete');
|
||||
});
|
||||
|
||||
test('source-scopes pages — multi-source isolation', async () => {
|
||||
await seedSource(engine, 'aim-src-a', { withPages: 2 });
|
||||
await seedSource(engine, 'aim-src-b', { withPages: 5 });
|
||||
const a = await assessDestructiveImpact(engine, 'aim-src-a');
|
||||
const b = await assessDestructiveImpact(engine, 'aim-src-b');
|
||||
expect(a!.pageCount).toBe(2);
|
||||
expect(b!.pageCount).toBe(5);
|
||||
});
|
||||
});
|
||||
|
||||
describe('checkDestructiveConfirmation (gate truth table)', () => {
|
||||
const populated: DestructiveImpact = {
|
||||
sourceId: 'has-data',
|
||||
sourceName: 'has-data',
|
||||
pageCount: 100,
|
||||
chunkCount: 500,
|
||||
embeddingCount: 500,
|
||||
fileCount: 0,
|
||||
summary: '⚠️ This will permanently delete: 100 pages, 500 chunks, 500 embeddings',
|
||||
};
|
||||
|
||||
const empty: DestructiveImpact = {
|
||||
sourceId: 'no-data',
|
||||
sourceName: 'no-data',
|
||||
pageCount: 0,
|
||||
chunkCount: 0,
|
||||
embeddingCount: 0,
|
||||
fileCount: 0,
|
||||
summary: 'Source "no-data" has no data (safe to remove).',
|
||||
};
|
||||
|
||||
test('dry-run always passes regardless of flags', () => {
|
||||
expect(checkDestructiveConfirmation(populated, { dryRun: true })).toBeNull();
|
||||
expect(checkDestructiveConfirmation(populated, { yes: true, dryRun: true })).toBeNull();
|
||||
});
|
||||
|
||||
test('empty source passes without --confirm-destructive', () => {
|
||||
expect(checkDestructiveConfirmation(empty, {})).toBeNull();
|
||||
expect(checkDestructiveConfirmation(empty, { yes: true })).toBeNull();
|
||||
});
|
||||
|
||||
test('--confirm-destructive passes regardless of --yes', () => {
|
||||
expect(checkDestructiveConfirmation(populated, { confirmDestructive: true })).toBeNull();
|
||||
expect(checkDestructiveConfirmation(populated, { yes: true, confirmDestructive: true })).toBeNull();
|
||||
});
|
||||
|
||||
test('--yes alone with data is REJECTED with guidance message', () => {
|
||||
const msg = checkDestructiveConfirmation(populated, { yes: true });
|
||||
expect(msg).not.toBeNull();
|
||||
expect(msg).toContain('--confirm-destructive');
|
||||
expect(msg).toContain('archive');
|
||||
});
|
||||
|
||||
test('no flags + populated source rejects', () => {
|
||||
const msg = checkDestructiveConfirmation(populated, {});
|
||||
expect(msg).not.toBeNull();
|
||||
expect(msg).toContain('--confirm-destructive');
|
||||
});
|
||||
});
|
||||
|
||||
describe('soft-delete + restore lifecycle (column-based v0.26.5)', () => {
|
||||
// ONE engine for the whole describe — cold init runs ~29 migrations, ~3s.
|
||||
// Each test uses a unique source id so they don't cross-pollute.
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = await setupBrain();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('softDeleteSource flips column shape + sets TTL', async () => {
|
||||
const id = 'sd-flips';
|
||||
await seedSource(engine, id, { withPages: 2 });
|
||||
const before = Date.now();
|
||||
const result = await softDeleteSource(engine, id);
|
||||
const after = Date.now();
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.id).toBe(id);
|
||||
expect(result!.pageCount).toBe(2);
|
||||
const ttlMs = SOFT_DELETE_TTL_HOURS * 60 * 60 * 1000;
|
||||
expect(result!.expiresAt.getTime()).toBeGreaterThanOrEqual(before + ttlMs - 1000);
|
||||
expect(result!.expiresAt.getTime()).toBeLessThanOrEqual(after + ttlMs + 1000);
|
||||
const rows = await engine.executeRaw<{ archived: boolean; archived_at: string }>(
|
||||
`SELECT archived, archived_at FROM sources WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
expect(rows[0].archived).toBe(true);
|
||||
expect(rows[0].archived_at).not.toBeNull();
|
||||
});
|
||||
|
||||
test('softDeleteSource is idempotent-as-null on already-archived', async () => {
|
||||
const id = 'sd-idem';
|
||||
await seedSource(engine, id, { withPages: 1 });
|
||||
await softDeleteSource(engine, id);
|
||||
expect(await softDeleteSource(engine, id)).toBeNull();
|
||||
});
|
||||
|
||||
test('softDeleteSource returns null for unknown source', async () => {
|
||||
expect(await softDeleteSource(engine, 'sd-unknown-xyz')).toBeNull();
|
||||
});
|
||||
|
||||
test('softDeleteSource flips federated:false in JSONB but archived state is column-based', async () => {
|
||||
const id = 'sd-jsonb';
|
||||
await seedSource(engine, id, { withPages: 1 });
|
||||
await softDeleteSource(engine, id);
|
||||
const rows = await engine.executeRaw<{ config: any; archived: boolean }>(
|
||||
`SELECT config, archived FROM sources WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
|
||||
expect(config.federated).toBe(false);
|
||||
expect(rows[0].archived).toBe(true);
|
||||
// Issue 5 contract: archived must NOT live in config any more.
|
||||
expect(config.archived).toBeUndefined();
|
||||
expect(config.archived_at).toBeUndefined();
|
||||
});
|
||||
|
||||
test('restoreSource clears the column state and re-federates by default', async () => {
|
||||
const id = 'sd-restore-fed';
|
||||
await seedSource(engine, id, { withPages: 1 });
|
||||
await softDeleteSource(engine, id);
|
||||
expect(await restoreSource(engine, id)).toBe(true);
|
||||
const rows = await engine.executeRaw<{ archived: boolean; archived_at: string | null; config: any }>(
|
||||
`SELECT archived, archived_at, config FROM sources WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
expect(rows[0].archived).toBe(false);
|
||||
expect(rows[0].archived_at).toBeNull();
|
||||
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
|
||||
expect(config.federated).toBe(true);
|
||||
});
|
||||
|
||||
test('restoreSource respects --no-federate (refederate=false)', async () => {
|
||||
const id = 'sd-no-fed';
|
||||
await seedSource(engine, id, { withPages: 1 });
|
||||
await softDeleteSource(engine, id);
|
||||
await restoreSource(engine, id, false);
|
||||
const rows = await engine.executeRaw<{ config: any }>(
|
||||
`SELECT config FROM sources WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
|
||||
expect(config.federated).toBe(false);
|
||||
});
|
||||
|
||||
test('restoreSource is idempotent-as-false on already-active', async () => {
|
||||
const id = 'sd-active';
|
||||
await seedSource(engine, id);
|
||||
expect(await restoreSource(engine, id)).toBe(false);
|
||||
});
|
||||
|
||||
test('listArchivedSources filters via the archived column, not JSONB', async () => {
|
||||
const archivedId = 'la-archived';
|
||||
const liveId = 'la-live';
|
||||
await seedSource(engine, archivedId, { withPages: 3 });
|
||||
await seedSource(engine, liveId, { withPages: 1 });
|
||||
await softDeleteSource(engine, archivedId);
|
||||
const archived = await listArchivedSources(engine);
|
||||
const ids = archived.map((a) => a.id);
|
||||
expect(ids).toContain(archivedId);
|
||||
expect(ids).not.toContain(liveId);
|
||||
const archivedRow = archived.find((a) => a.id === archivedId)!;
|
||||
expect(archivedRow.pageCount).toBe(3);
|
||||
});
|
||||
|
||||
test('purgeExpiredSources only deletes rows with archive_expires_at <= now()', async () => {
|
||||
const expiredId = 'pe-expired';
|
||||
const recoverableId = 'pe-recoverable';
|
||||
await seedSource(engine, expiredId, { withPages: 2 });
|
||||
await seedSource(engine, recoverableId, { withPages: 1 });
|
||||
await softDeleteSource(engine, expiredId);
|
||||
await softDeleteSource(engine, recoverableId);
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET archive_expires_at = now() - INTERVAL '1 hour' WHERE id = $1`,
|
||||
[expiredId],
|
||||
);
|
||||
const purged = await purgeExpiredSources(engine);
|
||||
expect(purged).toContain(expiredId);
|
||||
expect(purged).not.toContain(recoverableId);
|
||||
const remainingPages = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
|
||||
[expiredId],
|
||||
);
|
||||
expect(remainingPages[0].n).toBe(0);
|
||||
});
|
||||
|
||||
test('purgeExpiredSources is no-op when nothing is past TTL', async () => {
|
||||
// After all earlier tests, there may still be archived rows whose
|
||||
// archive_expires_at is in the future. Force-update any leftover-past
|
||||
// rows OUT of expiration before this assertion (we only want to test
|
||||
// the no-op return here, not interfere with prior test state).
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET archive_expires_at = now() + INTERVAL '72 hours' WHERE archived = true`,
|
||||
);
|
||||
const purged = await purgeExpiredSources(engine);
|
||||
expect(purged).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('formatters (display helpers)', () => {
|
||||
test('formatImpact renders the boxed preview with the source id and counts', () => {
|
||||
const impact: DestructiveImpact = {
|
||||
sourceId: 'media-corpus',
|
||||
sourceName: 'Media Corpus',
|
||||
pageCount: 5033,
|
||||
chunkCount: 22000,
|
||||
embeddingCount: 22000,
|
||||
fileCount: 0,
|
||||
summary: '⚠️ This will permanently delete: 5,033 pages, 22,000 chunks, 22,000 embeddings',
|
||||
};
|
||||
const out = formatImpact(impact);
|
||||
expect(out).toContain('Media Corpus');
|
||||
expect(out).toContain('media-corpus');
|
||||
expect(out).toContain('5,033');
|
||||
expect(out).toContain('22,000');
|
||||
expect(out).toContain('DESTRUCTIVE OPERATION');
|
||||
});
|
||||
|
||||
test('formatSoftDelete renders the post-archive guidance with restore command', () => {
|
||||
const out = formatSoftDelete({
|
||||
id: 'src-a',
|
||||
name: 'src-a',
|
||||
deletedAt: new Date(),
|
||||
expiresAt: new Date(Date.now() + 72 * 60 * 60 * 1000),
|
||||
pageCount: 100,
|
||||
});
|
||||
expect(out).toContain('archived');
|
||||
expect(out).toContain('restore src-a');
|
||||
expect(out).toContain('72');
|
||||
});
|
||||
});
|
||||
@@ -97,8 +97,9 @@ describeE2E('E2E: runCycle against real Postgres', () => {
|
||||
});
|
||||
|
||||
expect(report.schema_version).toBe('1');
|
||||
// Cycle ran all 8 phases (or skipped the ones that don't support dry-run).
|
||||
expect(report.phases.length).toBe(8);
|
||||
// Cycle ran all 9 phases (or skipped the ones that don't support dry-run).
|
||||
// v0.26.5 added the `purge` phase (9th, after `orphans`).
|
||||
expect(report.phases.length).toBe(9);
|
||||
|
||||
// Nothing got written.
|
||||
const afterPages = await conn.unsafe(`SELECT count(*)::int AS n FROM pages`);
|
||||
|
||||
@@ -271,7 +271,8 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row'
|
||||
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(1);
|
||||
|
||||
// Remove the source.
|
||||
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['remove', 'cascadetest', '--yes']);
|
||||
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
|
||||
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['remove', 'cascadetest', '--confirm-destructive']);
|
||||
|
||||
// Everything for that source is gone.
|
||||
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
|
||||
|
||||
@@ -181,7 +181,8 @@ describe('v0.18.0 — sources remove cascades to pages', () => {
|
||||
);
|
||||
expect(before[0].n).toBeGreaterThan(0);
|
||||
|
||||
await runSources(engine, ['remove', 'testsrc', '--yes']);
|
||||
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
|
||||
await runSources(engine, ['remove', 'testsrc', '--confirm-destructive']);
|
||||
|
||||
const after = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'testsrc'`,
|
||||
|
||||
@@ -0,0 +1,323 @@
|
||||
/**
|
||||
* v0.26.5 — page-level soft-delete contract tests.
|
||||
*
|
||||
* IRON RULE regression test for Q3 (the lynchpin eng-review decision):
|
||||
* delete_page → get_page returns null → get_page({include_deleted:true}) returns
|
||||
* the row with deleted_at populated → restore_page → get_page returns the row
|
||||
* again with deleted_at unset.
|
||||
*
|
||||
* Plus: BrainEngine surface tests (softDeletePage / restorePage /
|
||||
* purgeDeletedPages) for happy-path / boundary / cascade cases.
|
||||
*
|
||||
* Runs against PGLite — same SQL contract as Postgres but DATABASE_URL-free.
|
||||
* Postgres-specific paths (CONCURRENTLY index, two-stage CTE) covered by
|
||||
* separate Postgres E2E tests.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
|
||||
async function setupBrain(): Promise<PGLiteEngine> {
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
return engine;
|
||||
}
|
||||
|
||||
async function seedPage(engine: PGLiteEngine, slug: string): Promise<void> {
|
||||
await engine.putPage(slug, {
|
||||
type: 'note' as any,
|
||||
title: slug,
|
||||
compiled_truth: `Content of ${slug}`,
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
}
|
||||
|
||||
describe('softDeletePage', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = await setupBrain();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('happy path: sets deleted_at and returns slug', async () => {
|
||||
await seedPage(engine, 'people/alice');
|
||||
const result = await engine.softDeletePage('people/alice');
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.slug).toBe('people/alice');
|
||||
// The row stays in the DB.
|
||||
const rows = await engine.executeRaw<{ deleted_at: string | null }>(
|
||||
`SELECT deleted_at FROM pages WHERE slug = $1`,
|
||||
['people/alice'],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].deleted_at).not.toBeNull();
|
||||
});
|
||||
|
||||
test('returns null for unknown slug (idempotent-as-null)', async () => {
|
||||
expect(await engine.softDeletePage('does/not/exist')).toBeNull();
|
||||
});
|
||||
|
||||
test('returns null on already-soft-deleted page (idempotent-as-null)', async () => {
|
||||
await seedPage(engine, 'people/bob');
|
||||
const first = await engine.softDeletePage('people/bob');
|
||||
expect(first).not.toBeNull();
|
||||
const second = await engine.softDeletePage('people/bob');
|
||||
expect(second).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('restorePage', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = await setupBrain();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('clears deleted_at on a soft-deleted page', async () => {
|
||||
await seedPage(engine, 'people/carol');
|
||||
await engine.softDeletePage('people/carol');
|
||||
expect(await engine.restorePage('people/carol')).toBe(true);
|
||||
const rows = await engine.executeRaw<{ deleted_at: string | null }>(
|
||||
`SELECT deleted_at FROM pages WHERE slug = $1`,
|
||||
['people/carol'],
|
||||
);
|
||||
expect(rows[0].deleted_at).toBeNull();
|
||||
});
|
||||
|
||||
test('returns false for unknown slug', async () => {
|
||||
expect(await engine.restorePage('does/not/exist')).toBe(false);
|
||||
});
|
||||
|
||||
test('returns false on already-active page (idempotent-as-false)', async () => {
|
||||
await seedPage(engine, 'people/dave');
|
||||
expect(await engine.restorePage('people/dave')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('purgeDeletedPages (TTL boundary)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = await setupBrain();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('purges pages whose deleted_at is older than the cutoff', async () => {
|
||||
await seedPage(engine, 'people/eve');
|
||||
await seedPage(engine, 'people/frank');
|
||||
// Soft-delete both, then push one's deleted_at into the distant past.
|
||||
await engine.softDeletePage('people/eve');
|
||||
await engine.softDeletePage('people/frank');
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET deleted_at = now() - INTERVAL '73 hours' WHERE slug = $1`,
|
||||
['people/eve'],
|
||||
);
|
||||
const result = await engine.purgeDeletedPages(72);
|
||||
expect(result.count).toBe(1);
|
||||
expect(result.slugs).toContain('people/eve');
|
||||
expect(result.slugs).not.toContain('people/frank');
|
||||
// 'eve' is gone; 'frank' is still there (still inside recovery window).
|
||||
const rows = await engine.executeRaw<{ slug: string }>(`SELECT slug FROM pages`);
|
||||
const remaining = rows.map((r) => r.slug);
|
||||
expect(remaining).not.toContain('people/eve');
|
||||
expect(remaining).toContain('people/frank');
|
||||
});
|
||||
|
||||
test('does NOT touch active pages (deleted_at IS NULL)', async () => {
|
||||
// Bound to this test's seeded slug. Other tests in the same describe may
|
||||
// have soft-deleted state laying around; we don't care about those, just
|
||||
// that THIS test's active page is not deleted.
|
||||
await seedPage(engine, 'people/grace-active');
|
||||
await engine.purgeDeletedPages(0);
|
||||
const rows = await engine.executeRaw<{ slug: string; deleted_at: string | null }>(
|
||||
`SELECT slug, deleted_at FROM pages WHERE slug = $1`,
|
||||
['people/grace-active'],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].deleted_at).toBeNull();
|
||||
});
|
||||
|
||||
test('cascades to content_chunks via FK ON DELETE CASCADE', async () => {
|
||||
await seedPage(engine, 'people/heidi');
|
||||
// Force-add a chunk row so we can observe cascade.
|
||||
const pageRows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1`,
|
||||
['people/heidi'],
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source) VALUES ($1, 0, 'test', 'compiled_truth')`,
|
||||
[pageRows[0].id],
|
||||
);
|
||||
await engine.softDeletePage('people/heidi');
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET deleted_at = now() - INTERVAL '73 hours' WHERE slug = $1`,
|
||||
['people/heidi'],
|
||||
);
|
||||
await engine.purgeDeletedPages(72);
|
||||
const remaining = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM content_chunks WHERE page_id = $1`,
|
||||
[pageRows[0].id],
|
||||
);
|
||||
expect(remaining[0].n).toBe(0);
|
||||
});
|
||||
|
||||
test('clamps negative hours to 0 (no crash, no future-cutoff explosion)', async () => {
|
||||
await seedPage(engine, 'people/ivan');
|
||||
await engine.softDeletePage('people/ivan');
|
||||
// The contract being pinned: negative input must NOT pass through to the
|
||||
// SQL as a literal negative interval (which would purge from the future
|
||||
// and effectively delete every soft-deleted row). Implementation does
|
||||
// `Math.max(0, Math.floor(olderThanHours))`, so -72 collapses to 0. With
|
||||
// hours=0, the predicate `deleted_at < now()` may or may not match a row
|
||||
// soft-deleted in the same statement (timing-dependent), so this test
|
||||
// pins only the safety contract: it returns successfully with a finite
|
||||
// count and doesn't blow up the brain.
|
||||
const result = await engine.purgeDeletedPages(-72);
|
||||
expect(result.count).toBeGreaterThanOrEqual(0);
|
||||
expect(Number.isFinite(result.count)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('getPage / listPages includeDeleted contract (Q3 IRON RULE)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = await setupBrain();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('Q3 round-trip: delete → get returns null → get(include_deleted) returns row → restore → get returns row again', async () => {
|
||||
await seedPage(engine, 'people/judy');
|
||||
|
||||
// Step 1: page is visible by default.
|
||||
const before = await engine.getPage('people/judy');
|
||||
expect(before).not.toBeNull();
|
||||
expect(before!.deleted_at).toBeFalsy();
|
||||
|
||||
// Step 2: soft-delete, default getPage returns null.
|
||||
await engine.softDeletePage('people/judy');
|
||||
const afterDelete = await engine.getPage('people/judy');
|
||||
expect(afterDelete).toBeNull();
|
||||
|
||||
// Step 3: include_deleted: true surfaces the row with deleted_at populated.
|
||||
const surfaced = await engine.getPage('people/judy', { includeDeleted: true });
|
||||
expect(surfaced).not.toBeNull();
|
||||
expect(surfaced!.deleted_at).toBeInstanceOf(Date);
|
||||
|
||||
// Step 4: restore → default getPage returns the row again.
|
||||
expect(await engine.restorePage('people/judy')).toBe(true);
|
||||
const restored = await engine.getPage('people/judy');
|
||||
expect(restored).not.toBeNull();
|
||||
expect(restored!.deleted_at).toBeFalsy();
|
||||
});
|
||||
|
||||
test('listPages excludes soft-deleted by default', async () => {
|
||||
await seedPage(engine, 'people/kim');
|
||||
await seedPage(engine, 'people/larry');
|
||||
await engine.softDeletePage('people/kim');
|
||||
const pages = await engine.listPages({ limit: 100 });
|
||||
const slugs = pages.map((p) => p.slug);
|
||||
expect(slugs).not.toContain('people/kim');
|
||||
expect(slugs).toContain('people/larry');
|
||||
});
|
||||
|
||||
test('listPages includes soft-deleted when includeDeleted: true', async () => {
|
||||
await seedPage(engine, 'people/mia');
|
||||
await engine.softDeletePage('people/mia');
|
||||
const pages = await engine.listPages({ limit: 100, includeDeleted: true });
|
||||
const slugs = pages.map((p) => p.slug);
|
||||
expect(slugs).toContain('people/mia');
|
||||
const mia = pages.find((p) => p.slug === 'people/mia')!;
|
||||
expect(mia.deleted_at).toBeInstanceOf(Date);
|
||||
});
|
||||
});
|
||||
|
||||
describe('search visibility (soft-deleted pages hidden from searchKeyword)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = await setupBrain();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('searchKeyword hides soft-deleted pages', async () => {
|
||||
// Two pages, same distinctive term, then soft-delete one.
|
||||
await engine.putPage('people/nora', {
|
||||
type: 'note' as any,
|
||||
title: 'Nora',
|
||||
compiled_truth: 'gbrainquantum signature term occurs here',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.putPage('people/oscar', {
|
||||
type: 'note' as any,
|
||||
title: 'Oscar',
|
||||
compiled_truth: 'gbrainquantum signature term occurs here too',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
// Force chunk creation so search has something to index.
|
||||
await engine.upsertChunks('people/nora', [
|
||||
{ chunk_index: 0, chunk_text: 'gbrainquantum signature term occurs here', chunk_source: 'compiled_truth' as any },
|
||||
]);
|
||||
await engine.upsertChunks('people/oscar', [
|
||||
{ chunk_index: 0, chunk_text: 'gbrainquantum signature term occurs here too', chunk_source: 'compiled_truth' as any },
|
||||
]);
|
||||
|
||||
const before = await engine.searchKeyword('gbrainquantum');
|
||||
expect(before.length).toBe(2);
|
||||
|
||||
await engine.softDeletePage('people/nora');
|
||||
const after = await engine.searchKeyword('gbrainquantum');
|
||||
const slugs = after.map((r) => r.slug);
|
||||
expect(slugs).not.toContain('people/nora');
|
||||
expect(slugs).toContain('people/oscar');
|
||||
});
|
||||
|
||||
test('searchKeyword hides pages from archived sources', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ('archived-src', 'archived-src') ON CONFLICT DO NOTHING`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (source_id, slug, type, title) VALUES ('archived-src', 'archived-src/secret', 'note', 'Secret')`,
|
||||
);
|
||||
const pageRows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = 'archived-src/secret'`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source) VALUES ($1, 0, 'gbrainsemaphore unique term', 'compiled_truth')`,
|
||||
[pageRows[0].id],
|
||||
);
|
||||
// Trigger should populate search_vector via the schema trigger.
|
||||
const before = await engine.searchKeyword('gbrainsemaphore');
|
||||
expect(before.length).toBe(1);
|
||||
|
||||
// Archive the source.
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET archived = true, archived_at = now(), archive_expires_at = now() + INTERVAL '72 hours' WHERE id = 'archived-src'`,
|
||||
);
|
||||
const after = await engine.searchKeyword('gbrainsemaphore');
|
||||
expect(after.length).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -60,6 +60,9 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [
|
||||
// v0.19+ — forward-referenced by `CREATE INDEX idx_chunks_language
|
||||
// ON content_chunks(language) WHERE language IS NOT NULL`.
|
||||
{ kind: 'column', table: 'content_chunks', column: 'language' },
|
||||
// v0.26.5 — forward-referenced by `CREATE INDEX pages_deleted_at_purge_idx
|
||||
// ON pages (deleted_at) WHERE deleted_at IS NOT NULL`.
|
||||
{ kind: 'column', table: 'pages', column: 'deleted_at' },
|
||||
];
|
||||
|
||||
test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => {
|
||||
@@ -89,6 +92,9 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in
|
||||
DROP INDEX IF EXISTS idx_chunks_language;
|
||||
ALTER TABLE content_chunks DROP COLUMN IF EXISTS symbol_name;
|
||||
ALTER TABLE content_chunks DROP COLUMN IF EXISTS language;
|
||||
|
||||
DROP INDEX IF EXISTS pages_deleted_at_purge_idx;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS deleted_at;
|
||||
`);
|
||||
|
||||
// Run bootstrap in isolation (NOT initSchema). This is what we're testing.
|
||||
@@ -142,6 +148,8 @@ test('after bootstrap, PGLITE_SCHEMA_SQL replays without crashing on missing for
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique;
|
||||
ALTER TABLE links DROP COLUMN IF EXISTS link_source;
|
||||
ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id;
|
||||
DROP INDEX IF EXISTS pages_deleted_at_purge_idx;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS deleted_at;
|
||||
`);
|
||||
|
||||
// Bootstrap, then schema replay. Either step crashing fails the test.
|
||||
|
||||
@@ -2,6 +2,7 @@ import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
buildSourceFactorCase,
|
||||
buildHardExcludeClause,
|
||||
buildVisibilityClause,
|
||||
__test__,
|
||||
} from '../src/core/search/sql-ranking.ts';
|
||||
import {
|
||||
@@ -253,3 +254,40 @@ describe('resolveHardExcludes', () => {
|
||||
expect(r).not.toContain('envdir/');
|
||||
});
|
||||
});
|
||||
|
||||
// v0.26.5 — visibility clause for soft-deleted pages and archived sources.
|
||||
describe('buildVisibilityClause (v0.26.5)', () => {
|
||||
test('emits both predicates joined by AND with a leading AND', () => {
|
||||
const clause = buildVisibilityClause('p', 's');
|
||||
// Leading AND so callers can splice unconditionally.
|
||||
expect(clause.startsWith('AND ')).toBe(true);
|
||||
// Both predicates present: page-level deleted_at IS NULL + source-level NOT archived.
|
||||
expect(clause).toContain('p.deleted_at IS NULL');
|
||||
expect(clause).toContain('NOT s.archived');
|
||||
});
|
||||
|
||||
test('uses the supplied aliases verbatim', () => {
|
||||
expect(buildVisibilityClause('pp', 'src')).toBe('AND pp.deleted_at IS NULL AND NOT src.archived');
|
||||
});
|
||||
|
||||
test('does NOT bypass on detail level — visibility is a contract, not a temporal preference', () => {
|
||||
// Distinct from buildSourceFactorCase: there's no detail-gated short-circuit.
|
||||
// Soft-deleted content stays hidden regardless of caller's detail level.
|
||||
// Function signature has no detail param at all; this test pins that contract.
|
||||
expect(buildVisibilityClause.length).toBe(2);
|
||||
});
|
||||
|
||||
test('emits a stable string regardless of call order (idempotent for snapshot tests)', () => {
|
||||
const a = buildVisibilityClause('p', 's');
|
||||
const b = buildVisibilityClause('p', 's');
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
test('produces no JSONB containment in the output (column-based, not @>)', () => {
|
||||
// Issue 5 contract: archived was promoted from JSONB key to real column.
|
||||
// The visibility clause must not regress to JSONB containment.
|
||||
const clause = buildVisibilityClause('p', 's');
|
||||
expect(clause).not.toContain('@>');
|
||||
expect(clause).not.toContain('config');
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user