Files
Patrick Erichsen 62a697ef1e feat: add permanent Test ranking metrics import (#3259)
* feat: add Test ranking metrics import

* fix: harden ranking metric table replacement

* fix: reserve Test for ranking imports

* fix: bind ranking rollback to imported state

* fix: persist rollback guard before import

* fix: quiesce ranking metric writes during imports

* fix: bind ranking imports to target identities

* fix: lock ranking target identities during imports
2026-07-24 16:00:27 -05:00

6.1 KiB

summary, read_when
summary read_when
Local development fixture seeding ownership rules.
Working on local seed data
Editing dashboard empty states
Changing devSeed fixtures

Dev Seeding

Local fixture seeding is command-driven by default:

  • Worktree dev startup (bun run dev:worktree) seeds shared catalog fixtures under @local, including skill, plugin, scanner, and moderation fixtures, before starting the local app when VITE_CONVEX_URL points at local Convex and CONVEX_DEPLOYMENT is an anonymous/local deployment marker. It also imports the committed public corpus, refreshes cached global stats, and writes .codex/runtime/dev-worktree.seeded so routine restarts skip the expensive corpus pass. This is the documented first-run local setup path.
  • CLI seeding (bun run seed:dev) runs the same seed path manually without starting the preview and bypasses the first-run sentinel.
  • bun run seed is the shared seed pipeline used after local setup and by disposable PR previews. It installs the same moderation fixtures and committed public corpus, creates deterministic catalog presentation fixtures, then refreshes global stats. The presentation pass creates 16 synthetic official organizations with real corpus-backed skills and plugins; the first eight of each type are highlighted so Featured and Official creators render in local and PR previews. Without --preview-name it accepts only a local Convex deployment; remote use requires an explicit preview name plus a Convex Preview deploy key. Vercel recreates that preview deployment before invoking the shared seed, so the corpus import does not perform a destructive reset.
  • bun run seed:public-corpus is the lower-level corpus-only import command. Use it for corpus fixture work, not as the default local setup command. The importer keeps each dummy owner's batches serialized while running different owners concurrently, so owner creation remains deterministic without paying one network round trip per corpus row.
  • bun run validate:public-corpus validates the committed public corpus fixture without seeding.
  • bun run seed:test targets only the permanent academic-chihuahua-392 ClawHub Test deployment. It applies the deterministic moderation fixture overlay without importing the local/public corpus, cloning a current user, or resetting production-derived staging rows. The Convex action also requires CLAWHUB_ENV=test, CLAWHUB_DISABLE_CRONS=1, and CLAWHUB_DEPLOYMENT_NAME=academic-chihuahua-392.
  • bun run seed:test:import-snapshot -- --snapshot <sanitized.zip> is the destructive one-time baseline restore. It validates the archive, refuses every deployment except academic-chihuahua-392, and uses Convex --replace-all so cross-project table IDs restore correctly.
  • After creating a read-only production snapshot with bunx convex export --prod, run bun run seed:test:ranking-export -- --snapshot <snapshot.zip> --dataset-version ranking-metrics-YYYY-MM-DD-vN --output <dataset.json>. The sanitizer reads five fixed tables: public skills, public plugins, their daily aggregate rows, and active bookmarks used only to aggregate bookmark creation counts by skill/day. The emitted JSON contains no Convex ids or user, device, session, IP, auth, moderation, or private telemetry fields and is rejected unless it covers an exact 60-day window.
  • Before any mutating ranking import command, dispatch Reserve Test from exact current main with the dataset version and expected SHA. Wait for it to enter in_progress, pass its run ID as --lane-run-id, and cancel the reservation only after import, proof, and any cleanup or rollback are complete. The reservation is read-only and shares the deploy-test concurrency group, so a Test deploy cannot overlap the operation.
  • bun run seed:test:ranking-import -- --dataset <dataset.json> --backup-dir <empty-dir> --lane-run-id <run-id> targets only academic-chihuahua-392. It verifies that the reservation, local checkout, and current main are the same revision, then atomically replaces the three ranking tables from one ZIP. Before the baseline export it sets an expiring Test-only write lock; skill/package daily-stat writes and the user, publisher, skill, and package identity writes used for re-keying fail closed while that lock is active. It preserves deterministic feature and skills.sh fixture rows as overlays, replaces provenance only for the matching dataset version, and retains older import metadata. It records version/checksum/count/time-range metadata and leaves an exact three-table backup. Immediately before replacement it re-exports and hashes all seven source and target tables and revalidates the reservation, aborting without mutation if the lane, lock, or tables changed. Use --readback --dataset-version <version> for 24-hour/60-day proof, --cleanup with the original dataset to remove that version, or --rollback --backup-dir <dir> --lane-run-id <run-id> to restore the pre-import tables. Rollback also requires the current three-table state to match the exact post-operation digests recorded in that backup, so an older backup cannot overwrite a subsequent import or fixture change.
  • internal.devSeed.seedCurrentUserFixtures remains a dev-only internal action for explicit local development tools/tests that need fixtures cloned to a local user.

Current-user fixture seeding must not be exposed as a public Convex api function or browser UI action. Internal tooling may pass an ownerUserId, but that id must stay inside trusted local seed tooling rather than crossing a frontend boundary. Fixture slugs and package names must include a stable per-user seed key so multiple developers can use the same dev deployment without colliding.

Current-user fixture seeding is dev-only. It must reject production Convex deployments, and it should not be exposed as a first-run dashboard button unless the UX and ownership rules are intentionally revisited.

Without OPENAI_API_KEY, public corpus import may use zero vectors. That is acceptable for local setup, disposable PR previews, and layout QA, but semantic search quality will be weaker than an embedding-backed database.