mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Garry TanandClaude Opus 5
f66bc9f150
PR #2850 fixed the sub-gate case (both push triggers submit noExtract: false) and left the >100-change case open: performSync's size-gate defer branch only printed a "deferring link/timeline extraction" hint and left links_extracted_at unstamped. The autopilot cycle's extract phase is slug-scoped (runExtractCore({slugs: syncPagesAffected})), so the next cycle's up_to_date sync hands it an empty slug list and it processes 0 pages — any webhook push over 100 changed files left extraction permanently stale until an operator ran `gbrain extract --stale` by hand. Fix: the defer branch now durably queues the follow-up instead of hoping for one. performSync submits an `extract` Minion job {stale: true, sourceId?, deferred_commit} bound to the consumed commit via idempotency key (extract-stale:<sourceId|default>:<headCommit>) so webhook redeliveries and re-syncs of the same commit coalesce onto one job, with maxWaiting: 1 and an explicit 35-min timeout_ms covering the sweep's internal 30-min budget (the tight null-default would wall-clock-kill it, #1737 class). The size gate itself is unchanged — no synchronous extract runs inside the webhook request path. The `extract` handler routes {stale: true} jobs through the existing extractStaleFromDB (checkout-less, resumable, idempotent), scoped to job.data.sourceId — the webhook always carries the registered source's id, so an untrusted push can only ever trigger a sweep of its own source. When the 30-min budget leaves staleRemaining > 0 with forward progress, the handler chains a continuation job so large backlogs converge. Queue submission is best-effort: on failure the branch falls back to the pre-fix hint (pages stay stale and doctor-visible, never mis-stamped). New test/sync-deferred-extract-queue.serial.test.ts fails 2/5 on unmodified master (no job queued after a 101-file sync; the handler ignores {stale: true}); the other 3 are negative pins (sub-gate syncs queue nothing, --no-extract suppresses the job, same-commit re-sync doesn't pile up). All 5 pass with the fix, including end-to-end handler consumption: links materialize and the watermark is stamped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>