From 11eebc3605db4238e525f6db39eb4616c202dd66 Mon Sep 17 00:00:00 2001 From: sameerbopardikar Date: Mon, 20 Jul 2026 23:57:53 +0000 Subject: [PATCH] fix(conversation): extract iMessage facts with real timestamps (#2756) (#2958) Co-authored-by: Sameer Bopardikar <203024074+sameerbopardikar@users.noreply.github.com> --- docs/architecture/KEY_FILES.md | 4 +- src/commands/doctor.ts | 6 +-- src/commands/extract-conversation-facts.ts | 15 ++++++- src/commands/jobs.ts | 2 +- src/commands/sources.ts | 9 ++++- src/core/conversation-parser/builtins.ts | 39 ++++++++++++++++++- src/core/conversation-parser/parse.ts | 15 ++++++- test/conversation-parser-cli.test.ts | 2 +- test/conversation-parser/parse.test.ts | 38 +++++++++++++++++- test/e2e/conversation-parser-pglite.test.ts | 2 +- test/extract-conversation-facts.test.ts | 35 +++++++++++++++++ test/fixtures/conversation-formats/all.jsonl | 1 + .../imessage-time-only-12h.jsonl | 1 + 13 files changed, 153 insertions(+), 16 deletions(-) create mode 100644 test/fixtures/conversation-formats/imessage-time-only-12h.jsonl diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index c81458c2a..ba29acf19 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -188,9 +188,9 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts` — `gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's model tooling sees only brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich::')` → `getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1` — `runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). `sanitizeContext` (thin.ts) neutralizes the `…` data-envelope delimiters (injection escape, mirrors the `` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (a bare `enrich:${sid}` would return stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (via read-only `BudgetTracker.cap` getter); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge. The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity). - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping. - `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. -- `src/core/conversation-parser/` — 14-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (14 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, index 3 after `bold-paren-time`) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. +- `src/core/conversation-parser/` — 15-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (15 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, index 3 after `bold-paren-time`) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. - `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path). -- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email pages, splits into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and runs through `extractFactsFromTurn()` so anchor-rich facts surface in `gbrain search`. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide); **page-level TERMINAL audit row** to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS matches the terminal row so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, used as-is — nested `withBudgetTracker` REPLACES; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline**; **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email`) with CLI default reading `cycle.conversation_facts_backfill.types`; **fingerprint on sourceId only**; **string-encoded op-checkpoint** entries `"||"` for resume (durable audit is the facts terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job catches + persists + marks `completed` with `result.budget_exhausted=true`). The companion cycle phase `conversation_facts_backfill` (default OFF) iterates `listSources(engine)`, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source call. Two-layer cost AND walltime caps: per-source (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by `test/extract-conversation-facts.test.ts` (27 cases). Migration v94 adds partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (`transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite). `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when disabled; OK when caught up; WARN when >10 pages lack the terminal row, with paste-ready `gbrain doctor --remediate` step). `src/commands/sources.ts:runAudit` adds `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Schema-pack `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable) into the base seed; backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51` not pack extractable. `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. +- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and runs through `extractFactsFromTurn()` so anchor-rich facts surface in `gbrain search`. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because PHASE_SCOPE='source' is taxonomy-only); **two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap MAX_PAGE_BODY_BYTES=25MB with `pages_skipped_too_large` counter surfaced in doctor); **page-global row_num accumulator** (facts unique index is `(source_id, source_markdown_slug, row_num)` per migration v51 — per-segment row_num would collide); **page-level TERMINAL audit row** to facts table after all segments commit (source='cli:extract-conversation-facts:terminal'; doctor's NOT EXISTS matches the terminal row so partial-extraction pages stay in backlog); **optional `opts.budgetTracker?`** (when present, used as-is — nested `withBudgetTracker` REPLACES; when absent, core auto-wraps with `BudgetTracker({maxCostUsd})`); **body read covers compiled_truth + timeline**; **honors `facts.extraction_enabled` kill-switch** with `--override-disabled` escape; **--types LIST allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`) with CLI default reading `cycle.conversation_facts_backfill.types`; **fingerprint on sourceId only**; **string-encoded op-checkpoint** entries `"||"` for resume (durable audit is the facts terminal row); **`--background` via maybeBackground** (Minion handler `extract-conversation-facts` re-creates BudgetTracker from `data.max_cost_usd`; on `BudgetExhausted` mid-job catches + persists + marks `completed` with `result.budget_exhausted=true`). The companion cycle phase `conversation_facts_backfill` (default OFF) iterates `listSources(engine)`, creates ONE brain-wide tracker per tick + wraps the loop in `withBudgetTracker` + passes the tracker into every per-source call. Two-layer cost AND walltime caps: per-source (`max_cost_usd=$1`, `max_walltime_min=20`) AND brain-wide (`max_total_cost_usd=$5`, `max_total_walltime_min=30`). Pinned by `test/extract-conversation-facts.test.ts` (27 cases). Migration v94 adds partial index `idx_facts_extract_conversation_session ON facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%'` (`transaction:false` + invalid-index pre-drop on Postgres; plain CREATE INDEX on PGLite). `src/commands/doctor.ts:computeConversationFactsBacklogCheck` is 3-state (SKIPPED when disabled; OK when caught up; WARN when >10 pages lack the terminal row, with paste-ready `gbrain doctor --remediate` step). `src/commands/sources.ts:runAudit` adds `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Schema-pack `gbrain-base.yaml` promotes `conversation` (temporal, extractable) + `atom` (annotation, NOT extractable) into the base seed; backstop uses hardcoded `ELIGIBLE_TYPES` in `src/core/facts/eligibility.ts:51` not pack extractable. `ALL_PAGE_TYPES` in `src/core/types.ts` extended with the two new types. - `src/core/link-extraction.ts` — shared library for the graph layer. `extractEntityRefs` (canonical) matches `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks; `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled`. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **Opt-in global-basename wikilink resolution** (issue #972, default off): `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in `extractEntityRefs`); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise` (multi-match by design — emits one edge per matching page). The single shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical), used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — source-scoped so a bare `[[name]]` never resolves to a same-tail page in a different source. `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug`, emits candidates tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the old `nullResolver` ternary). All three surfaces (FS extract, DB extract, `put_page` auto-link) tag provenance with `link_source='wikilink-resolved'`; `put_page` includes it in its reconcilable-edge set so stale basename edges are removed when the wikilink or the flag goes away. Exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). `gbrain doctor`'s `link_resolution_opportunity` check surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% match. Migration v113 widens `links_link_source_check` to admit `'wikilink-resolved'`; v114 (#1941) then opens it to any kebab-case provenance (`^[a-z][a-z0-9]*(-[a-z0-9]+)*$`, ≤64 chars) so external derivers register their own tag (e.g. `citation-graph`) without a migration. `LINK_EXTRACTOR_VERSION_TS` also lives here (bump like `CHUNKER_VERSION` to invalidate prior extract-stale stamps). Pinned by `test/link-extraction.test.ts`, `test/extract-fs.test.ts`, `test/doctor.test.ts`, `test/e2e/global-basename-pglite.test.ts`. - `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id ]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). No in-memory dedup pre-load — candidates buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, `created` counter returns real rows inserted. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id ` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id ] [--catch-up] [--dry-run] [--json]` branch (`extractStaleFromDB`) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three new `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush: link/timeline flush throws propagate and abort the batch; stamp LAST so a crash leaves pages unstamped and they re-extract idempotently (`addLinksBatch` ON CONFLICT DO NOTHING + timeline dedup). Race fix: `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Schema parity in schema.sql + pglite-schema.ts + schema-embedded.ts + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` gains `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` for `pagesAffected` at the inline-extract call site, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` adds `'extract.stale'` to `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` adds `links_extraction_lag` to `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp regression + crash-contract), `test/sync-inline-extract-stamps.serial.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite regression in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. - `src/core/extract/receipt-writer.ts` + `src/core/extract/rollup-writer.ts` + `src/commands/extract-status.ts` + `src/commands/extract-explain.ts` + `src/commands/extract-benchmark.ts` + `src/core/schema-pack/scaffold-extractable.ts` — unified extract operator surface. Every shipped extractor (deterministic `facts.conversation` in `src/commands/extract-conversation-facts.ts` + three LLM-backed cycle phases at `src/core/cycle/{extract-atoms,synthesize-concepts,propose-takes,extract-facts}.ts`) writes ONE receipt page per run (`writeReceipt`) + UPSERTs a row to `extract_rollup_7d` (`upsertExtractRollup`). Receipt slug `extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}.md`; frontmatter stamps BOTH `type: extract_receipt` AND `dream_generated: true` (belt+suspenders against extraction-loop guard drift). `extract_receipt` joins `ALL_PAGE_TYPES` in `src/core/types.ts`; `extracts/` prefix gets a 0.3x source-boost demote in `src/core/search/source-boost.ts`. Migration v104 adds `extract_rollup_7d (kind, source_id, day, cost_usd, halt_count, eval_pass_count, eval_fail_count, round_completed_count, rollup_write_failures, updated_at)` with PK `(kind, source_id, day)` + `idx_extract_rollup_7d_day`. Rollup writes best-effort with process-scoped error-dedup so transient DB failures bump `rollup_write_failures` instead of crashing the cycle. `extract_health` doctor check reads last 7 days, warns at halt-rate > 10% AND when rollup_write_failures > 0; pre-v104 brains report `ok`. CLI: `gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]` (7-day rollup, sorted halt_rate desc + cost desc, top-5 + "more rows" hint, stable `schema_version: 1`); `gbrain extract --explain ` (resolution chain pack-declared vs built-in cycle phase, prompt_template + fixture_corpus paths with `✓`/`(missing)`, last 7d rollup); `gbrain extract benchmark --pack X --kind Y` (loads pack fixture corpus through strict path validation — rejects absolute paths, `..` traversal, null bytes, AND symlinks resolving outside pack root; ships as a stub-reporter). `src/core/schema-pack/manifest-v1.ts` widens `extractable` from `z.boolean()` to `z.union([z.boolean(), ExtractableSpecSchema])` (carries `prompt_template`, `fixture_corpus`, `eval_dimensions`, `benchmark_min_recall`, plus reserved `verifier_path` — parses but refuses at runtime); `extractableSpecsFromPack` + `getExtractableSpec` + `refuseVerifierPathInV042` in `src/core/schema-pack/extractable.ts`; `gbrain schema scaffold-extractable --pack ` declares the type extractable, generates 5 placeholder fixtures + a prompt template stub under `packs//{fixtures,prompts}/extract/`, refuses to overwrite without `--force`. Pinned by `test/extractable-spec-widening.test.ts` (22), `test/extract/receipt-writer.test.ts` (12, canonical PGLite block R3+R4), `test/extract/benchmark.test.ts` (17), `test/extract/status.test.ts` (15), `test/schema-pack/scaffold-extractable.test.ts` (15, privacy guards), `test/doctor-extract-health.test.ts` (8). diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 8ab668c67..3e1bcb4cc 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -3056,7 +3056,7 @@ export async function computeConversationFactsBacklogCheck( const typesRaw = await engine.getConfig( 'cycle.conversation_facts_backfill.types', ); - let types = ['conversation', 'meeting', 'slack', 'email']; + let types = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily']; if (typesRaw) { try { const parsed = JSON.parse(typesRaw); @@ -4927,8 +4927,8 @@ export async function buildChecks( try { const { readConversationBodyForParsing } = await import('../core/conversation-parser/body.ts'); const { parseConversation } = await import('../core/conversation-parser/parse.ts'); - const allowedTypes = ['conversation', 'meeting', 'slack', 'email'] as const; - // PageFilters supports singular `type` only; iterate the 4 types + const allowedTypes = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'] as const; + // PageFilters supports singular `type` only; iterate the allowed types // and cap at ~50/each to land at ~200 total max. const sample: import('../core/types.ts').Page[] = []; for (const t of allowedTypes) { diff --git a/src/commands/extract-conversation-facts.ts b/src/commands/extract-conversation-facts.ts index 693902230..0d6625604 100644 --- a/src/commands/extract-conversation-facts.ts +++ b/src/commands/extract-conversation-facts.ts @@ -140,7 +140,14 @@ export const DEFAULT_MAX_COST_USD = 5.0; * `--types` flag is an explicit per-run override; cycle config is * the single source of truth. */ -export const ALLOWED_TYPES = ['conversation', 'meeting', 'slack', 'email'] as const; +export const ALLOWED_TYPES = [ + 'conversation', + 'meeting', + 'slack', + 'email', + 'imessage', + 'imessage-daily', +] as const; export type AllowedType = (typeof ALLOWED_TYPES)[number]; /** @@ -756,6 +763,12 @@ async function processPage( source_markdown_slug: page.slug, source: PER_SEGMENT_SOURCE_PREFIX, source_session: sessionId, + // Preserve the conversation's valid time instead of defaulting every + // extracted fact to extraction time. Epoch-anchored parses have no + // trustworthy date, so they retain the existing now() fallback. + ...(seg.startIso && !seg.startIso.startsWith('1970-') + ? { valid_from: new Date(seg.startIso) } + : {}), context: fact.context ?? `from ${page.slug} segment ${seg.startIso}..${seg.endIso}`, })); diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index 46f1027f5..cb09f9aa9 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -1497,7 +1497,7 @@ export async function registerBuiltinHandlers( } const types = Array.isArray(job.data.types) ? (job.data.types as string[]).filter((t) => - ['conversation', 'meeting', 'slack', 'email'].includes(t), + ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'].includes(t), ) : undefined; const result = await runExtractConversationFactsCore(engine, { diff --git a/src/commands/sources.ts b/src/commands/sources.ts index 13c52918b..cb855b3f6 100644 --- a/src/commands/sources.ts +++ b/src/commands/sources.ts @@ -1183,7 +1183,14 @@ async function runAudit(engine: BrainEngine, args: string[]): Promise { // frontmatter.type and estimates per-page segment count from body // bytes. Estimated per-segment Sonnet cost is a rough heuristic // (~2000 in + 500 out tokens at $3/MTok in + $15/MTok out ≈ $0.013). - const FACTS_BACKFILL_ALLOWED = ['conversation', 'meeting', 'slack', 'email']; + const FACTS_BACKFILL_ALLOWED = [ + 'conversation', + 'meeting', + 'slack', + 'email', + 'imessage', + 'imessage-daily', + ]; const FACTS_BACKFILL_CHARS_PER_SEGMENT = 6500; // matches SEGMENT_TEXT_CHAR_LIMIT const FACTS_BACKFILL_USD_PER_SEGMENT = 0.013; let factsBackfillPages = 0; diff --git a/src/core/conversation-parser/builtins.ts b/src/core/conversation-parser/builtins.ts index 07f1cd99f..4928cbe26 100644 --- a/src/core/conversation-parser/builtins.ts +++ b/src/core/conversation-parser/builtins.ts @@ -1,7 +1,7 @@ /** * v0.41.16.0 — Built-in conversation parser pattern registry. * - * Fourteen hand-vetted patterns covering the chat-export formats this + * Fifteen hand-vetted patterns covering the chat-export formats this * codebase is most likely to encounter. Each pattern's regex was * derived from a public format reference (source_doc field) so future * maintainers can verify against the wild shape. @@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string { return stripped || raw.trim(); } -/** The 14 hand-vetted built-in patterns. */ +/** The 15 hand-vetted built-in patterns. */ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ // ------------------------------------------------------------------- // INLINE-DATE patterns (date in every line; less ambiguous; tried first). @@ -178,6 +178,41 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ 'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)', }, + { + // iMessage sync's time-only 12-hour shape. AM/PM is required so this + // cannot shadow bold-paren-time's 24-hour form or imessage-slack's + // full-date form. + id: 'bold-paren-time-12h', + origin: 'builtin', + regex: /^\*\*(.+?)\*\*\s*\((\d{1,2}):(\d{2})\s*(AM|PM|am|pm)\)\s*:\s*(.*)$/, + captures: { + speaker_group: 1, + hour_group: 2, + minute_group: 3, + ampm_group: 4, + text_group: 5, + }, + date_source: 'frontmatter', + time_format: '12h_ampm', + timezone_policy: 'utc_assumed_with_warn', + multi_line: false, + quick_reject: /^\*\*/, + test_positive: [ + '**Me** (9:04 AM): sounds good, see you then', + '**+155****0135** (9:39 AM): Will do', + '**Alice Example** (12:00 PM): noon message', + '**Bob Example** (5:38 pm): lowercase ampm', + ], + test_negative: [ + '**Alice** (00:00): 24h shape', + '**Alice Example** (2024-03-15 9:00 AM): full-date iMessage shape', + '**[18:37] G T:** telegram bracket', + 'Alice (9:00 AM): missing the bold', + ], + source_doc: + 'Time-only 12h AM/PM iMessage export shape: `**Speaker** (H:MM AM): text`', + }, + { // Fathom/phone-call raw transcripts in this workspace use a plain // `Speaker A: ...` / `Speaker B: ...` shape with no per-line time. diff --git a/src/core/conversation-parser/parse.ts b/src/core/conversation-parser/parse.ts index 2758571f6..98689bdb1 100644 --- a/src/core/conversation-parser/parse.ts +++ b/src/core/conversation-parser/parse.ts @@ -321,11 +321,22 @@ export function applyPattern( if (!body) return []; const out: MatchedMessage[] = []; const lines = body.split(/\r?\n/); + // Some multi-day conversation exports use markdown date headings instead + // of repeating a date on every message. Keep the caller's context immutable + // while advancing a local date anchor as those headings are encountered. + const runningCtx: DateContext = { ...dateCtx }; + const dateHeaderRe = /^#{1,4}\s+(\d{4}-\d{2}-\d{2})\s*$/; for (let i = 0; i < lines.length; i++) { const rawLine = lines[i]; const line = rawLine.trim(); if (!line) continue; + const dateHeader = dateHeaderRe.exec(line); + if (dateHeader) { + runningCtx.fallbackDate = dateHeader[1]; + continue; + } + // Quick-reject fast path. if (entry.quick_reject && !entry.quick_reject.test(line)) { // Continuation handling for orphan lines. @@ -339,7 +350,7 @@ export function applyPattern( const m = entry.regex.exec(line); if (m) { - const iso = buildIso(m, entry, dateCtx); + const iso = buildIso(m, entry, runningCtx); if (iso === null) continue; // reconstruction failed; skip line const rawSpeaker = m[entry.captures.speaker_group] ?? ''; const speaker = cleanSpeaker(rawSpeaker, entry.speaker_clean); @@ -380,7 +391,7 @@ function getNonBlankLines(body: string, headCap?: number): string[] { * window) and `scorePatternFull` (whole body) delegate here so the * quick_reject + regex loop lives in one place. Reused by * `parseConversation`'s fallback path which pre-splits ONCE and - * passes the array to all 12 candidates (saves 11 redundant body + * passes the array to all 15 candidates (saves 14 redundant body * splits per fallback pass). */ function scoreFromLines( diff --git a/test/conversation-parser-cli.test.ts b/test/conversation-parser-cli.test.ts index ef5686b0e..22a638207 100644 --- a/test/conversation-parser-cli.test.ts +++ b/test/conversation-parser-cli.test.ts @@ -77,7 +77,7 @@ describe('runConversationParser — help', () => { }); describe('runConversationParser — list-builtins', () => { - test('human output includes all 12 pattern ids', async () => { + test('human output includes all built-in pattern ids', async () => { const cap = captureStdio(); try { await runConversationParser(null, ['list-builtins']); diff --git a/test/conversation-parser/parse.test.ts b/test/conversation-parser/parse.test.ts index 34bb7d222..a43e96692 100644 --- a/test/conversation-parser/parse.test.ts +++ b/test/conversation-parser/parse.test.ts @@ -3,7 +3,7 @@ * * Covers: * - PR #1461's 6 telegram-bracket cases verbatim (REGRESSION pin) - * - All 12 built-in patterns hit their test_positive samples + * - All built-in patterns hit their test_positive samples * - Date derivation precedence (D8) * - Pattern priority scoring (D18) — overlap resolution * - Quick-reject fast path (D11) @@ -116,7 +116,7 @@ describe('parseConversation — REGRESSION PR #1461 (telegram-bracket)', () => { }); // --------------------------------------------------------------------------- -// All 12 built-ins must parse their test_positive samples +// All built-ins must parse their test_positive samples // --------------------------------------------------------------------------- describe('parseConversation — every built-in matches its test_positive sample', () => { @@ -261,6 +261,40 @@ describe('parseConversation — multi-line continuation (D5)', () => { }); }); +describe('parseConversation — iMessage time-only 12h and date headings (#2756)', () => { + test('parses the time-only 12-hour iMessage shape', () => { + const r = parseConversation('**Alice Example** (9:04 PM): hello', { + fallbackDate: '2024-03-15', + }); + expect(r.matched_pattern_id).toBe('bold-paren-time-12h'); + expect(r.messages).toHaveLength(1); + expect(r.messages[0].timestamp).toBe('2024-03-15T21:04:00Z'); + }); + + test('markdown date headings advance the running date without becoming message text', () => { + const body = [ + '## 2024-03-15', + '**Alice Example** (9:04 AM): first day', + '## 2024-03-16', + '**Bob Example** (10:05 PM): second day', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2024-03-01' }); + expect(r.matched_pattern_id).toBe('bold-paren-time-12h'); + expect(r.messages.map((m) => m.timestamp)).toEqual([ + '2024-03-15T09:04:00Z', + '2024-03-16T22:05:00Z', + ]); + expect(r.messages[0].text).toBe('first day'); + }); + + test('date headings do not mutate the caller-provided context', () => { + const ctx = { fallbackDate: '2024-03-01', source: 'explicit' as const }; + const pattern = BUILTIN_PATTERNS.find((p) => p.id === 'bold-paren-time-12h')!; + applyPattern('## 2024-03-16\n**Alice** (9:04 AM): hello', pattern, ctx); + expect(ctx.fallbackDate).toBe('2024-03-01'); + }); +}); + // --------------------------------------------------------------------------- // Timezone warning (D19) // --------------------------------------------------------------------------- diff --git a/test/e2e/conversation-parser-pglite.test.ts b/test/e2e/conversation-parser-pglite.test.ts index f7a02d0a3..a5c0524f4 100644 --- a/test/e2e/conversation-parser-pglite.test.ts +++ b/test/e2e/conversation-parser-pglite.test.ts @@ -2,7 +2,7 @@ * v0.41.16.0 — E2E test for the conversation parser cathedral against * a real PGLite brain. * - * For each of the 12 built-in formats: seed a page through + * For each built-in format: seed a page through * `importFromContent`, run `parseConversation` against the body, assert * the parser identifies the correct pattern AND produces at least one * message AND the message timestamp lands in the expected date range. diff --git a/test/extract-conversation-facts.test.ts b/test/extract-conversation-facts.test.ts index 865022c13..8fc281a1e 100644 --- a/test/extract-conversation-facts.test.ts +++ b/test/extract-conversation-facts.test.ts @@ -36,6 +36,7 @@ import { MAX_PAGE_BODY_BYTES, TERMINAL_AUDIT_SOURCE, PER_SEGMENT_SOURCE_PREFIX, + ALLOWED_TYPES, } from '../src/commands/extract-conversation-facts.ts'; // --------------------------------------------------------------------------- @@ -93,6 +94,11 @@ describe('parseConversationMessages', () => { }); }); +test('conversation-facts allowlist includes native iMessage page types (#2756)', () => { + expect(ALLOWED_TYPES).toContain('imessage'); + expect(ALLOWED_TYPES).toContain('imessage-daily'); +}); + // --------------------------------------------------------------------------- // splitIntoSegments — PR's 5 cases verbatim plus tuning regression. // --------------------------------------------------------------------------- @@ -319,6 +325,13 @@ describe('runExtractConversationFactsCore', () => { timeline: '', frontmatter: {}, }); + await engine.putPage('conversations/imessage/native-example', { + type: 'imessage', + title: 'Native iMessage export', + compiled_truth: SAMPLE_BODY, + timeline: '', + frontmatter: {}, + }); await engine.putPage('people/alice-example', { type: 'person', title: 'Alice Example', @@ -392,6 +405,17 @@ describe('runExtractConversationFactsCore', () => { expect(result.pages_considered).toBe(0); }); + test('native imessage page types are eligible by default', async () => { + const result = await runExtractConversationFactsCore(engine, { + sourceId: 'default', + slug: 'conversations/imessage/native-example', + dryRun: true, + sleepMs: 0, + }); + expect(result.pages_considered).toBe(1); + expect(result.pages_processed).toBe(1); + }); + test('sinceIso filters already-processed history', async () => { const result = await runExtractConversationFactsCore(engine, { sourceId: 'default', @@ -432,6 +456,17 @@ describe('runExtractConversationFactsCore', () => { ); expect(Number(perSegFacts[0]?.count ?? 0)).toBeGreaterThan(0); + const validTimes = await engine.executeRaw<{ valid_from: Date }>( + `SELECT valid_from FROM facts + WHERE source = $1 AND source_session = $2 + ORDER BY valid_from ASC`, + [PER_SEGMENT_SOURCE_PREFIX, `${PER_SEGMENT_SOURCE_PREFIX}:conversations/imessage/alice-example`], + ); + expect(validTimes.map((row) => new Date(row.valid_from).toISOString())).toEqual([ + '2024-03-15T09:00:00.000Z', + '2024-03-16T08:00:00.000Z', + ]); + // Terminal audit row present. const terminalRows = await engine.executeRaw<{ count: string | number }>( `SELECT COUNT(*) AS count FROM facts WHERE source = $1 AND source_session = $2`, diff --git a/test/fixtures/conversation-formats/all.jsonl b/test/fixtures/conversation-formats/all.jsonl index 5127a65b5..5f19b8024 100644 --- a/test/fixtures/conversation-formats/all.jsonl +++ b/test/fixtures/conversation-formats/all.jsonl @@ -1,5 +1,6 @@ {"fixture_id":"imessage-001","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Alice Example** (2024-03-15 9:00 AM): morning\n**Bob Example** (2024-03-15 9:01 AM): hey there\n**Alice Example** (2024-03-15 9:02 AM): how are you\n**Bob Example** (2024-03-15 9:03 AM): good thanks\n**Alice Example** (2024-03-15 9:04 AM): you?","expected_messages":5,"expected_participants":["Alice Example","Bob Example"]} {"fixture_id":"imessage-002","pattern":"imessage-slack","frontmatter":{"date":"2024-03-15"},"body":"**Charlie Example** (2024-03-15 2:00 PM): afternoon\n**Charlie Example** (2024-03-15 2:01 PM): are you there?\n**Diana Example** (2024-03-15 2:05 PM): yes\n**Charlie Example** (2024-03-15 2:06 PM): great","expected_messages":4,"expected_participants":["Charlie Example","Diana Example"]} +{"fixture_id":"imessage-time-only-12h-001","pattern":"bold-paren-time-12h","frontmatter":{"date":"2024-03-15"},"body":"## 2024-03-15\n**Alice Example** (9:04 AM): morning\n**Bob Example** (9:05 AM): hey there\n## 2024-03-16\n**Alice Example** (10:06 PM): second day\n**Bob Example** (10:07 PM): good night","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} {"fixture_id":"telegram-bracket-001","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-24","timezone":"America/Los_Angeles"},"body":"**[18:37] 👤 Alice Example:** hello world\n**[18:38] 👤 Bob Example:** hey\n**[18:39] 👤 Alice Example:** how are you\n**[18:40] 👤 Bob Example:** good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} {"fixture_id":"telegram-bracket-002","pattern":"telegram-bracket","frontmatter":{"date":"2026-05-25","timezone":"America/Los_Angeles"},"body":"**[06:00] 🤖 Zion Bot:** On it.\n**[06:01] 👤 Charlie Example:** thanks\n**[06:02] 🤖 Zion Bot:** anything else?\n**[06:03] 👤 Charlie Example:** no good","expected_messages":4,"expected_participants":["Zion Bot","Charlie Example"]} {"fixture_id":"whatsapp-iso-001","pattern":"whatsapp-iso","frontmatter":{"date":"2024-03-15"},"body":"[15/03/24, 18:37:00] Alice Example: hello\n[15/03/24, 18:37:30] Bob Example: hey\n[15/03/24, 18:38:00] Alice Example: how are you\n[15/03/24, 18:39:00] Bob Example: good","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]} diff --git a/test/fixtures/conversation-formats/imessage-time-only-12h.jsonl b/test/fixtures/conversation-formats/imessage-time-only-12h.jsonl new file mode 100644 index 000000000..fa8373a89 --- /dev/null +++ b/test/fixtures/conversation-formats/imessage-time-only-12h.jsonl @@ -0,0 +1 @@ +{"fixture_id":"imessage-time-only-12h-001","pattern":"bold-paren-time-12h","frontmatter":{"date":"2024-03-15"},"body":"## 2024-03-15\n**Alice Example** (9:04 AM): morning\n**Bob Example** (9:05 AM): hey there\n## 2024-03-16\n**Alice Example** (10:06 PM): second day\n**Bob Example** (10:07 PM): good night","expected_messages":4,"expected_participants":["Alice Example","Bob Example"]}