From 52389dbe5be71ec4778e17e859ebbe58506830c0 Mon Sep 17 00:00:00 2001 From: Masa <98894508+Masashi-Ono0611@users.noreply.github.com> Date: Thu, 13 Aug 2026 20:51:32 +0900 Subject: [PATCH] fix(conversation-parser): add markdown-heading turn pattern (## User / ## Assistant) (#4005) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(facts): make transcript pages facts-extraction eligible `gbrain extract-conversation-facts`'s ALLOWED_TYPES allowlist omitted the `transcript` page type, so gbrain's own nightly transcript-ingest pages were silently skipped by both the CLI `--types` validation and the `cycle.conversation_facts_backfill.types` config filter. Even with the type allowed, the built-in conversation-parser had no pattern for the `## User` / `## Assistant` markdown-heading turn shape that transcript ingest writes into `compiled_truth`, so parsing would still yield 0 segments. This PR makes an explicit decision: transcript pages ARE now facts-extraction eligible. That is a real behavioral change (a new, potentially large corpus starts flowing through the extraction + segment-cost path), not a no-op bugfix — flagging it plainly rather than padding out the change as narrower than it is. Changes: - `src/commands/extract-conversation-facts.ts`: add `'transcript'` to `ALLOWED_TYPES` / `ALLOWED_TYPE_ALIASES` (the single source of truth for this allowlist). - `src/core/conversation-parser/builtins.ts`: add the `markdown-heading-turn` builtin pattern recognizing heading-only `## User` / `## Assistant` / `## Human` / `## System` lines as turn openers, with D5 continuation-line body absorption. `quick_reject` is deliberately scoped to the role-prefix (not a bare `#{2,3}` heading check) so a message body that happens to paste unrelated markdown headings doesn't starve the D18 scorer's anchor-candidate ratio. - `src/commands/jobs.ts`, `src/commands/doctor.ts` (x2 checks), `src/commands/sources.ts`: these each carried their own hand-copied literal of the same allowed-types list (background-job type filter, `conversation_facts_backlog` doctor check, `conversation_format_coverage` doctor check, `facts_backfill_estimate`). Switched each to import `ALLOWED_TYPES` from the command module instead of re-listing it, so this class of drift (a type added in one place, silently excluded everywhere else) can't recur. - `docs/architecture/KEY_FILES.md`: updated the two stale mentions (pattern count 17→18, allowlist list) to current-state per this repo's own reference-doc convention. Known limitation (not fixed here, scope-bounded intentionally): parsing is context-free, same as every other multi-line builtin in this registry — a message body that contains a literal `## User` line (e.g. someone pasting a markdown transcript excerpt into their own message) would be read as a turn boundary. This is a pre-existing property of the whole parser (`applyPattern`'s per-line scan has no fence-awareness), not something this PR introduces or could fix without a much larger, separate change to the shared orchestrator affecting all 18 patterns. Flagging it here rather than silently shipping the same limitation as the other 17 builtins. Tests: 4 new tests (2 in test/extract-conversation-facts.test.ts, 2 in test/conversation-parser/parse.test.ts) covering the allowlist, the new pattern's positive match + continuation absorption, and that ordinary `## Summary`-style headings are correctly rejected. Full targeted suite (conversation-parser + facts-extraction + doctor backlog + build-llms freshness): 263 pass / 0 fail. typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0126D3zLWL5RE3CVxnPANiiU * fix(facts): read the type allowlist from core, not the command module CI caught this: the known-flags registry drifted for doctor, sources, and repos. The obvious remedy the guard prints -- regenerate and commit -- would have been a regression, so this takes the other route. The generator walks one level of a command module's relative imports and harvests every flag-shaped string it finds, help text included, and is deliberately over-inclusive. Importing extract-conversation-facts.ts just to read ALLOWED_TYPES therefore spliced that command's entire flag vocabulary (--types, --sleep, --slug, --segment-limit, --override-disabled, ...) into the allowlists of three commands that implement none of it: `gbrain doctor --types foo` would have passed validation and been silently ignored. That is the exact defect class #2185 exists to close. (jobs.ts is unaffected: it already imported the command module on one line for runExtractConversationFactsCore, so those flags were already in its registry entry before this branch.) ALLOWED_TYPES + ALLOWED_TYPE_ALIASES now live in src/core/conversation-facts-types.ts, a constants-only module with no CLI text to harvest. extract-conversation-facts.ts re-exports both so its existing importers are unchanged. Verified: registry regenerates to zero drift (was doctor/repos/sources), cli-flag-validation 24 pass, typecheck clean, 287 pass across the touched areas. Confirmed against a clean upstream/master worktree that the drift was introduced by this branch and is not pre-existing. * fix(conversation-parser): reduce to the parser pattern only Withdraws the `transcript` allowlist half of this branch. The premise was wrong: `transcript` is not an upstream page type. `ALL_PAGE_TYPES` does not contain it, `gbrain-base.yaml` declares `conversation` for "long-running chat/transcript pages" and marks it `extractable: true` precisely so extract-conversation-facts walks it, and `gbrain-base-v2.yaml` lists `transcript` as an alias of `source` (a media primitive). Pages typed `transcript` are a convention of my own ingest pipeline, not something upstream produces — the fix for that belongs on my side, by emitting `conversation`. That takes the four call-site de-duplications with it (they existed only to keep the allowlist in sync), and with them the flag-registry drift: no imports are added, so the registry regenerates to zero drift with no constants module needed. What remains is the half that stands on its own: a `conversation` page whose body uses `## User` / `## Assistant` headings matches none of the 17 builtins and parses to 0 segments. `markdown-heading-turn` is an 18th pattern in the same shape as the iMessage/Circleback additions before it. Verified: typecheck clean, 181 pass / 0 fail across the parser, extraction, flag-registry and llms-freshness suites, registry drift zero. --------- Co-authored-by: Claude Opus 5 (1M context) --- docs/architecture/KEY_FILES.md | 2 +- src/core/conversation-parser/builtins.ts | 44 ++++++++++++++++++++++-- src/core/conversation-parser/parse.ts | 2 +- test/conversation-parser/parse.test.ts | 27 +++++++++++++++ test/extract-conversation-facts.test.ts | 18 ++++++++++ 5 files changed, 89 insertions(+), 4 deletions(-) diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 9d90cd3f5..21342f776 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -201,7 +201,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/retrieval-upgrade-planner.ts` — `runSchemaTransition(engine, targetDim)` (exported) is the ONE atomic dimension-transition path, shared by `ze-switch` and `gbrain migrate embeddings`. In a single transaction it rebuilds ALL THREE dim-pinned text-embedding-space columns at `targetDim` — `content_chunks.embedding`, `query_cache.embedding`, `facts.embedding` — preserving each column's declared type (`vector` vs `halfvec`, probed from `information_schema`) and recreating its HNSW index with the matching opclass, gated on `hnswIndexExpected` (above the per-type dim ceiling pgvector refuses the index and exact scans remain the path). query_cache + facts are created at brain-birth width by `migrate.ts` and NO migration ever ALTERs them, so omitting either leaves it silently broken: a narrow `query_cache.embedding` makes every `store()`/`lookup()` fail inside the cache's own error-swallowing (permanent 0% hit rate), and a narrow `facts.embedding` fails every per-fact embed write (the doctor check that would warn is skipped on PGLite, the default engine). `content_chunks.embedding_image` / `embedding_multimodal` are the deliberate exception — separate multimodal models, dimensions independent of the text model. Pinned by `test/embedding-migration.test.ts` (all three widths + a real INSERT at the new width into each) and `test/e2e/migrate-embeddings-postgres.test.ts`. - `src/core/embedding-migration.ts` — provider-agnostic embedding migration core (#3390): `planEmbeddingMigration` (workload counts via the widened stale predicates with the TARGET signature + `includeNullSignature`, so a mid-migration re-plan counts only what remains; cost via `embedding-pricing.ts`; `null_signature_chunks` split out for #3391 visibility; reranker-on-outgoing-provider warning), `applyEmbeddingMigration` (env-override gate BEFORE any mutation → in-flight state marker `embedding_migration.state` → `runSchemaTransition` when the ACTUAL column width differs from target → DB-plane `embedding_model`/`embedding_dimensions` → `persistConfig` callback for the file plane → `invalidateStaleSignatureEmbeddings({includeNullSignature: true})` → `SemanticQueryCache.clear()`), `completeEmbeddingMigration` (clears the marker + stamps `embedding_migration.completed`; call only at zero backlog), `resolveMigrationTarget` (validates `provider:model` via `resolveRecipe`, dims via `embeddingDimsForModel` or explicit `--dim`), `migrationSignature` (matches `currentEmbeddingSignature()` shape). Engine-pure; every step idempotent under crash + re-run — the NULL-embedding column is the checkpoint. Reuses `runSchemaTransition` (now exported from `retrieval-upgrade-planner.ts`) so ze-switch and the migration share ONE dimension-transition path. `reconcilePageSignatures(engine, plan)` runs after the re-embed drain and BEFORE the completion probe: it stamps the target signature on every page that has zero NULL-embedding chunks, covering pages whose chunks straddle a `listStaleChunks` batch boundary (the embed loop only stamps when `stale.length === existing.length`, so a split page is embedded correctly but never stamped — without the reconcile a >1-batch brain reports "incomplete" and the re-run re-invalidates and re-pays for those pages). Sound only because apply() invalidated everything not already in the target space; pages with a remaining NULL chunk stay unstamped so a real embed failure still surfaces. Invalidation is ordered BEFORE the config writes so a crash on a same-dim swap leaves rows merely stale (empty results) rather than new-space queries scored against old-space vectors (silently wrong). Pinned by `test/embedding-migration.test.ts` (PGLite) + `test/e2e/migrate-embeddings-postgres.test.ts` (real pgvector). - `src/commands/migrate-embeddings.ts` — `gbrain migrate embeddings --to [--dim N] [--dry-run] [--yes] [--json] [--no-embed] [--pace[=mode]] [--ignore-env-override]` (alias: `gbrain retrieval-upgrade`, the command README/doctor promised since v0.36). Flow: plan → render (stderr when `--json` so stdout stays JSON-clean) → consent gate (TTY y/N prompt or `--yes`; non-TTY without `--yes` refuses exit 2, mirroring the reindex-code cost gate) → live probe (one embed against the TARGET model/dims BEFORE any mutation — bad key/model/dim fails with nothing changed) → `applyEmbeddingMigration` with `persistEmbeddingFileConfig` (writes `~/.gbrain/config.json` + reconfigures the in-process gateway — the gateway reads file/env, NOT the DB plane) → `runEmbedCore({stale, catchUp, singleFlight, includeNullSignature, pace})` → drain check → `completeEmbeddingMigration` or exit 1 with the resume hint (re-run the same command). Also surfaced as the `migrate_embeddings` op (scope admin, localOnly, hidden cliHints; handler hard-refuses `ctx.remote !== false` and returns `needs_confirmation` + plan without `yes: true`). Pinned by `test/migrate-embeddings-flow.serial.test.ts` (full lifecycle incl. interrupted-run resume on PGLite). -- `src/core/conversation-parser/` — 17-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (17 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-time-dash, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, ordered after the time-bearing bold patterns) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time,bold-time-dash}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. Maintainer guidance: [conversation parser patterns](conversation-parser-patterns.md). +- `src/core/conversation-parser/` — 18-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (18 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-time-dash, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export, markdown-heading-turn; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, ordered after the time-bearing bold patterns) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time,bold-time-dash}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. Maintainer guidance: [conversation parser patterns](conversation-parser-patterns.md). - `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path). - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES=25MB`); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd`; **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); and **`--background` via `maybeBackground`**. The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source, and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts::`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). `deleteOrphanFactsForPage(engine, sourceId, slug)` provides delete-orphans-first replay safety — wipes facts from a prior crashed run for this (sourceId, slug) before re-extracting, closing the "terminal audit row written after partial insertFacts failure" class. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Checkpoint state is a shared `cpMap: Map` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-workers.test.ts` + the existing extract-conversation-facts behavioral tests. with `src/commands/doctor.ts` durable outcome authority: page completion survives operation-checkpoint GC through versioned terminal audit rows (`cli:extract-conversation-facts:terminal:v2`), while recognized pages with no eligible segment use the separate `cli:extract-conversation-facts:non-extractable:v2` source. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed delete-first. `no_match`, errors, cancellation, and dry runs never become durable negatives. Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. - `src/core/link-extraction.ts` — shared library for the graph layer. `extractEntityRefs` (canonical) matches `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks; `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled`. #2576: markdown links, bare-slug prose refs, and slash-shaped wikilinks match ANY dir-shaped path (`ANY_DIR_SEGMENT`), not a directory whitelist — nonexistent targets are dropped by the persist paths' page-existence checks (`resolveCandidateSources`, put_page's allSlugs filter, `addLinksBatch` INNER JOINs) and counted as `skippedMissingTarget` in the extract summaries; the `DIR_PATTERN` whitelist survives only as the typed fast-path for pass-2b wikilinks (non-whitelisted `[[dir/...]]` get an equivalent direct typed candidate in pass 2c, plus the flag-gated suffix rescue for non-exact matches). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. **Opt-in global-basename wikilink resolution** (issue #972, default off): `WIKILINK_GENERIC_RE` catches bare `[[name]]` wikilinks outside `DIR_PATTERN` (third pass `2c` in `extractEntityRefs`); `EntityRef.needsResolution: true` tags refs from this pass (the ref's `slug` is the wikilink TARGET, `name` the optional display alias). `SlugResolver` gains optional `resolveBasenameMatches(name): Promise` (multi-match by design — emits one edge per matching page). The single shared basename matcher is `buildBasenameIndex(slugs)` + `queryBasenameIndex(index, name)` + `normalizeBasename` (keys raw/lower/slugified tail, stable-sorted shorter-first then lexical), used by `makeResolver`, the FS `resolveBasenameMatchesFromSlugs`, AND the doctor check so they cannot drift. `makeResolver(engine, {mode, sourceId})` builds the index lazily via `engine.getAllSlugs({sourceId})` — source-scoped so a bare `[[name]]` never resolves to a same-tail page in a different source. `extractPageLinks` gains `opts.globalBasename` (routes `needsResolution` refs through `resolveBasenameMatches` keyed on `ref.slug`, emits candidates tagged `linkType: 'wikilink_basename'` + `linkSource: 'wikilink-resolved'`, skips self-loops) and `opts.skipFrontmatter` (replaces the old `nullResolver` ternary). All three surfaces (FS extract, DB extract, `put_page` auto-link) tag provenance with `link_source='wikilink-resolved'`; `put_page` includes it in its reconcilable-edge set so stale basename edges are removed when the wikilink or the flag goes away. Exports `WIKILINK_BASENAME_LINK_TYPE` + `isGlobalBasenameEnabled(engine)` (resolution order: env `GBRAIN_LINK_RESOLUTION_GLOBAL_BASENAME` → DB config `link_resolution.global_basename` → default false). `gbrain doctor`'s `link_resolution_opportunity` check surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve AND ≥20% match. Migration v113 widens `links_link_source_check` to admit `'wikilink-resolved'`; v114 (#1941) then opens it to any kebab-case provenance (`^[a-z][a-z0-9]*(-[a-z0-9]+)*$`, ≤64 chars) so external derivers register their own tag (e.g. `citation-graph`) without a migration. `LINK_EXTRACTOR_VERSION_TS` also lives here (bump like `CHUNKER_VERSION` to invalidate prior extract-stale stamps). Pinned by `test/link-extraction.test.ts`, `test/extract-fs.test.ts`, `test/doctor.test.ts`, `test/e2e/global-basename-pglite.test.ts`. diff --git a/src/core/conversation-parser/builtins.ts b/src/core/conversation-parser/builtins.ts index 969edb036..d23721ee4 100644 --- a/src/core/conversation-parser/builtins.ts +++ b/src/core/conversation-parser/builtins.ts @@ -1,7 +1,7 @@ /** * v0.41.16.0 — Built-in conversation parser pattern registry. * - * Seventeen hand-vetted patterns covering the chat-export formats this + * Eighteen hand-vetted patterns covering the chat-export formats this * codebase is most likely to encounter. Each pattern's regex was * derived from a public format reference (source_doc field) so future * maintainers can verify against the wild shape. @@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string { return stripped || raw.trim(); } -/** The 17 hand-vetted built-in patterns. */ +/** The 18 hand-vetted built-in patterns. */ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ // ------------------------------------------------------------------- // INLINE-DATE patterns (date in every line; less ambiguous; tried first). @@ -670,6 +670,46 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [ test_negative: [' classic irc, no time', '[18:37] @alice: matrix'], source_doc: 'weechat default logger.format `%H:%M %p\\t%m`', }, + + { + id: 'markdown-heading-turn', + origin: 'builtin', + // gbrain transcript-ingest shape: a heading-only line ('## User' / + // '## Assistant' / '### Human') opens a turn; the message text is + // the continuation lines below the heading (D5), not anything on + // the heading line itself. No per-line timestamps — date comes + // from frontmatter / effective_date. The speaker set is closed + // (User/Assistant/Human/System only) so ordinary section headings + // like '## Summary' never match, and a heading with trailing prose + // ('## User said hello') is rejected rather than mis-captured. + regex: /^#{2,3}\s+(User|Assistant|Human|System)\s*:?\s*()$/, + captures: { + speaker_group: 1, + text_group: 2, + }, + date_source: 'frontmatter', + time_format: '24h', + timezone_policy: 'utc_assumed_with_warn', + multi_line: true, + score_continuations_as_body: true, + // Narrowed to a role-prefix superset (NOT bare `/^#{2,3}\s/`): a body + // that pastes unrelated markdown headings (e.g. a document with many + // '## Section' headings) would otherwise inflate the D18 scorer's + // anchor-candidate denominator without inflating the anchored count, + // starving the pattern's score toward 0 on otherwise-valid transcripts. + // Still a strict superset of `regex` per validatePatternEntry's + // invariant (every test_positive sample passes both). + quick_reject: /^#{2,3}\s+(?:User|Assistant|Human|System)\b/, + test_positive: ['## User', '## Assistant', '### Human', '## System', '## User:'], + test_negative: [ + '## Summary', + '#### User', + 'User: plain no heading', + '## User said hello', + ], + source_doc: + 'gbrain nightly transcript ingest: compiled_truth bodies use markdown headings per turn', + }, ]; /** diff --git a/src/core/conversation-parser/parse.ts b/src/core/conversation-parser/parse.ts index c12b38ba4..2e3ed937c 100644 --- a/src/core/conversation-parser/parse.ts +++ b/src/core/conversation-parser/parse.ts @@ -392,7 +392,7 @@ function getNonBlankLines(body: string, headCap?: number): string[] { * window) and `scorePatternFull` (whole body) delegate here so the * quick_reject + regex loop lives in one place. Reused by * `parseConversation`'s fallback path which pre-splits ONCE and - * passes the array to all 17 candidates (saves 16 redundant body + * passes the array to all 18 candidates (saves 17 redundant body * splits per fallback pass). */ function scoreFromLines( diff --git a/test/conversation-parser/parse.test.ts b/test/conversation-parser/parse.test.ts index c9cc02648..6a1ec436f 100644 --- a/test/conversation-parser/parse.test.ts +++ b/test/conversation-parser/parse.test.ts @@ -276,6 +276,33 @@ describe('parseConversation — disabledBuiltinIds', () => { // Multi-line continuation (D5) // --------------------------------------------------------------------------- +describe('parseConversation — markdown-heading-turn (gbrain transcript ingest)', () => { + test('parses ## User / ## Assistant heading-only turns with continuation body', () => { + const body = [ + '## User', + 'What is the capital of France?', + '## Assistant', + 'The capital of France is Paris.', + 'It is also its largest city.', + ].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-08-11' }); + expect(r.matched_pattern_id).toBe('markdown-heading-turn'); + expect(r.messages).toHaveLength(2); + expect(r.messages[0].speaker).toBe('User'); + expect(r.messages[0].text).toBe('What is the capital of France?'); + expect(r.messages[1].speaker).toBe('Assistant'); + expect(r.messages[1].text).toBe( + 'The capital of France is Paris.\nIt is also its largest city.', + ); + }); + + test('does not mistake an ordinary ## Summary heading for a turn', () => { + const body = ['## Summary', 'This is not a speaker turn.'].join('\n'); + const r = parseConversation(body, { fallbackDate: '2026-08-11' }); + expect(r.matched_pattern_id).not.toBe('markdown-heading-turn'); + }); +}); + describe('parseConversation — multi-line continuation (D5)', () => { test('iMessage continuation absorbs orphan lines', () => { const body = [ diff --git a/test/extract-conversation-facts.test.ts b/test/extract-conversation-facts.test.ts index 2434ba0b3..2a494dc89 100644 --- a/test/extract-conversation-facts.test.ts +++ b/test/extract-conversation-facts.test.ts @@ -146,6 +146,24 @@ test('conversation-facts allowlist includes native iMessage page types (#2756)', expect(ALLOWED_TYPES).toContain('imessage-daily'); }); +test('parses a markdown-heading turn body (## User / ## Assistant)', () => { + const body = [ + '## User', + 'What is the capital of France?', + '## Assistant', + 'The capital of France is Paris.', + 'It is also its largest city.', + ].join('\n'); + const msgs = parseConversationMessages(body, { fallbackDate: '2026-08-11' }); + expect(msgs).toHaveLength(2); + expect(msgs[0].speaker).toBe('User'); + expect(msgs[0].text).toBe('What is the capital of France?'); + expect(msgs[1].speaker).toBe('Assistant'); + expect(msgs[1].text).toBe( + 'The capital of France is Paris.\nIt is also its largest city.', + ); +}); + // --------------------------------------------------------------------------- // splitIntoSegments — PR's 5 cases verbatim plus tuning regression. // ---------------------------------------------------------------------------