fix(conversation-parser): add markdown-heading turn pattern (## User / ## Assistant) (#4005)

* fix(facts): make transcript pages facts-extraction eligible

`gbrain extract-conversation-facts`'s ALLOWED_TYPES allowlist omitted the
`transcript` page type, so gbrain's own nightly transcript-ingest pages
were silently skipped by both the CLI `--types` validation and the
`cycle.conversation_facts_backfill.types` config filter. Even with the
type allowed, the built-in conversation-parser had no pattern for the
`## User` / `## Assistant` markdown-heading turn shape that transcript
ingest writes into `compiled_truth`, so parsing would still yield 0
segments.

This PR makes an explicit decision: transcript pages ARE now
facts-extraction eligible. That is a real behavioral change (a new,
potentially large corpus starts flowing through the extraction +
segment-cost path), not a no-op bugfix — flagging it plainly rather than
padding out the change as narrower than it is.

Changes:
- `src/commands/extract-conversation-facts.ts`: add `'transcript'` to
  `ALLOWED_TYPES` / `ALLOWED_TYPE_ALIASES` (the single source of truth
  for this allowlist).
- `src/core/conversation-parser/builtins.ts`: add the `markdown-heading-turn`
  builtin pattern recognizing heading-only `## User` / `## Assistant` /
  `## Human` / `## System` lines as turn openers, with D5 continuation-line
  body absorption. `quick_reject` is deliberately scoped to the role-prefix
  (not a bare `#{2,3}` heading check) so a message body that happens to
  paste unrelated markdown headings doesn't starve the D18 scorer's
  anchor-candidate ratio.
- `src/commands/jobs.ts`, `src/commands/doctor.ts` (x2 checks),
  `src/commands/sources.ts`: these each carried their own hand-copied
  literal of the same allowed-types list (background-job type filter,
  `conversation_facts_backlog` doctor check, `conversation_format_coverage`
  doctor check, `facts_backfill_estimate`). Switched each to import
  `ALLOWED_TYPES` from the command module instead of re-listing it, so this
  class of drift (a type added in one place, silently excluded everywhere
  else) can't recur.
- `docs/architecture/KEY_FILES.md`: updated the two stale mentions (pattern
  count 17→18, allowlist list) to current-state per this repo's own
  reference-doc convention.

Known limitation (not fixed here, scope-bounded intentionally): parsing is
context-free, same as every other multi-line builtin in this registry — a
message body that contains a literal `## User` line (e.g. someone pasting
a markdown transcript excerpt into their own message) would be read as a
turn boundary. This is a pre-existing property of the whole parser
(`applyPattern`'s per-line scan has no fence-awareness), not something
this PR introduces or could fix without a much larger, separate change to
the shared orchestrator affecting all 18 patterns. Flagging it here rather
than silently shipping the same limitation as the other 17 builtins.

Tests: 4 new tests (2 in test/extract-conversation-facts.test.ts, 2 in
test/conversation-parser/parse.test.ts) covering the allowlist, the new
pattern's positive match + continuation absorption, and that ordinary
`## Summary`-style headings are correctly rejected. Full targeted suite
(conversation-parser + facts-extraction + doctor backlog + build-llms
freshness): 263 pass / 0 fail. typecheck clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126D3zLWL5RE3CVxnPANiiU

* fix(facts): read the type allowlist from core, not the command module

CI caught this: the known-flags registry drifted for doctor, sources, and
repos. The obvious remedy the guard prints -- regenerate and commit -- would
have been a regression, so this takes the other route.

The generator walks one level of a command module's relative imports and
harvests every flag-shaped string it finds, help text included, and is
deliberately over-inclusive. Importing extract-conversation-facts.ts just to
read ALLOWED_TYPES therefore spliced that command's entire flag vocabulary
(--types, --sleep, --slug, --segment-limit, --override-disabled, ...) into
the allowlists of three commands that implement none of it: `gbrain doctor
--types foo` would have passed validation and been silently ignored. That is
the exact defect class #2185 exists to close.

(jobs.ts is unaffected: it already imported the command module on one line
for runExtractConversationFactsCore, so those flags were already in its
registry entry before this branch.)

ALLOWED_TYPES + ALLOWED_TYPE_ALIASES now live in
src/core/conversation-facts-types.ts, a constants-only module with no CLI
text to harvest. extract-conversation-facts.ts re-exports both so its
existing importers are unchanged.

Verified: registry regenerates to zero drift (was doctor/repos/sources),
cli-flag-validation 24 pass, typecheck clean, 287 pass across the touched
areas. Confirmed against a clean upstream/master worktree that the drift was
introduced by this branch and is not pre-existing.

* fix(conversation-parser): reduce to the parser pattern only

Withdraws the `transcript` allowlist half of this branch. The premise was
wrong: `transcript` is not an upstream page type. `ALL_PAGE_TYPES` does not
contain it, `gbrain-base.yaml` declares `conversation` for "long-running
chat/transcript pages" and marks it `extractable: true` precisely so
extract-conversation-facts walks it, and `gbrain-base-v2.yaml` lists
`transcript` as an alias of `source` (a media primitive). Pages typed
`transcript` are a convention of my own ingest pipeline, not something
upstream produces — the fix for that belongs on my side, by emitting
`conversation`.

That takes the four call-site de-duplications with it (they existed only to
keep the allowlist in sync), and with them the flag-registry drift: no
imports are added, so the registry regenerates to zero drift with no
constants module needed.

What remains is the half that stands on its own: a `conversation` page whose
body uses `## User` / `## Assistant` headings matches none of the 17 builtins
and parses to 0 segments. `markdown-heading-turn` is an 18th pattern in the
same shape as the iMessage/Circleback additions before it.

Verified: typecheck clean, 181 pass / 0 fail across the parser, extraction,
flag-registry and llms-freshness suites, registry drift zero.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/core/conversation-parser/builtins.ts

@@ -1,7 +1,7 @@
 /**
  * v0.41.16.0 — Built-in conversation parser pattern registry.
  *
- * Seventeen hand-vetted patterns covering the chat-export formats this
+ * Eighteen hand-vetted patterns covering the chat-export formats this
  * codebase is most likely to encounter. Each pattern's regex was
  * derived from a public format reference (source_doc field) so future
  * maintainers can verify against the wild shape.
@@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string {
   return stripped || raw.trim();
 }
 
-/** The 17 hand-vetted built-in patterns. */
+/** The 18 hand-vetted built-in patterns. */
 export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
   // -------------------------------------------------------------------
   // INLINE-DATE patterns (date in every line; less ambiguous; tried first).
@@ -670,6 +670,46 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
     test_negative: ['<alice> classic irc, no time', '[18:37] @alice: matrix'],
     source_doc: 'weechat default logger.format `%H:%M %p\\t%m`',
   },
+
+  {
+    id: 'markdown-heading-turn',
+    origin: 'builtin',
+    // gbrain transcript-ingest shape: a heading-only line ('## User' /
+    // '## Assistant' / '### Human') opens a turn; the message text is
+    // the continuation lines below the heading (D5), not anything on
+    // the heading line itself. No per-line timestamps — date comes
+    // from frontmatter / effective_date. The speaker set is closed
+    // (User/Assistant/Human/System only) so ordinary section headings
+    // like '## Summary' never match, and a heading with trailing prose
+    // ('## User said hello') is rejected rather than mis-captured.
+    regex: /^#{2,3}\s+(User|Assistant|Human|System)\s*:?\s*()$/,
+    captures: {
+      speaker_group: 1,
+      text_group: 2,
+    },
+    date_source: 'frontmatter',
+    time_format: '24h',
+    timezone_policy: 'utc_assumed_with_warn',
+    multi_line: true,
+    score_continuations_as_body: true,
+    // Narrowed to a role-prefix superset (NOT bare `/^#{2,3}\s/`): a body
+    // that pastes unrelated markdown headings (e.g. a document with many
+    // '## Section' headings) would otherwise inflate the D18 scorer's
+    // anchor-candidate denominator without inflating the anchored count,
+    // starving the pattern's score toward 0 on otherwise-valid transcripts.
+    // Still a strict superset of `regex` per validatePatternEntry's
+    // invariant (every test_positive sample passes both).
+    quick_reject: /^#{2,3}\s+(?:User|Assistant|Human|System)\b/,
+    test_positive: ['## User', '## Assistant', '### Human', '## System', '## User:'],
+    test_negative: [
+      '## Summary',
+      '#### User',
+      'User: plain no heading',
+      '## User said hello',
+    ],
+    source_doc:
+      'gbrain nightly transcript ingest: compiled_truth bodies use markdown headings per turn',
+  },
 ];
 
 /**

src/core/conversation-parser/parse.ts

@@ -392,7 +392,7 @@ function getNonBlankLines(body: string, headCap?: number): string[] {
  * window) and `scorePatternFull` (whole body) delegate here so the
  * quick_reject + regex loop lives in one place. Reused by
  * `parseConversation`'s fallback path which pre-splits ONCE and
- * passes the array to all 17 candidates (saves 16 redundant body
+ * passes the array to all 18 candidates (saves 17 redundant body
  * splits per fallback pass).
  */
 function scoreFromLines(

test/conversation-parser/parse.test.ts

@@ -276,6 +276,33 @@ describe('parseConversation — disabledBuiltinIds', () => {
 // Multi-line continuation (D5)
 // ---------------------------------------------------------------------------
 
+describe('parseConversation — markdown-heading-turn (gbrain transcript ingest)', () => {
+  test('parses ## User / ## Assistant heading-only turns with continuation body', () => {
+    const body = [
+      '## User',
+      'What is the capital of France?',
+      '## Assistant',
+      'The capital of France is Paris.',
+      'It is also its largest city.',
+    ].join('\n');
+    const r = parseConversation(body, { fallbackDate: '2026-08-11' });
+    expect(r.matched_pattern_id).toBe('markdown-heading-turn');
+    expect(r.messages).toHaveLength(2);
+    expect(r.messages[0].speaker).toBe('User');
+    expect(r.messages[0].text).toBe('What is the capital of France?');
+    expect(r.messages[1].speaker).toBe('Assistant');
+    expect(r.messages[1].text).toBe(
+      'The capital of France is Paris.\nIt is also its largest city.',
+    );
+  });
+
+  test('does not mistake an ordinary ## Summary heading for a turn', () => {
+    const body = ['## Summary', 'This is not a speaker turn.'].join('\n');
+    const r = parseConversation(body, { fallbackDate: '2026-08-11' });
+    expect(r.matched_pattern_id).not.toBe('markdown-heading-turn');
+  });
+});
+
 describe('parseConversation — multi-line continuation (D5)', () => {
   test('iMessage continuation absorbs orphan lines', () => {
     const body = [

test/extract-conversation-facts.test.ts

@@ -146,6 +146,24 @@ test('conversation-facts allowlist includes native iMessage page types (#2756)',
   expect(ALLOWED_TYPES).toContain('imessage-daily');
 });
 
+test('parses a markdown-heading turn body (## User / ## Assistant)', () => {
+  const body = [
+    '## User',
+    'What is the capital of France?',
+    '## Assistant',
+    'The capital of France is Paris.',
+    'It is also its largest city.',
+  ].join('\n');
+  const msgs = parseConversationMessages(body, { fallbackDate: '2026-08-11' });
+  expect(msgs).toHaveLength(2);
+  expect(msgs[0].speaker).toBe('User');
+  expect(msgs[0].text).toBe('What is the capital of France?');
+  expect(msgs[1].speaker).toBe('Assistant');
+  expect(msgs[1].text).toBe(
+    'The capital of France is Paris.\nIt is also its largest city.',
+  );
+});
+
 // ---------------------------------------------------------------------------
 // splitIntoSegments — PR's 5 cases verbatim plus tuning regression.
 // ---------------------------------------------------------------------------