Files
MasaandClaude Opus 5 52389dbe5b fix(conversation-parser): add markdown-heading turn pattern (## User / ## Assistant) (#4005)
* fix(facts): make transcript pages facts-extraction eligible

`gbrain extract-conversation-facts`'s ALLOWED_TYPES allowlist omitted the
`transcript` page type, so gbrain's own nightly transcript-ingest pages
were silently skipped by both the CLI `--types` validation and the
`cycle.conversation_facts_backfill.types` config filter. Even with the
type allowed, the built-in conversation-parser had no pattern for the
`## User` / `## Assistant` markdown-heading turn shape that transcript
ingest writes into `compiled_truth`, so parsing would still yield 0
segments.

This PR makes an explicit decision: transcript pages ARE now
facts-extraction eligible. That is a real behavioral change (a new,
potentially large corpus starts flowing through the extraction +
segment-cost path), not a no-op bugfix — flagging it plainly rather than
padding out the change as narrower than it is.

Changes:
- `src/commands/extract-conversation-facts.ts`: add `'transcript'` to
  `ALLOWED_TYPES` / `ALLOWED_TYPE_ALIASES` (the single source of truth
  for this allowlist).
- `src/core/conversation-parser/builtins.ts`: add the `markdown-heading-turn`
  builtin pattern recognizing heading-only `## User` / `## Assistant` /
  `## Human` / `## System` lines as turn openers, with D5 continuation-line
  body absorption. `quick_reject` is deliberately scoped to the role-prefix
  (not a bare `#{2,3}` heading check) so a message body that happens to
  paste unrelated markdown headings doesn't starve the D18 scorer's
  anchor-candidate ratio.
- `src/commands/jobs.ts`, `src/commands/doctor.ts` (x2 checks),
  `src/commands/sources.ts`: these each carried their own hand-copied
  literal of the same allowed-types list (background-job type filter,
  `conversation_facts_backlog` doctor check, `conversation_format_coverage`
  doctor check, `facts_backfill_estimate`). Switched each to import
  `ALLOWED_TYPES` from the command module instead of re-listing it, so this
  class of drift (a type added in one place, silently excluded everywhere
  else) can't recur.
- `docs/architecture/KEY_FILES.md`: updated the two stale mentions (pattern
  count 17→18, allowlist list) to current-state per this repo's own
  reference-doc convention.

Known limitation (not fixed here, scope-bounded intentionally): parsing is
context-free, same as every other multi-line builtin in this registry — a
message body that contains a literal `## User` line (e.g. someone pasting
a markdown transcript excerpt into their own message) would be read as a
turn boundary. This is a pre-existing property of the whole parser
(`applyPattern`'s per-line scan has no fence-awareness), not something
this PR introduces or could fix without a much larger, separate change to
the shared orchestrator affecting all 18 patterns. Flagging it here rather
than silently shipping the same limitation as the other 17 builtins.

Tests: 4 new tests (2 in test/extract-conversation-facts.test.ts, 2 in
test/conversation-parser/parse.test.ts) covering the allowlist, the new
pattern's positive match + continuation absorption, and that ordinary
`## Summary`-style headings are correctly rejected. Full targeted suite
(conversation-parser + facts-extraction + doctor backlog + build-llms
freshness): 263 pass / 0 fail. typecheck clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0126D3zLWL5RE3CVxnPANiiU

* fix(facts): read the type allowlist from core, not the command module

CI caught this: the known-flags registry drifted for doctor, sources, and
repos. The obvious remedy the guard prints -- regenerate and commit -- would
have been a regression, so this takes the other route.

The generator walks one level of a command module's relative imports and
harvests every flag-shaped string it finds, help text included, and is
deliberately over-inclusive. Importing extract-conversation-facts.ts just to
read ALLOWED_TYPES therefore spliced that command's entire flag vocabulary
(--types, --sleep, --slug, --segment-limit, --override-disabled, ...) into
the allowlists of three commands that implement none of it: `gbrain doctor
--types foo` would have passed validation and been silently ignored. That is
the exact defect class #2185 exists to close.

(jobs.ts is unaffected: it already imported the command module on one line
for runExtractConversationFactsCore, so those flags were already in its
registry entry before this branch.)

ALLOWED_TYPES + ALLOWED_TYPE_ALIASES now live in
src/core/conversation-facts-types.ts, a constants-only module with no CLI
text to harvest. extract-conversation-facts.ts re-exports both so its
existing importers are unchanged.

Verified: registry regenerates to zero drift (was doctor/repos/sources),
cli-flag-validation 24 pass, typecheck clean, 287 pass across the touched
areas. Confirmed against a clean upstream/master worktree that the drift was
introduced by this branch and is not pre-existing.

* fix(conversation-parser): reduce to the parser pattern only

Withdraws the `transcript` allowlist half of this branch. The premise was
wrong: `transcript` is not an upstream page type. `ALL_PAGE_TYPES` does not
contain it, `gbrain-base.yaml` declares `conversation` for "long-running
chat/transcript pages" and marks it `extractable: true` precisely so
extract-conversation-facts walks it, and `gbrain-base-v2.yaml` lists
`transcript` as an alias of `source` (a media primitive). Pages typed
`transcript` are a convention of my own ingest pipeline, not something
upstream produces — the fix for that belongs on my side, by emitting
`conversation`.

That takes the four call-site de-duplications with it (they existed only to
keep the allowlist in sync), and with them the flag-registry drift: no
imports are added, so the registry regenerates to zero drift with no
constants module needed.

What remains is the half that stands on its own: a `conversation` page whose
body uses `## User` / `## Assistant` headings matches none of the 17 builtins
and parses to 0 segments. `markdown-heading-turn` is an 18th pattern in the
same shape as the iMessage/Circleback additions before it.

Verified: typecheck clean, 181 pass / 0 fail across the parser, extraction,
flag-registry and llms-freshness suites, registry drift zero.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 04:51:32 -07:00

987 lines
39 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* v0.41.16.0 — Conversation parser orchestrator tests.
*
* Covers:
* - PR #1461's 6 telegram-bracket cases verbatim (REGRESSION pin)
* - All built-in patterns hit their test_positive samples
* - Date derivation precedence (D8)
* - Pattern priority scoring (D18) — overlap resolution
* - Quick-reject fast path (D11)
* - Multi-line continuation (D5)
* - Disabled-builtin honored
* - Timezone warning (D19) emitted when frontmatter timezone missing
*
* Pure-function tests; no PGLite, no LLM. The LLM polish/fallback
* tests live in `llm-base.test.ts`, `llm-polish.test.ts`,
* `llm-fallback.test.ts` (T4).
*/
import { describe, expect, test } from 'bun:test';
import {
parseConversation,
deriveDateContext,
applyPattern,
scorePattern,
scorePatternFull,
} from '../../src/core/conversation-parser/parse.ts';
import {
BUILTIN_PATTERNS,
validatePatternEntry,
} from '../../src/core/conversation-parser/builtins.ts';
import type { Page } from '../../src/core/types.ts';
// Helper to construct a minimal Page for date-derivation tests.
function makePage(
frontmatter: Record<string, unknown> = {},
effective_date?: Date,
): Page {
return {
id: 1,
slug: 'test/page',
type: 'conversation',
title: 'Test',
compiled_truth: '',
timeline: '',
frontmatter,
content_hash: undefined,
created_at: new Date(),
updated_at: new Date(),
effective_date: effective_date ?? null,
} as Page;
}
// ---------------------------------------------------------------------------
// REGRESSION: PR #1461's 6 telegram-bracket cases verbatim
// ---------------------------------------------------------------------------
describe('parseConversation — REGRESSION PR #1461 (telegram-bracket)', () => {
test('bracket-time with 👤 emoji speaker prefix', () => {
const body = '**[18:37] \u{1f464} G T:** hello world';
const r = parseConversation(body, { fallbackDate: '2026-05-24' });
expect(r.messages).toHaveLength(1);
expect(r.messages[0].speaker).toBe('G T');
expect(r.messages[0].text).toBe('hello world');
expect(r.messages[0].timestamp).toBe('2026-05-24T18:37:00Z');
expect(r.matched_pattern_id).toBe('telegram-bracket');
});
test('bracket-time with 🤖 robot emoji', () => {
const body = '**[06:00] \u{1f916} Zion:** On it.';
const r = parseConversation(body, { fallbackDate: '2026-05-25' });
expect(r.messages).toHaveLength(1);
expect(r.messages[0].speaker).toBe('Zion');
expect(r.messages[0].text).toBe('On it.');
expect(r.messages[0].timestamp).toBe('2026-05-25T06:00:00Z');
});
test('bracket-time multi-line continuation', () => {
const body = [
'**[09:00] \u{1f464} Alice Example:** first line',
'second line of same message',
'**[09:05] \u{1f464} Bob Example:** separate message',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-05-20' });
expect(r.messages).toHaveLength(2);
expect(r.messages[0].text).toBe('first line\nsecond line of same message');
expect(r.messages[1].text).toBe('separate message');
});
test('bracket-time falls back to 1970-01-01 without fallbackDate', () => {
const body = '**[14:30] \u{1f464} Alice Example:** test';
const r = parseConversation(body);
expect(r.messages).toHaveLength(1);
expect(r.messages[0].timestamp).toBe('1970-01-01T14:30:00Z');
});
test('mixed formats in one body: iMessage + bracket-time', () => {
const body = [
'**Alice Example** (2024-03-15 9:00 AM): format 1',
'**[10:30] \u{1f464} Bob Example:** format 2',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
// D18 scoring picks the dominant pattern. Both have one hit; ties
// resolve to declared priority (imessage-slack=0, telegram-bracket=1).
// The imessage line matches; the telegram line becomes a
// continuation. This is a known D18 tradeoff for very-mixed bodies.
// In practice every chat export is homogeneous, so this is a
// degenerate test case.
expect(r.messages.length).toBeGreaterThanOrEqual(1);
expect(r.matched_pattern_id).toBe('imessage-slack');
});
test('bracket-time without emoji prefix', () => {
const body = '**[22:15] Plain Name:** no emoji';
const r = parseConversation(body, { fallbackDate: '2026-01-01' });
expect(r.messages).toHaveLength(1);
expect(r.messages[0].speaker).toBe('Plain Name');
expect(r.messages[0].text).toBe('no emoji');
});
});
// ---------------------------------------------------------------------------
// All built-ins must parse their test_positive samples
// ---------------------------------------------------------------------------
describe('parseConversation — every built-in matches its test_positive sample', () => {
for (const entry of BUILTIN_PATTERNS) {
test(`pattern ${entry.id}: first test_positive parses`, () => {
const body = entry.test_positive[0];
// Multi-line patterns need a body line on the next line.
const fullBody =
entry.multi_line && entry.captures.text_group === 0
? `${body}\nsome body text`
: body;
const r = parseConversation(fullBody, {
fallbackDate: '2024-03-15',
});
// Either matches a message OR (for some multi-line patterns) the
// first line is the anchor and the body is consumed as text.
expect(r.messages.length).toBeGreaterThanOrEqual(1);
expect(r.matched_pattern_id).toBe(entry.id);
});
}
});
test('validatePatternEntry rejects invalid capture indexes', () => {
const base = BUILTIN_PATTERNS[0];
const aboveRange = {
...base,
id: 'invalid-text-capture',
captures: { ...base.captures, text_group: 99 },
};
const zeroSpeaker = {
...base,
id: 'invalid-speaker-capture',
captures: { ...base.captures, speaker_group: 0 },
};
const negativeText = {
...base,
id: 'negative-text-capture',
captures: { ...base.captures, text_group: -1 },
};
expect(() => validatePatternEntry(aboveRange)).toThrow(
"captures group 99 but regex only emits",
);
expect(() => validatePatternEntry(zeroSpeaker)).toThrow(
'speaker_group must be an integer >= 1',
);
expect(() => validatePatternEntry(negativeText)).toThrow(
'text_group must be an integer >= 0',
);
});
// ---------------------------------------------------------------------------
// Date derivation precedence (D8)
// ---------------------------------------------------------------------------
describe('deriveDateContext (D8 precedence chain)', () => {
test('explicit fallbackDate wins', () => {
const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15'));
const ctx = deriveDateContext({ fallbackDate: '2025-12-25', page });
expect(ctx.fallbackDate).toBe('2025-12-25');
expect(ctx.source).toBe('explicit');
});
test('frontmatter.date wins over effective_date', () => {
const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15'));
const ctx = deriveDateContext({ page });
expect(ctx.fallbackDate).toBe('2024-01-01');
expect(ctx.source).toBe('frontmatter_date');
});
test('effective_date wins when no frontmatter.date', () => {
const page = makePage({}, new Date('2023-06-15T00:00:00Z'));
const ctx = deriveDateContext({ page });
expect(ctx.fallbackDate).toBe('2023-06-15');
expect(ctx.source).toBe('effective_date');
});
test('epoch_default when nothing set', () => {
const ctx = deriveDateContext({});
expect(ctx.fallbackDate).toBe('1970-01-01');
expect(ctx.source).toBe('epoch_default');
});
test('frontmatter.timezone surfaces', () => {
const page = makePage({
date: '2024-01-01',
timezone: 'America/Los_Angeles',
});
const ctx = deriveDateContext({ page });
expect(ctx.timezone).toBe('America/Los_Angeles');
});
test('invalid frontmatter.date falls through', () => {
const page = makePage({ date: 'not-a-date' });
const ctx = deriveDateContext({ page });
expect(ctx.source).toBe('epoch_default');
});
test('frontmatter.date slices full ISO to YYYY-MM-DD', () => {
const page = makePage({ date: '2024-03-15T18:37:00Z' });
const ctx = deriveDateContext({ page });
expect(ctx.fallbackDate).toBe('2024-03-15');
});
});
// ---------------------------------------------------------------------------
// Pattern priority scoring (D18)
// ---------------------------------------------------------------------------
describe('scorePattern (D18 priority scoring)', () => {
test('telegram-bracket scores 1.0 on a pure telegram body', () => {
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
'**[18:39] \u{1f464} Alice:** three',
].join('\n');
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
expect(scorePattern(body, tg)).toBe(1);
});
test('imessage-slack scores 0 on pure telegram body', () => {
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
].join('\n');
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
expect(scorePattern(body, im)).toBe(0);
});
test('mixed body: D18 picks the higher-scoring pattern', () => {
// 3 telegram lines + 1 imessage line. Telegram should win.
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
'**[18:39] \u{1f464} Alice:** three',
'**Charlie** (2024-03-15 9:00 AM): one imessage',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(r.matched_pattern_id).toBe('telegram-bracket');
});
});
// ---------------------------------------------------------------------------
// Disabled-builtin honored
// ---------------------------------------------------------------------------
describe('parseConversation — disabledBuiltinIds', () => {
test('disabling top pattern falls through to next', () => {
const body = '**[18:37] \u{1f464} Alice:** hello';
const rDefault = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(rDefault.matched_pattern_id).toBe('telegram-bracket');
const rDisabled = parseConversation(body, {
fallbackDate: '2024-03-15',
disabledBuiltinIds: ['telegram-bracket'],
});
// No other built-in matches this exact shape → no_match.
expect(rDisabled.phase).toBe('no_match');
});
});
// ---------------------------------------------------------------------------
// Multi-line continuation (D5)
// ---------------------------------------------------------------------------
describe('parseConversation — markdown-heading-turn (gbrain transcript ingest)', () => {
test('parses ## User / ## Assistant heading-only turns with continuation body', () => {
const body = [
'## User',
'What is the capital of France?',
'## Assistant',
'The capital of France is Paris.',
'It is also its largest city.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-08-11' });
expect(r.matched_pattern_id).toBe('markdown-heading-turn');
expect(r.messages).toHaveLength(2);
expect(r.messages[0].speaker).toBe('User');
expect(r.messages[0].text).toBe('What is the capital of France?');
expect(r.messages[1].speaker).toBe('Assistant');
expect(r.messages[1].text).toBe(
'The capital of France is Paris.\nIt is also its largest city.',
);
});
test('does not mistake an ordinary ## Summary heading for a turn', () => {
const body = ['## Summary', 'This is not a speaker turn.'].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-08-11' });
expect(r.matched_pattern_id).not.toBe('markdown-heading-turn');
});
});
describe('parseConversation — multi-line continuation (D5)', () => {
test('iMessage continuation absorbs orphan lines', () => {
const body = [
'**Alice Example** (2024-03-15 9:00 AM): first line',
'continuation line',
'another continuation',
'**Bob Example** (2024-03-15 9:05 AM): second message',
].join('\n');
const r = parseConversation(body);
expect(r.messages).toHaveLength(2);
expect(r.messages[0].text).toBe(
'first line\ncontinuation line\nanother continuation',
);
expect(r.messages[1].text).toBe('second message');
});
});
describe('parseConversation — iMessage time-only 12h and date headings (#2756)', () => {
test('parses the time-only 12-hour iMessage shape', () => {
const r = parseConversation('**Alice Example** (9:04 PM): hello', {
fallbackDate: '2024-03-15',
});
expect(r.matched_pattern_id).toBe('bold-paren-time-12h');
expect(r.messages).toHaveLength(1);
expect(r.messages[0].timestamp).toBe('2024-03-15T21:04:00Z');
});
test('markdown date headings advance the running date without becoming message text', () => {
const body = [
'## 2024-03-15',
'**Alice Example** (9:04 AM): first day',
'## 2024-03-16',
'**Bob Example** (10:05 PM): second day',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2024-03-01' });
expect(r.matched_pattern_id).toBe('bold-paren-time-12h');
expect(r.messages.map((m) => m.timestamp)).toEqual([
'2024-03-15T09:04:00Z',
'2024-03-16T22:05:00Z',
]);
expect(r.messages[0].text).toBe('first day');
});
test('date headings do not mutate the caller-provided context', () => {
const ctx = { fallbackDate: '2024-03-01', source: 'explicit' as const };
const pattern = BUILTIN_PATTERNS.find((p) => p.id === 'bold-paren-time-12h')!;
applyPattern('## 2024-03-16\n**Alice** (9:04 AM): hello', pattern, ctx);
expect(ctx.fallbackDate).toBe('2024-03-01');
});
});
// ---------------------------------------------------------------------------
// Timezone warning (D19)
// ---------------------------------------------------------------------------
describe('parseConversation — timezone warning (D19)', () => {
test('telegram-bracket emits warning when no timezone in frontmatter', () => {
const body = '**[18:37] \u{1f464} Alice:** hello';
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(r.timezone_warning).toBeDefined();
expect(r.timezone_warning).toContain('telegram-bracket');
expect(r.timezone_warning).toContain('UTC');
});
test('telegram-bracket does NOT warn when timezone is present', () => {
const body = '**[18:37] \u{1f464} Alice:** hello';
const page = makePage({
date: '2024-03-15',
timezone: 'America/Los_Angeles',
});
const r = parseConversation(body, { page });
expect(r.timezone_warning).toBeUndefined();
});
test('imessage-slack does NOT warn (inline_utc policy)', () => {
const body = '**Alice Example** (2024-03-15 9:00 AM): hello';
const r = parseConversation(body);
expect(r.timezone_warning).toBeUndefined();
});
});
// ---------------------------------------------------------------------------
// Empty body + degenerate cases
// ---------------------------------------------------------------------------
describe('parseConversation — degenerate inputs', () => {
test('empty body returns no_match + empty messages', () => {
expect(parseConversation('')).toEqual({
messages: [],
phase: 'no_match',
});
});
test('non-conversational text returns no_match', () => {
const r = parseConversation('This is just prose with no chat shape.');
expect(r.phase).toBe('no_match');
expect(r.messages).toHaveLength(0);
});
});
// ---------------------------------------------------------------------------
// applyPattern — direct unit tests for the matcher
// ---------------------------------------------------------------------------
describe('applyPattern — quick_reject fast path (D11)', () => {
test('telegram quick_reject skips iMessage lines fast', () => {
const body = '**Alice Example** (2024-03-15 9:00 AM): hello';
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
const r = applyPattern(body, tg, {
fallbackDate: '2024-03-15',
source: 'explicit',
});
// Quick_reject /^\*\*\[/ rejects '**Alice' (no `[`). Zero matches.
expect(r).toHaveLength(0);
});
});
// ---------------------------------------------------------------------------
// scorePattern boundary cases
// ---------------------------------------------------------------------------
describe('scorePattern — boundary', () => {
test('empty body scores 0', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
expect(scorePattern('', tg)).toBe(0);
});
test('only blank lines scores 0', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
expect(scorePattern('\n\n \n', tg)).toBe(0);
});
// T5 reshape (Codex P2 #6): pins BEHAVIOR not the constant value.
// The prior test ("100 matching lines score 1.0") would pass with
// head=10 or head=1000 — it didn't prove anything about the cap.
test('head cap ignores lines past line 10 (10 match + 1 non-match scores 1.0)', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
const matching = Array.from(
{ length: 10 },
(_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`,
);
const body = [...matching, 'plain text outside the head window'].join('\n');
// First 10 lines all match → 10/10. Line 11 was ignored.
expect(scorePattern(body, tg)).toBe(1);
});
test('head cap stops at line 10 (9 non-match + 1 match at line 10 + 100 match after scores 0.1)', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
const nonMatches = Array.from({ length: 9 }, (_, i) => `non-matching prose line ${i}`);
const matchingLate = Array.from(
{ length: 100 },
(_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`,
);
// Line 10 (index 9 in the matching array) IS a match; lines 11-109 are too
// but are past the head cap and don't count.
const body = [...nonMatches, matchingLate[0], ...matchingLate.slice(1)].join('\n');
// Head sees 9 non-matches + 1 match = 1/10 = 0.1. Pre-fix: same result.
// Post-fix: same result (this test pins head-cap behavior, not the new
// fallback path — that's tested separately below).
expect(scorePattern(body, tg)).toBe(0.1);
});
});
// ---------------------------------------------------------------------------
// scorePatternFull — direct unit tests (v0.41.18+ T3 #5)
// ---------------------------------------------------------------------------
describe('scorePatternFull — full-body scoring (v0.41.18+ Codex P1 #1)', () => {
test('empty body scores 0', () => {
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
expect(scorePatternFull('', im)).toBe(0);
});
test('preamble + 20 matching lines scores 20/(preamble + 20)', () => {
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
const preamble = ['## Summary', 'Three sentences.', '> Source: ref', '## Transcript'];
const matches = Array.from(
{ length: 20 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): message ${i}`,
);
const body = [...preamble, ...matches].join('\n');
// 24 total non-blank, 20 match → 20/24 ≈ 0.833
expect(scorePatternFull(body, im)).toBeCloseTo(20 / 24, 5);
});
test('preamble-only-no-match scores 0', () => {
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
const body = '## Summary\nProse paragraph.\n> Blockquote\n## Heading';
expect(scorePatternFull(body, im)).toBe(0);
});
});
// ---------------------------------------------------------------------------
// bold-paren-time pattern (v0.41.18+ D-FOLLOWUP-1.B; closes user-facing
// half of #1533 — the 112 Circleback meeting files at
// ~/git/brain/meetings/*.md with `source: circleback` frontmatter)
// ---------------------------------------------------------------------------
describe('bold-paren-time pattern (Circleback meeting transcripts)', () => {
test('matches **Speaker** (HH:MM): text with frontmatter date', () => {
const body = [
'**Alice Example** (00:00): Hey, can you hear me?',
'**Participant 2** (02:22): Yeah, just joined.',
'**Alice Example** (15:09): That makes sense.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.messages).toHaveLength(3);
expect(r.messages[0]).toEqual({
speaker: 'Alice Example',
timestamp: '2026-03-19T00:00:00Z',
text: 'Hey, can you hear me?',
});
expect(r.messages[2]).toEqual({
speaker: 'Alice Example',
timestamp: '2026-03-19T15:09:00Z',
text: 'That makes sense.',
});
});
test('matches **Speaker** (HH:MM:SS): text shape (Circleback seconds variant)', () => {
const body = [
'**Participant 1** (00:00:00): opening line',
'**Participant 2** (00:00:19): quick reply',
'**Participant 1** (01:23:45): later in the meeting',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-04-01' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.messages).toHaveLength(3);
// Seconds segment is non-capturing; minute_group still captures the
// minutes component. Time-format is wall-clock 24h on frontmatter date.
expect(r.messages[0].timestamp).toBe('2026-04-01T00:00:00Z');
expect(r.messages[1].timestamp).toBe('2026-04-01T00:00:00Z');
expect(r.messages[2].timestamp).toBe('2026-04-01T01:23:00Z');
});
test('imessage-slack shape still wins over bold-paren-time on overlap', () => {
// Both patterns start with `**` and have parens. The imessage-
// slack regex requires a full date+time inside; bold-paren-time
// requires just `(HH:MM)`. The dates-with-AM/PM shape MUST fall
// through to imessage-slack, not bold-paren-time.
const body = '**Alice Example** (2024-03-15 9:00 AM): hello world';
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(1);
expect(r.messages[0].text).toBe('hello world');
});
test('meeting page with preamble + bold-paren-time transcript hits fallback', () => {
// Real Circleback shape: ## Summary + blockquote + ## Transcript
// before the bold-paren-time chat. Same fallback gate that
// closes #1533 must work for this pattern too.
const preamble = [
'## Summary',
'Meeting covered Q1 roadmap discussion.',
'> Source: circleback meeting #7411053',
'## Topics Discussed',
'- Roadmap',
'- Hiring',
'## Transcript',
];
const transcript = Array.from(
{ length: 20 },
(_, i) => `**Participant 2** (${String(Math.floor(i / 6)).padStart(2, '0')}:${String((i * 11) % 60).padStart(2, '0')}): transcript line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.messages).toHaveLength(20);
});
});
// ---------------------------------------------------------------------------
// bold-time-dash pattern (normalized Slack Markdown)
// ---------------------------------------------------------------------------
describe('bold-time-dash pattern (normalized Slack Markdown)', () => {
test('parses anchors, dash variants, and multi-line continuation text', () => {
const body = [
'# Team channel — 2026-04-09',
'**Alice Example** 09:15 — first line',
'- detailed bullet one',
'- detailed bullet two',
'**Summary Bot** 09:18 second message',
'> continuation of second message',
'**Bob Example** 10:01 - final message',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-04-09' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-time-dash');
expect(r.messages).toHaveLength(3);
expect(r.messages[0]).toEqual({
speaker: 'Alice Example',
timestamp: '2026-04-09T09:15:00Z',
text: 'first line\n- detailed bullet one\n- detailed bullet two',
});
expect(r.messages[1]).toEqual({
speaker: 'Summary Bot',
timestamp: '2026-04-09T09:18:00Z',
text: 'second message\n> continuation of second message',
});
expect(r.messages[2]).toEqual({
speaker: 'Bob Example',
timestamp: '2026-04-09T10:01:00Z',
text: 'final message',
});
});
test('parses one anchor with a long Markdown continuation body', () => {
const continuation = Array.from(
{ length: 30 },
(_, index) => `- supporting detail ${index + 1}`,
);
const body = [
'**Alice Example** 09:15 — summary',
...continuation,
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-04-09' });
expect(r.matched_pattern_id).toBe('bold-time-dash');
expect(r.messages).toHaveLength(1);
expect(r.messages[0].text.split('\n')).toHaveLength(31);
expect(r.messages[0].text.endsWith('- supporting detail 30')).toBe(true);
});
test('does not treat one stray anchor in long prose as a conversation', () => {
const before = Array.from(
{ length: 150 },
(_, index) => `Prose paragraph before ${index + 1}.`,
);
const after = Array.from(
{ length: 150 },
(_, index) => `Prose paragraph after ${index + 1}.`,
);
const body = [
...before,
'**Deadline** 09:15 — quoted schedule entry',
...after,
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-04-09' });
expect(r.phase).toBe('no_match');
expect(r.messages).toEqual([]);
});
test('uses date headings to advance the frontmatter date anchor', () => {
const body = [
'## 2026-04-09',
'**Alice Example** 23:59 — day one',
'## 2026-04-10',
'**Bob Example** 00:01 — day two',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-04-09' });
expect(r.matched_pattern_id).toBe('bold-time-dash');
expect(r.messages.map((message) => message.timestamp)).toEqual([
'2026-04-09T23:59:00Z',
'2026-04-10T00:01:00Z',
]);
});
test('uses page date and preserves the time-only timezone policy', () => {
const body = '**Alice Example** 09:15 — hello';
const withoutTimezone = parseConversation(body, {
page: makePage({ date: '2026-04-09' }),
});
const withTimezone = parseConversation(body, {
page: makePage({
date: '2026-04-09',
timezone: 'America/Los_Angeles',
}),
});
expect(withoutTimezone.messages[0].timestamp).toBe(
'2026-04-09T09:15:00Z',
);
expect(withoutTimezone.timezone_warning).toContain('bold-time-dash');
// Current time-only policy records the captured wall-clock fields with Z;
// timezone metadata suppresses the warning but does not convert the time.
expect(withTimezone.messages[0].timestamp).toBe('2026-04-09T09:15:00Z');
expect(withTimezone.timezone_warning).toBeUndefined();
});
test('does not shadow existing bold transcript formats', () => {
const opts = { fallbackDate: '2026-04-09' };
expect(
parseConversation('**Alice Example** (00:00): hello', opts)
.matched_pattern_id,
).toBe('bold-paren-time');
expect(
parseConversation('**Alice Example** (9:15 AM): hello', opts)
.matched_pattern_id,
).toBe('bold-paren-time-12h');
expect(
parseConversation('**Alice Example:** hello', opts).matched_pattern_id,
).toBe('bold-name-no-time');
expect(
parseConversation(
'**Alice Example** (2026-04-09 9:15 AM): hello',
opts,
).matched_pattern_id,
).toBe('imessage-slack');
});
test('rejects invalid 24-hour times', () => {
const body = [
'**Alice Example** 24:00 — invalid hour',
'**Bob Example** 09:60 — invalid minute',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-04-09' });
expect(r.phase).toBe('no_match');
expect(r.messages).toEqual([]);
});
});
// ---------------------------------------------------------------------------
// bold-name-no-time pattern (Circleback / Granola / Zoom transcripts with NO
// per-line timestamp — `**Speaker:** text`). Additive pattern; the colon
// inside the bold markers + the `(?!\[)` lookahead are what keep it from
// shadowing bold-paren-time / telegram-bracket (NOT declaration order).
// ---------------------------------------------------------------------------
describe('bold-name-no-time pattern (Circleback/Granola/Zoom, no timestamp)', () => {
test('parses **Speaker:** text transcript with frontmatter date anchor', () => {
const body = [
'**Alice Example:** Okay, start on. And then weirdly like zoom doesnt...',
'**Participant 2:** he tried to reset it remotely the other night. Let me ask him.',
'**Alice Example:** I mean its really just like we need to get zoom to fix this.',
'**Participant 2:** Okay, let me.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-name-no-time');
expect(r.messages).toHaveLength(4);
expect(r.messages[0]).toEqual({
speaker: 'Alice Example',
timestamp: '2026-05-28T00:00:00Z',
text: 'Okay, start on. And then weirdly like zoom doesnt...',
});
expect(r.messages[1].speaker).toBe('Participant 2');
expect(r.messages[1].text).toBe(
'he tried to reset it remotely the other night. Let me ask him.',
);
expect(r.messages[3]).toEqual({
speaker: 'Participant 2',
timestamp: '2026-05-28T00:00:00Z',
text: 'Okay, let me.',
});
// No-time pattern anchors at 00:00:00 of the frontmatter date
// (same convention as irc-classic). No wall-clock time fabricated.
});
test('scores above the 0.05 floor on a pure bold-name transcript (epoch default)', () => {
const body = [
'**Alice Example:** line one',
'**Participant 2:** line two',
'**Alice Example:** line three',
].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-name-no-time');
expect(r.messages).toHaveLength(3);
// No fallbackDate → epoch default, still anchors at 00:00:00.
expect(r.messages[0].timestamp).toBe('1970-01-01T00:00:00Z');
});
// REGRESSION: must NOT shadow bold-paren-time. A `**Name** (00:00): text`
// line has its colon OUTSIDE the bold markers, so it must still parse via
// bold-paren-time (the safety is the regex, not declaration order).
test('REGRESSION: **Speaker** (HH:MM): text still matches bold-paren-time', () => {
const body = [
'**Alice Example** (00:00): Hey, can you hear me?',
'**Participant 2** (02:22): Yeah, just joined.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.matched_pattern_id).not.toBe('bold-name-no-time');
expect(r.messages).toHaveLength(2);
expect(r.messages[0].timestamp).toBe('2026-03-19T00:00:00Z');
});
// REGRESSION: a pure telegram-bracket transcript has the colon inside the
// bold markers BUT the speaker starts with `[`, so the `(?!\[)` lookahead
// rejects it AND telegram-bracket (lower declaration index) wins anyway.
test('REGRESSION: telegram-bracket transcript still matches telegram-bracket', () => {
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(r.matched_pattern_id).toBe('telegram-bracket');
});
// F1 (Codex): the broad regex matches any `**Label:** text`. A prose notes
// page with bold labels CLUSTERED in its first 10 lines scores 0.3 on the
// head pass (3/10) — NOT < SCORING_HEAD_TRIGGER_THRESHOLD (0.3), so the
// full-body fallback never fires — and that 0.3 clears the 0.05 floor.
// Without score_full_body this page would mis-parse as a conversation.
// score_full_body recomputes the winner over the FULL body: 3 labels /
// 83 lines ≈ 0.036 < 0.05 → no_match. This is the exact bug Codex caught
// and the test the naive "0.05 floor protects us" assumption would fail.
test('F1: prose with bold labels clustered in the head returns no_match', () => {
const lines = [
'**Attendees:** Alice Example, Bob Example, Participant 2',
'**Date:** 2026-05-28',
'**Goal:** decide on the Q3 roadmap and unblock the vendor migration',
];
// 80 plain prose lines (no bold-label shape) → 83 total, only 3 match.
// First 10 lines = 3 labels + 7 prose → head score 0.3 (skips rescore).
// Full body = 3/83 ≈ 0.036 < 0.05 floor.
for (let i = 0; i < 80; i++) {
lines.push(
`This is an ordinary prose sentence number ${i} describing the meeting in detail.`,
);
}
const body = lines.join('\n');
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
expect(r.phase).toBe('no_match');
});
});
describe('speaker-letter-no-time pattern (raw transcript sidecars)', () => {
test('parses plain Speaker A / Speaker B transcripts', () => {
const body = [
'Speaker A: That is exactly the issue.',
'Speaker B: Yeah, I know.',
'Speaker A: Let me ask him.',
'Speaker B: Sounds good.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-06-01' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('speaker-letter-no-time');
expect(r.messages).toHaveLength(4);
expect(r.messages[0]).toEqual({
speaker: 'Speaker A',
timestamp: '2026-06-01T00:00:00Z',
text: 'That is exactly the issue.',
});
expect(r.messages[1].speaker).toBe('Speaker B');
expect(r.messages[3].text).toBe('Sounds good.');
});
test('does not parse ordinary prose labels as transcript lines', () => {
const body = [
'Owner: Elliot',
'Decision: Ship the parser fix',
'Next step: rerun extraction',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-06-01' });
expect(r.phase).toBe('no_match');
});
});
// ---------------------------------------------------------------------------
// parseConversation — full-body fallback (v0.41.18+ #1533 + Codex P1 #1, #2, #8)
// ---------------------------------------------------------------------------
describe('parseConversation — full-body fallback', () => {
// T3 #1: IRON-RULE regression pin for #1533. Pre-fix this returns
// no_match because head 10 sees only preamble.
test('#1533: meeting page with ## Summary + blockquote + ## Transcript before chat hits fallback', () => {
const preamble = [
'## Summary',
'This meeting covered Q1 roadmap discussion.',
'Three engineers participated in the call.',
'Action items were captured during the conversation.',
'> Source: [meeting recording](https://example.com/rec/123)',
'## Topics Discussed',
'- Product roadmap for Q1',
'- Engineering team allocation',
'- Customer feedback synthesis',
'## Transcript',
];
const transcript = Array.from(
{ length: 20 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(20);
});
// T3 #2: diagnostic now reports total_non_blank - matched, not total.
test('#1533: unmatched_line_count subtracts matched messages after fallback', () => {
const preamble = [
'## Summary',
'Prose A.',
'Prose B.',
'> Blockquote',
'## Transcript',
];
const transcript = Array.from(
{ length: 20 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body, { diagnostic: true });
expect(r.phase).toBe('regex_match');
expect(r.unmatched_line_count).toBe(5); // 25 total non-blank - 20 messages = 5
});
// T3 #3: a 50-line essay with no chat shape stays no_match.
test('pure-prose 50-line essay stays no_match (fallback found nothing to anchor)', () => {
const body = Array.from(
{ length: 50 },
(_, i) => `This is the ${i + 1}th paragraph of a pure-prose article.`,
).join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('no_match');
expect(r.messages).toHaveLength(0);
});
// T3 #4: proves "full-body" not just "wider window" — 300-line preamble
// far exceeds any reasonable head-bump alternative.
test('300-line preamble + 50 chat lines hits fallback (any preamble length)', () => {
const preamble = Array.from(
{ length: 300 },
(_, i) => `Preamble paragraph ${i + 1} with prose content here.`,
);
const transcript = Array.from(
{ length: 50 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): chat line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(50);
});
// T3 #6 (Codex P1 #1 + #8): stray-head-match doesn't suppress fallback.
// Pre-fix: irc-classic 0.1 in head → no fallback → irc-classic wins with 1
// message. Post-fix: 0.1 < 0.3 trigger → fallback re-scores → imessage-slack
// wins (50/60 ≈ 0.83 vs irc-classic 1/60 ≈ 0.017).
test('Codex P1 #1: stray irc-classic match in head does not suppress fallback', () => {
const preamble = [
'## Meeting Notes',
'<presenter> Garry Tan opening remarks', // stray irc-classic match
'- agenda item 1',
'- agenda item 2',
'- agenda item 3',
'- agenda item 4',
'- agenda item 5',
'- agenda item 6',
'- agenda item 7',
'## Transcript',
];
const transcript = Array.from(
{ length: 50 },
(_, i) => `**Garry Tan** (2024-01-29 12:00 PM): real transcript line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
// The critical assertion: imessage-slack wins, NOT irc-classic.
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(50);
});
// T3 #7 (Codex P1 #2): essay with one stray chat-shape line stays
// no_match. 1/301 ≈ 0.003, below SCORING_MIN_ACCEPTANCE (0.05).
test('Codex P1 #2: 300-line essay with one stray chat line stays no_match (acceptance floor)', () => {
const prose = Array.from(
{ length: 150 },
(_, i) => `Essay paragraph ${i + 1} of pure prose with no chat shape.`,
);
const strayChatLine = '**Author Name** (2024-01-01 9:00 AM): stray quoted snippet';
const morePros = Array.from(
{ length: 150 },
(_, i) => `Essay continuation paragraph ${i + 151}.`,
);
const body = [...prose, strayChatLine, ...morePros].join('\n');
const r = parseConversation(body);
// Pre-fix: regex_match with messages.length === 1.
// Post-fix: no_match because 1/301 < 0.05 acceptance floor.
expect(r.phase).toBe('no_match');
expect(r.messages).toHaveLength(0);
});
});