mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7e1c4f312f | ||
|
|
e5b3e7aba7 |
@@ -23,7 +23,7 @@ health_checks:
|
||||
label: "Auth provider"
|
||||
checks:
|
||||
- type: http
|
||||
url: "$CLAWVISOR_URL/health"
|
||||
url: "$CLAWVISOR_URL/ready"
|
||||
label: "ClawVisor"
|
||||
- type: env_exists
|
||||
name: GOOGLE_CLIENT_ID
|
||||
@@ -135,7 +135,7 @@ Tell the user:
|
||||
|
||||
Validate:
|
||||
```bash
|
||||
curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL"
|
||||
curl -sf "$CLAWVISOR_URL/ready" && echo "PASS: ClawVisor reachable" || echo "FAIL"
|
||||
```
|
||||
|
||||
**STOP until ClawVisor validates.**
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
---
|
||||
id: contacts-to-brain
|
||||
name: Contacts-to-Brain
|
||||
version: 0.1.0
|
||||
description: Google Contacts become canonical people/ pages, enriching brain entities with ground-truth name/email/phone/org data.
|
||||
category: sense
|
||||
requires: [credential-gateway]
|
||||
secrets:
|
||||
- name: CLAWVISOR_URL
|
||||
description: ClawVisor gateway URL (Option A — recommended, handles OAuth for you)
|
||||
where: https://clawvisor.com — create an agent, activate Google Contacts service
|
||||
- name: CLAWVISOR_AGENT_TOKEN
|
||||
description: ClawVisor agent token (Option A)
|
||||
where: https://clawvisor.com — agent settings, copy the agent token
|
||||
- name: GOOGLE_CLIENT_ID
|
||||
description: Google OAuth2 client ID (Option B — direct API access, you manage tokens)
|
||||
where: https://console.cloud.google.com/apis/credentials — create OAuth 2.0 Client ID
|
||||
- name: GOOGLE_CLIENT_SECRET
|
||||
description: Google OAuth2 client secret (Option B)
|
||||
where: https://console.cloud.google.com/apis/credentials — same page as client ID
|
||||
health_checks:
|
||||
- type: any_of
|
||||
label: "Auth provider"
|
||||
checks:
|
||||
- type: http
|
||||
url: "$CLAWVISOR_URL/ready"
|
||||
label: "ClawVisor"
|
||||
- type: env_exists
|
||||
name: GOOGLE_CLIENT_ID
|
||||
label: "Google OAuth"
|
||||
setup_time: 15 min
|
||||
cost_estimate: "$0 (both options are free)"
|
||||
---
|
||||
|
||||
# Contacts-to-Brain: Your Address Book Becomes Ground Truth
|
||||
|
||||
Calendar attendees and email senders are the bulk of `people/<slug>` brain pages.
|
||||
Your Google Contacts is the ground-truth directory — canonical name, email, phone,
|
||||
organization — for those same entities. Syncing it closes the "who is this person"
|
||||
loop automatically and feeds enrichment.
|
||||
|
||||
## IMPORTANT: Instructions for the Agent
|
||||
|
||||
**You are the installer.** Follow these steps precisely.
|
||||
|
||||
**Why this matters:** email-to-brain and calendar-to-brain create people pages
|
||||
from whatever name string the API returned — sometimes an email prefix, sometimes
|
||||
a nickname. Contacts carries the authoritative record. After this recipe runs,
|
||||
"j.smith@acme-example.com" and "Jon S" resolve to the same person page with the
|
||||
right display name, phone, and company.
|
||||
|
||||
**The output is staging files, not direct writes:** the deterministic collector
|
||||
dumps contact records to `brain/contacts/.staging/`; YOU (the agent) merge them
|
||||
into `people/<slug>` pages using judgment — the Notability Gate in
|
||||
`skills/_brain-filing-rules.md` applies. Not every contact deserves a page.
|
||||
|
||||
**Do not skip steps. Verify after each step.**
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Google Contacts (People API, paginated)
|
||||
↓ (ClawVisor credential gateway: list_contacts / get_contact / search_contacts)
|
||||
Contacts Sync Script (deterministic Node.js)
|
||||
↓ Outputs:
|
||||
├── brain/contacts/.raw/contacts-{date}.json (raw API responses, provenance)
|
||||
└── brain/contacts/.staging/{slug}.md (one markdown record per contact)
|
||||
↓
|
||||
Agent reads staging files
|
||||
↓ Judgment calls (Notability Gate):
|
||||
├── Merge into existing people/<slug> pages (name/email/phone/org enrichment)
|
||||
├── Create new pages ONLY for notable contacts not already in brain
|
||||
└── Skip the rest (staging is not the brain)
|
||||
```
|
||||
|
||||
## Opinionated Defaults
|
||||
|
||||
**Staging record format** (one file per contact, deterministic):
|
||||
```markdown
|
||||
# Alice Example
|
||||
|
||||
- **Emails:** alice@acme-example.com, alice@gmail.com
|
||||
- **Phone:** +1 555 0100
|
||||
- **Organization:** Acme Example — VP Engineering
|
||||
- **Source:** Google Contacts (resourceName people/c123, synced 2026-07-21)
|
||||
```
|
||||
|
||||
**Enrichment, not duplication:** if `people/alice-example.md` already exists,
|
||||
append missing fields to it with a `[Source: Google Contacts]` citation. Do NOT
|
||||
create a second page. Slug-match by normalized name, then by email against
|
||||
existing page content.
|
||||
|
||||
**Notability Gate (from `skills/_brain-filing-rules.md`):** a contact with no
|
||||
brain presence gets a new page only if they appear elsewhere in the brain
|
||||
(calendar attendee, email correspondent) or the user confirms they matter.
|
||||
When in doubt, DON'T create — a junk page degrades search quality.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **GBrain installed and configured** (`gbrain doctor` passes)
|
||||
2. **Node.js 18+** (for the sync script)
|
||||
3. **Google Contacts access** via ONE of:
|
||||
- **Option A: ClawVisor** (recommended, handles OAuth for you, no token management)
|
||||
- **Option B: Google OAuth2 directly** (you manage tokens, no extra service needed)
|
||||
|
||||
## Setup Flow
|
||||
|
||||
### Step 1: Choose and Configure Contacts Access
|
||||
|
||||
Ask the user: "How do you want to connect to Google Contacts?
|
||||
|
||||
**Option A: ClawVisor (recommended)**
|
||||
ClawVisor handles OAuth, token refresh, and encryption. If you already use
|
||||
ClawVisor for email-to-brain or calendar-to-brain, this uses the same setup —
|
||||
just activate the Google Contacts service on your existing agent.
|
||||
|
||||
**Option B: Google OAuth2 directly**
|
||||
Connect to the Google People API directly. No extra service needed, but you
|
||||
manage OAuth tokens yourself."
|
||||
|
||||
#### Option A: ClawVisor Setup
|
||||
|
||||
Tell the user:
|
||||
"I need your ClawVisor URL and agent token.
|
||||
1. Go to https://clawvisor.com
|
||||
2. Create an agent (or use existing)
|
||||
3. Activate the **Google Contacts** service
|
||||
4. Create a standing task with purpose: 'Full contacts access for people
|
||||
enrichment: list contacts, read contact details, search contacts across
|
||||
all connected Google accounts.'
|
||||
IMPORTANT: Be EXPANSIVE in the task purpose. Narrow purposes block requests.
|
||||
5. Copy the gateway URL and agent token"
|
||||
|
||||
Validate:
|
||||
```bash
|
||||
curl -sf "$CLAWVISOR_URL/ready" && echo "PASS: ClawVisor reachable" || echo "FAIL"
|
||||
```
|
||||
|
||||
**STOP until ClawVisor validates.**
|
||||
|
||||
#### Option B: Google OAuth2 Setup
|
||||
|
||||
Same flow as `recipes/credential-gateway.md` Option B, with the contacts scope:
|
||||
|
||||
1. https://console.cloud.google.com/apis/credentials — create an OAuth client ID
|
||||
(Desktop app), consent screen scope: `https://www.googleapis.com/auth/contacts.readonly`
|
||||
2. Enable the People API: https://console.cloud.google.com/apis/library/people.googleapis.com
|
||||
3. Run the OAuth flow; store tokens in `~/.gbrain/google-tokens.json` (auto-refresh on expiry)
|
||||
|
||||
Validate:
|
||||
```bash
|
||||
[ -n "$GOOGLE_CLIENT_ID" ] && [ -n "$GOOGLE_CLIENT_SECRET" ] \
|
||||
&& echo "PASS: Google OAuth credentials set" \
|
||||
|| echo "FAIL: Missing GOOGLE_CLIENT_ID or GOOGLE_CLIENT_SECRET"
|
||||
```
|
||||
|
||||
**STOP until OAuth flow completes and tokens are stored.**
|
||||
|
||||
### Step 2: Set Up the Contacts Sync Script
|
||||
|
||||
```bash
|
||||
mkdir -p contacts-sync
|
||||
cd contacts-sync
|
||||
npm init -y
|
||||
```
|
||||
|
||||
The sync script needs these capabilities:
|
||||
|
||||
1. **Paginated retrieval** — `list_contacts` (People API `people.connections.list`)
|
||||
returns pages of up to 100; follow `nextPageToken` until exhausted. Request
|
||||
fields: names, emailAddresses, phoneNumbers, organizations, metadata.
|
||||
2. **Deterministic staging output** — one markdown file per contact at
|
||||
`brain/contacts/.staging/{slug}.md`, slug from normalized display name
|
||||
(fall back to email prefix). Same contact = same file on every run (idempotent).
|
||||
3. **Raw JSON preservation** — save raw API responses to
|
||||
`brain/contacts/.raw/contacts-{date}.json` for provenance.
|
||||
4. **Skip empty records** — contacts with no name AND no email are noise; drop them.
|
||||
|
||||
### Step 3: Run the Full Sync
|
||||
|
||||
```bash
|
||||
node contacts-sync.mjs
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
ls brain/contacts/.staging/ | head -10
|
||||
```
|
||||
|
||||
Should show one file per contact, e.g. `alice-example.md`, `charlie-example.md`.
|
||||
|
||||
### Step 4: Enrich People Pages (Agent Judgment)
|
||||
|
||||
This is YOUR job (the agent). For each staging record:
|
||||
|
||||
1. **Check brain**: `gbrain search "contact name"` — do they have a
|
||||
`people/<slug>` page? Also search by email address.
|
||||
2. **Existing page** → merge the ground-truth fields (canonical name, emails,
|
||||
phone, organization) into the page, each with a
|
||||
`[Source: Google Contacts]` citation. Fix a wrong/partial display name.
|
||||
3. **No page** → apply the Notability Gate: create a page only if the contact
|
||||
already appears in the brain (calendar, email) or is clearly relevant.
|
||||
Otherwise skip.
|
||||
4. **Back-link** per the Iron Law in `skills/_brain-filing-rules.md`: an
|
||||
organization with a brain page gets a link from the person's page and back.
|
||||
|
||||
After enrichment, import and embed:
|
||||
```bash
|
||||
gbrain sync --no-pull --no-embed && gbrain embed --stale
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
gbrain search "alice-example" --limit 3
|
||||
```
|
||||
|
||||
Should return the enriched people page with contact details.
|
||||
|
||||
### Step 5: Set Up Weekly Sync
|
||||
|
||||
Contacts change slowly; once a week is plenty:
|
||||
```bash
|
||||
# Cron: every Sunday at 9 AM
|
||||
0 9 * * 0 cd /path/to/contacts-sync && node contacts-sync.mjs
|
||||
```
|
||||
|
||||
After each sync, re-run the Step 4 enrichment pass over CHANGED staging files
|
||||
only (compare mtime or diff against git), then:
|
||||
```bash
|
||||
gbrain sync --no-pull --no-embed && gbrain embed --stale
|
||||
```
|
||||
|
||||
### Step 6: Log Setup Completion
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.gbrain/integrations/contacts-to-brain
|
||||
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.1.0","status":"ok","details":{"contacts":"CONTACT_COUNT"}}' >> ~/.gbrain/integrations/contacts-to-brain/heartbeat.jsonl
|
||||
```
|
||||
|
||||
Tell the user: "Contacts-to-brain is set up. [N] contacts staged; [M] people
|
||||
pages enriched with ground-truth contact data. Weekly sync keeps it current."
|
||||
|
||||
## Implementation Guide
|
||||
|
||||
### Pagination
|
||||
|
||||
```
|
||||
list_all_contacts():
|
||||
contacts = []
|
||||
token = null
|
||||
do:
|
||||
page = list_contacts({ pageSize: 100, pageToken: token,
|
||||
personFields: 'names,emailAddresses,phoneNumbers,organizations,metadata' })
|
||||
contacts += page.connections
|
||||
token = page.nextPageToken
|
||||
while token
|
||||
return contacts
|
||||
```
|
||||
|
||||
### Slug Normalization
|
||||
|
||||
```
|
||||
slugify(contact):
|
||||
name = contact.names?[0]?.displayName
|
||||
if not name:
|
||||
name = contact.emailAddresses?[0]?.value.split('@')[0]
|
||||
return name.toLowerCase().normalize('NFD')
|
||||
.replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '')
|
||||
```
|
||||
|
||||
Collision (two contacts, same slug): suffix with the email domain
|
||||
(`alice-example-acme-example`) rather than overwrite.
|
||||
|
||||
### What the Agent Should Test After Setup
|
||||
|
||||
1. **Idempotency:** run the sync twice. `git status` on `brain/contacts/.staging/`
|
||||
shows no changes on the second run.
|
||||
2. **Pagination:** with 250+ contacts, verify the staging count matches the
|
||||
Google Contacts count (not capped at 100).
|
||||
3. **Notability Gate:** verify a one-off contact with no brain presence did NOT
|
||||
get a `people/` page.
|
||||
4. **Enrichment merge:** verify an existing people page gained contact fields
|
||||
without losing its prior content, each with a `[Source: Google Contacts]`
|
||||
citation.
|
||||
|
||||
## Cost Estimate
|
||||
|
||||
| Component | Monthly Cost |
|
||||
|-----------|-------------|
|
||||
| ClawVisor (free tier) | $0 |
|
||||
| Google People API | $0 (within free quota) |
|
||||
| **Total** | **$0** |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**No contacts returned:**
|
||||
- Check ClawVisor has the Google Contacts service activated
|
||||
- Check the standing task purpose is expansive enough
|
||||
- Option B: verify the People API is enabled and the token carries
|
||||
`contacts.readonly`
|
||||
|
||||
**Duplicate people pages after enrichment:**
|
||||
- The agent matched by name but the brain page slug differs — search by email
|
||||
address too before creating, then merge and delete the duplicate
|
||||
|
||||
**Contacts with no name:**
|
||||
- The sync script falls back to the email prefix; records with neither name
|
||||
nor email are dropped as noise
|
||||
@@ -23,7 +23,7 @@ health_checks:
|
||||
label: "Auth provider"
|
||||
checks:
|
||||
- type: http
|
||||
url: "$CLAWVISOR_URL/health"
|
||||
url: "$CLAWVISOR_URL/ready"
|
||||
label: "ClawVisor"
|
||||
- type: env_exists
|
||||
name: GOOGLE_CLIENT_ID
|
||||
@@ -75,7 +75,7 @@ Tell the user:
|
||||
3. Activate the services you need:
|
||||
- **Gmail** (for email-to-brain)
|
||||
- **Google Calendar** (for calendar-to-brain)
|
||||
- **Google Contacts** (for enrichment)
|
||||
- **Google Contacts** (for contacts-to-brain)
|
||||
4. Create a standing task with a broad purpose. CRITICAL: be EXPANSIVE.
|
||||
|
||||
Good purpose: 'Full executive assistant access to Gmail, Calendar, and
|
||||
@@ -88,7 +88,7 @@ Tell the user:
|
||||
|
||||
Validate:
|
||||
```bash
|
||||
curl -sf "$CLAWVISOR_URL/health" \
|
||||
curl -sf "$CLAWVISOR_URL/ready" \
|
||||
&& echo "PASS: ClawVisor reachable" \
|
||||
|| echo "FAIL: ClawVisor not reachable — check the URL"
|
||||
```
|
||||
@@ -171,7 +171,7 @@ can now access your Google services."
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. **ClawVisor:** `curl $CLAWVISOR_URL/health` returns OK.
|
||||
1. **ClawVisor:** `curl $CLAWVISOR_URL/ready` returns OK.
|
||||
2. **Google OAuth:** Tokens exist at `~/.gbrain/google-tokens.json`.
|
||||
3. **Gmail access:** Run the email collector — it should pull recent messages.
|
||||
4. **Calendar access:** Run the calendar sync — it should pull today's events.
|
||||
|
||||
@@ -23,7 +23,7 @@ health_checks:
|
||||
label: "Auth provider"
|
||||
checks:
|
||||
- type: http
|
||||
url: "$CLAWVISOR_URL/health"
|
||||
url: "$CLAWVISOR_URL/ready"
|
||||
label: "ClawVisor"
|
||||
- type: env_exists
|
||||
name: GOOGLE_CLIENT_ID
|
||||
@@ -130,7 +130,7 @@ Tell the user:
|
||||
|
||||
Validate:
|
||||
```bash
|
||||
curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL"
|
||||
curl -sf "$CLAWVISOR_URL/ready" && echo "PASS: ClawVisor reachable" || echo "FAIL"
|
||||
```
|
||||
|
||||
**STOP until ClawVisor validates.**
|
||||
@@ -328,7 +328,7 @@ threads you already replied to. Sent mail acts as a negative filter.
|
||||
## Troubleshooting
|
||||
|
||||
**No emails collected:**
|
||||
- Check ClawVisor health: `curl $CLAWVISOR_URL/health`
|
||||
- Check ClawVisor health: `curl $CLAWVISOR_URL/ready`
|
||||
- Check standing task is active and has Gmail service enabled
|
||||
- Check task purpose is expansive enough (narrow purposes block requests)
|
||||
|
||||
|
||||
@@ -334,6 +334,10 @@ export async function scanIntegrity(
|
||||
if (!page) continue;
|
||||
// Skip grandfathered pages (opted out of brain-integrity enforcement)
|
||||
if ((page.frontmatter as Record<string, unknown> | undefined)?.validate === false) continue;
|
||||
// Skip code pages: indexed source files aren't prose. 'tweet' in an
|
||||
// identifier or comment is not a bare-tweet citation gap, and auto-repair
|
||||
// would inject wikilink brackets into source code.
|
||||
if (page.type === 'code') continue;
|
||||
pagesScanned++;
|
||||
bareHits.push(...findBareTweetHits(page.compiled_truth, slug));
|
||||
externalHits.push(...findExternalLinks(page.compiled_truth, slug));
|
||||
@@ -363,6 +367,9 @@ async function scanIntegrityBatch(
|
||||
// YAML) diverges from the sequential path's strict === false check. Intentional
|
||||
// — gbrain lint should reject stringly-typed validate at write time.
|
||||
const validateCondition = sql`AND (frontmatter->>'validate' IS NULL OR frontmatter->>'validate' != 'false')`;
|
||||
// Mirror of the sequential path's `page.type === 'code'` skip: code pages
|
||||
// (indexed source files) are never prose-integrity candidates.
|
||||
const codeCondition = sql`AND type IS DISTINCT FROM 'code'`;
|
||||
|
||||
// v0.32.8: scan ONE row per (source_id, slug) pair, not one per slug.
|
||||
// Pre-fix used DISTINCT ON (slug) which collapsed multi-source rows into
|
||||
@@ -372,7 +379,7 @@ async function scanIntegrityBatch(
|
||||
const rows = await sql`
|
||||
SELECT slug, compiled_truth, frontmatter
|
||||
FROM pages
|
||||
WHERE 1=1 ${typeCondition} ${validateCondition}
|
||||
WHERE 1=1 ${typeCondition} ${validateCondition} ${codeCondition}
|
||||
ORDER BY source_id, slug
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
@@ -461,6 +468,9 @@ async function cmdAuto(args: string[]): Promise<void> {
|
||||
|
||||
const page = await engine.getPage(slug, { sourceId: source_id });
|
||||
if (!page) continue;
|
||||
// Never auto-repair code pages — injecting tweet citations into
|
||||
// indexed source files corrupts them. Same gate as scanIntegrity.
|
||||
if (page.type === 'code') continue;
|
||||
|
||||
pagesProcessed++;
|
||||
progress.tick(1, slug);
|
||||
|
||||
@@ -5208,9 +5208,18 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const { rows: [h] } = await this.db.query(`
|
||||
WITH entity_pages AS (
|
||||
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
|
||||
),
|
||||
narrative_pages AS (
|
||||
-- Composition-aware score: code source files and calendar daily
|
||||
-- files are orphans-by-design (no inbound wikilinks, no Timeline
|
||||
-- fence). Excluding them from the orphan/link-density/timeline
|
||||
-- denominators keeps a bulk code/calendar import from cratering
|
||||
-- brain_score.
|
||||
SELECT id FROM pages WHERE type IS NULL OR type NOT IN ('code', 'calendar-index')
|
||||
)
|
||||
SELECT
|
||||
(SELECT count(*) FROM pages) as page_count,
|
||||
(SELECT count(*) FROM narrative_pages) as narrative_page_count,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
|
||||
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
|
||||
0 as stale_pages,
|
||||
@@ -5222,7 +5231,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
) as dead_links,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
|
||||
(SELECT count(*) FROM links) as link_count,
|
||||
(SELECT count(DISTINCT page_id) FROM timeline_entries) as pages_with_timeline,
|
||||
(SELECT count(DISTINCT te.page_id) FROM timeline_entries te
|
||||
WHERE te.page_id IN (SELECT id FROM narrative_pages)) as pages_with_timeline,
|
||||
(SELECT count(*) FROM entity_pages e
|
||||
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
|
||||
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as link_coverage,
|
||||
@@ -5244,12 +5254,19 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const { rows: islandedRows } = await this.db.query(`
|
||||
SELECT p.slug
|
||||
FROM pages p
|
||||
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
||||
-- Narrative pages only (same type filter as the narrative_pages CTE):
|
||||
-- code/calendar-index pages are orphans-by-design and must not count
|
||||
-- against the noOrphans component (#1144).
|
||||
WHERE (p.type IS NULL OR p.type NOT IN ('code', 'calendar-index'))
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
|
||||
`);
|
||||
|
||||
const r = h as Record<string, unknown>;
|
||||
const pageCount = Number(r.page_count);
|
||||
// Composition-aware denominators (excludes code/calendar-index pages);
|
||||
// a code-only brain has nothing narrative to penalize → full marks.
|
||||
const narrativePageCount = Number(r.narrative_page_count);
|
||||
const embedCoverage = Number(r.embed_coverage);
|
||||
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
|
||||
const orphanOverrides = await loadOrphanPolicyOverrides(this);
|
||||
@@ -5259,9 +5276,9 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const linkCount = Number(r.link_count);
|
||||
const pagesWithTimeline = Number(r.pages_with_timeline);
|
||||
|
||||
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
|
||||
const timelineCoverageDensity = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
|
||||
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
|
||||
const linkDensity = narrativePageCount > 0 ? Math.min(linkCount / narrativePageCount, 1) : 1;
|
||||
const timelineCoverageDensity = narrativePageCount > 0 ? Math.min(pagesWithTimeline / narrativePageCount, 1) : 1;
|
||||
const noOrphans = narrativePageCount > 0 ? 1 - (orphanPages / narrativePageCount) : 1;
|
||||
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
|
||||
// Bug 11 — per-component points. Sum equals brainScore by construction
|
||||
// so `doctor` can render a breakdown that adds up to the total.
|
||||
|
||||
@@ -5328,9 +5328,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
const [h] = await sql`
|
||||
WITH entity_pages AS (
|
||||
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
|
||||
),
|
||||
narrative_pages AS (
|
||||
-- Composition-aware score: code source files and calendar daily
|
||||
-- files are orphans-by-design (no inbound wikilinks, no Timeline
|
||||
-- fence). Excluding them from the orphan/link-density/timeline
|
||||
-- denominators keeps a bulk code/calendar import from cratering
|
||||
-- brain_score.
|
||||
SELECT id FROM pages WHERE type IS NULL OR type NOT IN ('code', 'calendar-index')
|
||||
)
|
||||
SELECT
|
||||
(SELECT count(*) FROM pages) as page_count,
|
||||
(SELECT count(*) FROM narrative_pages) as narrative_page_count,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
|
||||
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
|
||||
0 as stale_pages,
|
||||
@@ -5340,7 +5349,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
) as dead_links,
|
||||
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
|
||||
(SELECT count(*) FROM links) as link_count,
|
||||
(SELECT count(DISTINCT page_id) FROM timeline_entries) as pages_with_timeline,
|
||||
(SELECT count(DISTINCT te.page_id) FROM timeline_entries te
|
||||
WHERE te.page_id IN (SELECT id FROM narrative_pages)) as pages_with_timeline,
|
||||
(SELECT count(*) FROM entity_pages e
|
||||
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
|
||||
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as link_coverage,
|
||||
@@ -5361,11 +5371,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
const islandedRows = await sql<{ slug: string }[]>`
|
||||
SELECT p.slug
|
||||
FROM pages p
|
||||
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
||||
-- Narrative pages only (same type filter as the narrative_pages CTE):
|
||||
-- code/calendar-index pages are orphans-by-design and must not count
|
||||
-- against the noOrphans component (#1144).
|
||||
WHERE (p.type IS NULL OR p.type NOT IN ('code', 'calendar-index'))
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
|
||||
`;
|
||||
|
||||
const pageCount = Number(h.page_count);
|
||||
// Composition-aware denominators (excludes code/calendar-index pages);
|
||||
// a code-only brain has nothing narrative to penalize → full marks.
|
||||
const narrativePageCount = Number(h.narrative_page_count);
|
||||
const embedCoverage = Number(h.embed_coverage);
|
||||
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
|
||||
const orphanOverrides = await loadOrphanPolicyOverrides(this);
|
||||
@@ -5375,9 +5392,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
const pagesWithTimeline = Number(h.pages_with_timeline);
|
||||
|
||||
// brain_score: 0-100 weighted average
|
||||
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
|
||||
const timelineCoverageWhole = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
|
||||
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
|
||||
const linkDensity = narrativePageCount > 0 ? Math.min(linkCount / narrativePageCount, 1) : 1;
|
||||
const timelineCoverageWhole = narrativePageCount > 0 ? Math.min(pagesWithTimeline / narrativePageCount, 1) : 1;
|
||||
const noOrphans = narrativePageCount > 0 ? 1 - (orphanPages / narrativePageCount) : 1;
|
||||
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
|
||||
// Per-component points. Sum equals brainScore by construction.
|
||||
//
|
||||
|
||||
@@ -119,6 +119,40 @@ describe('Bug 11 — orphan_pages is "no inbound links"', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1144 — brain_score is composition-aware', () => {
|
||||
test('code and calendar-index pages do not count as orphans or dilute density metrics', async () => {
|
||||
// Two linked narrative pages + a flood of orphan-by-design pages.
|
||||
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: 'a', frontmatter: {} });
|
||||
await engine.putPage('people/bob', { type: 'person', title: 'Bob', compiled_truth: 'b', frontmatter: {} });
|
||||
const aId = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug='people/alice'`)).rows[0].id;
|
||||
const bId = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug='people/bob'`)).rows[0].id;
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
|
||||
[aId, bId],
|
||||
);
|
||||
for (let i = 0; i < 10; i++) {
|
||||
await engine.putPage(`code/src/f${i}.py`, { type: 'code', title: `f${i}.py`, compiled_truth: 'def f(): pass', frontmatter: {} });
|
||||
}
|
||||
await engine.putPage('daily/calendar/2026-01-01', { type: 'calendar-index', title: '2026-01-01', compiled_truth: 'events', frontmatter: {} });
|
||||
|
||||
const h = await engine.getHealth();
|
||||
// 11 unlinked code/calendar pages exist, but both narrative pages are
|
||||
// linked → 0 orphans, full no-orphans marks.
|
||||
expect(h.orphan_pages).toBe(0);
|
||||
expect(h.no_orphans_score).toBe(15);
|
||||
// Link density: 1 link / 2 narrative pages, not 1 / 13 total pages.
|
||||
expect(h.link_density_score).toBe(Math.round(0.5 * 25));
|
||||
});
|
||||
|
||||
test('a brain with ONLY non-narrative pages gets full composition marks', async () => {
|
||||
await engine.putPage('code/src/only.py', { type: 'code', title: 'only.py', compiled_truth: 'x = 1', frontmatter: {} });
|
||||
const h = await engine.getHealth();
|
||||
expect(h.no_orphans_score).toBe(15);
|
||||
expect(h.link_density_score).toBe(25);
|
||||
expect(h.timeline_coverage_score).toBe(15);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Bug 11 — doctor renders brain_score breakdown', () => {
|
||||
test('doctor source contains brain_score breakdown rendering', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
|
||||
|
||||
@@ -763,3 +763,48 @@ describeBoth('Engine parity — federated sourceIds[] secondary reads (#2200)',
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// #1144 — brain_score is composition-aware: code / calendar-index pages are
|
||||
// orphans-by-design and must not feed the orphan/link-density/timeline
|
||||
// denominators. Both engines must agree.
|
||||
async function seedComposition(eng: BrainEngine) {
|
||||
await eng.putPage('people/comp-alice', { type: 'person', title: 'Alice', compiled_truth: 'a', timeline: '' });
|
||||
await eng.putPage('people/comp-bob', { type: 'person', title: 'Bob', compiled_truth: 'b', timeline: '' });
|
||||
await eng.addLink('people/comp-alice', 'people/comp-bob', 'knows', 'mentions', 'markdown');
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await eng.putPage(`code/src/comp-f${i}.py`, { type: 'code', title: `f${i}.py`, compiled_truth: 'def f(): pass', timeline: '' });
|
||||
}
|
||||
await eng.putPage('daily/calendar/2026-01-01', { type: 'calendar-index', title: '2026-01-01', compiled_truth: 'events', timeline: '' });
|
||||
}
|
||||
|
||||
describeBoth('Engine parity — getHealth composition-aware brain_score (#1144)', () => {
|
||||
let pgEngine: BrainEngine;
|
||||
let pgliteEngine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
pgEngine = await setupDB();
|
||||
await seedComposition(pgEngine);
|
||||
pgliteEngine = new PGLiteEngine();
|
||||
await pgliteEngine.connect({});
|
||||
await pgliteEngine.initSchema();
|
||||
await seedComposition(pgliteEngine);
|
||||
}, 90_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await pgliteEngine.disconnect();
|
||||
await teardownDB();
|
||||
}, 30_000);
|
||||
|
||||
test('orphan/link-density/timeline components identical and exclude code + calendar-index pages', async () => {
|
||||
const pg = await pgEngine.getHealth();
|
||||
const pglite = await pgliteEngine.getHealth();
|
||||
for (const k of ['orphan_pages', 'no_orphans_score', 'link_density_score', 'timeline_coverage_score', 'brain_score'] as const) {
|
||||
expect(pg[k]).toBe(pglite[k]);
|
||||
}
|
||||
// 6 non-narrative pages exist but both narrative pages are linked → 0 orphans.
|
||||
expect(pg.orphan_pages).toBe(0);
|
||||
expect(pg.no_orphans_score).toBe(15);
|
||||
// 1 link / 2 narrative pages, not / 8 total pages.
|
||||
expect(pg.link_density_score).toBe(Math.round(0.5 * 25));
|
||||
});
|
||||
});
|
||||
|
||||
@@ -140,6 +140,35 @@ describeE2E('scanIntegrity batch parity (E2E, Postgres-only)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('code pages', () => {
|
||||
test('type:code page is skipped on both paths (#1143)', async () => {
|
||||
const engine = getEngine();
|
||||
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice tweeted about something.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
await engine.putPage('code/src/bot.py', {
|
||||
type: 'code',
|
||||
title: 'src/bot.py (python)',
|
||||
compiled_truth: '# the user tweeted about this feature\ndef send_tweet():\n pass',
|
||||
timeline: '',
|
||||
frontmatter: { language: 'python', file: 'src/bot.py' },
|
||||
});
|
||||
|
||||
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
|
||||
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
|
||||
|
||||
expect(batchResult.pagesScanned).toBe(seqResult.pagesScanned);
|
||||
expect(batchResult.pagesScanned).toBe(1);
|
||||
expect(batchResult.bareHits.map(h => h.slug)).not.toContain('code/src/bot.py');
|
||||
expect(seqResult.bareHits.map(h => h.slug)).not.toContain('code/src/bot.py');
|
||||
});
|
||||
});
|
||||
|
||||
describe('topPages', () => {
|
||||
test('topPages ordering matches between paths', async () => {
|
||||
const engine = getEngine();
|
||||
|
||||
@@ -201,6 +201,15 @@ describe('scanIntegrity', () => {
|
||||
timeline: '',
|
||||
frontmatter: { validate: false },
|
||||
});
|
||||
// Indexed source file — 'tweeted about' in a comment must NOT be flagged
|
||||
// as a bare-tweet citation gap (#1143: auto-repair would corrupt source).
|
||||
await engine.putPage('code/src/bot.py', {
|
||||
type: 'code',
|
||||
title: 'src/bot.py (python)',
|
||||
compiled_truth: '# the user tweeted about this feature\ndef send_tweet():\n pass',
|
||||
timeline: '',
|
||||
frontmatter: { language: 'python', file: 'src/bot.py' },
|
||||
});
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -223,6 +232,13 @@ describe('scanIntegrity', () => {
|
||||
expect(slugs).not.toContain('people/legacy');
|
||||
});
|
||||
|
||||
test('skips type:code pages (#1143 — bare-tweet false positives on source files)', async () => {
|
||||
const res = await scanIntegrity(engine);
|
||||
const slugs = res.bareHits.map(h => h.slug);
|
||||
expect(slugs).not.toContain('code/src/bot.py');
|
||||
expect(res.pagesScanned).toBe(2);
|
||||
});
|
||||
|
||||
test('honors limit', async () => {
|
||||
const res = await scanIntegrity(engine, { limit: 1 });
|
||||
expect(res.pagesScanned).toBe(1);
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
/**
|
||||
* #1098 — ClawVisor health checks must hit /ready (surfaces db/vault
|
||||
* degradation), never /health (bare liveness, masks those failure modes).
|
||||
* #1099 — the credential-gateway recipe tells users to activate Google
|
||||
* Contacts; the canonical consumer recipe must actually ship.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
const RECIPES = join(import.meta.dir, '../recipes');
|
||||
|
||||
describe('#1098 — ClawVisor health checks use /ready', () => {
|
||||
for (const f of ['email-to-brain.md', 'calendar-to-brain.md', 'credential-gateway.md', 'contacts-to-brain.md']) {
|
||||
test(`${f} references $CLAWVISOR_URL/ready, never /health`, () => {
|
||||
const text = readFileSync(join(RECIPES, f), 'utf-8');
|
||||
expect(text).not.toContain('$CLAWVISOR_URL/health');
|
||||
expect(text).toContain('$CLAWVISOR_URL/ready');
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe('#1099 — canonical contacts-to-brain recipe ships', () => {
|
||||
test('recipes/contacts-to-brain.md exists with the sibling frontmatter shape', () => {
|
||||
const p = join(RECIPES, 'contacts-to-brain.md');
|
||||
expect(existsSync(p)).toBe(true);
|
||||
const text = readFileSync(p, 'utf-8');
|
||||
expect(text).toContain('id: contacts-to-brain');
|
||||
expect(text).toContain('requires: [credential-gateway]');
|
||||
expect(text).toContain('health_checks:');
|
||||
expect(text).toContain('contacts.readonly');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user