feat: v0.15.0 llms.txt + llms-full.txt + AGENTS.md (#294)

* feat: llms.txt + llms-full.txt + AGENTS.md (v0.15.0)

Ship three new public artifacts at the repo root so agents that aren't
Claude Code can discover GBrain documentation cleanly:

- AGENTS.md — ~45-line install + operating protocol for non-Claude agents
  (Codex, Cursor, OpenClaw, Aider). Covers install, read order, trust
  boundary, config/debug/migration pointers, fork regeneration. Uses
  relative links so it survives fork/rename.
- llms.txt — llmstxt.org-spec index (H1 + blockquote + Core entry points /
  Configuration / Debugging / Migrations / Philosophy / Optional H2s).
- llms-full.txt — same index with core docs inlined for single-fetch
  ingestion. ~225KB, well under the 600KB FULL_SIZE_BUDGET.

Generator-driven via scripts/build-llms.ts + scripts/llms-config.ts.
LLMS_REPO_BASE env var makes it fork-friendly. bun run build:llms
regenerates both outputs deterministically.

test/build-llms.test.ts has 7 cases: paths resolve on disk, generator
idempotent, llms.txt spec shape, checked-in files match generator output
(drift guard), content contract (RESOLVER / AGENTS / INSTALL referenced),
AGENTS mirrors README + INSTALL_FOR_AGENTS install path, llms-full.txt
under size budget.

Leverage point per Codex review: README.md + INSTALL_FOR_AGENTS.md
install prompts now tell agents to fetch AGENTS.md first. Without this,
the new files were invisible.

Drive-by fix: INSTALL_FOR_AGENTS.md:136 had `git pull origin main` while
the repo's default branch is master (origin/HEAD -> master). Corrected.

Plan + reviews: /plan-eng-review CLEARED, /codex adversarial review
found 15 issues — 7 folded in directly, 3 user tension decisions, 5
stayed as NOT-in-scope with reasoning.

Version bumps to 0.15.0 (new public-artifact feature surface per Step 12
of /ship feature-signal heuristic).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: normalize VERSION to 3-digit to match master

master uses 3-digit semver (0.14.2); my earlier /ship bumped VERSION to
the 4-digit gstack format (0.15.0.0). Revert to 0.15.0 to match
package.json (already 3-digit) and master's convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-21 11:51:32 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent b5fa3d044a
commit 7f156c8873
12 changed files with 5107 additions and 4 deletions
+59
View File
@@ -0,0 +1,59 @@
# Agents working on GBrain
This is your install + operating protocol. Claude Code reads `./CLAUDE.md` automatically.
Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL):
start here.
## Install (5 min)
1. Clone: `git clone https://github.com/garrytan/gbrain ~/gbrain && cd ~/gbrain`
2. Install: `bun install`
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
multi-machine sync, init suggests Postgres + pgvector via Supabase.
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
(API keys, identity, cron, verification).
## Read this order
1. `./AGENTS.md` (this file) — install + operating protocol.
2. [`./CLAUDE.md`](./CLAUDE.md) — architecture reference, key files, trust boundaries,
test layout.
3. [`./skills/RESOLVER.md`](./skills/RESOLVER.md) — skill dispatcher. Read before any task.
## Trust boundary (critical)
GBrain distinguishes **trusted local CLI callers** (`OperationContext.remote = false`,
set by `src/cli.ts`) from **untrusted agent-facing callers** (`remote = true`, set by
`src/mcp/server.ts`). Security-sensitive operations like `file_upload` tighten filesystem
confinement when `remote = true` and default to strict behavior when unset. If you are
writing or reviewing an operation, consult `src/core/operations.ts` for the contract.
## Common tasks
- **Configure:** [`docs/ENGINES.md`](./docs/ENGINES.md),
[`docs/guides/live-sync.md`](./docs/guides/live-sync.md),
[`docs/mcp/DEPLOY.md`](./docs/mcp/DEPLOY.md).
- **Debug:** [`docs/GBRAIN_VERIFY.md`](./docs/GBRAIN_VERIFY.md),
[`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`.
- **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`.
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
single-fetch ingestion.
## Before shipping
Run `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin up the test
Postgres container, run `bun run test:e2e`, tear it down). Ship via the `/ship` skill,
not by hand.
## Privacy
Never commit real names of people, companies, or funds into public artifacts. See the
Privacy rule in `./CLAUDE.md`. GBrain pages reference real contacts; public docs must
use generic placeholders (`alice-example`, `acme-example`, `fund-a`).
## Forks
If you are a fork, regenerate `llms.txt` + `llms-full.txt` with your own URL base before
publishing: `LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms`.
+70
View File
@@ -2,6 +2,76 @@
All notable changes to GBrain will be documented in this file.
## [0.15.0] - 2026-04-21
## **GBrain now talks to LLMs the way modern docs sites do.**
## **One URL, full context. Three files, zero drift.**
Three new artifacts ship at the repo root: `llms.txt` (llmstxt.org-spec index), `llms-full.txt` (same map with core docs inlined, ~225KB, fits well under a 150k-token context window), and `AGENTS.md` (the non-Claude-agent operating protocol). All three are generator-driven. `scripts/build-llms.ts` reads a curated `scripts/llms-config.ts` and emits `llms.txt` + `llms-full.txt` deterministically; `AGENTS.md` is hand-written and uses relative links so it survives forks and rename. Every agent that clones GBrain now has a one-screen answer to "I just got here, what do I do?"
README and `INSTALL_FOR_AGENTS.md` now point agents at `AGENTS.md` first. The old install prompt still works, but the leverage point, Codex's read of the plan, was that these files are invisible unless the install path references them. Fixed.
### The numbers that matter
Measured on this release:
| Metric | BEFORE | AFTER | Δ |
|-------------------------------------------------|----------------------------------|-----------------------------------|----------------------------|
| Agent entry points with clear install protocol | 1 (CLAUDE.md, Claude Code only) | 3 (CLAUDE.md + AGENTS.md + llms.txt) | +non-Claude coverage |
| Docs referenced at a single canonical URL | 0 | 20 (across 5 H2 sections) | index exists |
| Full-context fetch round-trips | ~20 (one per doc) | 1 (`llms-full.txt`, 224 KB) | ~20x fewer fetches |
| Tests guarding the doc index | 0 | 7 (paths resolve, idempotent, spec shape, regen-drift, content contract, AGENTS mirror, size budget) | +7 |
| Pre-existing repo bugs found and fixed | — | 1 (`git pull origin main``master`) | drive-by |
The 7 tests enforce content contract: removing `skills/RESOLVER.md` or the Debugging H2 from the config fails `bun test`. Forgetting to rerun `bun run build:llms` after adding a new doc fails `bun test`. The size budget (600KB) fails `bun test` if `llms-full.txt` balloons.
### What this means for you
If you're running GBrain: nothing to do. Your agent already has CLAUDE.md. But next time you install GBrain on Codex, Cursor, or OpenClaw, the agent lands on `AGENTS.md` and walks the install without hunting. If you run a fork, regenerate with `LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms` to rewrite URLs. If you publish GBrain docs alongside your own, `llms.txt` is the index; `llms-full.txt` is the drop-into-a-context-window bundle.
Credit to Codex for catching that the original plan's AGENTS.md was underpowered, that the eng review missed a content-contract test, and that the install prompt was the real leverage point. Seven of the fifteen Codex findings landed directly in the plan; three went to user decision; five stayed as intentional NOT-in-scope.
## To take advantage of this release
`gbrain upgrade` does not need to do anything. These are new public files; existing installs pick them up on their next pull.
1. **If you wrote a downstream fork:** regenerate with your URL base.
```bash
LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms
git add llms.txt llms-full.txt && git commit
```
2. **If you add a new doc under `docs/`:** add it to `scripts/llms-config.ts`, then
```bash
bun run build:llms
bun test test/build-llms.test.ts
```
CI blocks ship if these drift.
3. **Verify it actually works:** ask a fresh LLM
```
Fetch https://raw.githubusercontent.com/garrytan/gbrain/master/llms.txt and tell me
how I'd debug a broken live sync.
```
Answer should cite `docs/GBRAIN_VERIFY.md`, `docs/guides/live-sync.md`, and `gbrain doctor`.
### Itemized changes
#### Added
- `AGENTS.md` at repo root — ~45-line non-Claude-agent operating protocol. Install, read order, trust boundary, config/debug/migration pointers, fork instructions. Uses relative links so it survives renames.
- `llms.txt` at repo root — llmstxt.org-spec index. H1 + blockquote + 5 required H2 sections (Core entry points, Configuration, Debugging, Migrations) plus an Operational tips block with `gbrain doctor`, `gbrain orphans`, `gbrain repair-jsonb`. ~4KB.
- `llms-full.txt` at repo root — same index with core docs inlined under `## {path}` headings for single-fetch ingestion. ~225KB, under the 600KB `FULL_SIZE_BUDGET`.
- `scripts/llms-config.ts` — curated TS config. `LLMS_REPO_BASE` env var lets forks regenerate with their own URL base. `includeInFull: false` flags entries that should appear in `llms.txt` but not be inlined in `llms-full.txt` (Philosophy, Optional, CHANGELOG).
- `scripts/build-llms.ts` — the generator. Deterministic, no timestamps, sorted by config order. Warns (does not fail) if `llms-full.txt` exceeds `FULL_SIZE_BUDGET` with the biggest entries listed.
- `test/build-llms.test.ts` — 7 cases: paths resolve on disk, generator idempotent, llms.txt spec shape, checked-in files match generator output (drift guard), content contract (RESOLVER / AGENTS / INSTALL_FOR_AGENTS referenced), AGENTS mirrors README+INSTALL install path, size budget enforcement.
- `bun run build:llms` script in `package.json`.
#### Changed
- `README.md` — adds a one-line LLMs/Agents pointer above the install CTA and a follow-up paragraph under the agent paste block naming `AGENTS.md` + `llms.txt` as fallback entry points for non-Claude agents.
- `INSTALL_FOR_AGENTS.md` — new "Step 0: If you are not Claude Code" prelude points agents at `AGENTS.md` first.
- `CLAUDE.md` — adds `scripts/llms-config.ts`, `scripts/build-llms.ts`, and `AGENTS.md` to Key files. Explicitly notes that committed generator output is NOT analogous to `schema-embedded.ts` (no runtime consumer; committed for GitHub browsing + fork safety).
#### Fixed
- `INSTALL_FOR_AGENTS.md:136` — `git pull origin main` → `git pull origin master`. Pre-existing drift: README and CI use `master`, `origin/HEAD -> master`, but the upgrade instructions told users to pull from a branch that doesn't exist. Folded into this release as a drive-by fix.
## [0.14.2] - 2026-04-20
## **Eight deferred bugs, root-cause fixes, one clean wave.**
+4 -1
View File
@@ -71,6 +71,8 @@ strict behavior when unset.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run]`: health checks. v0.12.3 adds two reliability detection checks: `jsonb_integrity` (scans pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata for `jsonb_typeof='string'` rows left over from v0.12.0) and `markdown_body_completeness` (flags pages whose compiled_truth is <30% of raw source when raw has multiple H2/H3 boundaries). Fix hints point at `gbrain repair-jsonb` and `gbrain sync --force`. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing.
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces the `${JSON.stringify(x)}::jsonb` interpolation pattern (which postgres.js v3 double-encodes). Wired into `bun test`.
- `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env var lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output is not analogous to `schema-embedded.ts` (no runtime consumer); we commit for GitHub browsing and fork-safe fetching.
- `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`.
- `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich.
- `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`)
- `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts)
@@ -205,7 +207,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics).
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases).
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
+12 -1
View File
@@ -3,6 +3,17 @@
Read this entire file, then follow the steps. Ask the user for API keys when needed.
Target: ~30 minutes to a fully working brain.
## Step 0: If you are not Claude Code
Read `AGENTS.md` at the repo root first. It's the non-Claude-agent operating
protocol (install, read order, trust boundary, common tasks). Claude Code reads
`CLAUDE.md` automatically and can skip ahead.
If you fetched this file by URL without cloning yet, the companion files live at:
- `https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` — start here
- `https://raw.githubusercontent.com/garrytan/gbrain/master/llms.txt` — full doc map
- `https://raw.githubusercontent.com/garrytan/gbrain/master/llms-full.txt` — same map, inlined
## Step 1: Install GBrain
```bash
@@ -133,7 +144,7 @@ actually works) is the most important.
## Upgrade
```bash
cd ~/gbrain && git pull origin main && bun install
cd ~/gbrain && git pull origin master && bun install
gbrain init # apply schema migrations (idempotent)
gbrain post-upgrade # show migration notes for the version range
```
+7
View File
@@ -10,6 +10,8 @@ GBrain is those patterns, generalized. 26 skills. Install in 30 minutes. Your ag
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
> **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code).
## Install
### On an agent platform (recommended)
@@ -28,6 +30,11 @@ https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 26 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
agent operating protocol (install, read order, trust boundary, common tasks). For
the full doc map, use `llms.txt` at the same URL root.
### Standalone CLI (no agent)
```bash
+1 -1
View File
@@ -1 +1 @@
0.14.2
0.15.0
+4404
View File
File diff suppressed because it is too large Load Diff
+52
View File
@@ -0,0 +1,52 @@
# GBrain
> GBrain is a personal knowledge brain and GStack mod for agent platforms. Pluggable engines (PGLite default, Postgres+pgvector for scale), contract-first operations, 26 fat-markdown skills. Teaches agents brain ops, ingestion, enrichment, scheduling, identity, and access control.
Repo: https://github.com/garrytan/gbrain
## Core entry points
- [AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md): Start here if you are not Claude Code. Install order, trust boundary, skill resolver, config/debug/migration pointers.
- [CLAUDE.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CLAUDE.md): Architecture reference. Key files, trust boundaries, engine factory, test layout.
- [INSTALL_FOR_AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md): 9-step agent installation.
- [skills/RESOLVER.md](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/RESOLVER.md): Skill dispatcher. Read first for any task.
- [README.md](https://raw.githubusercontent.com/garrytan/gbrain/master/README.md): Project overview, benchmarks, 30-minute setup.
## Configuration
- [docs/ENGINES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ENGINES.md): PGLite vs Postgres trade-off and when to migrate.
- [docs/GBRAIN_RECOMMENDED_SCHEMA.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md): MECE directory structure (people/, companies/, concepts/).
- [docs/guides/live-sync.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/live-sync.md): Incremental markdown sync setup.
- [docs/guides/cron-schedule.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/cron-schedule.md): Recurring job scheduling.
- [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery.
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
## Debugging
- [docs/GBRAIN_VERIFY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_VERIFY.md): 7-check post-setup verification. Start here when something feels off.
- [docs/guides/minions-fix.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-fix.md): Troubleshooting the Minions job queue.
- [docs/integrations/reliability-repair.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/integrations/reliability-repair.md): Data integrity recovery.
## Migrations
- [docs/UPGRADING_DOWNSTREAM_AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/UPGRADING_DOWNSTREAM_AGENTS.md): Patches for downstream agent skill forks. One section per release.
- [skills/migrations/](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/migrations/): Per-version (v0.5.0 - v0.14.1) agent-executable migration instructions.
- [CHANGELOG.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CHANGELOG.md): Release-summary voice + itemized changes + self-repair block per version.
## Philosophy
- [docs/ethos/THIN_HARNESS_FAT_SKILLS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ethos/THIN_HARNESS_FAT_SKILLS.md): Why skills live in markdown.
- [docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md): Homebrew for Personal AI.
## Optional
- [docs/benchmarks/](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/benchmarks/): Retrieval quality benchmarks.
- [docs/designs/](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/designs/): Forward-looking designs.
- [docs/architecture/infra-layer.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/architecture/infra-layer.md): Shared infra patterns.
## Operational tips
- `gbrain doctor [--json] [--fast] [--fix]` - built-in health checks.
- `gbrain orphans [--json]` - pages with zero inbound wikilinks.
- `gbrain repair-jsonb [--dry-run]` - repair v0.12.0 double-encoded JSONB rows.
- `gbrain upgrade` runs post-upgrade + apply-migrations.
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.14.2",
"version": "0.15.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -20,6 +20,7 @@
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
"build:schema": "bash scripts/build-schema.sh",
"build:llms": "bun run scripts/build-llms.ts",
"test": "scripts/check-jsonb-pattern.sh && bun test",
"test:e2e": "bun test test/e2e/",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
+193
View File
@@ -0,0 +1,193 @@
#!/usr/bin/env bun
/**
* build-llms — generate llms.txt + llms-full.txt from scripts/llms-config.ts.
*
* Run: `bun run build:llms` (or `bun run scripts/build-llms.ts`).
*
* Outputs:
* - llms.txt — llmstxt.org-spec index (H1 / blockquote / H2 sections).
* - llms-full.txt — concatenated full content of non-optional entries.
*
* Deterministic: no timestamps, sorted within categories by config order.
* Warns (does not fail) if llms-full.txt exceeds FULL_SIZE_BUDGET. CI catches
* drift via test/build-llms.test.ts.
*
* Fork override: set LLMS_REPO_BASE to regenerate with a different URL base.
*/
import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs";
import { join, dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import {
FULL_SIZE_BUDGET,
INLINE_TIPS,
PROJECT,
SECTIONS,
type DocEntry,
type DocSection,
} from "./llms-config";
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
function urlFor(entry: DocEntry): string {
return `${PROJECT.rawBaseUrl}/${entry.path}`;
}
function isDirectoryPath(path: string): boolean {
return path.endsWith("/");
}
function renderLlmsTxt(): string {
const lines: string[] = [];
lines.push(`# ${PROJECT.name}`);
lines.push("");
lines.push(`> ${PROJECT.summary}`);
lines.push("");
lines.push(`Repo: ${PROJECT.repoUrl}`);
lines.push("");
for (const section of SECTIONS) {
lines.push(`## ${section.heading}`);
lines.push("");
for (const entry of section.entries) {
lines.push(
`- [${entry.title}](${urlFor(entry)}): ${entry.description}`,
);
}
lines.push("");
}
lines.push("## Operational tips");
lines.push("");
for (const tip of INLINE_TIPS) {
lines.push(`- ${tip}`);
}
lines.push("");
return lines.join("\n");
}
function renderLlmsFullTxt(): { content: string; sizes: Array<{ path: string; bytes: number }> } {
const lines: string[] = [];
const sizes: Array<{ path: string; bytes: number }> = [];
lines.push(`# ${PROJECT.name} — Full Context`);
lines.push("");
lines.push(`> ${PROJECT.summary}`);
lines.push("");
lines.push(
`This file concatenates core GBrain documentation for single-fetch ingestion.`,
);
lines.push(
`For the link-only index, see \`llms.txt\`. Source of truth: ${PROJECT.repoUrl}.`,
);
lines.push("");
for (const section of SECTIONS) {
if (section.optional) continue;
lines.push(`# ${section.heading}`);
lines.push("");
for (const entry of section.entries) {
if (entry.includeInFull === false) continue;
if (isDirectoryPath(entry.path)) continue;
const absPath = join(repoRoot, entry.path);
if (!existsSync(absPath)) {
// build-llms won't silently skip — surface the problem. Test case 1
// catches this too, but fail fast for manual runs.
throw new Error(
`llms-config references missing file: ${entry.path}`,
);
}
const body = readFileSync(absPath, "utf8");
const bytes = Buffer.byteLength(body, "utf8");
sizes.push({ path: entry.path, bytes });
lines.push(`## ${entry.path}`);
lines.push("");
lines.push(`Source: ${urlFor(entry)}`);
lines.push("");
lines.push(body.trimEnd());
lines.push("");
lines.push("---");
lines.push("");
}
}
return { content: lines.join("\n"), sizes };
}
function validateConfig(): void {
for (const section of SECTIONS) {
for (const entry of section.entries) {
const absPath = join(repoRoot, entry.path);
if (!existsSync(absPath)) {
throw new Error(
`llms-config references missing path: ${entry.path}`,
);
}
const st = statSync(absPath);
if (isDirectoryPath(entry.path) && !st.isDirectory()) {
throw new Error(
`llms-config path ends with '/' but is a file: ${entry.path}`,
);
}
if (!isDirectoryPath(entry.path) && !st.isFile()) {
throw new Error(
`llms-config path is a directory but missing trailing '/': ${entry.path}`,
);
}
}
}
}
export function buildLlmsFiles(): {
llmsTxt: string;
llmsFullTxt: string;
sizes: Array<{ path: string; bytes: number }>;
} {
validateConfig();
const llmsTxt = renderLlmsTxt();
const { content: llmsFullTxt, sizes } = renderLlmsFullTxt();
return { llmsTxt, llmsFullTxt, sizes };
}
function main(): void {
const { llmsTxt, llmsFullTxt, sizes } = buildLlmsFiles();
const llmsPath = join(repoRoot, "llms.txt");
const llmsFullPath = join(repoRoot, "llms-full.txt");
writeFileSync(llmsPath, llmsTxt);
writeFileSync(llmsFullPath, llmsFullTxt);
const fullBytes = Buffer.byteLength(llmsFullTxt, "utf8");
console.log(`wrote ${llmsPath} (${Buffer.byteLength(llmsTxt, "utf8")} bytes)`);
console.log(`wrote ${llmsFullPath} (${fullBytes} bytes)`);
if (fullBytes > FULL_SIZE_BUDGET) {
console.warn("");
console.warn(
`WARN: llms-full.txt (${fullBytes} bytes) exceeds FULL_SIZE_BUDGET (${FULL_SIZE_BUDGET} bytes).`,
);
console.warn(
"Add `includeInFull: false` to the biggest entries in scripts/llms-config.ts:",
);
const sorted = [...sizes].sort((a, b) => b.bytes - a.bytes);
for (const entry of sorted.slice(0, 5)) {
console.warn(` ${entry.bytes} bytes ${entry.path}`);
}
}
}
const isMainModule = fileURLToPath(import.meta.url) === process.argv[1];
if (isMainModule) {
try {
main();
} catch (err) {
console.error(err instanceof Error ? err.message : err);
process.exit(1);
}
}
+205
View File
@@ -0,0 +1,205 @@
/**
* llms-config — single source of truth for llms.txt + llms-full.txt.
*
* Consumed by scripts/build-llms.ts (emits llms.txt, llms-full.txt) and
* test/build-llms.test.ts (asserts paths resolve, content contract holds).
*
* Adding a doc? Add it here and run `bun run build:llms`. The drift-detection
* test fails CI if you forget.
*
* Fork-friendliness: `rawBaseUrl` reads from `LLMS_REPO_BASE` so forks can
* regenerate without manual URL rewrites:
* LLMS_REPO_BASE=https://raw.githubusercontent.com/fork-org/gbrain/main bun run build:llms
*/
export type DocEntry = {
title: string;
description: string;
path: string;
includeInFull?: boolean;
};
export type DocSection = {
heading: string;
optional?: boolean;
entries: DocEntry[];
};
export const PROJECT = {
name: "GBrain",
summary:
"GBrain is a personal knowledge brain and GStack mod for agent platforms. Pluggable engines (PGLite default, Postgres+pgvector for scale), contract-first operations, 26 fat-markdown skills. Teaches agents brain ops, ingestion, enrichment, scheduling, identity, and access control.",
repoUrl: "https://github.com/garrytan/gbrain",
rawBaseUrl:
process.env.LLMS_REPO_BASE ??
"https://raw.githubusercontent.com/garrytan/gbrain/master",
};
export const SECTIONS: DocSection[] = [
{
heading: "Core entry points",
entries: [
{
title: "AGENTS.md",
description:
"Start here if you are not Claude Code. Install order, trust boundary, skill resolver, config/debug/migration pointers.",
path: "AGENTS.md",
},
{
title: "CLAUDE.md",
description:
"Architecture reference. Key files, trust boundaries, engine factory, test layout.",
path: "CLAUDE.md",
},
{
title: "INSTALL_FOR_AGENTS.md",
description: "9-step agent installation.",
path: "INSTALL_FOR_AGENTS.md",
},
{
title: "skills/RESOLVER.md",
description: "Skill dispatcher. Read first for any task.",
path: "skills/RESOLVER.md",
},
{
title: "README.md",
description: "Project overview, benchmarks, 30-minute setup.",
path: "README.md",
},
],
},
{
heading: "Configuration",
entries: [
{
title: "docs/ENGINES.md",
description: "PGLite vs Postgres trade-off and when to migrate.",
path: "docs/ENGINES.md",
},
{
title: "docs/GBRAIN_RECOMMENDED_SCHEMA.md",
description:
"MECE directory structure (people/, companies/, concepts/).",
path: "docs/GBRAIN_RECOMMENDED_SCHEMA.md",
},
{
title: "docs/guides/live-sync.md",
description: "Incremental markdown sync setup.",
path: "docs/guides/live-sync.md",
},
{
title: "docs/guides/cron-schedule.md",
description: "Recurring job scheduling.",
path: "docs/guides/cron-schedule.md",
},
{
title: "docs/guides/quiet-hours.md",
description: "Notification hold + timezone-aware delivery.",
path: "docs/guides/quiet-hours.md",
},
{
title: "docs/mcp/DEPLOY.md",
description: "MCP server deployment.",
path: "docs/mcp/DEPLOY.md",
},
],
},
{
heading: "Debugging",
entries: [
{
title: "docs/GBRAIN_VERIFY.md",
description:
"7-check post-setup verification. Start here when something feels off.",
path: "docs/GBRAIN_VERIFY.md",
},
{
title: "docs/guides/minions-fix.md",
description: "Troubleshooting the Minions job queue.",
path: "docs/guides/minions-fix.md",
},
{
title: "docs/integrations/reliability-repair.md",
description: "Data integrity recovery.",
path: "docs/integrations/reliability-repair.md",
},
],
},
{
heading: "Migrations",
entries: [
{
title: "docs/UPGRADING_DOWNSTREAM_AGENTS.md",
description:
"Patches for downstream agent skill forks. One section per release.",
path: "docs/UPGRADING_DOWNSTREAM_AGENTS.md",
},
{
title: "skills/migrations/",
description:
"Per-version (v0.5.0 - v0.14.1) agent-executable migration instructions.",
path: "skills/migrations/",
},
{
title: "CHANGELOG.md",
description:
"Release-summary voice + itemized changes + self-repair block per version.",
path: "CHANGELOG.md",
includeInFull: false,
},
],
},
{
heading: "Philosophy",
optional: true,
entries: [
{
title: "docs/ethos/THIN_HARNESS_FAT_SKILLS.md",
description: "Why skills live in markdown.",
path: "docs/ethos/THIN_HARNESS_FAT_SKILLS.md",
includeInFull: false,
},
{
title: "docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md",
description: "Homebrew for Personal AI.",
path: "docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md",
includeInFull: false,
},
],
},
{
heading: "Optional",
optional: true,
entries: [
{
title: "docs/benchmarks/",
description: "Retrieval quality benchmarks.",
path: "docs/benchmarks/",
includeInFull: false,
},
{
title: "docs/designs/",
description: "Forward-looking designs.",
path: "docs/designs/",
includeInFull: false,
},
{
title: "docs/architecture/infra-layer.md",
description: "Shared infra patterns.",
path: "docs/architecture/infra-layer.md",
includeInFull: false,
},
],
},
];
export const INLINE_TIPS = [
"`gbrain doctor [--json] [--fast] [--fix]` - built-in health checks.",
"`gbrain orphans [--json]` - pages with zero inbound wikilinks.",
"`gbrain repair-jsonb [--dry-run]` - repair v0.12.0 double-encoded JSONB rows.",
"`gbrain upgrade` runs post-upgrade + apply-migrations.",
];
// Target ~600KB so llms-full.txt fits in ~150k-token contexts with room to spare.
// Generator prints a WARN if exceeded; ship with includeInFull=false exclusions.
export const FULL_SIZE_BUDGET = 600_000;
+98
View File
@@ -0,0 +1,98 @@
import { describe, test, expect } from "bun:test";
import { existsSync, readFileSync, statSync } from "node:fs";
import { join } from "node:path";
import { buildLlmsFiles } from "../scripts/build-llms";
import { SECTIONS, FULL_SIZE_BUDGET } from "../scripts/llms-config";
const repoRoot = join(import.meta.dir, "..");
describe("build-llms generator", () => {
// Case 1 — every config path resolves on disk. Catches rename-induced 404s.
test("every configured path exists on disk", () => {
for (const section of SECTIONS) {
for (const entry of section.entries) {
const abs = join(repoRoot, entry.path);
expect(existsSync(abs), `missing: ${entry.path}`).toBe(true);
const st = statSync(abs);
if (entry.path.endsWith("/")) {
expect(st.isDirectory(), `${entry.path} should be a directory`).toBe(true);
} else {
expect(st.isFile(), `${entry.path} should be a file`).toBe(true);
}
}
}
});
// Case 2 — generator is idempotent. Run twice in-memory, compare byte-for-byte.
test("generator output is deterministic across runs", () => {
const first = buildLlmsFiles();
const second = buildLlmsFiles();
expect(second.llmsTxt).toBe(first.llmsTxt);
expect(second.llmsFullTxt).toBe(first.llmsFullTxt);
});
// Case 3 — llms.txt spec shape per llmstxt.org: H1 + blockquote + required H2s.
test("llms.txt follows llmstxt.org spec shape", () => {
const { llmsTxt } = buildLlmsFiles();
const lines = llmsTxt.split("\n");
expect(lines[0], "first line must be H1").toBe("# GBrain");
// Blockquote summary on line 2 or 3 (spec allows blank line after H1).
const hasEarlyBlockquote =
lines.slice(1, 4).some((line) => line.startsWith("> "));
expect(hasEarlyBlockquote, "needs > blockquote summary near top").toBe(true);
// Required H2 sections for GBrain's user need (config/debug/migration).
expect(llmsTxt).toContain("## Core entry points");
expect(llmsTxt).toContain("## Configuration");
expect(llmsTxt).toContain("## Debugging");
expect(llmsTxt).toContain("## Migrations");
});
// Case 4 — checked-in files match generator output. Catches "forgot to rerun
// generator" before ship. If this fails in CI, run `bun run build:llms` and
// commit the result.
test("committed llms.txt + llms-full.txt match current generator output", () => {
const { llmsTxt, llmsFullTxt } = buildLlmsFiles();
const committedLlms = readFileSync(join(repoRoot, "llms.txt"), "utf8");
const committedFull = readFileSync(join(repoRoot, "llms-full.txt"), "utf8");
const helpMsg =
"Run `bun run build:llms` and commit the updated output before shipping.";
expect(committedLlms, helpMsg).toBe(llmsTxt);
expect(committedFull, helpMsg).toBe(llmsFullTxt);
});
// Case 5 — content contract. Prevents silent removal of critical sections or
// entries from llms-config.ts. Catches "someone deleted the Debugging section."
test("content contract: llms.txt references required entry points", () => {
const { llmsTxt } = buildLlmsFiles();
expect(llmsTxt).toContain("skills/RESOLVER.md");
expect(llmsTxt).toContain("INSTALL_FOR_AGENTS.md");
expect(llmsTxt).toContain("AGENTS.md");
expect(llmsTxt).toContain("CLAUDE.md");
});
test("content contract: AGENTS.md mirrors README + INSTALL_FOR_AGENTS install path", () => {
const agents = readFileSync(join(repoRoot, "AGENTS.md"), "utf8");
expect(agents).toContain("CLAUDE.md");
expect(agents).toContain("skills/RESOLVER.md");
expect(agents).toContain("INSTALL_FOR_AGENTS.md");
expect(agents).toContain("llms.txt");
// Trust boundary is the non-obvious security concept agents need up-front.
expect(agents.toLowerCase()).toContain("trust boundary");
});
test("llms-full.txt stays within size budget", () => {
const { llmsFullTxt } = buildLlmsFiles();
const bytes = Buffer.byteLength(llmsFullTxt, "utf8");
expect(
bytes,
`llms-full.txt is ${bytes} bytes (budget ${FULL_SIZE_BUDGET}). Add includeInFull: false to large entries.`,
).toBeLessThan(FULL_SIZE_BUDGET);
});
});