Compare commits

...
Author SHA1 Message Date
root d134a1fb7e feat: storage tiering — git-tracked vs supabase-only directories
Brain repos scaling to 200K+ files. Bulk data (tweets, articles, transcripts)
bloats git repos and slows operations. New storage config in gbrain.yml lets
users declare git-tracked and supabase-only directories.

Changes:
- New config: storage.git_tracked and storage.supabase_only in gbrain.yml
- gbrain sync auto-manages .gitignore for supabase-only paths
- gbrain export --restore-only restores missing supabase-only files from DB
- New gbrain storage status command shows tier breakdown
- Config validation warns on conflicts
- 8 tests passing, full docs at docs/storage-tiering.md

Backward compatible — systems without gbrain.yml work unchanged.
2026-04-25 12:34:10 +00:00
8 changed files with 658 additions and 7 deletions
+188
View File
@@ -0,0 +1,188 @@
# Storage Tiering: git-tracked vs supabase-only directories
## Overview
GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.
## Configuration
Add a `storage` section to your `gbrain.yml` file in the brain repository root:
```yaml
storage:
# Directories that are git-tracked (version controlled, human-edited)
git_tracked:
- people/
- companies/
- deals/
- concepts/
- yc/
- ideas/
- projects/
# Directories persisted via Supabase only (bulk ingest, machine-generated)
# These are written to disk as a local cache but not committed to git.
# `gbrain export` restores them from Supabase when missing.
supabase_only:
- media/x/
- media/articles/
- meetings/transcripts/
```
## Behavior Changes
### 1. `gbrain sync` - Automatic .gitignore Management
When storage configuration is present, `sync` automatically manages `.gitignore` entries:
- Adds missing `supabase_only` directory patterns to `.gitignore`
- Prevents duplicate entries
- Adds a comment section for auto-managed entries
- Only runs when not in `--dry-run` mode
Example `.gitignore` addition:
```gitignore
# Auto-managed by gbrain (supabase-only directories)
media/x/
media/articles/
meetings/transcripts/
```
### 2. `gbrain export` - Enhanced Restore Capabilities
New flags for targeted export:
```bash
# Restore only missing supabase-only files from database
gbrain export --restore-only --repo /path/to/brain
# Filter by page type
gbrain export --restore-only --type media --repo /path/to/brain
# Filter by slug prefix
gbrain export --restore-only --slug-prefix media/x/ --repo /path/to/brain
# Combine filters
gbrain export --restore-only --type media --slug-prefix media/x/ --repo /path/to/brain
```
The `--restore-only` flag:
- Only exports pages that match `supabase_only` patterns
- Only includes pages where the file is missing from disk
- Ideal for container restart recovery scenarios
### 3. `gbrain storage status` - Storage Health Dashboard
New command to inspect storage tier configuration and health:
```bash
# Human-readable status
gbrain storage status --repo /path/to/brain
# JSON output for scripts
gbrain storage status --repo /path/to/brain --json
```
Output includes:
- Total page counts by storage tier
- Disk usage breakdown
- Missing files that need restoration
- Configuration validation warnings
- Current storage tier directory listing
Example output:
```
Storage Status
==============
Repository: /data/brain
Total pages: 15,243
Storage Tiers:
─────────────
Git tracked: 2,156 pages
Supabase only: 12,887 pages
Unspecified: 200 pages
Disk Usage:
──────────
Git tracked: 45.2 MB
Supabase only: 2.1 GB
Missing Files (need restore):
────────────────────────────
media/x/tweet-1234567890
media/x/tweet-0987654321
... and 47 more
Use: gbrain export --restore-only --repo "/data/brain"
Configuration:
─────────────
Git tracked directories:
• people/
• companies/
• deals/
Supabase-only directories:
• media/x/
• media/articles/
• meetings/transcripts/
```
## Validation
The system validates storage configuration and warns about:
- Directories appearing in both `git_tracked` and `supabase_only`
- Directory paths not ending with `/` (consistency recommendation)
- `supabase_only` directories not present in `.gitignore`
## Use Cases
### 1. Brain Repository Scaling
Perfect for brain repositories approaching 100K+ files where:
- Core knowledge (people, companies, deals) remains git-tracked
- Bulk data (tweets, articles, transcripts) moves to supabase-only
- Development stays fast with smaller git repos
- Full data remains available via database
### 2. Container-based Deployments
Essential for ephemeral container environments:
- Git repo contains only essential files
- Container restarts don't lose supabase-only data
- `gbrain export --restore-only` quickly restores bulk files when needed
- Local disk acts as cache layer
### 3. Multi-Environment Consistency
Enables consistent data access across environments:
- Development: small git clone, restore bulk data on demand
- Production: full dataset via database, selective local caching
- CI/CD: fast tests with git-tracked data only
## Migration Strategy
1. **Assess current repository**: Use `gbrain storage status` to understand current distribution
2. **Plan directory structure**: Identify which directories should be git-tracked vs supabase-only
3. **Create gbrain.yml**: Add storage configuration to repository root
4. **Test with dry-run**: Use `gbrain sync --dry-run` to verify behavior
5. **Update .gitignore**: Let `gbrain sync` auto-manage entries
6. **Verify exports**: Test `gbrain export --restore-only` for container scenarios
## Best Practices
- **Directory naming**: Always end storage paths with `/` for consistency
- **Start small**: Begin with clearly machine-generated directories in `supabase_only`
- **Monitor warnings**: Address configuration validation warnings promptly
- **Test restore**: Regularly test `--restore-only` in staging environments
- **Document decisions**: Comment your `gbrain.yml` to explain tier choices
## Compatibility
- **Backward compatible**: Systems without `gbrain.yml` work unchanged
- **Progressive enhancement**: Add configuration when needed
- **Database unchanged**: All data remains in Supabase regardless of tier
- **Existing workflows**: All existing `sync` and `export` behavior preserved
+14
View File
@@ -0,0 +1,14 @@
storage:
git_tracked:
- people/
- companies/
- deals/
- concepts/
- yc/
- ideas/
- projects/
supabase_only:
- media/x/
- media/articles/
- meetings/transcripts/
+10 -1
View File
@@ -19,7 +19,7 @@ for (const op of operations) {
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test']);
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage']);
async function main() {
// Parse global flags (--quiet / --progress-json / --progress-interval)
@@ -511,6 +511,11 @@ async function handleCliOnly(command: string, args: string[]) {
await runSources(engine, args);
break;
}
case 'storage': {
const { runStorage } = await import('./commands/storage.ts');
await runStorage(engine, args);
break;
}
}
} finally {
if (command !== 'serve') await engine.disconnect();
@@ -581,6 +586,8 @@ IMPORT/EXPORT
sync --watch [--interval N] Continuous sync (loops until stopped)
sync --install-cron Install persistent sync daemon
export [--dir ./out/] Export to markdown
export --restore-only [--repo <p>] Restore missing supabase-only files
[--type T] [--slug-prefix S] With optional filters
FILES
files list [slug] List stored files
@@ -643,6 +650,8 @@ ADMIN
features [--json] [--auto-fix] Scan usage + recommend unused features
autopilot [--repo] [--interval N] Self-maintaining brain daemon
config [show|get|set] <key> [val] Brain config
storage status [--repo <path>] Storage tier status and health
[--json] (git-tracked vs supabase-only)
serve MCP server (stdio)
call <tool> '<json>' Raw tool invocation
version Version info
+52 -5
View File
@@ -1,16 +1,59 @@
import { writeFileSync, mkdirSync } from 'fs';
import { writeFileSync, mkdirSync, existsSync } from 'fs';
import { join, dirname } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { serializeMarkdown } from '../core/markdown.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { loadStorageConfig, isSupabaseOnly, getStorageTier } from '../core/storage-config.ts';
import type { PageType } from '../core/types.ts';
export async function runExport(engine: BrainEngine, args: string[]) {
const dirIdx = args.indexOf('--dir');
const outDir = dirIdx !== -1 ? args[dirIdx + 1] : './export';
const pages = await engine.listPages({ limit: 100000 });
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
const repoIdx = args.indexOf('--repo');
const repoPath = repoIdx !== -1 ? args[repoIdx + 1] : null;
const typeIdx = args.indexOf('--type');
const typeFilter = typeIdx !== -1 ? args[typeIdx + 1] as PageType : undefined;
const slugPrefixIdx = args.indexOf('--slug-prefix');
const slugPrefix = slugPrefixIdx !== -1 ? args[slugPrefixIdx + 1] : undefined;
const restoreOnly = args.includes('--restore-only');
// Load storage configuration if repo path is provided
const storageConfig = repoPath ? loadStorageConfig(repoPath) : null;
// Build filters
const filters: any = { limit: 100000 };
if (typeFilter) {
filters.type = typeFilter;
}
let pages = await engine.listPages(filters);
// Apply slug prefix filter
if (slugPrefix) {
pages = pages.filter(page => page.slug.startsWith(slugPrefix));
}
// Apply restore-only filter
if (restoreOnly && repoPath && storageConfig) {
pages = pages.filter(page => {
// Only include supabase-only pages that are missing from disk
if (!isSupabaseOnly(page.slug, storageConfig)) {
return false;
}
const filePath = join(repoPath, page.slug + '.md');
return !existsSync(filePath);
});
}
if (restoreOnly) {
console.log(`Restoring ${pages.length} supabase-only pages to ${outDir}/`);
} else {
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
}
// Progress on stderr so stdout stays clean for scripts parsing counts.
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
@@ -52,5 +95,9 @@ export async function runExport(engine: BrainEngine, args: string[]) {
progress.finish();
// Stdout summary preserved so scripts that grep for "Exported N pages" keep working.
console.log(`Exported ${exported} pages to ${outDir}/`);
if (restoreOnly) {
console.log(`Restored ${exported} pages to ${outDir}/`);
} else {
console.log(`Exported ${exported} pages to ${outDir}/`);
}
}
+195
View File
@@ -0,0 +1,195 @@
import { existsSync, statSync } from 'fs';
import { join } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { loadStorageConfig, validateStorageConfig, getStorageTier } from '../core/storage-config.ts';
import type { StorageConfig, StorageTier } from '../core/storage-config.ts';
interface StorageStatusResult {
config: StorageConfig | null;
repoPath: string | null;
totalPages: number;
pagesByTier: Record<StorageTier, number>;
missingFiles: Array<{
slug: string;
expectedPath: string;
}>;
diskUsageByTier: Record<StorageTier, number>;
warnings: string[];
}
export async function runStorage(engine: BrainEngine, args: string[]) {
const subcommand = args[0];
if (!subcommand || subcommand === 'status') {
await runStorageStatus(engine, args.slice(1));
} else {
console.error(`Unknown storage subcommand: ${subcommand}`);
console.error('Available subcommands: status');
process.exit(1);
}
}
async function runStorageStatus(engine: BrainEngine, args: string[]) {
// Try to determine repo path from args or sync configuration
let repoPath: string | null = null;
const repoIdx = args.indexOf('--repo');
if (repoIdx !== -1 && args[repoIdx + 1]) {
repoPath = args[repoIdx + 1];
} else {
// Try to get from sync configuration
try {
const sources = await engine.executeRaw<{local_path: string | null}>(
`SELECT local_path FROM sources WHERE id = $1`,
['default']
);
if (sources[0]?.local_path) {
repoPath = sources[0].local_path;
}
} catch {
// Fall back to current directory if no sources configured
repoPath = process.cwd();
}
}
const result = await getStorageStatus(engine, repoPath);
if (args.includes('--json')) {
console.log(JSON.stringify(result, null, 2));
return;
}
// Human-readable output
console.log('Storage Status');
console.log('==============\n');
if (!result.config) {
console.log('No gbrain.yml configuration found.');
if (result.repoPath) {
console.log(`Checked: ${result.repoPath}/gbrain.yml`);
}
console.log('\nAll pages are stored in git by default.');
console.log(`Total pages: ${result.totalPages}`);
return;
}
console.log(`Repository: ${result.repoPath}`);
console.log(`Total pages: ${result.totalPages}\n`);
console.log('Storage Tiers:');
console.log('─────────────');
console.log(`Git tracked: ${result.pagesByTier.git_tracked.toLocaleString()} pages`);
console.log(`Supabase only: ${result.pagesByTier.supabase_only.toLocaleString()} pages`);
console.log(`Unspecified: ${result.pagesByTier.unspecified.toLocaleString()} pages`);
if (result.diskUsageByTier.git_tracked > 0 || result.diskUsageByTier.supabase_only > 0) {
console.log('\nDisk Usage:');
console.log('──────────');
if (result.diskUsageByTier.git_tracked > 0) {
console.log(`Git tracked: ${formatBytes(result.diskUsageByTier.git_tracked)}`);
}
if (result.diskUsageByTier.supabase_only > 0) {
console.log(`Supabase only: ${formatBytes(result.diskUsageByTier.supabase_only)}`);
}
if (result.diskUsageByTier.unspecified > 0) {
console.log(`Unspecified: ${formatBytes(result.diskUsageByTier.unspecified)}`);
}
}
if (result.missingFiles.length > 0) {
console.log('\nMissing Files (need restore):');
console.log('────────────────────────────');
for (const missing of result.missingFiles.slice(0, 10)) {
console.log(` ${missing.slug}`);
}
if (result.missingFiles.length > 10) {
console.log(` ... and ${result.missingFiles.length - 10} more`);
}
console.log(`\nUse: gbrain export --restore-only --repo "${result.repoPath}"`);
}
if (result.warnings.length > 0) {
console.log('\nWarnings:');
console.log('─────────');
for (const warning of result.warnings) {
console.log(` ⚠️ ${warning}`);
}
}
console.log('\nConfiguration:');
console.log('─────────────');
console.log('Git tracked directories:');
for (const dir of result.config.git_tracked) {
console.log(`${dir}`);
}
console.log('\nSupabase-only directories:');
for (const dir of result.config.supabase_only) {
console.log(`${dir}`);
}
}
async function getStorageStatus(engine: BrainEngine, repoPath: string | null): Promise<StorageStatusResult> {
const config = repoPath ? loadStorageConfig(repoPath) : null;
const warnings = config ? validateStorageConfig(config) : [];
// Get all pages from database
const pages = await engine.listPages({ limit: 1000000 });
// Categorize pages by storage tier
const pagesByTier: Record<StorageTier, number> = {
git_tracked: 0,
supabase_only: 0,
unspecified: 0
};
const diskUsageByTier: Record<StorageTier, number> = {
git_tracked: 0,
supabase_only: 0,
unspecified: 0
};
const missingFiles: Array<{ slug: string; expectedPath: string; }> = [];
for (const page of pages) {
const tier = config ? getStorageTier(page.slug, config) : 'unspecified';
pagesByTier[tier]++;
// Check if file exists and calculate disk usage
if (repoPath) {
const filePath = join(repoPath, page.slug + '.md');
if (existsSync(filePath)) {
try {
const stats = statSync(filePath);
diskUsageByTier[tier] += stats.size;
} catch {
// Ignore errors reading file stats
}
} else if (config && tier === 'supabase_only') {
// This is a supabase-only file that's missing from disk
missingFiles.push({
slug: page.slug,
expectedPath: filePath
});
}
}
}
return {
config,
repoPath,
totalPages: pages.length,
pagesByTier,
missingFiles,
diskUsageByTier,
warnings
};
}
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B';
const k = 1024;
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
}
+41 -1
View File
@@ -1,4 +1,4 @@
import { existsSync } from 'fs';
import { existsSync, readFileSync, writeFileSync } from 'fs';
import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import type { BrainEngine } from '../core/engine.ts';
@@ -14,6 +14,7 @@ import {
import type { SyncManifest } from '../core/sync.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { loadStorageConfig } from '../core/storage-config.ts';
export interface SyncResult {
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
@@ -599,6 +600,45 @@ export async function runSync(engine: BrainEngine, args: string[]) {
}
}
/**
* Auto-manage .gitignore entries for supabase-only directories
*/
async function manageGitignore(repoPath: string): Promise<void> {
const storageConfig = loadStorageConfig(repoPath);
if (!storageConfig || storageConfig.supabase_only.length === 0) {
return;
}
const gitignorePath = join(repoPath, '.gitignore');
let gitignoreContent = '';
// Read existing .gitignore
if (existsSync(gitignorePath)) {
gitignoreContent = readFileSync(gitignorePath, 'utf-8');
}
// Check which supabase-only directories are missing from .gitignore
const linesToAdd: string[] = [];
const existingLines = new Set(gitignoreContent.split('\n').map(line => line.trim()));
for (const dir of storageConfig.supabase_only) {
if (!existingLines.has(dir) && !existingLines.has(`/${dir}`)) {
linesToAdd.push(dir);
}
}
// Add missing entries
if (linesToAdd.length > 0) {
if (gitignoreContent && !gitignoreContent.endsWith('\n')) {
gitignoreContent += '\n';
}
gitignoreContent += '\n# Auto-managed by gbrain (supabase-only directories)\n';
gitignoreContent += linesToAdd.join('\n') + '\n';
writeFileSync(gitignorePath, gitignoreContent);
}
}
function printSyncResult(result: SyncResult) {
switch (result.status) {
case 'up_to_date':
+82
View File
@@ -0,0 +1,82 @@
import { readFileSync, existsSync } from 'fs';
import { join } from 'path';
import matter from 'gray-matter';
export interface StorageConfig {
git_tracked: string[];
supabase_only: string[];
}
export interface GBrainYamlConfig {
storage?: StorageConfig;
}
/**
* Load gbrain.yml configuration from the brain repository root.
* Returns null if no configuration file exists.
*/
export function loadStorageConfig(repoPath?: string): StorageConfig | null {
if (!repoPath) return null;
const yamlPath = join(repoPath, 'gbrain.yml');
if (!existsSync(yamlPath)) return null;
try {
const content = readFileSync(yamlPath, 'utf-8');
const parsed = matter(content);
const config = parsed.data as GBrainYamlConfig;
return config.storage || null;
} catch (error) {
console.warn(`Warning: Failed to parse gbrain.yml: ${error instanceof Error ? error.message : String(error)}`);
return null;
}
}
/**
* Validate storage configuration for conflicts and issues.
*/
export function validateStorageConfig(config: StorageConfig): string[] {
const warnings: string[] = [];
// Check for overlap between git_tracked and supabase_only
const gitSet = new Set(config.git_tracked);
const supabaseSet = new Set(config.supabase_only);
for (const path of config.supabase_only) {
if (gitSet.has(path)) {
warnings.push(`Directory "${path}" appears in both git_tracked and supabase_only`);
}
}
// Check if directories end with / for consistency
const allPaths = [...config.git_tracked, ...config.supabase_only];
for (const path of allPaths) {
if (!path.endsWith('/')) {
warnings.push(`Directory path "${path}" should end with "/" for consistency`);
}
}
return warnings;
}
/**
* Check if a slug matches any of the storage tier patterns.
*/
export function isGitTracked(slug: string, config: StorageConfig): boolean {
return config.git_tracked.some(dir => slug.startsWith(dir));
}
export function isSupabaseOnly(slug: string, config: StorageConfig): boolean {
return config.supabase_only.some(dir => slug.startsWith(dir));
}
/**
* Determine storage tier for a slug.
*/
export type StorageTier = 'git_tracked' | 'supabase_only' | 'unspecified';
export function getStorageTier(slug: string, config: StorageConfig): StorageTier {
if (isGitTracked(slug, config)) return 'git_tracked';
if (isSupabaseOnly(slug, config)) return 'supabase_only';
return 'unspecified';
}
+76
View File
@@ -0,0 +1,76 @@
import { test, expect, describe } from 'bun:test';
import {
validateStorageConfig,
isGitTracked,
isSupabaseOnly,
getStorageTier
} from '../src/core/storage-config.ts';
import type { StorageConfig } from '../src/core/storage-config.ts';
describe('Storage Configuration', () => {
const testConfig: StorageConfig = {
git_tracked: ['people/', 'companies/', 'deals/'],
supabase_only: ['media/x/', 'media/articles/', 'meetings/transcripts/']
};
describe('validateStorageConfig', () => {
test('should return no warnings for valid config', () => {
const warnings = validateStorageConfig(testConfig);
expect(warnings).toEqual([]);
});
test('should warn about overlap between git_tracked and supabase_only', () => {
const invalidConfig: StorageConfig = {
git_tracked: ['people/', 'media/'],
supabase_only: ['media/', 'articles/']
};
const warnings = validateStorageConfig(invalidConfig);
expect(warnings).toContain('Directory "media/" appears in both git_tracked and supabase_only');
});
test('should warn about paths not ending with /', () => {
const invalidConfig: StorageConfig = {
git_tracked: ['people', 'companies/'],
supabase_only: ['media/x/', 'articles']
};
const warnings = validateStorageConfig(invalidConfig);
expect(warnings).toContain('Directory path "people" should end with "/" for consistency');
expect(warnings).toContain('Directory path "articles" should end with "/" for consistency');
});
});
describe('Storage tier detection', () => {
test('should identify git-tracked pages', () => {
expect(isGitTracked('people/john-doe', testConfig)).toBe(true);
expect(isGitTracked('companies/acme-corp', testConfig)).toBe(true);
expect(isGitTracked('deals/series-a', testConfig)).toBe(true);
});
test('should identify supabase-only pages', () => {
expect(isSupabaseOnly('media/x/tweet-123', testConfig)).toBe(true);
expect(isSupabaseOnly('media/articles/blog-post', testConfig)).toBe(true);
expect(isSupabaseOnly('meetings/transcripts/standup', testConfig)).toBe(true);
});
test('should return false for non-matching paths', () => {
expect(isGitTracked('media/x/tweet-123', testConfig)).toBe(false);
expect(isSupabaseOnly('people/john-doe', testConfig)).toBe(false);
});
test('should correctly determine storage tier', () => {
expect(getStorageTier('people/john-doe', testConfig)).toBe('git_tracked');
expect(getStorageTier('media/x/tweet-123', testConfig)).toBe('supabase_only');
expect(getStorageTier('projects/random-thing', testConfig)).toBe('unspecified');
});
test('should handle edge cases', () => {
// Exact match shouldn't match (needs prefix)
expect(isGitTracked('people', testConfig)).toBe(false);
expect(isGitTracked('people/', testConfig)).toBe(true);
// Partial match shouldn't match
expect(isGitTracked('peoplex/test', testConfig)).toBe(false);
expect(isSupabaseOnly('mediax/test', testConfig)).toBe(false);
});
});
});