mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 17:02:19 +00:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d134a1fb7e |
@@ -0,0 +1,188 @@
|
||||
# Storage Tiering: git-tracked vs supabase-only directories
|
||||
|
||||
## Overview
|
||||
|
||||
GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add a `storage` section to your `gbrain.yml` file in the brain repository root:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Directories that are git-tracked (version controlled, human-edited)
|
||||
git_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
- concepts/
|
||||
- yc/
|
||||
- ideas/
|
||||
- projects/
|
||||
|
||||
# Directories persisted via Supabase only (bulk ingest, machine-generated)
|
||||
# These are written to disk as a local cache but not committed to git.
|
||||
# `gbrain export` restores them from Supabase when missing.
|
||||
supabase_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
## Behavior Changes
|
||||
|
||||
### 1. `gbrain sync` - Automatic .gitignore Management
|
||||
|
||||
When storage configuration is present, `sync` automatically manages `.gitignore` entries:
|
||||
|
||||
- Adds missing `supabase_only` directory patterns to `.gitignore`
|
||||
- Prevents duplicate entries
|
||||
- Adds a comment section for auto-managed entries
|
||||
- Only runs when not in `--dry-run` mode
|
||||
|
||||
Example `.gitignore` addition:
|
||||
```gitignore
|
||||
# Auto-managed by gbrain (supabase-only directories)
|
||||
media/x/
|
||||
media/articles/
|
||||
meetings/transcripts/
|
||||
```
|
||||
|
||||
### 2. `gbrain export` - Enhanced Restore Capabilities
|
||||
|
||||
New flags for targeted export:
|
||||
|
||||
```bash
|
||||
# Restore only missing supabase-only files from database
|
||||
gbrain export --restore-only --repo /path/to/brain
|
||||
|
||||
# Filter by page type
|
||||
gbrain export --restore-only --type media --repo /path/to/brain
|
||||
|
||||
# Filter by slug prefix
|
||||
gbrain export --restore-only --slug-prefix media/x/ --repo /path/to/brain
|
||||
|
||||
# Combine filters
|
||||
gbrain export --restore-only --type media --slug-prefix media/x/ --repo /path/to/brain
|
||||
```
|
||||
|
||||
The `--restore-only` flag:
|
||||
- Only exports pages that match `supabase_only` patterns
|
||||
- Only includes pages where the file is missing from disk
|
||||
- Ideal for container restart recovery scenarios
|
||||
|
||||
### 3. `gbrain storage status` - Storage Health Dashboard
|
||||
|
||||
New command to inspect storage tier configuration and health:
|
||||
|
||||
```bash
|
||||
# Human-readable status
|
||||
gbrain storage status --repo /path/to/brain
|
||||
|
||||
# JSON output for scripts
|
||||
gbrain storage status --repo /path/to/brain --json
|
||||
```
|
||||
|
||||
Output includes:
|
||||
- Total page counts by storage tier
|
||||
- Disk usage breakdown
|
||||
- Missing files that need restoration
|
||||
- Configuration validation warnings
|
||||
- Current storage tier directory listing
|
||||
|
||||
Example output:
|
||||
```
|
||||
Storage Status
|
||||
==============
|
||||
|
||||
Repository: /data/brain
|
||||
Total pages: 15,243
|
||||
|
||||
Storage Tiers:
|
||||
─────────────
|
||||
Git tracked: 2,156 pages
|
||||
Supabase only: 12,887 pages
|
||||
Unspecified: 200 pages
|
||||
|
||||
Disk Usage:
|
||||
──────────
|
||||
Git tracked: 45.2 MB
|
||||
Supabase only: 2.1 GB
|
||||
|
||||
Missing Files (need restore):
|
||||
────────────────────────────
|
||||
media/x/tweet-1234567890
|
||||
media/x/tweet-0987654321
|
||||
... and 47 more
|
||||
|
||||
Use: gbrain export --restore-only --repo "/data/brain"
|
||||
|
||||
Configuration:
|
||||
─────────────
|
||||
Git tracked directories:
|
||||
• people/
|
||||
• companies/
|
||||
• deals/
|
||||
|
||||
Supabase-only directories:
|
||||
• media/x/
|
||||
• media/articles/
|
||||
• meetings/transcripts/
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
The system validates storage configuration and warns about:
|
||||
|
||||
- Directories appearing in both `git_tracked` and `supabase_only`
|
||||
- Directory paths not ending with `/` (consistency recommendation)
|
||||
- `supabase_only` directories not present in `.gitignore`
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. Brain Repository Scaling
|
||||
|
||||
Perfect for brain repositories approaching 100K+ files where:
|
||||
- Core knowledge (people, companies, deals) remains git-tracked
|
||||
- Bulk data (tweets, articles, transcripts) moves to supabase-only
|
||||
- Development stays fast with smaller git repos
|
||||
- Full data remains available via database
|
||||
|
||||
### 2. Container-based Deployments
|
||||
|
||||
Essential for ephemeral container environments:
|
||||
- Git repo contains only essential files
|
||||
- Container restarts don't lose supabase-only data
|
||||
- `gbrain export --restore-only` quickly restores bulk files when needed
|
||||
- Local disk acts as cache layer
|
||||
|
||||
### 3. Multi-Environment Consistency
|
||||
|
||||
Enables consistent data access across environments:
|
||||
- Development: small git clone, restore bulk data on demand
|
||||
- Production: full dataset via database, selective local caching
|
||||
- CI/CD: fast tests with git-tracked data only
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. **Assess current repository**: Use `gbrain storage status` to understand current distribution
|
||||
2. **Plan directory structure**: Identify which directories should be git-tracked vs supabase-only
|
||||
3. **Create gbrain.yml**: Add storage configuration to repository root
|
||||
4. **Test with dry-run**: Use `gbrain sync --dry-run` to verify behavior
|
||||
5. **Update .gitignore**: Let `gbrain sync` auto-manage entries
|
||||
6. **Verify exports**: Test `gbrain export --restore-only` for container scenarios
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Directory naming**: Always end storage paths with `/` for consistency
|
||||
- **Start small**: Begin with clearly machine-generated directories in `supabase_only`
|
||||
- **Monitor warnings**: Address configuration validation warnings promptly
|
||||
- **Test restore**: Regularly test `--restore-only` in staging environments
|
||||
- **Document decisions**: Comment your `gbrain.yml` to explain tier choices
|
||||
|
||||
## Compatibility
|
||||
|
||||
- **Backward compatible**: Systems without `gbrain.yml` work unchanged
|
||||
- **Progressive enhancement**: Add configuration when needed
|
||||
- **Database unchanged**: All data remains in Supabase regardless of tier
|
||||
- **Existing workflows**: All existing `sync` and `export` behavior preserved
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
storage:
|
||||
git_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
- concepts/
|
||||
- yc/
|
||||
- ideas/
|
||||
- projects/
|
||||
|
||||
supabase_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
+10
-1
@@ -19,7 +19,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage']);
|
||||
|
||||
async function main() {
|
||||
// Parse global flags (--quiet / --progress-json / --progress-interval)
|
||||
@@ -511,6 +511,11 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runSources(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'storage': {
|
||||
const { runStorage } = await import('./commands/storage.ts');
|
||||
await runStorage(engine, args);
|
||||
break;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
if (command !== 'serve') await engine.disconnect();
|
||||
@@ -581,6 +586,8 @@ IMPORT/EXPORT
|
||||
sync --watch [--interval N] Continuous sync (loops until stopped)
|
||||
sync --install-cron Install persistent sync daemon
|
||||
export [--dir ./out/] Export to markdown
|
||||
export --restore-only [--repo <p>] Restore missing supabase-only files
|
||||
[--type T] [--slug-prefix S] With optional filters
|
||||
|
||||
FILES
|
||||
files list [slug] List stored files
|
||||
@@ -643,6 +650,8 @@ ADMIN
|
||||
features [--json] [--auto-fix] Scan usage + recommend unused features
|
||||
autopilot [--repo] [--interval N] Self-maintaining brain daemon
|
||||
config [show|get|set] <key> [val] Brain config
|
||||
storage status [--repo <path>] Storage tier status and health
|
||||
[--json] (git-tracked vs supabase-only)
|
||||
serve MCP server (stdio)
|
||||
call <tool> '<json>' Raw tool invocation
|
||||
version Version info
|
||||
|
||||
+52
-5
@@ -1,16 +1,59 @@
|
||||
import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { writeFileSync, mkdirSync, existsSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { serializeMarkdown } from '../core/markdown.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { loadStorageConfig, isSupabaseOnly, getStorageTier } from '../core/storage-config.ts';
|
||||
import type { PageType } from '../core/types.ts';
|
||||
|
||||
export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
const dirIdx = args.indexOf('--dir');
|
||||
const outDir = dirIdx !== -1 ? args[dirIdx + 1] : './export';
|
||||
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
|
||||
|
||||
const repoIdx = args.indexOf('--repo');
|
||||
const repoPath = repoIdx !== -1 ? args[repoIdx + 1] : null;
|
||||
|
||||
const typeIdx = args.indexOf('--type');
|
||||
const typeFilter = typeIdx !== -1 ? args[typeIdx + 1] as PageType : undefined;
|
||||
|
||||
const slugPrefixIdx = args.indexOf('--slug-prefix');
|
||||
const slugPrefix = slugPrefixIdx !== -1 ? args[slugPrefixIdx + 1] : undefined;
|
||||
|
||||
const restoreOnly = args.includes('--restore-only');
|
||||
|
||||
// Load storage configuration if repo path is provided
|
||||
const storageConfig = repoPath ? loadStorageConfig(repoPath) : null;
|
||||
|
||||
// Build filters
|
||||
const filters: any = { limit: 100000 };
|
||||
if (typeFilter) {
|
||||
filters.type = typeFilter;
|
||||
}
|
||||
|
||||
let pages = await engine.listPages(filters);
|
||||
|
||||
// Apply slug prefix filter
|
||||
if (slugPrefix) {
|
||||
pages = pages.filter(page => page.slug.startsWith(slugPrefix));
|
||||
}
|
||||
|
||||
// Apply restore-only filter
|
||||
if (restoreOnly && repoPath && storageConfig) {
|
||||
pages = pages.filter(page => {
|
||||
// Only include supabase-only pages that are missing from disk
|
||||
if (!isSupabaseOnly(page.slug, storageConfig)) {
|
||||
return false;
|
||||
}
|
||||
const filePath = join(repoPath, page.slug + '.md');
|
||||
return !existsSync(filePath);
|
||||
});
|
||||
}
|
||||
if (restoreOnly) {
|
||||
console.log(`Restoring ${pages.length} supabase-only pages to ${outDir}/`);
|
||||
} else {
|
||||
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
|
||||
}
|
||||
|
||||
// Progress on stderr so stdout stays clean for scripts parsing counts.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
@@ -52,5 +95,9 @@ export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
|
||||
progress.finish();
|
||||
// Stdout summary preserved so scripts that grep for "Exported N pages" keep working.
|
||||
console.log(`Exported ${exported} pages to ${outDir}/`);
|
||||
if (restoreOnly) {
|
||||
console.log(`Restored ${exported} pages to ${outDir}/`);
|
||||
} else {
|
||||
console.log(`Exported ${exported} pages to ${outDir}/`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
import { existsSync, statSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadStorageConfig, validateStorageConfig, getStorageTier } from '../core/storage-config.ts';
|
||||
import type { StorageConfig, StorageTier } from '../core/storage-config.ts';
|
||||
|
||||
interface StorageStatusResult {
|
||||
config: StorageConfig | null;
|
||||
repoPath: string | null;
|
||||
totalPages: number;
|
||||
pagesByTier: Record<StorageTier, number>;
|
||||
missingFiles: Array<{
|
||||
slug: string;
|
||||
expectedPath: string;
|
||||
}>;
|
||||
diskUsageByTier: Record<StorageTier, number>;
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
export async function runStorage(engine: BrainEngine, args: string[]) {
|
||||
const subcommand = args[0];
|
||||
|
||||
if (!subcommand || subcommand === 'status') {
|
||||
await runStorageStatus(engine, args.slice(1));
|
||||
} else {
|
||||
console.error(`Unknown storage subcommand: ${subcommand}`);
|
||||
console.error('Available subcommands: status');
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
async function runStorageStatus(engine: BrainEngine, args: string[]) {
|
||||
// Try to determine repo path from args or sync configuration
|
||||
let repoPath: string | null = null;
|
||||
const repoIdx = args.indexOf('--repo');
|
||||
if (repoIdx !== -1 && args[repoIdx + 1]) {
|
||||
repoPath = args[repoIdx + 1];
|
||||
} else {
|
||||
// Try to get from sync configuration
|
||||
try {
|
||||
const sources = await engine.executeRaw<{local_path: string | null}>(
|
||||
`SELECT local_path FROM sources WHERE id = $1`,
|
||||
['default']
|
||||
);
|
||||
if (sources[0]?.local_path) {
|
||||
repoPath = sources[0].local_path;
|
||||
}
|
||||
} catch {
|
||||
// Fall back to current directory if no sources configured
|
||||
repoPath = process.cwd();
|
||||
}
|
||||
}
|
||||
|
||||
const result = await getStorageStatus(engine, repoPath);
|
||||
|
||||
if (args.includes('--json')) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
// Human-readable output
|
||||
console.log('Storage Status');
|
||||
console.log('==============\n');
|
||||
|
||||
if (!result.config) {
|
||||
console.log('No gbrain.yml configuration found.');
|
||||
if (result.repoPath) {
|
||||
console.log(`Checked: ${result.repoPath}/gbrain.yml`);
|
||||
}
|
||||
console.log('\nAll pages are stored in git by default.');
|
||||
console.log(`Total pages: ${result.totalPages}`);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Repository: ${result.repoPath}`);
|
||||
console.log(`Total pages: ${result.totalPages}\n`);
|
||||
|
||||
console.log('Storage Tiers:');
|
||||
console.log('─────────────');
|
||||
console.log(`Git tracked: ${result.pagesByTier.git_tracked.toLocaleString()} pages`);
|
||||
console.log(`Supabase only: ${result.pagesByTier.supabase_only.toLocaleString()} pages`);
|
||||
console.log(`Unspecified: ${result.pagesByTier.unspecified.toLocaleString()} pages`);
|
||||
|
||||
if (result.diskUsageByTier.git_tracked > 0 || result.diskUsageByTier.supabase_only > 0) {
|
||||
console.log('\nDisk Usage:');
|
||||
console.log('──────────');
|
||||
if (result.diskUsageByTier.git_tracked > 0) {
|
||||
console.log(`Git tracked: ${formatBytes(result.diskUsageByTier.git_tracked)}`);
|
||||
}
|
||||
if (result.diskUsageByTier.supabase_only > 0) {
|
||||
console.log(`Supabase only: ${formatBytes(result.diskUsageByTier.supabase_only)}`);
|
||||
}
|
||||
if (result.diskUsageByTier.unspecified > 0) {
|
||||
console.log(`Unspecified: ${formatBytes(result.diskUsageByTier.unspecified)}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (result.missingFiles.length > 0) {
|
||||
console.log('\nMissing Files (need restore):');
|
||||
console.log('────────────────────────────');
|
||||
for (const missing of result.missingFiles.slice(0, 10)) {
|
||||
console.log(` ${missing.slug}`);
|
||||
}
|
||||
if (result.missingFiles.length > 10) {
|
||||
console.log(` ... and ${result.missingFiles.length - 10} more`);
|
||||
}
|
||||
console.log(`\nUse: gbrain export --restore-only --repo "${result.repoPath}"`);
|
||||
}
|
||||
|
||||
if (result.warnings.length > 0) {
|
||||
console.log('\nWarnings:');
|
||||
console.log('─────────');
|
||||
for (const warning of result.warnings) {
|
||||
console.log(` ⚠️ ${warning}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('\nConfiguration:');
|
||||
console.log('─────────────');
|
||||
console.log('Git tracked directories:');
|
||||
for (const dir of result.config.git_tracked) {
|
||||
console.log(` • ${dir}`);
|
||||
}
|
||||
console.log('\nSupabase-only directories:');
|
||||
for (const dir of result.config.supabase_only) {
|
||||
console.log(` • ${dir}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function getStorageStatus(engine: BrainEngine, repoPath: string | null): Promise<StorageStatusResult> {
|
||||
const config = repoPath ? loadStorageConfig(repoPath) : null;
|
||||
const warnings = config ? validateStorageConfig(config) : [];
|
||||
|
||||
// Get all pages from database
|
||||
const pages = await engine.listPages({ limit: 1000000 });
|
||||
|
||||
// Categorize pages by storage tier
|
||||
const pagesByTier: Record<StorageTier, number> = {
|
||||
git_tracked: 0,
|
||||
supabase_only: 0,
|
||||
unspecified: 0
|
||||
};
|
||||
|
||||
const diskUsageByTier: Record<StorageTier, number> = {
|
||||
git_tracked: 0,
|
||||
supabase_only: 0,
|
||||
unspecified: 0
|
||||
};
|
||||
|
||||
const missingFiles: Array<{ slug: string; expectedPath: string; }> = [];
|
||||
|
||||
for (const page of pages) {
|
||||
const tier = config ? getStorageTier(page.slug, config) : 'unspecified';
|
||||
pagesByTier[tier]++;
|
||||
|
||||
// Check if file exists and calculate disk usage
|
||||
if (repoPath) {
|
||||
const filePath = join(repoPath, page.slug + '.md');
|
||||
if (existsSync(filePath)) {
|
||||
try {
|
||||
const stats = statSync(filePath);
|
||||
diskUsageByTier[tier] += stats.size;
|
||||
} catch {
|
||||
// Ignore errors reading file stats
|
||||
}
|
||||
} else if (config && tier === 'supabase_only') {
|
||||
// This is a supabase-only file that's missing from disk
|
||||
missingFiles.push({
|
||||
slug: page.slug,
|
||||
expectedPath: filePath
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
config,
|
||||
repoPath,
|
||||
totalPages: pages.length,
|
||||
pagesByTier,
|
||||
missingFiles,
|
||||
diskUsageByTier,
|
||||
warnings
|
||||
};
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes === 0) return '0 B';
|
||||
|
||||
const k = 1024;
|
||||
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
|
||||
}
|
||||
+41
-1
@@ -1,4 +1,4 @@
|
||||
import { existsSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { loadStorageConfig } from '../core/storage-config.ts';
|
||||
|
||||
export interface SyncResult {
|
||||
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
|
||||
@@ -599,6 +600,45 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-manage .gitignore entries for supabase-only directories
|
||||
*/
|
||||
async function manageGitignore(repoPath: string): Promise<void> {
|
||||
const storageConfig = loadStorageConfig(repoPath);
|
||||
if (!storageConfig || storageConfig.supabase_only.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const gitignorePath = join(repoPath, '.gitignore');
|
||||
let gitignoreContent = '';
|
||||
|
||||
// Read existing .gitignore
|
||||
if (existsSync(gitignorePath)) {
|
||||
gitignoreContent = readFileSync(gitignorePath, 'utf-8');
|
||||
}
|
||||
|
||||
// Check which supabase-only directories are missing from .gitignore
|
||||
const linesToAdd: string[] = [];
|
||||
const existingLines = new Set(gitignoreContent.split('\n').map(line => line.trim()));
|
||||
|
||||
for (const dir of storageConfig.supabase_only) {
|
||||
if (!existingLines.has(dir) && !existingLines.has(`/${dir}`)) {
|
||||
linesToAdd.push(dir);
|
||||
}
|
||||
}
|
||||
|
||||
// Add missing entries
|
||||
if (linesToAdd.length > 0) {
|
||||
if (gitignoreContent && !gitignoreContent.endsWith('\n')) {
|
||||
gitignoreContent += '\n';
|
||||
}
|
||||
gitignoreContent += '\n# Auto-managed by gbrain (supabase-only directories)\n';
|
||||
gitignoreContent += linesToAdd.join('\n') + '\n';
|
||||
|
||||
writeFileSync(gitignorePath, gitignoreContent);
|
||||
}
|
||||
}
|
||||
|
||||
function printSyncResult(result: SyncResult) {
|
||||
switch (result.status) {
|
||||
case 'up_to_date':
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import matter from 'gray-matter';
|
||||
|
||||
export interface StorageConfig {
|
||||
git_tracked: string[];
|
||||
supabase_only: string[];
|
||||
}
|
||||
|
||||
export interface GBrainYamlConfig {
|
||||
storage?: StorageConfig;
|
||||
}
|
||||
|
||||
/**
|
||||
* Load gbrain.yml configuration from the brain repository root.
|
||||
* Returns null if no configuration file exists.
|
||||
*/
|
||||
export function loadStorageConfig(repoPath?: string): StorageConfig | null {
|
||||
if (!repoPath) return null;
|
||||
|
||||
const yamlPath = join(repoPath, 'gbrain.yml');
|
||||
if (!existsSync(yamlPath)) return null;
|
||||
|
||||
try {
|
||||
const content = readFileSync(yamlPath, 'utf-8');
|
||||
const parsed = matter(content);
|
||||
const config = parsed.data as GBrainYamlConfig;
|
||||
return config.storage || null;
|
||||
} catch (error) {
|
||||
console.warn(`Warning: Failed to parse gbrain.yml: ${error instanceof Error ? error.message : String(error)}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate storage configuration for conflicts and issues.
|
||||
*/
|
||||
export function validateStorageConfig(config: StorageConfig): string[] {
|
||||
const warnings: string[] = [];
|
||||
|
||||
// Check for overlap between git_tracked and supabase_only
|
||||
const gitSet = new Set(config.git_tracked);
|
||||
const supabaseSet = new Set(config.supabase_only);
|
||||
|
||||
for (const path of config.supabase_only) {
|
||||
if (gitSet.has(path)) {
|
||||
warnings.push(`Directory "${path}" appears in both git_tracked and supabase_only`);
|
||||
}
|
||||
}
|
||||
|
||||
// Check if directories end with / for consistency
|
||||
const allPaths = [...config.git_tracked, ...config.supabase_only];
|
||||
for (const path of allPaths) {
|
||||
if (!path.endsWith('/')) {
|
||||
warnings.push(`Directory path "${path}" should end with "/" for consistency`);
|
||||
}
|
||||
}
|
||||
|
||||
return warnings;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if a slug matches any of the storage tier patterns.
|
||||
*/
|
||||
export function isGitTracked(slug: string, config: StorageConfig): boolean {
|
||||
return config.git_tracked.some(dir => slug.startsWith(dir));
|
||||
}
|
||||
|
||||
export function isSupabaseOnly(slug: string, config: StorageConfig): boolean {
|
||||
return config.supabase_only.some(dir => slug.startsWith(dir));
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine storage tier for a slug.
|
||||
*/
|
||||
export type StorageTier = 'git_tracked' | 'supabase_only' | 'unspecified';
|
||||
|
||||
export function getStorageTier(slug: string, config: StorageConfig): StorageTier {
|
||||
if (isGitTracked(slug, config)) return 'git_tracked';
|
||||
if (isSupabaseOnly(slug, config)) return 'supabase_only';
|
||||
return 'unspecified';
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
import { test, expect, describe } from 'bun:test';
|
||||
import {
|
||||
validateStorageConfig,
|
||||
isGitTracked,
|
||||
isSupabaseOnly,
|
||||
getStorageTier
|
||||
} from '../src/core/storage-config.ts';
|
||||
import type { StorageConfig } from '../src/core/storage-config.ts';
|
||||
|
||||
describe('Storage Configuration', () => {
|
||||
const testConfig: StorageConfig = {
|
||||
git_tracked: ['people/', 'companies/', 'deals/'],
|
||||
supabase_only: ['media/x/', 'media/articles/', 'meetings/transcripts/']
|
||||
};
|
||||
|
||||
describe('validateStorageConfig', () => {
|
||||
test('should return no warnings for valid config', () => {
|
||||
const warnings = validateStorageConfig(testConfig);
|
||||
expect(warnings).toEqual([]);
|
||||
});
|
||||
|
||||
test('should warn about overlap between git_tracked and supabase_only', () => {
|
||||
const invalidConfig: StorageConfig = {
|
||||
git_tracked: ['people/', 'media/'],
|
||||
supabase_only: ['media/', 'articles/']
|
||||
};
|
||||
const warnings = validateStorageConfig(invalidConfig);
|
||||
expect(warnings).toContain('Directory "media/" appears in both git_tracked and supabase_only');
|
||||
});
|
||||
|
||||
test('should warn about paths not ending with /', () => {
|
||||
const invalidConfig: StorageConfig = {
|
||||
git_tracked: ['people', 'companies/'],
|
||||
supabase_only: ['media/x/', 'articles']
|
||||
};
|
||||
const warnings = validateStorageConfig(invalidConfig);
|
||||
expect(warnings).toContain('Directory path "people" should end with "/" for consistency');
|
||||
expect(warnings).toContain('Directory path "articles" should end with "/" for consistency');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Storage tier detection', () => {
|
||||
test('should identify git-tracked pages', () => {
|
||||
expect(isGitTracked('people/john-doe', testConfig)).toBe(true);
|
||||
expect(isGitTracked('companies/acme-corp', testConfig)).toBe(true);
|
||||
expect(isGitTracked('deals/series-a', testConfig)).toBe(true);
|
||||
});
|
||||
|
||||
test('should identify supabase-only pages', () => {
|
||||
expect(isSupabaseOnly('media/x/tweet-123', testConfig)).toBe(true);
|
||||
expect(isSupabaseOnly('media/articles/blog-post', testConfig)).toBe(true);
|
||||
expect(isSupabaseOnly('meetings/transcripts/standup', testConfig)).toBe(true);
|
||||
});
|
||||
|
||||
test('should return false for non-matching paths', () => {
|
||||
expect(isGitTracked('media/x/tweet-123', testConfig)).toBe(false);
|
||||
expect(isSupabaseOnly('people/john-doe', testConfig)).toBe(false);
|
||||
});
|
||||
|
||||
test('should correctly determine storage tier', () => {
|
||||
expect(getStorageTier('people/john-doe', testConfig)).toBe('git_tracked');
|
||||
expect(getStorageTier('media/x/tweet-123', testConfig)).toBe('supabase_only');
|
||||
expect(getStorageTier('projects/random-thing', testConfig)).toBe('unspecified');
|
||||
});
|
||||
|
||||
test('should handle edge cases', () => {
|
||||
// Exact match shouldn't match (needs prefix)
|
||||
expect(isGitTracked('people', testConfig)).toBe(false);
|
||||
expect(isGitTracked('people/', testConfig)).toBe(true);
|
||||
|
||||
// Partial match shouldn't match
|
||||
expect(isGitTracked('peoplex/test', testConfig)).toBe(false);
|
||||
expect(isSupabaseOnly('mediax/test', testConfig)).toBe(false);
|
||||
});
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user