diff --git a/.agents/skills/clawhub-security-dataset/SKILL.md b/.agents/skills/clawhub-security-dataset/SKILL.md new file mode 100644 index 00000000..02b1016f --- /dev/null +++ b/.agents/skills/clawhub-security-dataset/SKILL.md @@ -0,0 +1,127 @@ +--- +name: clawhub-security-dataset +description: Use when generating, publishing, validating, or changing the ClawHub security/eval dataset export, nightly snapshot workflow, Hugging Face dataset upload, Convex export ingestion, or sanitizer/redaction logic. +--- + +# ClawHub Security Dataset + +This skill covers the maintainer-facing ClawHub security/eval dataset export. +The dataset is public; raw Convex exports, broad production credentials, storage +ids, internal document ids, and obvious secrets are not. + +## First Pass + +- Read `AGENTS.md`, `convex/_generated/ai/guidelines.md`, and the current + Linear coordination issue when this is issue-backed work. +- Check recent commits and security branches before editing: + +```sh +git fetch --all --prune +git log --oneline --decorate --date-order -18 origin/main +``` + +- Inspect the current command and data path: + +```sh +node -e "const p=require('./package.json'); for (const [k,v] of Object.entries(p.scripts)) if(k.includes('dataset')) console.log(k+'='+v)" +rg -n "securityDataset|dataset:snapshot|hf-dataset|convex-export|storageId|redaction" convex scripts .github/workflows package.json +``` + +## Command Map + +Dry-run a tiny live export against the configured Convex deployment: + +```sh +bun run dataset:snapshot -- --dry-run --limit 10 +``` + +Dry-run production only when the target deployment and auth boundary are clear: + +```sh +bun run dataset:snapshot:prod:dry-run +``` + +Build sanitized files from a local Convex export ZIP: + +```sh +bun run dataset:snapshot -- \ + --convex-export-zip /path/to/export.zip \ + --source-snapshot-id \ + --out-dir .data/security-dataset/snapshots \ + --hf-dataset +``` + +Useful flags: + +- `--prod` or `--deployment ` selects the Convex runtime. Do not use both. +- `--push` pushes current Convex functions before `convex run`; use only when + that is intentional. +- `--limit ` caps source artifacts for proof runs. +- `--source-kind skill|package|all` narrows source rows. +- `--created-after ` and `--created-before ` constrain by created + time. +- `--hf-dataset`, `--hf-repo`, `--hf-revision`, and `--hf-commit` control the + Hugging Face-shaped split files and manifest metadata. + +## Security Rules + +- Prefer the live sanitized Convex export path for nightly automation unless a + genuinely narrow Convex backup-reader credential exists. +- Do not automate backup downloads with broad deploy/admin credentials. +- Do not upload raw Convex backups, raw `_storage` blobs, raw `storageId`, or + internal Convex document ids to Hugging Face. +- The backup ZIP importer must resolve `_storage/` only so sanitizer + logic can redact and cap public output content. +- Reuse the shared sanitizer/normalizer for live exports and backup ZIP imports. + Do not add a second redaction policy without a manifest version bump. +- Hugging Face writes should use the dataset repo `OpenClaw/clawhub-security-signals` + and OIDC/trusted publishing when available. + +## Output Shape + +The maintained output is: + +- normalized JSONL: `artifacts.jsonl`, `scan_results.jsonl`, + `static_findings.jsonl`, `clawscan_findings.jsonl`, `labels.jsonl`, + `splits.jsonl` +- optional HF rows under `hf-dataset/data/{train,validation,test,eval_holdout}.jsonl` +- `manifest.json` with source id, exporter git SHA, redaction policy version, + row counts, output sizes, selected splits/configs, and HF repo/revision/commit + metadata when applicable + +## Validation + +Run focused tests after changing this subsystem: + +```sh +bun test scripts/security-dataset +``` + +For workflow or publish-path changes, also produce a dry-run proof with a small +limit and inspect the generated manifest/output: + +```sh +bun run dataset:snapshot -- --dry-run --limit 10 --hf-dataset +``` + +Before claiming output is safe, search generated sanitized files for forbidden +markers: + +```sh +grep -R -E 'storageId|skillVersions:|packageReleases:|_storage/' +grep -R -E 'gh[pousr]_[A-Za-z0-9_]{20,}|sk-[A-Za-z0-9_-]{20,}|AKIA[0-9A-Z]{16}|-----BEGIN [A-Z0-9 ]*(PRIVATE KEY|CERTIFICATE)-----' +``` + +If credentials are available, validate a manual workflow run against a test HF +revision before enabling or trusting the scheduled production upload. + +## Common Mistakes + +- Treating Convex Team Access Tokens as read-only backup credentials without + verifying Custom Roles or an equivalent permission boundary. +- Blessing a backup-based workflow after the team decided a live sanitized + export is safer without narrow backup-read auth. +- Documenting a successful dry run as a successful HF upload. +- Forgetting that `--dry-run` does not write snapshot files. +- Removing manual/admin export commands before the nightly path has equivalent + proof and a maintained entrypoint. diff --git a/README.md b/README.md index 8ac45802..5a1735a7 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,21 @@ export CLAWHUB_DISABLE_TELEMETRY=1 Details: [`docs/telemetry.md`](docs/telemetry.md). +## Security dataset export + +Maintainers can generate a sanitized ClawHub security/eval dataset snapshot with: + +```bash +bun run dataset:snapshot -- --dry-run --limit 10 +bun run dataset:snapshot -- --convex-export-zip /path/to/convex-export.zip --hf-dataset +``` + +The exporter writes normalized JSONL files, optional Hugging Face split files, and a manifest with +row counts, source metadata, output sizes, and the redaction policy version. Do not use broad Convex +admin credentials for automated backup reads; prefer a live sanitized export path or a genuinely +narrow backup-reader credential. Future agents should read the repo-local `clawhub-security-dataset` +skill before changing the exporter, workflow, or Hugging Face publishing path. + ## Repo layout - `src/` — TanStack Start app (routes, components, styles).