diff --git a/.agents/skills/clawhub-security-dataset/SKILL.md b/.agents/skills/clawhub-security-dataset/SKILL.md deleted file mode 100644 index 02b1016f..00000000 --- a/.agents/skills/clawhub-security-dataset/SKILL.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -name: clawhub-security-dataset -description: Use when generating, publishing, validating, or changing the ClawHub security/eval dataset export, nightly snapshot workflow, Hugging Face dataset upload, Convex export ingestion, or sanitizer/redaction logic. ---- - -# ClawHub Security Dataset - -This skill covers the maintainer-facing ClawHub security/eval dataset export. -The dataset is public; raw Convex exports, broad production credentials, storage -ids, internal document ids, and obvious secrets are not. - -## First Pass - -- Read `AGENTS.md`, `convex/_generated/ai/guidelines.md`, and the current - Linear coordination issue when this is issue-backed work. -- Check recent commits and security branches before editing: - -```sh -git fetch --all --prune -git log --oneline --decorate --date-order -18 origin/main -``` - -- Inspect the current command and data path: - -```sh -node -e "const p=require('./package.json'); for (const [k,v] of Object.entries(p.scripts)) if(k.includes('dataset')) console.log(k+'='+v)" -rg -n "securityDataset|dataset:snapshot|hf-dataset|convex-export|storageId|redaction" convex scripts .github/workflows package.json -``` - -## Command Map - -Dry-run a tiny live export against the configured Convex deployment: - -```sh -bun run dataset:snapshot -- --dry-run --limit 10 -``` - -Dry-run production only when the target deployment and auth boundary are clear: - -```sh -bun run dataset:snapshot:prod:dry-run -``` - -Build sanitized files from a local Convex export ZIP: - -```sh -bun run dataset:snapshot -- \ - --convex-export-zip /path/to/export.zip \ - --source-snapshot-id \ - --out-dir .data/security-dataset/snapshots \ - --hf-dataset -``` - -Useful flags: - -- `--prod` or `--deployment ` selects the Convex runtime. Do not use both. -- `--push` pushes current Convex functions before `convex run`; use only when - that is intentional. -- `--limit ` caps source artifacts for proof runs. -- `--source-kind skill|package|all` narrows source rows. -- `--created-after ` and `--created-before ` constrain by created - time. -- `--hf-dataset`, `--hf-repo`, `--hf-revision`, and `--hf-commit` control the - Hugging Face-shaped split files and manifest metadata. - -## Security Rules - -- Prefer the live sanitized Convex export path for nightly automation unless a - genuinely narrow Convex backup-reader credential exists. -- Do not automate backup downloads with broad deploy/admin credentials. -- Do not upload raw Convex backups, raw `_storage` blobs, raw `storageId`, or - internal Convex document ids to Hugging Face. -- The backup ZIP importer must resolve `_storage/` only so sanitizer - logic can redact and cap public output content. -- Reuse the shared sanitizer/normalizer for live exports and backup ZIP imports. - Do not add a second redaction policy without a manifest version bump. -- Hugging Face writes should use the dataset repo `OpenClaw/clawhub-security-signals` - and OIDC/trusted publishing when available. - -## Output Shape - -The maintained output is: - -- normalized JSONL: `artifacts.jsonl`, `scan_results.jsonl`, - `static_findings.jsonl`, `clawscan_findings.jsonl`, `labels.jsonl`, - `splits.jsonl` -- optional HF rows under `hf-dataset/data/{train,validation,test,eval_holdout}.jsonl` -- `manifest.json` with source id, exporter git SHA, redaction policy version, - row counts, output sizes, selected splits/configs, and HF repo/revision/commit - metadata when applicable - -## Validation - -Run focused tests after changing this subsystem: - -```sh -bun test scripts/security-dataset -``` - -For workflow or publish-path changes, also produce a dry-run proof with a small -limit and inspect the generated manifest/output: - -```sh -bun run dataset:snapshot -- --dry-run --limit 10 --hf-dataset -``` - -Before claiming output is safe, search generated sanitized files for forbidden -markers: - -```sh -grep -R -E 'storageId|skillVersions:|packageReleases:|_storage/' -grep -R -E 'gh[pousr]_[A-Za-z0-9_]{20,}|sk-[A-Za-z0-9_-]{20,}|AKIA[0-9A-Z]{16}|-----BEGIN [A-Z0-9 ]*(PRIVATE KEY|CERTIFICATE)-----' -``` - -If credentials are available, validate a manual workflow run against a test HF -revision before enabling or trusting the scheduled production upload. - -## Common Mistakes - -- Treating Convex Team Access Tokens as read-only backup credentials without - verifying Custom Roles or an equivalent permission boundary. -- Blessing a backup-based workflow after the team decided a live sanitized - export is safer without narrow backup-read auth. -- Documenting a successful dry run as a successful HF upload. -- Forgetting that `--dry-run` does not write snapshot files. -- Removing manual/admin export commands before the nightly path has equivalent - proof and a maintained entrypoint. diff --git a/README.md b/README.md index 5a1735a7..8ac45802 100644 --- a/README.md +++ b/README.md @@ -81,21 +81,6 @@ export CLAWHUB_DISABLE_TELEMETRY=1 Details: [`docs/telemetry.md`](docs/telemetry.md). -## Security dataset export - -Maintainers can generate a sanitized ClawHub security/eval dataset snapshot with: - -```bash -bun run dataset:snapshot -- --dry-run --limit 10 -bun run dataset:snapshot -- --convex-export-zip /path/to/convex-export.zip --hf-dataset -``` - -The exporter writes normalized JSONL files, optional Hugging Face split files, and a manifest with -row counts, source metadata, output sizes, and the redaction policy version. Do not use broad Convex -admin credentials for automated backup reads; prefer a live sanitized export path or a genuinely -narrow backup-reader credential. Future agents should read the repo-local `clawhub-security-dataset` -skill before changing the exporter, workflow, or Hugging Face publishing path. - ## Repo layout - `src/` — TanStack Start app (routes, components, styles).