Compare commits

...
Author SHA1 Message Date
Dallin Romney f2ea75c921 fix: preserve scan report access 2026-06-12 10:08:39 -07:00
Dallin Romney 1e3bc9b02a refactor: extract github app common helpers 2026-06-12 10:06:06 -07:00
Dallin Romney af9cfce93b docs: add github sync runtime proof checklist 2026-06-12 10:06:06 -07:00
Dallin Romney be29a89a72 feat: gate github sync ui rollout 2026-06-12 10:06:06 -07:00
Dallin Romney 273d11c5a9 fix: address github sync review feedback 2026-06-12 10:06:06 -07:00
Dallin Romney cf6441add5 fix: preserve version scan blocks 2026-06-12 10:06:06 -07:00
Dallin Romney 869e340ecc fix: address final github sync review 2026-06-12 10:05:18 -07:00
Dallin Romney 2a2e6c538c fix: continue github sync candidate batches 2026-06-12 10:03:00 -07:00
Dallin Romney 9a2342d627 fix: enforce version scan gates for skill artifacts 2026-06-12 10:03:00 -07:00
Dallin Romney 85be25bccb fix: batch github app repository linking 2026-06-12 10:00:27 -07:00
Dallin Romney e953810d8b fix: enforce github app install target 2026-06-12 10:00:27 -07:00
Dallin Romney bdeb8d0120 fix: retry pending github app install callbacks 2026-06-12 10:00:27 -07:00
Dallin Romney 47454585be fix: address github app sync review findings 2026-06-12 10:00:27 -07:00
Dallin Romney bbc5e2c385 feat: add github app sync UI 2026-06-12 10:00:27 -07:00
Dallin Romney 8683f18d3a fix: harden github app sync handshake 2026-06-12 09:59:38 -07:00
Dallin Romney a01323e835 feat: add github app skill sync 2026-06-12 09:59:38 -07:00
Jesse Merhi b2447d750a Show unfiltered browse totals 2026-06-12 19:04:57 +10:00
Momo 8ce11888aa fix: tolerate duplicate github auth rows (#2612) 2026-06-12 14:49:49 +08:00
Momoandclawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> 3e66b50065 feat: expose skill description setup metadata (#2571)
* feat: expose skill description setup metadata

* feat: expose skill description setup metadata

* fix: narrow skill setup metadata contract

* fix: preserve required setup env metadata

* fix: import public skill file access helper

---------

Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
2026-06-12 11:19:38 +08:00
Patrick Erichsen 4f2d20e1e0 chore: remove clawscan cleanup migration
Remove the temporary ClawScan cleanup migration and narrow the schema after production cleanup.
2026-06-11 19:55:33 -07:00
Patrick Erichsen 1f66c9cde6 chore: add clawscan cleanup migration (#2610) 2026-06-11 19:12:31 -07:00
Patrick Erichsen dfac8d83ce fix: allow npm-compatible publisher handles
Allow publisher/org handles to use npm-compatible dots and underscores, update route validation and user-facing copy, and use neutral scoped package examples in docs/tests.
2026-06-11 18:45:25 -07:00
Patrick Erichsen a63d153b1d chore: release clawhub cli 0.21.0 2026-06-11 17:58:19 -07:00
Patrick Erichsen 9d49df109d ci: make package publish artifacts matrix-safe (#2606) 2026-06-11 17:56:04 -07:00
Patrick Erichsen 59c1ee3a7b chore: remove clawscan note remnants
Delete all remaining deprecated ClawScan note compatibility code, schema fields, tests, and spec text.
2026-06-11 17:54:45 -07:00
Patrick Erichsen a810d838bd fix: reserve docs owner handle (#2604) 2026-06-11 17:47:58 -07:00
Patrick Erichsen e319df0760 Move ClawHub emails to React Email (#2597)
* feat: update ClawHub email templates

* feat: move ClawHub emails to React Email

* fix: align local auth email assertion
2026-06-11 17:25:19 -07:00
Patrick Erichsen a1becd0c0f chore: remove autoban remediation tooling
Remove the obsolete one-off autoban remediation command/API, keep deprecated clawscan-note records from leaking through APIs, and retain legacy queue compatibility for old scan jobs.
2026-06-11 17:13:26 -07:00
Patrick Erichsen 2e7f3e752e fix: use no-reply sender for staff emails (#2602) 2026-06-11 16:58:38 -07:00
Patrick Erichsen 3b53ddbdbe chore: rename ClawHub admin package (#2599) 2026-06-11 16:49:17 -07:00
Patrick Erichsen 9a0671034c feat: expose package trusted publisher CLI commands (#2601)
* feat: expose package trusted publisher CLI commands

* docs: document package trusted publisher commands

* test: isolate clawhub artifact default help
2026-06-11 16:43:00 -07:00
Patrick Erichsen 6bc45b30ba fix: block malicious skill versions on download
Ensures skill file/card/download APIs block the exact requested malicious skill version, even when the current/latest version is clean.
2026-06-11 16:17:59 -07:00
Jesse MerhiandPatrick Erichsen 2c9aa36dcb feat: remove the souls feature end to end
Removes the SOULS content type and the SoulHub/onlycrabs.ai dual-site mode:
six Convex tables, the /api/v1/souls HTTP API, /souls routes, soul OG
images, GitHub soul backups, seeds, the VITE_FEATURE_SOULS flag, and the
site-mode machinery. Surviving skills-only code paths are de-branched and
simplified (tag resolution, publish form, nav/footer, ban flow, search).

Product decisions: /souls URLs and /api/v1/souls return plain 404s (no
redirect or 410 tombstone); reserved slugs souls/soulhub/onlycrabs stay.

Deploy prerequisite: clear the six soul tables and four storage blobs in
the prod Convex dashboard first (see PR description runbook).

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2026-06-11 10:45:58 -07:00
Patrick Erichsen 6fd3dcabf9 docs: add clawhub 0.20.2 release notes 2026-06-11 10:36:13 -07:00
Momoandclawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> 7166883d7d fix: tolerate duplicate CLI device user codes (#2589)
* fix: tolerate duplicate CLI device user codes

* fix: harden duplicate device code handling

* fix: tolerate duplicate CLI device user codes

---------

Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
2026-06-11 18:18:38 +08:00
Jason (Json) 4496acf9f0 fix: remove homepage suggestion chips (#2590) 2026-06-11 15:46:16 +10:00
Jesse Merhi d6500794c6 feat: rank skills by installs and dedupe install telemetry (#2535) 2026-06-11 14:22:11 +10:00
joshavant a1c91b265e chore: remove temporary auth identity cleanup tool 2026-06-10 19:19:55 -05:00
joshavant cd47bbe060 chore: add temporary auth identity cleanup tool 2026-06-10 19:12:34 -05:00
joshavant 31c69ff518 chore: remove temporary auth identity cleanup tool 2026-06-10 18:57:04 -05:00
Patrick Erichsen c22a2d295b fix: normalize ClawScan review visibility 2026-06-10 16:51:30 -07:00
joshavant f0b0185649 chore: add temporary auth identity cleanup tool 2026-06-10 18:46:50 -05:00
joshavant 7742ac7f52 chore: remove temporary identity remediation tool 2026-06-10 15:34:22 -05:00
joshavant e4ef192ce6 chore: add temporary identity remediation tool 2026-06-10 15:32:09 -05:00
joshavant 207ab1758e chore: remove github auth remediation tool 2026-06-10 14:34:21 -05:00
Patrick Erichsen 8446d65224 chore: remove migration lifecycle skill (#2592) 2026-06-10 12:32:57 -07:00
joshavant 58051dffcb chore: add github auth remediation tool 2026-06-10 14:14:08 -05:00
joshavant 0b6466184b fix: reject malformed github oauth ids 2026-06-10 13:14:04 -05:00
joshavant b3f98b4686 fix: prevent GitHub email account linking 2026-06-10 12:03:16 -05:00
Patrick Erichsen 79d9a86e6f ci: dispatch plugin inspector scans on pin bumps (#2584)
* ci: dispatch plugin inspector bulk scans on pin bump

* ci: rename plugin inspector bulk scan workflow

* fix: include CLI plugin inspector pin in scan dispatch

* chore: remove stale plugin inspector cron comment

* fix: dispatch inspector scans from main pushes
2026-06-09 21:45:29 -07:00
Patrick Erichsen 27faf50992 feat: surface author remediation for plugin validation
Adds author-remediation metadata and ClawHub validation UI/CLI/email handling for Plugin Inspector findings.
2026-06-09 21:09:25 -07:00
Patrick Erichsen e9571f3aa4 docs: remove stale skill folder guidance
Remove obsolete AGENTS.md guidance that treated repo skill additions as a standing contributor-policy issue.
2026-06-09 21:08:16 -07:00
Patrick Erichsen 18ec428fc5 docs: correct org delete moderation note (#2582) 2026-06-09 21:06:50 -07:00
Patrick Erichsen 0724abccbc Add guarded admin email and reserve clawhub handle (#2580)
* feat: add guarded admin email command

* fix: reserve clawhub publisher handle

* fix: harden admin org deletion and email audit
2026-06-09 20:56:48 -07:00
Patrick Erichsen 7b71c15ee1 fix: delegate browser automation scanning to SkillSpector
Remove ClawHub static-scanner browser automation rules because SkillSpector owns that analysis lane. Keep static moderation focused on non-browser-specific concrete source/sink evidence and update policy/spec wording accordingly.

Linear: CLAW-252
2026-06-09 20:55:08 -07:00
Patrick Erichsen d2921c0600 feat: add owner skill deletion (#2548) 2026-06-09 20:50:17 -07:00
Patrick Erichsen f5bf1b61e6 chore: remove stale maintenance backfills
Remove completed/stale maintenance backfills and their obsolete package artifact HTTP/CLI/schema surface. Keep the targeted legacy publisher ownership repair path.
2026-06-09 18:47:30 -07:00
Patrick Erichsen 6447101397 chore: remove clawhub ui proof skill 2026-06-09 16:29:58 -07:00
Patrick Erichsen d12adf2f37 chore: remove dependency registry cleanup 2026-06-09 16:29:13 -07:00
Patrick Erichsen 741599d20e fix: remove redundant ban email support note 2026-06-09 15:20:26 -07:00
Patrick Erichsen 9244087d00 chore: add admin alias for ClawHub operator CLI 2026-06-09 15:20:26 -07:00
Patrick Erichsen 23edf7d19d fix: split dependency cleanup pagination (#2573) 2026-06-09 11:32:52 -07:00
Jesse Merhi 1876698093 fix: simplify package listing cursors
* fix: simplify package listing cursors

* test: avoid asserting package cursor internals
2026-06-10 00:54:52 +10:00
Jason (Json)andJesse Merhi ad640d2903 fix: sort plugins by downloads with family index
* fix: sort plugin downloads by family index

* fix: reset legacy package download cursors

---------

Co-authored-by: Jesse Merhi <79823012+jesse-merhi@users.noreply.github.com>
2026-06-09 23:18:36 +10:00
Jesse Merhi c0143b1cd1 fix: retire dependency registry scans 2026-06-09 22:28:13 +10:00
clawsweeper[bot]chlinyclawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>momothemage
8970a46acf fix: use consistent installedAt timestamp for origin.json and lockfile (#2569)
Summary:
- The PR captures one `installedAt` value per install/update path and reuses it for both `.clawhub/origin.json` and the lockfile, with regression tests for timestamp equality.
- Reproducibility: yes. Source inspection of current `main` shows separate timestamp writes in `cmdInstall` an ... d report provides the CLI and `jq` reproduction path, though I did not execute it in this read-only review.

Automerge notes:
- PR branch already contained follow-up commit before automerge: test: add regression tests for installedAt timestamp equality
- PR branch already contained follow-up commit before automerge: fix: use consistent installedAt timestamp for origin.json and lockfile

Validation:
- ClawSweeper review passed for head b941ee37d6.
- Required merge gates passed before the squash merge.

Prepared head SHA: b941ee37d6
Review: https://github.com/openclaw/clawhub/pull/2569#issuecomment-4657649958

Co-authored-by: chliny <chliny11@gmail.com>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: momothemage
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
2026-06-09 08:16:53 +00:00
Jesse Merhi a3de2360b4 docs: clarify ClawHub vulnerability disclosure policy (#2567) 2026-06-09 17:38:56 +10:00
clawsweeper[bot]momothemageclawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>momothemage
620445f080 fix: slim package manage context (#2564)
Summary:
- The branch narrows `packages.getManageContext` to package/release identifier fields, adds unit and local-auth payload-capture coverage, and adjusts a WebCrypto digest helper.
- Reproducibility: yes. Source inspection of current main shows `getManageContext` returning the full package document and public release object, and the PR adds a focused test for the slim response shape.

Automerge notes:
- PR branch already contained follow-up commit before automerge: fix: slim package manage context

Validation:
- ClawSweeper review passed for head 5b5834bfc6.
- Required merge gates passed before the squash merge.

Prepared head SHA: 5b5834bfc6
Review: https://github.com/openclaw/clawhub/pull/2564#issuecomment-4655984471

Co-authored-by: momothemage <niuzhengnan@163.com>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: momothemage
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
2026-06-09 04:35:14 +00:00
Patrick Erichsen 89a89af183 fix: suppress inspector gap findings in clawhub 2026-06-08 21:29:31 -07:00
Patrick Erichsen 2d367c60ed Revert "Revert "chore: disable plugin inspector nightly schedule (#2558)" (#2…" (#2563)
This reverts commit 8b39d00247.
2026-06-08 20:51:15 -07:00
Patrick Erichsen 8b39d00247 Revert "chore: disable plugin inspector nightly schedule (#2558)" (#2562)
This reverts commit e09bf7f0fd.
2026-06-08 20:35:53 -07:00
Patrick Erichsen e6a8bce394 Fix package publish validation flow (#2561)
* feat: support prebuilt package publish artifacts

* fix: finalize package publish validation flow

* fix: polish plugin inspector findings email
2026-06-08 20:35:25 -07:00
Patrick Erichsen e09bf7f0fd chore: disable plugin inspector nightly schedule (#2558) 2026-06-08 20:09:23 -07:00
Patrick Erichsen c9bb130235 feat: support prebuilt package publish artifacts 2026-06-08 19:52:32 -07:00
Patrick Erichsen be70ae5e9f Publish public Plugin Inspector findings (#2525)
* feat: gate plugin publishes with inspector warnings

* fix: harden plugin publish inspector gate

* feat: publish public plugin inspector findings

* fix: refine plugin inspector publish errors

* feat: add package validation command

* fix: refine plugin validation findings UI

* fix: scan only latest plugin releases nightly

* feat: add dry run for nightly plugin inspection

* test: fix plugin validation tab e2e matcher
2026-06-08 19:41:48 -07:00
Vincent Koc 9321c83b93 fix: block moderated skill version metadata
Block exact skill-version metadata and scan responses when the requested version is the moderated source version. Apply the same guard to package-compatible skill version metadata and cover public-null fallback cases.
2026-06-09 11:23:55 +09:00
Patrick Erichsen 2a3e08e1bd fix: include clawscan summaries in blocked version emails (#2554) 2026-06-08 18:14:51 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 0ead9a60cf build(deps-dev): bump the development-minor-and-patch group with 3 updates (#2553)
Bumps the development-minor-and-patch group with 3 updates: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node), [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) and [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint).


Updates `@types/node` from 25.9.1 to 25.9.2
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `oxfmt` from 0.53.0 to 0.54.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.54.0/npm/oxfmt)

Updates `oxlint` from 1.68.0 to 1.69.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.69.0/npm/oxlint)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-version: 25.9.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
- dependency-name: oxfmt
  dependency-version: 0.54.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
- dependency-name: oxlint
  dependency-version: 1.69.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-08 15:02:38 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> f0274e41c4 build(deps): bump the production-minor-and-patch group with 13 updates (#2552)
Bumps the production-minor-and-patch group with 13 updates:

| Package | From | To |
| --- | --- | --- |
| [@radix-ui/react-avatar](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/avatar) | `1.1.11` | `1.1.12` |
| [@radix-ui/react-dialog](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/dialog) | `1.1.15` | `1.1.16` |
| [@radix-ui/react-dropdown-menu](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/dropdown-menu) | `2.1.16` | `2.1.17` |
| [@radix-ui/react-label](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/label) | `2.1.8` | `2.1.9` |
| [@radix-ui/react-select](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/select) | `2.2.6` | `2.3.0` |
| [@radix-ui/react-separator](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/separator) | `1.1.8` | `1.1.9` |
| [@radix-ui/react-slot](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/slot) | `1.2.4` | `1.2.5` |
| [@radix-ui/react-toggle-group](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/toggle-group) | `1.1.11` | `1.1.12` |
| [@radix-ui/react-tooltip](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/tooltip) | `1.2.8` | `1.2.9` |
| [@tanstack/react-router](https://github.com/TanStack/router/tree/HEAD/packages/react-router) | `1.170.12` | `1.170.15` |
| [@tanstack/react-start](https://github.com/TanStack/router/tree/HEAD/packages/react-start) | `1.168.21` | `1.168.25` |
| [semver](https://github.com/npm/node-semver) | `7.8.2` | `7.8.3` |
| [undici](https://github.com/nodejs/undici) | `7.27.1` | `7.27.2` |


Updates `@radix-ui/react-avatar` from 1.1.11 to 1.1.12
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/avatar/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/avatar)

Updates `@radix-ui/react-dialog` from 1.1.15 to 1.1.16
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/dialog/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/dialog)

Updates `@radix-ui/react-dropdown-menu` from 2.1.16 to 2.1.17
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/dropdown-menu/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/dropdown-menu)

Updates `@radix-ui/react-label` from 2.1.8 to 2.1.9
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/label/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/label)

Updates `@radix-ui/react-select` from 2.2.6 to 2.3.0
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/select/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/select)

Updates `@radix-ui/react-separator` from 1.1.8 to 1.1.9
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/separator/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/separator)

Updates `@radix-ui/react-slot` from 1.2.4 to 1.2.5
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/slot/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/slot)

Updates `@radix-ui/react-toggle-group` from 1.1.11 to 1.1.12
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/toggle-group/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/toggle-group)

Updates `@radix-ui/react-tooltip` from 1.2.8 to 1.2.9
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/tooltip/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/tooltip)

Updates `@tanstack/react-router` from 1.170.12 to 1.170.15
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-router@1.170.15/packages/react-router)

Updates `@tanstack/react-start` from 1.168.21 to 1.168.25
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-start/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-start@1.168.25/packages/react-start)

Updates `semver` from 7.8.2 to 7.8.3
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v7.8.2...v7.8.3)

Updates `undici` from 7.27.1 to 7.27.2
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.27.1...v7.27.2)

---
updated-dependencies:
- dependency-name: "@radix-ui/react-avatar"
  dependency-version: 1.1.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-dialog"
  dependency-version: 1.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-dropdown-menu"
  dependency-version: 2.1.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-label"
  dependency-version: 2.1.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-select"
  dependency-version: 2.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-separator"
  dependency-version: 1.1.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-slot"
  dependency-version: 1.2.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-toggle-group"
  dependency-version: 1.1.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-tooltip"
  dependency-version: 1.2.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-router"
  dependency-version: 1.170.15
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-start"
  dependency-version: 1.168.25
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: semver
  dependency-version: 7.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: undici
  dependency-version: 7.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-08 15:02:35 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 315cf9603f build(deps): bump github/codeql-action in the github-actions group (#2551)
Bumps the github-actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 4.36.1 to 4.36.2
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/87557b9c84dde89fdd9b10e88954ac2f4248e463...8aad20d150bbac5944a9f9d289da16a4b0d87c1e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.36.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-08 14:54:30 -07:00
Peter Steinberger 2c623b3beb fix: keep dev persona handles stable 2026-06-08 20:27:25 +01:00
Peter Steinberger 01612b4f15 style: fix changelog spacing 2026-06-08 20:11:37 +01:00
Peter Steinberger 295db69eba chore: require Node.js 22 for CLI packages 2026-06-08 20:01:44 +01:00
Patrick Erichsen 1fa4f9049d feat: add plugin export endpoint (#2547)
Add an authenticated /api/v1/plugins/export route that mirrors the skills export shape, supports an optional plugin family filter, defaults to both code and bundle plugins, and emits ZIP archives with manifest, error, and per-plugin metadata entries.

Autoreview findings addressed:

- [P1] Do not mark partially consumed plugin pages done

  Keep merged plugin export family cursor state active while buffered rows remain, and cover the pagination regression.

- [P1] Block release-level security states in plugin export

  Apply the package release download security block before reading release storage blobs.

- [P2] Avoid colliding with exported plugin metadata

  Move generated plugin metadata under __clawhub_export/ so plugin file paths cannot overwrite it.
2026-06-07 22:52:04 -07:00
Patrick Erichsen 5c19da48d0 fix: reserve admin owner handle (#2546) 2026-06-07 20:27:58 -07:00
Patrick Erichsen fc0a85e150 fix: make account recovery purge idempotent (#2543) 2026-06-06 19:08:23 -07:00
Patrick Erichsen 3a2568f751 fix: schedule account deletion digest sync safely 2026-06-06 18:41:44 -07:00
Patrick Erichsen 8e1bad4ac1 feat: improve account appeal notifications (#2520)
* feat: improve account appeal notifications

Adds Resend-backed account-action emails for bans/restores, public-safe ban reason summaries, scanner-triggered appeal scan guidance, and inline banned-account sign-in appeal messaging.

Autoreview findings addressed:

- [P2] Wire restored-account email into unban flow

- [P2] Add the scan upload test to the CI selection

* feat: quarantine malicious artifact uploads

* fix: warn on repeated malicious rejections

* fix: route banned sign-ins to appeal page

* feat: add stored scan report downloads

* chore: bump clawhub cli to 0.20.0

* test: cover malicious skill ban journey

* test: align deletion cleanup expectations after merge

* fix: hide deleted account packages before cleanup

* fix: finalize appeals moderation flow
2026-06-06 16:55:23 -07:00
Patrick Erichsen 8f7c21ac02 test: cover account recreation after deletion (#2540) 2026-06-06 16:47:19 -07:00
Patrick Erichsen 999131e2e7 fix: purge auth-locked deleted accounts
* fix: purge auth-locked deleted accounts

* test: wait for account deletion resources

* fix: mark packages inactive during deletion purge

* test: expect package soft delete before purge

* test: ignore post-delete auth timeout noise
2026-06-06 16:07:19 -07:00
Patrick Erichsen 571cc70c7a test: limit local-auth playwright retries (#2538) 2026-06-06 14:58:18 -07:00
Patrick Erichsen 3ead985f82 fix: restore post-merge CI gates (#2537) 2026-06-06 14:50:46 -07:00
Patrick Erichsen 1117aa4340 feat: add cohort temporal abuse scoring (#2536) 2026-06-06 14:26:46 -07:00
Patrick Erichsen 9ce5e702b5 feat: hard delete account-owned resources (#2533)
* feat: hard delete account-owned resources

* fix: expose account recovery purge dry-run candidates
2026-06-06 14:17:50 -07:00
Patrick Erichsen c632b697bb fix: hide banned user publisher profiles (#2528)
* fix: hide banned user publisher profiles

* docs: require real ClawHub screenshot proof

* test: stabilize local auth profile visibility checks
2026-06-05 17:33:03 -07:00
Patrick Erichsen d2525b179a fix: stop copying publisher ids to skill embeddings (#2532) 2026-06-05 17:28:55 -07:00
Patrick Erichsen 8ebae39140 feat: add temporal publisher abuse scan (#2531) 2026-06-05 17:14:57 -07:00
Patrick Erichsen 821494b9dd fix(cli): accept legacy verify json flag
Accept legacy skill verify --json usage as a hidden compatibility no-op, add a regression e2e for the flattened verifier response, and prepare clawhub@0.19.2.
2026-06-05 16:30:19 -07:00
Patrick Erichsen 94413a60fb feat: show plugin downloads in lists
Show plugin download counts in lists, add downloads sorting, and remove plugin family badges from plugin list/grid rows.
2026-06-05 16:18:00 -07:00
Patrick Erichsen 9cbf98297c fix: exclude official orgs from publisher abuse (#2527) 2026-06-05 15:59:10 -07:00
Patrick Erichsen 94ded18dec feat: add org deletion cascade
Closes #2516.
2026-06-05 14:52:31 -07:00
Patrick Erichsen 9897850074 fix: add targeted legacy publisher ownership repair
Adds a targeted legacy owner-publisher projection repair mutation for one user and covers the repair path in maintenance tests.
2026-06-05 14:24:29 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 159118dc59 build(deps): bump the github-actions group with 3 updates (#2473)
Bumps the github-actions group with 3 updates: [useblacksmith/begin-testbox](https://github.com/useblacksmith/begin-testbox), [github/codeql-action](https://github.com/github/codeql-action) and [trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog).


Updates `useblacksmith/begin-testbox` from d0e04585c26905fdd92c94a09c159544c7ee1b67 to 233448af4bfdc6fca509a7f0974411ac6d8a8043
- [Commits](https://github.com/useblacksmith/begin-testbox/compare/d0e04585c26905fdd92c94a09c159544c7ee1b67...233448af4bfdc6fca509a7f0974411ac6d8a8043)

Updates `github/codeql-action` from 4.36.0 to 4.36.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/7211b7c8077ea37d8641b6271f6a365a22a5fbfa...87557b9c84dde89fdd9b10e88954ac2f4248e463)

Updates `trufflesecurity/trufflehog` from 3.95.3 to 3.95.5
- [Release notes](https://github.com/trufflesecurity/trufflehog/releases)
- [Commits](https://github.com/trufflesecurity/trufflehog/compare/v3.95.3...v3.95.5)

---
updated-dependencies:
- dependency-name: useblacksmith/begin-testbox
  dependency-version: 233448af4bfdc6fca509a7f0974411ac6d8a8043
  dependency-type: direct:production
  dependency-group: github-actions
- dependency-name: github/codeql-action
  dependency-version: 4.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
- dependency-name: trufflesecurity/trufflehog
  dependency-version: 3.95.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-05 14:11:39 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 843f19e640 build(deps): bump commander from 14.0.3 to 15.0.0 (#2476)
Bumps [commander](https://github.com/tj/commander.js) from 14.0.3 to 15.0.0.
- [Release notes](https://github.com/tj/commander.js/releases)
- [Changelog](https://github.com/tj/commander.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tj/commander.js/compare/v14.0.3...v15.0.0)

---
updated-dependencies:
- dependency-name: commander
  dependency-version: 15.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-05 14:07:07 -07:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Peter Steinberger
4b882b2f43 build(deps): bump the production-minor-and-patch group across 1 directory with 12 updates (#2510)
* build(deps): bump the production-minor-and-patch group across 1 directory with 12 updates

Bumps the production-minor-and-patch group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [@convex-dev/auth](https://github.com/get-convex/convex-auth) | `0.0.92` | `0.0.93` |
| [@shikijs/rehype](https://github.com/shikijs/shiki/tree/HEAD/packages/rehype) | `4.1.0` | `4.2.0` |
| [@tanstack/react-router](https://github.com/TanStack/router/tree/HEAD/packages/react-router) | `1.170.8` | `1.170.12` |
| [@tanstack/react-start](https://github.com/TanStack/router/tree/HEAD/packages/react-start) | `1.168.13` | `1.168.21` |
| [convex](https://github.com/get-convex/convex-backend/tree/HEAD/npm-packages/convex) | `1.39.1` | `1.40.0` |
| [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.16.0` | `1.17.0` |
| [react](https://github.com/facebook/react/tree/HEAD/packages/react) | `19.2.6` | `19.2.7` |
| [react-dom](https://github.com/facebook/react/tree/HEAD/packages/react-dom) | `19.2.6` | `19.2.7` |
| [semver](https://github.com/npm/node-semver) | `7.8.1` | `7.8.2` |
| [shiki](https://github.com/shikijs/shiki/tree/HEAD/packages/shiki) | `4.1.0` | `4.2.0` |
| [undici](https://github.com/nodejs/undici) | `7.26.0` | `7.27.1` |
| [@clack/prompts](https://github.com/bombshell-dev/clack/tree/HEAD/packages/prompts) | `1.4.0` | `1.5.1` |



Updates `@convex-dev/auth` from 0.0.92 to 0.0.93
- [Changelog](https://github.com/get-convex/convex-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/get-convex/convex-auth/compare/v0.0.92...v0.0.93)

Updates `@shikijs/rehype` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/shikijs/shiki/releases)
- [Commits](https://github.com/shikijs/shiki/commits/v4.2.0/packages/rehype)

Updates `@tanstack/react-router` from 1.170.8 to 1.170.12
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-router@1.170.12/packages/react-router)

Updates `@tanstack/react-start` from 1.168.13 to 1.168.21
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-start/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-start@1.168.21/packages/react-start)

Updates `convex` from 1.39.1 to 1.40.0
- [Release notes](https://github.com/get-convex/convex-backend/releases)
- [Changelog](https://github.com/get-convex/convex-backend/blob/main/npm-packages/convex/CHANGELOG.md)
- [Commits](https://github.com/get-convex/convex-backend/commits/HEAD/npm-packages/convex)

Updates `lucide-react` from 1.16.0 to 1.17.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.17.0/packages/lucide-react)

Updates `react` from 19.2.6 to 19.2.7
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v19.2.7/packages/react)

Updates `react-dom` from 19.2.6 to 19.2.7
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v19.2.7/packages/react-dom)

Updates `semver` from 7.8.1 to 7.8.2
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v7.8.1...v7.8.2)

Updates `shiki` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/shikijs/shiki/releases)
- [Commits](https://github.com/shikijs/shiki/commits/v4.2.0/packages/shiki)

Updates `undici` from 7.26.0 to 7.27.1
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.26.0...v7.27.1)

Updates `@clack/prompts` from 1.4.0 to 1.5.1
- [Release notes](https://github.com/bombshell-dev/clack/releases)
- [Changelog](https://github.com/bombshell-dev/clack/blob/main/packages/prompts/CHANGELOG.md)
- [Commits](https://github.com/bombshell-dev/clack/commits/@clack/prompts@1.5.1/packages/prompts)

---
updated-dependencies:
- dependency-name: "@clack/prompts"
  dependency-version: 1.5.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: "@convex-dev/auth"
  dependency-version: 0.0.93
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@shikijs/rehype"
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-router"
  dependency-version: 1.170.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-start"
  dependency-version: 1.168.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: convex
  dependency-version: 1.40.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: lucide-react
  dependency-version: 1.17.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: react
  dependency-version: 19.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: react-dom
  dependency-version: 19.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: semver
  dependency-version: 7.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: shiki
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: undici
  dependency-version: 7.27.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* ci: update convex local auth runner

* ci: keep local auth deployment marker prefixed

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-05 13:50:43 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 634667c2c8 build(deps-dev): bump the development-minor-and-patch group across 1 directory with 5 updates (#2475)
Bumps the development-minor-and-patch group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [@vitest/coverage-v8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8) | `4.1.7` | `4.1.8` |
| [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) | `0.51.0` | `0.53.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) | `1.66.0` | `1.68.0` |
| [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.0.14` | `8.0.16` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.7` | `4.1.8` |



Updates `@vitest/coverage-v8` from 4.1.7 to 4.1.8
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/coverage-v8)

Updates `oxfmt` from 0.51.0 to 0.53.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.53.0/npm/oxfmt)

Updates `oxlint` from 1.66.0 to 1.68.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.68.0/npm/oxlint)

Updates `vite` from 8.0.14 to 8.0.16
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.0.16/packages/vite)

Updates `vitest` from 4.1.7 to 4.1.8
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/vitest)

---
updated-dependencies:
- dependency-name: "@vitest/coverage-v8"
  dependency-version: 4.1.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
- dependency-name: oxfmt
  dependency-version: 0.53.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
- dependency-name: oxlint
  dependency-version: 1.68.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
- dependency-name: vite
  dependency-version: 8.0.16
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
- dependency-name: vitest
  dependency-version: 4.1.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-05 13:26:14 -07:00
Patrick Erichsen bd30b182d7 feat: show skill scan queue progress 2026-06-05 11:35:08 -07:00
Patrick Erichsen 2d719feef5 fix: sync github status through skill digests 2026-06-04 21:02:54 -07:00
Patrick Erichsen d3e1059266 fix: remove org badge from publisher profiles 2026-06-04 21:02:54 -07:00
Patrick Erichsen 1a968d8243 fix: clean up publisher catalog rows (#2513)
* fix: clean up publisher catalog rows

* fix: type github sync soft deletes
2026-06-04 20:21:59 -07:00
Patrick Erichsen 59667cc1a5 fix: keep pending github skills visible (#2514) 2026-06-04 20:20:13 -07:00
Patrick Erichsen 5b0dd96530 fix: soft delete removed github skills (#2512) 2026-06-04 19:46:57 -07:00
Patrick Erichsen 8e3858a31d chore: release clawhub cli 0.19.1
Release clawhub CLI 0.19.1 so npm includes GitHub-backed skill install support.
2026-06-04 19:26:34 -07:00
Patrick Erichsen fe4acc8e10 feat: surface github sync issues (#2509) 2026-06-04 19:15:39 -07:00
Patrick Erichsen f2e6e87756 test: gate public routes in ci smoke (#2508) 2026-06-04 19:09:39 -07:00
Patrick Erichsen 3de69919de Add official org selector to GitHub sync settings (#2505)
* feat: select official org for github sync sources

* fix: rename github sync publisher field
2026-06-04 19:08:23 -07:00
Patrick Erichsen 7629433248 fix: skip publisher handle conflicts in ownership repair (#2507) 2026-06-04 19:06:38 -07:00
Patrick Erichsen b82ad43eae fix: align org publisher catalog tabs (#2506) 2026-06-04 19:05:33 -07:00
Patrick Erichsen b7ee5c8e62 fix: backfill legacy publisher ownership (#2503)
* fix: backfill legacy publisher ownership

* test: update publisher official mocks
2026-06-04 18:57:42 -07:00
Patrick Erichsen 1c8543ade6 feat: add official org mod commands (#2504) 2026-06-04 18:47:12 -07:00
Patrick Erichsen 7cd0ef362d fix: select publisher previews by installs (#2502) 2026-06-04 18:39:47 -07:00
Patrick Erichsen 9cf9e12bdd GitHub-backed skill source sync (#2498)
* feat: add github-backed skill seed schema

* feat: render github-backed skill manifests

* feat: add github-backed skill install resolver

* feat: add github-backed skill source sync

* fix: handle github resolver blocks

* fix: harden github source sync

* fix: expose github install blocks

* feat: support GitHub-backed skill sync

* fix: handle blocked github skill content

* fix: harden github sync rollout

* fix: bound github sync payloads

* feat: allow forcing pending github skill installs

* fix: remove official publisher seed rollout

* chore: format clawhub e2e test
2026-06-04 18:20:43 -07:00
Patrick Erichsen 112e25e28c fix: retry incomplete github sign-ins (#2501) 2026-06-04 17:18:32 -07:00
Jesse Merhi d786374b77 feat: add shared download metering (#2485) 2026-06-04 16:48:08 -07:00
Jesse Merhi 3fbe27560e feat: add publisher abuse review dashboard (#2471) 2026-06-05 09:05:50 +10:00
Patrick Erichsen 33539b6df3 fix: point account recovery to github issues (#2499) 2026-06-04 14:41:52 -07:00
clawsweeper[bot]clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>momothemage
354f12a033 fix(httpApiV1): surface 400-class delete validation errors instead of opaque 500 (#2496)
Summary:
- The branch narrows the soft-delete bad-request substring whitelist from any `reserved` message to the ClawHu ... rvation phrase and adds regression tests for the intended 400 path and an unrelated reserved-word 500 path.
- Reproducibility: yes. by source inspection: current main maps any cleaned soft-delete error containing `rese ...  while preserving the package route-reservation case. I did not execute tests during this read-only review.

Automerge notes:
- No ClawSweeper repair was needed after automerge opt-in.

Validation:
- ClawSweeper review passed for head a58687c54e.
- Required merge gates passed before the squash merge.

Prepared head SHA: a58687c54e
Review: https://github.com/openclaw/clawhub/pull/2496#issuecomment-4620039541

Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: momothemage
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
2026-06-04 07:43:22 +00:00
Momo b0ea80df6c fix(httpApiV1): surface 400-class delete validation errors instead of opaque 500 (#2488)
Merged via squash.

Prepared head SHA: cc6ac9d31a
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Reviewed-by: @momothemage
2026-06-04 15:32:30 +08:00
Patrick Erichsen 909a47106e chore: bump clawhub cli to 0.19.0 (#2486) 2026-06-03 16:19:42 -07:00
Patrick Erichsen e8cfbddf17 feat: add clawhub scan command (#2479)
* docs: design clawhub scan command

* docs: plan clawhub scan command

* feat: add clawhub scan command

* fix: prune ephemeral scan uploads

* fix: avoid scan route slug collision
2026-06-03 16:05:11 -07:00
Patrick Erichsen 162528abe4 feat: show plugin download counts (#2484) 2026-06-03 14:46:20 -07:00
Vyctor H. Brzezowski 74aa61086e fix: gate local Codex workers (#2472) 2026-06-03 12:26:20 -07:00
Momo 858a121d33 fix(plugins): rewrite relative README image URLs to source-host raw URLs (#2412)
Merged via squash.

Prepared head SHA: 545db16f01
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Reviewed-by: @momothemage
2026-06-03 17:05:58 +08:00
Patrick Erichsen 953358a322 fix: use GitHub App auth for GitHub account lookups
* fix: use GitHub App auth for GitHub account lookups

* style: format GitHub account age backfill
2026-06-03 00:24:23 -07:00
Patrick Erichsen 0a79612fe5 feat: add security audit download export (#2477) 2026-06-02 22:20:05 -07:00
Peter Steinberger 9b5d2e088d fix: speed GitHub account-age backfill scan 2026-06-02 13:01:59 +01:00
Peter Steinberger ce62df9d08 fix: batch GitHub account-age backfill writes 2026-06-02 11:52:21 +01:00
Peter Steinberger 0abdbf4a50 fix: harden GitHub account age lookup 2026-06-02 11:32:16 +01:00
Jesse Merhi dcbc38999f fix: require multipart package publishes
Require multipart package publishes so public clients cannot provide trusted file or artifact metadata.

Preserve released CLI multipart field names (`files` and `clawpack`), derive stored file and npm-pack artifact metadata from uploaded bytes, and keep large ClawPack tarballs working through ticketed staged uploads.

Tests:
- bunx vitest run convex/uploads.test.ts --reporter=dot
- bunx vitest run convex/httpApi.handlers.test.ts convex/httpApiV1.handlers.test.ts --testNamePattern "package publish|multipart package publish|multipart ClawPack|staged ClawPack|browser session auth|cliUploadUrl|upload tickets" --reporter=dot
- bun run --cwd packages/clawhub test:src -- src/cli/commands/packages.test.ts --testNamePattern "ClawPack tarballs over|stages ClawPack|publishes a ClawPack|cleans generated ClawPack"
- bun run format:check
- bun run lint
- bunx tsc -p packages/schema/tsconfig.json --noEmit --pretty false
- bunx tsc -p packages/clawhub/tsconfig.json --noEmit --pretty false
- bunx tsc --noEmit --pretty false
- bun run ci:unit
- git diff --check origin/main
2026-06-02 09:59:53 +10:00
Jesse Merhi 01aa28ccda feat: make recommended the default skill ranking
Make Recommended the default public skill ranking while preserving the v1 API no-sort default. Adds recommended/default API support, digest rank indexes/backfill safety, OpenAPI/docs updates, and regression coverage.
2026-06-02 08:20:37 +10:00
Jesse Merhi cb6ced7906 fix: keep skill card metadata grouped (#2465)
Move skill card tag/platform pills below the description and keep author/update/stat metadata grouped at the bottom of grid cards.

Thanks @jesse-merhi.
2026-06-02 05:02:40 +10:00
Vyctor H. Brzezowski ded9ff4235 fix: stabilize auth loading states (#2447) 2026-06-01 10:01:04 -07:00
Val AlexanderandBunsDev 9fc2da4dc4 fix: clarify paginated plugin count (#2393)
* fix: clarify paginated plugin count

* fix: correct shown-count pluralization

Agent-Logs-Url: https://github.com/openclaw/clawhub/sessions/32575bd4-9dc7-4b8f-9536-bdc96f008783

Co-authored-by: BunsDev <68980965+BunsDev@users.noreply.github.com>
2026-05-31 01:00:02 -05:00
Patrick Erichsen 05d5fc1151 docs: explain skill catalog publishing (#2455)
* docs: explain skill publishing workflow

* docs: polish ClawHub publishing guide
2026-05-30 22:14:59 -05:00
Patrick Erichsen 9aaab158cb feat(cli): print stored auth token (#2454) 2026-05-30 21:11:51 -05:00
Patrick Erichsen 6fc5bb7cd8 feat: add reusable skill publish workflow (#2450)
* feat: add reusable skill publish workflow

* test: cover exact skill sync roots

* fix: keep skill workflow scans exact
2026-05-30 20:26:51 -05:00
Patrick Erichsen 9aa3f37ee1 feat: export skill bundle content for security datasets (#2448) 2026-05-30 12:11:19 -05:00
Patrick Erichsen 9a20795b54 feat: add NVIDIA as official publisher (#2445) 2026-05-30 00:37:40 -05:00
Patrick Erichsen ff75a7e9ae fix(auth): avoid false sign-in failure on redirect (#2443) 2026-05-29 16:13:19 -05:00
Patrick Erichsen 4c965f4957 fix: repair pending vt skill versions (#2442) 2026-05-29 16:10:36 -05:00
Patrick Erichsen f71139e9ae feat: add VT pending repair command (#2441) 2026-05-29 15:07:29 -05:00
ShadowandPatrick Erichsen a20e2efd68 feat: add ban appeals service endpoints (#2408)
Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2026-05-29 14:55:56 -05:00
Dallin Romney 83cc4d0f87 fix: surface organization creation errors (#2440) 2026-05-29 14:39:44 -05:00
Peter Steinberger 1f7f483b1d fix: route diffs language pack slug 2026-05-29 19:45:51 +01:00
Vyctor H. Brzezowski 309723f7a8 fix(cli): report unknown root commands (#2433) 2026-05-29 11:03:53 -05:00
Peter Steinberger 23932ec7de fix(packages): store npm pack entries sequentially 2026-05-29 15:21:55 +01:00
Peter Steinberger b292f7eaf5 fix(auth): let admins publish without github age lookup 2026-05-29 13:41:49 +01:00
Peter Steinberger cbbb6e7a61 fix(packages): allow large npm pack entries over HTTP 2026-05-29 13:36:54 +01:00
Peter Steinberger 42e9690e78 fix(packages): allow large files inside npm pack artifacts 2026-05-29 13:32:05 +01:00
Patrick Erichsen ff48b2cc70 feat!: remove ClawScan note feature (#2432)
BREAKING CHANGE: ClawScan publisher notes are no longer accepted by publish APIs, CLI commands, schema packages, or UI flows.
2026-05-28 16:00:45 -05:00
Peter Steinberger 562810f29b docs: position README banner 2026-05-28 20:47:46 +01:00
Peter Steinberger 327231535f docs: add README banner 2026-05-28 19:43:01 +01:00
Patrick Erichsen 51967bca7f feat: export skillspector issue details (#2430) 2026-05-28 10:32:28 -05:00
Patrick Erichsen 0132f1f530 fix: preserve Codex diagnostic messages (#2429)
* fix: remove global ClawScan claim ceiling

* fix: preserve codex diagnostic messages
2026-05-28 08:46:13 -05:00
Patrick Erichsen 05f27f640e feat: export skillspector dataset signals (#2424) 2026-05-28 08:40:54 -05:00
Vyctor H. Brzezowski 6adf379f32 feat: add publish flow guidance shortcuts (#2423)
* feat: add publish flow guidance shortcuts

* style: polish publish follow-up surfaces

* fix: route soul publish guide link
2026-05-27 20:55:43 -05:00
Patrick Erichsen 7d6efae74b fix: remove global ClawScan claim ceiling (#2422) 2026-05-27 17:35:48 -05:00
Vyctor H. BrzezowskiandPatrick Erichsen d854449610 fix: restrict membership management to org publishers (#2285)
* fix: restrict membership management to org publishers

* fix(api): ignore stale personal publisher memberships

* fix(api): reject stale personal publisher publish targets

* fix(api): reject stale personal publisher memberships

* fix(api): use personal publisher links for package access

* fix(api): guard personal publisher owner scopes

* fix: enforce publisher ownership for skill reads

* fix: narrow personal publisher dashboard owner

* fix: ignore stale personal package memberships

* fix: allow own legacy personal skill destination

* fix: preserve legacy personal package dashboards

* fix: preserve legacy personal skill dashboards

* fix: avoid redundant personal publisher boolean coercion

* fix: preserve legacy personal publisher access

* fix: include legacy direct packages in personal dashboard

* fix: close stale personal publisher ownership gaps

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2026-05-27 14:34:13 -07:00
Vyctor H. BrzezowskiandPatrick Erichsen 87f2b846ef fix: hide package resources when owners are banned (#2283)
* fix: hide package resources when owners are banned

* fix(api): attribute unban package restores to moderator

* fix(web): clarify package effects in ban confirmations

* fix(api): continue package ban batches during in-flight bans

* fix: keep package unban restore scoped to ban batch

* fix: retimestamp package releases during repeated bans

* fix: start package ban batches after user ban commit

* fix: preserve manual package moderation after unban

* fix: cover personal publisher package sanctions

* fix: restore personal publisher packages in autoban remediation

* fix: bound package publish token revocation batches

* fix: clear package ban reason during remediation restore

* fix: block direct package ban restores

* fix: scan linked personal publisher packages during sanctions

* fix: tighten package sanction restore batches

* fix: block personal publisher publishes after owner ban

* fix: allow initial package ban cleanup before commit

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2026-05-27 14:17:39 -07:00
Vyctor H. BrzezowskiandPatrick Erichsen 97023d3123 fix(api): guard moderated skill files and tags (#2287)
* fix(api): guard moderated skill files and tags

* fix(api): guard public list latest version ownership

* fix(api): guard stale latest version outputs

* fix(api): keep legacy digest latest versions

* fix(api): guard public latest-version readers

* fix: avoid ambiguous array allocation in skill export

* fix: drop legacy markerless digest versions

* fix: verify markerless digest versions

* test: mark package catalog digest versions

* fix: keep skill list tag resolution on digest path

* test: mark resolved skill versions with owner

* fix: repair digest capability backfill skip

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2026-05-27 13:34:54 -07:00
Patrick Erichsen a920323a86 ci: fan out skill card worker (#2419) 2026-05-27 14:50:18 -05:00
Patrick Erichsen b8eaada68d fix: frame skillspector findings as advisory (#2418) 2026-05-27 14:29:38 -05:00
Vyctor H. BrzezowskiandPatrick Erichsen 18acbc1209 fix: abort stale unban skill restore batches (#2284)
* fix: abort stale unban skill restore batches

* fix(api): keep ban restore markers monotonic

* fix(api): restore legacy ban-hidden skills on unban

* fix: retimestamp legacy ban-hidden skills on re-ban

* fix: abort stale scheduled skill ban pages

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2026-05-27 12:14:52 -07:00
Patrick Erichsen 57bc9f2a46 feat: add bulk skill rescan admin tool (#2413)
* feat: add bulk skill rescan backend queue

* feat: expose bulk skill rescan admin API

* feat: add bulk skill rescan mod command

* style: format bulk rescan changes

* fix: print bulk rescan cli progress

* fix: include legacy active skills in bulk rescans
2026-05-27 14:03:03 -05:00
Vyctor H. BrzezowskiandPatrick Erichsen 6f893b54f4 fix: block direct skill transfers under moderation (#2282)
* fix: block direct skill transfers under moderation

* fix(api): block accepted transfers for moderated skills

* fix(api): block malware-flagged skill transfers

* fix: cover moderated skill transfer bypasses

* test: support ownership heal transfer sync

* fix: close moderated transfer backfill gaps

* fix: block soft-deleted transfer guard state

---------

Co-authored-by: Patrick Erichsen <patrick.a.erichsen@gmail.com>
2026-05-27 12:01:51 -07:00
Patrick Erichsen 8bb6a0d584 chore: remove repo-local generic skills (#2417) 2026-05-27 13:48:05 -05:00
Vyctor H. Brzezowski 321df223b2 fix(web): polish skill and plugin upload forms (#2415)
* feat: add shared publish form UI pieces

* fix: improve skill publish form UX

* fix: align plugin publish form UX

* fix: refine publish form polish follow-ups

* fix: refine publish upload states

* fix(web): finalize publish upload polish

* test: use ClawHub logo for owner avatar fixture

* fix(web): avoid premature skill publish field errors

* test: adapt local auth publish flow to owner picker

* fix(web): keep plugin publish locked after success
2026-05-27 11:10:50 -07:00
Vyctor H. Brzezowski 707d390923 fix: revalidate package publish owners before insert (#2281)
* fix: revalidate package publish owners before insert

* fix(api): recheck package publisher membership before insert
2026-05-27 11:03:11 -07:00
Patrick Erichsen 3c4608156c fix(cli): remove redundant skill verify json flag (#2401) 2026-05-27 12:52:15 -05:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 9c97d643ac build(deps): bump github/codeql-action (#2306)
Bumps the github-actions group with 1 update in the / directory: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 4.35.4 to 4.36.0
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/68bde559dea0fdcac2102bfdf6230c5f70eb485e...7211b7c8077ea37d8641b6271f6a365a22a5fbfa)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.35.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-27 12:36:25 -05:00
Patrick Erichsen a66df774f2 feat: make UI proof full-stack by default (#2210)
* feat: add full-stack UI proof backend

* docs: prefer here.now links for issue screenshots
2026-05-27 12:33:09 -05:00
Patrick Erichsen 30bf8f252a feat: add org member removal to clawhub-mod (#2416) 2026-05-27 12:27:16 -05:00
Patrick Erichsen 5ed0ddd066 fix: keep mod org creation actor out of members (#2360) 2026-05-27 12:03:29 -05:00
Jesse Merhi ce1be46c60 docs: clarify security report scope (#2410) 2026-05-27 14:06:25 +10:00
Patrick Erichsen 667bc55299 Add bulk skill security verdicts endpoint (#2404)
* feat: add bulk skill security verdicts endpoint

* fix: keep bulk verdict lookup lightweight

* fix: harden bulk verdict review findings
2026-05-26 10:05:56 -07:00
Jesse Merhi 90de729fe1 fix: simplify official publisher policy (#2380) 2026-05-26 21:44:43 +10:00
Jesse Merhi 8a2c0c06fd Add publisher abuse dry run (#2381)
* feat: add publisher abuse dry run

* chore: remove publisher abuse management UI

* chore: remove unused management styles

* fix: harden publisher abuse review queue

* fix: refresh passing abuse nominations

* fix: bound abuse queue to actionable labels

* fix: keep abuse runs model-consistent

* fix: ignore zero-skill publishers in abuse scoring

* fix: expand filtered abuse queue scan

* fix: score publisher abuse with skill stats

* fix: harden abuse scoring cohort stats

* fix: bound publisher abuse score runs

* fix: keep abuse scoring compatible with existing publisher stats

* fix: recompute missing publisher skill totals

* fix: avoid aggregate fallback for mixed publisher abuse stats

* fix: harden publisher abuse score runs

* fix: derive missing publisher abuse skill counts

* fix: handle unknown abuse publisher package counts

* fix: reopen actionable abuse nominations

* fix: bound abuse cron fallback scoring

* fix: keep publisher stat deltas bounded

* fix: skip unknown-base cron abuse scores

* fix: skip mixed unknown cron abuse scores

* fix: bound manual abuse fallback scoring

* fix: prevent overlapping abuse score runs

* fix: preserve abuse triage notes

* fix: score legacy mixed publishers in abuse cron

* fix: keep publisher abuse dry run internal
2026-05-26 14:35:29 +10:00
Patrick Erichsen 07fed45f42 fix: reshape skill verify security signals (#2402) 2026-05-25 18:49:56 -07:00
Patrick Erichsen cc16d7fbd9 fix: route static scan findings through clawscan (#2398)
* fix: route static scan findings through clawscan

* fix: preserve clawscan-only security verdicts after merge
2026-05-25 18:33:33 -07:00
Patrick Erichsen 1f56a71430 fix: stabilize local-auth publish lifecycle e2e (#2400) 2026-05-25 18:04:35 -07:00
540 changed files with 92007 additions and 20827 deletions
+1 -1
View File
@@ -55,7 +55,7 @@ Pick the smallest proof that matches the touched surface:
| Formatting/lint/static repo health | `bun run ci:static` |
| Unit-tested source behavior | focused `bunx vitest run ...`, then `bun run ci:unit` when PR-ready |
| Convex code | read `convex/_generated/ai/guidelines.md` first; run focused tests and the deploy/typecheck path that covers the change |
| Packages/CLI/mod tool | `bun run ci:packages` or the package-specific `verify` script |
| Packages/CLI/admin tool | `bun run ci:packages` or the package-specific `verify` script |
| Runtime/build/package surface | `bun run ci:types-build`, `bun run ci:e2e-http`, or the matching broader gate |
| UI behavior | use `clawhub-ui-proof` with `proof:ui`; publish proof before final PR comments when needed |
| Linux/CI-parity validation | use `crabbox`, normally through the repo scripts |
+146 -29
View File
@@ -1,85 +1,198 @@
---
name: clawhub-moderation
description: "Use for ClawHub staff moderation actions with the repo-local clawhub-mod tool: unhide skills, ban or unban users, change roles, and verify moderation state."
description: "Use for ClawHub staff moderation actions with the repo-local ClawHub admin tool: skills, users, org publishers, plugin packages, trusted publishers, official publishers, and guarded staff email."
---
# ClawHub Moderation
Use the repo-local `clawhub-mod` tool from a checked-out ClawHub repo. It wraps
Use the repo-local admin tool from a checked-out ClawHub repo. It wraps
the existing ClawHub CLI auth/config and HTTP API surfaces. Do not call Convex
internal mutations directly for staff actions.
## Safety Rules
- Require an explicit target from the user: skill slug, user handle, or user id.
- Require a reason for `skills unhide`, `users ban`, and `users unban`.
- Require a reason for destructive, restorative, ownership, or moderation writes.
- Before any write, show the exact command and ask for confirmation unless the
user already said to proceed or supplied `--yes`.
- For `email send`, the user must explicitly ask for the email and sign off on
the final recipient, subject, and body. Dry-run is fine for drafting. Never
send until both are true, and only use `--send --confirm-user-request
--confirm-user-signoff` after that approval.
- Prefer handles for humans. Use `--id` only when the user provides a user id.
- Never bypass API-token auth, server role checks, or audit logging.
- After the write, verify state with the CLI/API and report the result.
## Commands
## Command Map
Run from the ClawHub repo root:
```sh
bun run mod -- --help
bun run admin -- --help
```
Authenticate or validate the current token:
```sh
bun run mod -- login
bun run mod -- whoami
bun run admin -- login
bun run admin -- whoami
```
Unhide a skill after moderator review:
Current top-level command groups:
```sh
bun run mod -- skills unhide <slug> --reason "<reason>" --yes
```text
auth
users
plugins|plugin
packages|package
org
email
skills|skill
```
List and triage skill reports:
### Skills
```sh
bun run mod -- skills reports --status open
bun run mod -- skills triage-report <report-id> --status confirmed --action hide --note "<note>" --yes
`bun run admin -- skills --help` exposes:
```text
unhide <slug>
rescan <slug>
reports
triage-report <report-id>
```
Ban a user:
Examples:
```sh
bun run mod -- users ban <handleOrId> --reason "<reason>" --yes
bun run admin -- skills unhide <slug> --reason "<reason>" --yes
bun run admin -- skills rescan <slug> --reason "<reason>" --yes
bun run admin -- skills reports --status open
bun run admin -- skills triage-report <report-id> --status confirmed --action hide --note "<note>" --yes
```
Unban a user:
### Users
```sh
bun run mod -- users unban <handleOrId> --reason "<reason>" --yes
`bun run admin -- users --help` exposes:
```text
ban <handleOrId>
unban <handleOrId>
set-role <handleOrId> <role>
reclassify-ban <handleOrId>
remediate-autobans
```
Change a user role:
Examples:
```sh
bun run mod -- users set-role <handleOrId> <user|moderator|admin> --yes
bun run admin -- users ban <handleOrId> --reason "<reason>" --yes
bun run admin -- users unban <handleOrId> --reason "<reason>" --yes
bun run admin -- users set-role <handleOrId> <user|moderator|admin> --yes
bun run admin -- users reclassify-ban <handleOrId> --reason "<reason>" --apply --yes
bun run admin -- users remediate-autobans --apply --reason "<reason>"
```
Use `--id` when `<handleOrId>` is a user id. Use `--fuzzy` only when the user
has asked for fuzzy handle resolution or the exact handle is ambiguous.
The old top-level aliases still exist for user commands:
### Org Publishers
`bun run admin -- org --help` exposes:
```text
official
create <handle>
remove-member <handle> <member>
delete <handle>
repair-scoped-packages <csv>
```
Examples:
```sh
bun run mod -- ban-user <handleOrId> --reason "<reason>" --yes
bun run mod -- unban-user <handleOrId> --reason "<reason>" --yes
bun run admin -- org official list
bun run admin -- org official add <handle> --reason "<reason>" --yes
bun run admin -- org official remove <handle> --reason "<reason>" --yes
bun run admin -- org create <handle> --display-name "<name>" --member <user-handle> --role owner
bun run admin -- org remove-member <handle> <member-handle>
bun run admin -- org delete <handle> --reason "<reason>" # dry-run
bun run admin -- org delete <handle> --reason "<reason>" --apply
bun run admin -- org repair-scoped-packages <csv> # dry-run
bun run admin -- org repair-scoped-packages <csv> --apply
```
`org create` requires `--member`; it must not add the moderator running the
command as an implicit owner. `org delete` only works for empty org publishers
and defaults to dry-run.
### Plugin Packages
`bun run admin -- packages --help` exposes:
```text
moderate <name>
status|moderation-status <name>
queue|moderation-queue
reports
triage-report <report-id>
transfer <name>
repair-name <name>
migrations
set-migration <bundled-plugin-id>
trusted-publisher
```
Examples:
```sh
bun run admin -- packages status <name>
bun run admin -- packages transfer <name> --to <owner> --reason "<reason>" # dry-run
bun run admin -- packages transfer <name> --to <owner> --reason "<reason>" --apply
bun run admin -- packages repair-name <name> --next-name <name> --reason "<reason>"
bun run admin -- packages trusted-publisher get <name>
bun run admin -- packages trusted-publisher set <name> --repository <owner/repo> --workflow-filename <file>
```
### Staff Email
`bun run admin -- email send --help` exposes:
```text
--to <email>
--user <handle>
--subject <subject>
--body-file <path>
--body <text>
--send
--confirm-user-request
--confirm-user-signoff
--json
```
Draft only:
```sh
bun run admin -- email send --user <handle> --subject "<subject>" --body-file <path>
```
Send only after explicit request and sign-off:
```sh
bun run admin -- email send --user <handle> --subject "<subject>" --body-file <path> --send --confirm-user-request --confirm-user-signoff
```
The server sends through the production noreply provider and writes an audit log
only after admin auth succeeds.
## Verification
- For skills, inspect the page/API status after `skills unhide`.
- For users, prefer `bun run mod -- whoami` for the current token and user
search/admin surfaces for target accounts where available.
- For users, prefer user search/admin surfaces for target accounts where
available.
- For orgs and packages, use the public publisher/plugin pages and the relevant
CLI status command after a write.
- For email, verify the CLI response and audit expectation; do not send a second
email just to test delivery.
- If verification is blocked by auth or missing admin access, report the command
result and the verification blocker plainly.
@@ -88,9 +201,13 @@ bun run mod -- unban-user <handleOrId> --reason "<reason>" --yes
- `skills unhide` is a moderator manual restore. It clears skill hidden state,
applies a clean manual override to top-level moderation fields, preserves
version-level scanner records, updates public stats, and writes audit logs.
- There is no standalone `skills hide` command in `clawhub-mod`; use report
- There is no standalone `skills hide` command in `clawhub-admin`; use report
triage with `--action hide` when resolving a report that should hide a skill.
- `ban-user` is disruptive: it revokes API tokens, marks the user deleted,
- `users ban` is disruptive: it revokes API tokens, marks the user deleted,
hides owned skills, soft-deletes comments, and writes audit logs.
- `unban-user` is admin-only. It clears ban state and restores skills that were
- `users unban` is admin-only. It clears ban state and restores skills that were
hidden by the matching ban flow; revoked API tokens stay revoked.
- `packages transfer` preserves the package row, stats, releases, and history;
it changes the owner publisher.
- `org delete` soft-deletes an empty org publisher and retains member rows for
history; it refuses orgs with active skills or packages.
-75
View File
@@ -1,75 +0,0 @@
---
name: clawhub-ui-proof
description: Use when ClawHub UI changes need visual proof, before/after comparison, new-feature screenshots, temporary Playwright scenarios, or Crabbox desktop recordings.
---
# ClawHub UI Proof
Use `proof:ui` for human-readable UI evidence. The agent should write a
temporary scenario for the feature instead of manually clicking through the UI.
## Pick A Mode
- Use `--mode before-after` for bug fixes, regressions, changed copy, changed
layout, or anything where main-vs-candidate comparison helps. This is the
default and runs baseline `origin/main` plus the candidate worktree.
- Use `--mode feature` for new pages, new workflows, or new UI states that do
not exist on main. This runs only the candidate lane.
- Do not use `proof:ui` to inspect contributor-provided screenshots, videos, or
logs. Review those artifacts directly and cite what they prove or fail to
prove.
## Scenario Shape
Create a temporary scenario under `.artifacts/proof-scenarios/`:
```js
export default async function scenario({ baseURL, expect, page, proof }) {
await proof.step("01 skills list", async () => {
await page.goto(`${baseURL}/skills`);
await expect(page.getByText("Skills")).toBeVisible();
});
}
```
Each `proof.step()` captures a screenshot after the step. The runner compares
`origin/main` to the current worktree by default in `before-after` mode.
## Commands
Dry-run the plan first. Before/after mode is the default:
```sh
bun run proof:ui -- --mode before-after --scenario .artifacts/proof-scenarios/my-fix.pw.ts --dry-run
```
For new feature proof, run candidate-only:
```sh
bun run proof:ui -- --mode feature --scenario .artifacts/proof-scenarios/my-feature.pw.ts --dry-run
```
Run real desktop proof on a Crabbox-owned provider:
```sh
bun run proof:ui -- --mode before-after --scenario .artifacts/proof-scenarios/my-fix.pw.ts --provider hetzner
```
Artifacts are written under `.artifacts/clawhub-ui-proof/<timestamp>/` with
screenshots, videos when available, `summary.json`, and `report.md`. Feature
mode has only candidate artifacts. Promote only broadly useful scenarios into
committed `e2e/proofs/`.
## Publish To A PR
When UI proof should appear on a GitHub PR, publish the completed proof run
instead of posting local paths:
```sh
bun run proof:publish -- --proof-dir .artifacts/clawhub-ui-proof/<timestamp> --target-pr <number>
```
`proof:publish` copies the selected screenshots, video preview GIFs when
present, MP4s, `summary.json`, and `report.md` to the `qa-artifacts` branch,
then upserts a marker-backed PR comment with inline screenshots/previews and
linked MP4s. Use `--dry-run` first when drafting or checking the comment body.
-51
View File
@@ -1,51 +0,0 @@
---
name: crabbox
description: Use when ClawHub needs remote Linux validation, CI-parity checks, broad Bun gates, hosted-service checks, desktop/VNC inspection, or Crabbox lease cleanup.
---
# Crabbox
Crabbox is ClawHub's agent-facing isolation layer. Use direct `blacksmith`
commands only as a backend emergency fallback; normal agents should go through
the repo scripts below.
## Fast Checks
Run from the repo root:
```sh
bun run crabbox:run -- --help
bun run crabbox:warmup -- --provider blacksmith-testbox --blacksmith-org openclaw --blacksmith-workflow .github/workflows/ci-check-testbox.yml --blacksmith-job check
```
The wrapper prefers `../crabbox/bin/crabbox` when present and rejects stale
binaries that do not support the Blacksmith Testbox provider. For desktop UI
proof, use a Crabbox-owned provider such as `hetzner` or `aws`; the
`blacksmith-testbox` provider cannot expose VNC, screenshots, or desktop
artifacts.
## Common Remote Validation
Broad ClawHub gates:
```sh
bun run crabbox:run -- --provider blacksmith-testbox --shell -- "bun run ci:static"
bun run crabbox:run -- --provider blacksmith-testbox --shell -- "VITE_CONVEX_URL=https://example.invalid bun run coverage"
```
Reusable desktop lease:
```sh
bun run crabbox:warmup -- --provider hetzner --desktop --browser --class standard --idle-timeout 60m --ttl 120m
bun run crabbox:run -- --provider hetzner --id <cbx_id-or-slug> --keep --shell -- "bun run test"
bun run crabbox:stop -- --provider hetzner <cbx_id-or-slug>
```
## Cleanup
Stop leases created for the task before handoff unless the user asked to keep
one open for WebVNC inspection:
```sh
bun run crabbox:stop -- --provider <provider> <id-or-slug>
```
+12 -3
View File
@@ -1,9 +1,6 @@
# Frontend
VITE_CONVEX_URL=
VITE_CONVEX_SITE_URL=
VITE_SOULHUB_SITE_URL=
VITE_SOULHUB_HOST=
VITE_SITE_MODE=
VITE_ENABLE_DEV_AUTH=
SITE_URL=http://localhost:3000
CONVEX_SITE_URL=
@@ -16,5 +13,17 @@ AUTH_GITHUB_SECRET=
JWT_PRIVATE_KEY=
JWKS=
# Local dev personas
DEV_AUTH_ENABLED=
DEV_AUTH_CONVEX_DEPLOYMENT=
DEV_AUTH_SITE_URL=
DEV_AUTH_SECRET=
# Embeddings
OPENAI_API_KEY=
# Transactional email
RESEND_API_KEY=
CLAWHUB_SECURITY_EMAIL=security@notifications.openclaw.ai
CLAWHUB_SECURITY_EMAIL_FROM=ClawHub Security <noreply@notifications.openclaw.ai>
CLAWHUB_NOREPLY_FROM=ClawHub <noreply@notifications.openclaw.ai>
+15
View File
@@ -1,5 +1,19 @@
version: 2
updates:
- package-ecosystem: "bun"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
time: "09:00"
timezone: "America/Los_Angeles"
open-pull-requests-limit: 3
labels:
- dependencies
- needs-cli-release
allow:
- dependency-name: "@openclaw/plugin-inspector"
- package-ecosystem: "bun"
directory: "/"
schedule:
@@ -9,6 +23,7 @@ updates:
timezone: "America/Los_Angeles"
open-pull-requests-limit: 10
ignore:
- dependency-name: "@openclaw/plugin-inspector"
- dependency-name: "@auth/core"
update-types:
- "version-update:semver-minor"
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Begin Testbox
uses: useblacksmith/begin-testbox@d0e04585c26905fdd92c94a09c159544c7ee1b67
uses: useblacksmith/begin-testbox@233448af4bfdc6fca509a7f0974411ac6d8a8043
with:
testbox_id: ${{ inputs.testbox_id }}
+2 -2
View File
@@ -88,13 +88,13 @@ jobs:
- name: Initialize CodeQL
if: ${{ github.event_name != 'workflow_dispatch' || inputs.profile == 'all' || inputs.profile == matrix.category }}
uses: github/codeql-action/init@68bde559dea0fdcac2102bfdf6230c5f70eb485e # v4
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
with:
languages: ${{ matrix.language }}
config-file: ${{ matrix.config_file }}
- name: Analyze
if: ${{ github.event_name != 'workflow_dispatch' || inputs.profile == 'all' || inputs.profile == matrix.category }}
uses: github/codeql-action/analyze@68bde559dea0fdcac2102bfdf6230c5f70eb485e # v4
uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
with:
category: "/codeql-light/${{ matrix.category }}"
+237 -6
View File
@@ -61,11 +61,24 @@ on:
description: Optional source path inside the repository for monorepo package publishes.
required: false
type: string
clawhub_version:
description: Legacy npm CLI version input. Kept for compatibility; the workflow now runs the checked-out source.
package_artifact_name:
description: Optional Actions artifact name containing a prebuilt ClawPack .tgz to publish.
required: false
type: string
default: latest
package_artifact_path:
description: Optional path to the .tgz inside package_artifact_name. Defaults to the only .tgz in the artifact.
required: false
type: string
inspector_artifact_name:
description: Artifact name for plugin inspector reports. Set a unique value when calling this workflow from a matrix.
required: false
type: string
default: plugin-inspector-report
publish_json_artifact_name:
description: Artifact name for the package publish JSON output. Set a unique value when calling this workflow from a matrix.
required: false
type: string
default: clawhub-package-publish-json
secrets:
clawhub_token:
required: false
@@ -87,6 +100,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
actions: read
contents: read
id-token: write
outputs:
@@ -218,7 +232,56 @@ jobs:
PY
echo "CLAWHUB_CONFIG_PATH=$RUNNER_TEMP/clawhub-config.json" >> "$GITHUB_ENV"
- name: Download prebuilt package artifact
if: inputs.package_artifact_name != ''
uses: actions/download-artifact@v8
with:
name: ${{ inputs.package_artifact_name }}
path: ${{ runner.temp }}/prebuilt-package-artifact
- name: Resolve prebuilt package artifact
id: resolve_artifact
env:
INPUT_PACKAGE_ARTIFACT_NAME: ${{ inputs.package_artifact_name }}
INPUT_PACKAGE_ARTIFACT_PATH: ${{ inputs.package_artifact_path }}
run: |
python3 - <<'PY'
import os
from pathlib import Path
artifact_name = os.environ["INPUT_PACKAGE_ARTIFACT_NAME"].strip()
output_path = Path(os.environ["GITHUB_OUTPUT"])
if not artifact_name:
with output_path.open("a", encoding="utf-8") as fh:
fh.write("package_artifact_path=\n")
raise SystemExit(0)
artifact_root = Path(os.environ["RUNNER_TEMP"]) / "prebuilt-package-artifact"
requested_path = os.environ["INPUT_PACKAGE_ARTIFACT_PATH"].strip()
if requested_path:
candidate = (artifact_root / requested_path).resolve()
if artifact_root.resolve() not in candidate.parents and candidate != artifact_root.resolve():
raise SystemExit(f"Prebuilt artifact path escapes downloaded artifact: {requested_path}")
if not candidate.is_file():
raise SystemExit(f"Prebuilt package artifact path not found: {requested_path}")
else:
candidates = sorted(path for path in artifact_root.rglob("*.tgz") if path.is_file())
if not candidates:
raise SystemExit(f"Prebuilt package artifact {artifact_name!r} did not contain a .tgz file.")
if len(candidates) > 1:
joined = ", ".join(str(path.relative_to(artifact_root)) for path in candidates)
raise SystemExit(
"Prebuilt package artifact contains multiple .tgz files; set package_artifact_path. "
f"Found: {joined}"
)
candidate = candidates[0]
with output_path.open("a", encoding="utf-8") as fh:
fh.write(f"package_artifact_path={candidate}\n")
PY
- name: Resolve publish command
id: resolve_publish
env:
INPUT_SOURCE: ${{ inputs.source }}
INPUT_REF: ${{ inputs.ref }}
@@ -230,9 +293,11 @@ jobs:
INPUT_SOURCE_COMMIT: ${{ inputs.source_commit }}
INPUT_SOURCE_REF: ${{ inputs.source_ref }}
INPUT_SOURCE_PATH: ${{ inputs.source_path }}
PREBUILT_PACKAGE_ARTIFACT_PATH: ${{ steps.resolve_artifact.outputs.package_artifact_path }}
INPUT_SITE: ${{ inputs.site }}
INPUT_REGISTRY: ${{ inputs.registry }}
CLAWHUB_TOKEN: ${{ secrets.clawhub_token }}
GITHUB_TOKEN: ${{ github.token }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_REF: ${{ github.ref }}
@@ -243,6 +308,78 @@ jobs:
import os
import shlex
from pathlib import Path
from urllib.error import HTTPError
from urllib.parse import quote, urlparse
from urllib.request import Request, urlopen
def split_ref_path(value):
if not value:
return "", ""
if ":" not in value:
return value, ""
ref, path = value.split(":", 1)
return ref, path.strip("/")
def github_commit_exists(repo, ref):
token = os.environ.get("GITHUB_TOKEN", "").strip()
headers = {
"Accept": "application/vnd.github+json",
"User-Agent": "clawhub-package-publish",
}
if token:
headers["Authorization"] = f"Bearer {token}"
request = Request(
f"https://api.github.com/repos/{repo}/commits/{quote(ref, safe='')}",
headers=headers,
)
try:
with urlopen(request, timeout=10) as response:
return 200 <= response.status < 300
except HTTPError as error:
if error.code in (404, 422):
return False
raise
def resolve_github_url_ref_and_path(repo, kind, segments):
min_path_segments = 1 if kind == "blob" else 0
max_ref_segments = len(segments) - min_path_segments
for ref_segment_count in range(max_ref_segments, 0, -1):
ref = "/".join(segments[:ref_segment_count])
path = "/".join(segments[ref_segment_count:]).strip("/")
if kind == "blob" and not path:
continue
if not github_commit_exists(repo, ref):
continue
if kind == "blob":
path = "/".join(path.split("/")[:-1]).strip("/")
return ref, path
raise SystemExit(f"GitHub ref not found in source URL for {repo}")
def parse_github_source(value):
raw = value.strip()
if raw.startswith("github:"):
raw = raw[len("github:"):]
if raw.startswith("https://") or raw.startswith("http://"):
parsed = urlparse(raw)
if parsed.netloc.lower() != "github.com":
return None
parts = [part for part in parsed.path.strip("/").split("/") if part]
if len(parts) < 2:
return None
repo_name = parts[1][:-4] if parts[1].endswith(".git") else parts[1]
repo = f"{parts[0]}/{repo_name}"
if len(parts) >= 4 and parts[2] in {"tree", "blob"}:
ref, path = resolve_github_url_ref_and_path(repo, parts[2], parts[3:])
return {"repo": repo, "ref": ref, "path": path}
return {"repo": repo, "ref": "", "path": ""}
source_part, at, ref_part = raw.partition("@")
repo_parts = source_part.split("/")
if len(repo_parts) != 2 or not repo_parts[0] or not repo_parts[1]:
return None
repo_name = repo_parts[1][:-4] if repo_parts[1].endswith(".git") else repo_parts[1]
ref, path = split_ref_path(ref_part if at else "")
return {"repo": f"{repo_parts[0]}/{repo_name}", "ref": ref, "path": path}
source = os.environ["INPUT_SOURCE"].strip()
if not source:
@@ -254,6 +391,33 @@ jobs:
is_local_source = source.startswith(".") or source.startswith("/") or Path(source).exists()
if ref and "@" not in source and not source.startswith("http") and not is_local_source:
source = f"{source}@{ref}"
source_path = os.environ["INPUT_SOURCE_PATH"].strip()
prebuilt_artifact_path = os.environ["PREBUILT_PACKAGE_ARTIFACT_PATH"].strip()
inspect_checkout_repository = ""
inspect_checkout_ref = ""
inspect_local_root = str(Path(os.environ["GITHUB_WORKSPACE"]).resolve())
inspect_subdir = source_path
if prebuilt_artifact_path:
inspect_local_root = str((Path(os.environ["RUNNER_TEMP"]) / "prebuilt-package-inspect").resolve())
inspect_subdir = ""
elif is_local_source:
inspect_local_root = str(Path(source).resolve())
else:
github_source = parse_github_source(source)
source_ref_differs_from_checkout = (
bool(github_source and github_source["ref"])
and github_source["ref"] != os.environ["GITHUB_SHA"]
)
if github_source and (
github_source["repo"] != os.environ["GITHUB_REPOSITORY"]
or source_ref_differs_from_checkout
):
inspect_checkout_repository = github_source["repo"]
inspect_checkout_ref = github_source["ref"]
inspect_local_root = str((Path(os.environ["GITHUB_WORKSPACE"]) / "clawhub-publish-source").resolve())
inspect_subdir = source_path or github_source["path"]
elif github_source:
inspect_subdir = source_path or github_source["path"]
cli_entry = (
Path(os.environ["GITHUB_WORKSPACE"])
@@ -266,12 +430,13 @@ jobs:
if not cli_entry.exists():
raise SystemExit(f"Missing ClawHub CLI entrypoint at {cli_entry}")
cmd_source = prebuilt_artifact_path or source
cmd = [
"bun",
str(cli_entry),
"package",
"publish",
source,
cmd_source,
"--site",
os.environ["INPUT_SITE"],
"--registry",
@@ -294,7 +459,16 @@ jobs:
source_repo = os.environ["INPUT_SOURCE_REPO"].strip()
source_commit = os.environ["INPUT_SOURCE_COMMIT"].strip()
source_ref = os.environ["INPUT_SOURCE_REF"].strip()
source_path = os.environ["INPUT_SOURCE_PATH"].strip()
if prebuilt_artifact_path:
if not source_repo and not source_commit:
source_repo = os.environ["GITHUB_REPOSITORY"].strip()
source_commit = os.environ["GITHUB_SHA"].strip()
elif not source_repo or not source_commit:
raise SystemExit(
"Prebuilt artifact mode requires source_repo and source_commit together when overriding source attribution."
)
if not source_ref:
source_ref = os.environ["GITHUB_REF"].strip()
if source_repo:
cmd += ["--source-repo", source_repo]
if source_commit:
@@ -318,8 +492,65 @@ jobs:
path.write_text("#!/usr/bin/env bash\nset -euo pipefail\n" + shell_line + "\n", encoding="utf-8")
path.chmod(0o755)
print(shell_line)
output_path = Path(os.environ["GITHUB_OUTPUT"])
with output_path.open("a", encoding="utf-8") as fh:
fh.write(f"inspect_checkout_repository={inspect_checkout_repository}\n")
fh.write(f"inspect_checkout_ref={inspect_checkout_ref}\n")
fh.write(f"inspect_local_root={inspect_local_root}\n")
fh.write(f"inspect_subdir={inspect_subdir}\n")
PY
- name: Extract prebuilt package artifact for plugin validation
if: steps.resolve_artifact.outputs.package_artifact_path != ''
env:
PREBUILT_PACKAGE_ARTIFACT_PATH: ${{ steps.resolve_artifact.outputs.package_artifact_path }}
INSPECT_LOCAL_ROOT: ${{ steps.resolve_publish.outputs.inspect_local_root }}
run: |
set -euo pipefail
mkdir -p "$INSPECT_LOCAL_ROOT"
tar -xzf "$PREBUILT_PACKAGE_ARTIFACT_PATH" -C "$INSPECT_LOCAL_ROOT" --strip-components=1
- name: Checkout publish source for plugin inspector
if: steps.resolve_publish.outputs.inspect_checkout_repository != ''
uses: actions/checkout@v6
with:
repository: ${{ steps.resolve_publish.outputs.inspect_checkout_repository }}
ref: ${{ steps.resolve_publish.outputs.inspect_checkout_ref }}
path: clawhub-publish-source
- name: Run plugin validation
env:
INSPECT_LOCAL_ROOT: ${{ steps.resolve_publish.outputs.inspect_local_root }}
INSPECT_SUBDIR: ${{ steps.resolve_publish.outputs.inspect_subdir }}
run: |
set -euo pipefail
inspect_root="$(python3 - <<'PY'
import os
from pathlib import Path
root = Path(os.environ["INSPECT_LOCAL_ROOT"]).resolve()
subdir = os.environ["INSPECT_SUBDIR"].strip()
inspect_root = (root / subdir).resolve() if subdir else root
if inspect_root != root and root not in inspect_root.parents:
raise SystemExit(f"Inspector source path escapes publish source: {subdir}")
print(inspect_root)
PY
)"
if [ ! -f "$inspect_root/package.json" ] && [ ! -f "$inspect_root/openclaw.plugin.json" ]; then
echo "::warning::Plugin Inspector skipped because $inspect_root is not a plugin root."
exit 0
fi
bun "$GITHUB_WORKSPACE/clawhub-source/packages/clawhub/src/cli.ts" package validate "$inspect_root" --out "$RUNNER_TEMP/plugin-inspector"
- name: Upload plugin inspector reports
if: always()
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.inspector_artifact_name }}
path: ${{ runner.temp }}/plugin-inspector
if-no-files-found: ignore
- name: Run package publish
run: |
set -euo pipefail
@@ -349,6 +580,6 @@ jobs:
- name: Upload publish JSON artifact
uses: actions/upload-artifact@v7
with:
name: clawhub-package-publish-json
name: ${{ inputs.publish_json_artifact_name }}
path: ${{ runner.temp }}/package-publish.json
if-no-files-found: error
@@ -0,0 +1,63 @@
name: Plugin Inspector Bulk Scan
on:
workflow_dispatch:
inputs:
batch_size:
description: "Maximum plugin releases to scan"
required: false
default: "25"
dry_run:
description: "Preview impact without persisting findings or sending emails"
required: false
default: "false"
type: boolean
dry_run_max_batches:
description: "Maximum preview batches to scan when dry_run is enabled"
required: false
default: "20"
source_pr:
description: "Merged PR number that triggered this scan, when dispatched automatically"
required: false
default: ""
source_sha:
description: "Merged commit SHA that triggered this scan, when dispatched automatically"
required: false
default: ""
permissions:
contents: read
jobs:
scan:
name: Scan published plugins
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Run plugin inspector bulk scan
env:
CLAWHUB_SITE_URL: ${{ vars.CLAWHUB_SITE_URL || 'https://clawhub.ai' }}
CLAWHUB_PLUGIN_INSPECTOR_WORKER_TOKEN: ${{ secrets.CLAWHUB_PLUGIN_INSPECTOR_WORKER_TOKEN }}
PLUGIN_INSPECTOR_BATCH_SIZE: ${{ inputs.batch_size || '25' }}
PLUGIN_INSPECTOR_DRY_RUN: ${{ inputs.dry_run && '1' || '0' }}
PLUGIN_INSPECTOR_DRY_RUN_MAX_BATCHES: ${{ inputs.dry_run_max_batches || '20' }}
PLUGIN_INSPECTOR_SOURCE_PR: ${{ inputs.source_pr || '' }}
PLUGIN_INSPECTOR_SOURCE_SHA: ${{ inputs.source_sha || '' }}
PLUGIN_INSPECTOR_ARTIFACT_DIR: plugin-inspector-bulk-scan-reports
run: bun scripts/package-inspector-nightly-scan.ts
- name: Upload inspector reports
if: always()
uses: actions/upload-artifact@v4
with:
name: plugin-inspector-bulk-scan-reports
path: plugin-inspector-bulk-scan-reports
if-no-files-found: warn
@@ -0,0 +1,55 @@
name: Plugin Inspector Pin Bump Dispatch
on:
push:
branches:
- main
paths:
- package.json
- packages/clawhub/package.json
- bun.lock
permissions:
actions: write
contents: read
jobs:
dispatch-plugin-inspector-bulk-scan:
name: Dispatch Plugin Inspector bulk scan
runs-on: ubuntu-latest
steps:
- name: Checkout main commit
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}
fetch-depth: 0
- name: Fetch previous main commit
env:
BASE_SHA: ${{ github.event.before }}
run: git fetch --no-tags --depth=1 origin "$BASE_SHA"
- name: Detect pinned Plugin Inspector change
id: detect
env:
BASE_SHA: ${{ github.event.before }}
HEAD_SHA: ${{ github.sha }}
run: node scripts/github/plugin-inspector-pin-change.mjs --base "$BASE_SHA" --head "$HEAD_SHA"
- name: Dispatch Plugin Inspector bulk scan
if: ${{ steps.detect.outputs.changed == 'true' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
gh workflow run plugin-inspector-bulk-scan.yml \
--ref main \
-f batch_size=25 \
-f dry_run=false \
-f dry_run_max_batches=20 \
-f source_sha=${{ github.sha }}
- name: Explain skipped dispatch
if: ${{ steps.detect.outputs.changed != 'true' }}
env:
DISPATCH_SKIP_REASON: ${{ steps.detect.outputs.reason }}
run: printf '%s\n' "$DISPATCH_SKIP_REASON"
@@ -0,0 +1,84 @@
name: ClawHub Scheduled Live Checks
on:
schedule:
- cron: "17 5 * * *"
workflow_dispatch:
inputs:
github-repo:
description: GitHub skills repo to use for the source-backed canary
required: false
default: openclaw/agent-skills
github-skill:
description: Skill slug to verify from the GitHub skills repo
required: false
default: handoff
concurrency:
group: clawhub-scheduled-live-checks-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
env:
VITE_CONVEX_URL: https://example.invalid
jobs:
github-backed-skills:
name: GitHub-backed skills canary
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/setup-bun
- name: Run GitHub-backed skills live canary
env:
CLAWHUB_LIVE_GITHUB_CANARY: "1"
CLAWHUB_LIVE_GITHUB_REPO: ${{ inputs.github-repo || 'openclaw/agent-skills' }}
CLAWHUB_LIVE_GITHUB_SKILL: ${{ inputs.github-skill || 'handoff' }}
GITHUB_TOKEN: ${{ github.token }}
run: bunx vitest run convex/githubSkillSync.live.test.ts
open-failure-issue:
name: Open failure issue
needs: github-backed-skills
if: ${{ always() && needs.github-backed-skills.result == 'failure' }}
runs-on: ubuntu-latest
permissions:
issues: write
steps:
- name: Open or update failure issue
env:
GH_TOKEN: ${{ github.token }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
WORKFLOW_NAME: ${{ github.workflow }}
run: |
set -euo pipefail
marker_token="clawhub-scheduled-live-checks-failure"
marker="<!-- $marker_token -->"
title="ClawHub scheduled live checks failing"
issue_number="$(gh issue list \
--repo "$GITHUB_REPOSITORY" \
--state open \
--search "$marker_token in:body" \
--json number \
--jq '.[0].number // empty')"
body_file="$(mktemp)"
cat > "$body_file" <<EOF
$marker
The scheduled ClawHub live checks failed.
Workflow: $WORKFLOW_NAME
Run: $RUN_URL
EOF
if [[ -n "$issue_number" ]]; then
gh issue comment "$issue_number" --repo "$GITHUB_REPOSITORY" --body-file "$body_file"
else
gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$body_file"
fi
+1 -1
View File
@@ -51,7 +51,7 @@ jobs:
id: trufflehog
# Use a concrete released ref that resolves in upstream action registry.
# v3 (major tag) is not published by trufflesecurity/trufflehog.
uses: trufflesecurity/trufflehog@v3.95.3
uses: trufflesecurity/trufflehog@v3.95.5
with:
path: ./
base: ${{ steps.scan_range.outputs.base }}
+15 -12
View File
@@ -7,38 +7,41 @@ on:
workflow_dispatch:
inputs:
batch-limit:
description: "Maximum Skill Card jobs to run in parallel per batch"
description: "Maximum Skill Card jobs to run in parallel per worker shard"
required: true
default: "5"
default: "6"
max-jobs:
description: "Optional total jobs cap for this run"
description: "Optional total jobs cap per worker shard"
required: false
default: ""
max-runtime-minutes:
description: "Stop claiming new batches after this many minutes"
required: true
default: "30"
concurrency:
group: skill-card-worker
cancel-in-progress: false
default: "40"
permissions:
contents: read
jobs:
skill-card-worker:
name: Skill Card worker shard ${{ matrix.shard }}
runs-on: blacksmith-8vcpu-ubuntu-2404
timeout-minutes: 45
timeout-minutes: 60
environment: Production
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2, 3, 4, 5, 6, 7]
env:
CONVEX_URL: ${{ vars.CONVEX_URL || vars.VITE_CONVEX_URL || 'https://wry-manatee-359.convex.cloud' }}
# Shared Convex worker credential used by security and Skill Card workers.
SECURITY_SCAN_WORKER_TOKEN: ${{ secrets.SECURITY_SCAN_WORKER_TOKEN }}
SKILL_CARD_WORKER_LIMIT: ${{ github.event.inputs['batch-limit'] || '5' }}
SKILL_CARD_WORKER_LIMIT: ${{ github.event.inputs['batch-limit'] || '6' }}
SKILL_CARD_WORKER_MAX_JOBS: ${{ github.event.inputs['max-jobs'] || '' }}
SKILL_CARD_WORKER_MAX_RUNTIME_MINUTES: ${{ github.event.inputs['max-runtime-minutes'] || '30' }}
SKILL_CARD_WORKER_LEASE_MINUTES: "30"
SKILL_CARD_WORKER_MAX_RUNTIME_MINUTES: ${{ github.event.inputs['max-runtime-minutes'] || '40' }}
SKILL_CARD_WORKER_LEASE_MINUTES: "60"
SKILL_CARD_WORKER_SHARD: ${{ matrix.shard }}
SKILL_CARD_WORKER_ID: "github-actions:${{ github.run_id }}:${{ github.run_attempt }}:${{ matrix.shard }}"
NVIDIA_TRUSTWORTHY_AI_DIR: ${{ github.workspace }}/.artifacts/nvidia-trustworthy-ai
steps:
- uses: actions/checkout@v6
+292
View File
@@ -0,0 +1,292 @@
name: Skill Publish
on:
workflow_call:
inputs:
skill_path:
description: Optional path to one skill folder. When set, only this skill is processed.
required: false
type: string
default: ""
root:
description: Directory containing skill folders for bulk catalog publishing.
required: false
type: string
default: skills
dry_run:
description: Preview only. When true, no publish mutation is performed.
required: false
type: boolean
default: true
owner:
description: Optional owner/publisher handle for org publishing.
required: false
type: string
default: ""
tags:
description: Optional comma-separated tags override.
required: false
type: string
default: latest
bump:
description: Version bump for updated skills. One of patch, minor, or major.
required: false
type: string
default: patch
registry:
description: ClawHub registry URL.
required: false
type: string
default: https://clawhub.ai
site:
description: ClawHub site URL.
required: false
type: string
default: https://clawhub.ai
ref:
description: Optional caller repository ref to check out.
required: false
type: string
default: ""
secrets:
clawhub_token:
required: false
outputs:
publish_json:
description: Structured JSON output from clawhub sync.
value: ${{ jobs.publish.outputs.publish_json }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
permissions: {}
jobs:
publish:
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
id-token: write
outputs:
publish_json: ${{ steps.capture.outputs.publish_json }}
steps:
- uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.sha }}
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: 1.3.10
- name: Resolve ClawHub workflow source
id: clawhub_source
run: |
python3 - <<'PY'
import base64
import json
import os
from pathlib import Path
from urllib.request import Request, urlopen
request_token = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_TOKEN", "").strip()
request_url = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_URL", "").strip()
if not request_token or not request_url:
raise SystemExit("GitHub OIDC token request env vars are missing; id-token: write is required.")
audience = "clawhub-workflow-source"
joiner = "&" if "?" in request_url else "?"
token_url = f"{request_url}{joiner}audience={audience}"
request = Request(token_url, headers={"Authorization": f"Bearer {request_token}"})
with urlopen(request) as response:
payload = json.load(response)
token = str(payload.get("value", "")).strip()
if not token:
raise SystemExit("GitHub OIDC token response did not include a token value.")
try:
encoded_payload = token.split(".")[1]
except IndexError as exc:
raise SystemExit("GitHub OIDC token was not a valid JWT.") from exc
padding = "=" * (-len(encoded_payload) % 4)
claims = json.loads(base64.urlsafe_b64decode(encoded_payload + padding).decode("utf-8"))
workflow_ref = str(claims.get("job_workflow_ref", "")).strip()
workflow_sha = str(claims.get("job_workflow_sha", "")).strip()
repo, marker, _ = workflow_ref.partition("/.github/workflows/")
if not marker or not repo or not workflow_sha:
raise SystemExit(
"Unable to resolve reusable workflow source from GitHub OIDC claims: "
f"job_workflow_ref={workflow_ref!r} job_workflow_sha={workflow_sha!r}"
)
output_path = Path(os.environ["GITHUB_OUTPUT"])
with output_path.open("a", encoding="utf-8") as fh:
fh.write(f"repository={repo}\n")
fh.write(f"ref={workflow_sha}\n")
PY
- uses: actions/checkout@v6
with:
repository: ${{ steps.clawhub_source.outputs.repository }}
ref: ${{ steps.clawhub_source.outputs.ref }}
path: clawhub-source
- name: Install ClawHub CLI dependencies
working-directory: clawhub-source
run: bun install --frozen-lockfile
- name: Validate publish mode inputs
env:
DRY_RUN: ${{ inputs.dry_run }}
CLAWHUB_TOKEN: ${{ secrets.clawhub_token }}
run: |
if [[ "$DRY_RUN" == "true" ]]; then
exit 0
fi
if [[ -n "$CLAWHUB_TOKEN" ]]; then
exit 0
fi
echo "::error::Real skill publishes need secrets.clawhub_token. GitHub OIDC trusted publishing for skills is not supported yet."
exit 1
- name: Write ClawHub config
env:
CLAWHUB_TOKEN: ${{ secrets.clawhub_token }}
CLAWHUB_REGISTRY: ${{ inputs.registry }}
run: |
if [[ -z "$CLAWHUB_TOKEN" ]]; then
echo "No ClawHub token provided, skipping config file creation."
exit 0
fi
python3 - <<'PY'
import json
import os
from pathlib import Path
path = Path(os.environ["RUNNER_TEMP"]) / "clawhub-config.json"
path.write_text(
json.dumps(
{
"registry": os.environ["CLAWHUB_REGISTRY"],
"token": os.environ["CLAWHUB_TOKEN"],
},
indent=2,
)
+ "\n",
encoding="utf-8",
)
print(path)
PY
echo "CLAWHUB_CONFIG_PATH=$RUNNER_TEMP/clawhub-config.json" >> "$GITHUB_ENV"
- name: Resolve sync command
env:
INPUT_SKILL_PATH: ${{ inputs.skill_path }}
INPUT_ROOT: ${{ inputs.root }}
INPUT_DRY_RUN: ${{ inputs.dry_run }}
INPUT_OWNER: ${{ inputs.owner }}
INPUT_TAGS: ${{ inputs.tags }}
INPUT_BUMP: ${{ inputs.bump }}
INPUT_SITE: ${{ inputs.site }}
INPUT_REGISTRY: ${{ inputs.registry }}
INPUT_REF: ${{ inputs.ref }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_REF: ${{ github.ref }}
run: |
python3 - <<'PY'
import os
import shlex
import subprocess
from pathlib import Path
skill_path = os.environ["INPUT_SKILL_PATH"].strip()
root = os.environ["INPUT_ROOT"].strip() or "skills"
scan_root = skill_path or root
source_commit = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
source_ref = os.environ["INPUT_REF"].strip() or os.environ["GITHUB_REF"].strip()
cli_entry = (
Path(os.environ["GITHUB_WORKSPACE"])
/ "clawhub-source"
/ "packages"
/ "clawhub"
/ "src"
/ "cli.ts"
)
if not cli_entry.exists():
raise SystemExit(f"Missing ClawHub CLI entrypoint at {cli_entry}")
cmd = [
"bun",
str(cli_entry),
"--workdir",
scan_root,
"--dir",
".",
"sync",
"--all",
"--json",
"--no-clawdbot-roots",
"--site",
os.environ["INPUT_SITE"],
"--registry",
os.environ["INPUT_REGISTRY"],
"--bump",
os.environ["INPUT_BUMP"].strip() or "patch",
"--source-repo",
os.environ["GITHUB_REPOSITORY"],
"--source-commit",
source_commit,
]
if os.environ["INPUT_DRY_RUN"] == "true":
cmd.append("--dry-run")
owner = os.environ["INPUT_OWNER"].strip()
tags = os.environ["INPUT_TAGS"].strip()
if owner:
cmd += ["--owner", owner]
if tags:
cmd += ["--tags", tags]
if source_ref:
cmd += ["--source-ref", source_ref]
path = Path(os.environ["RUNNER_TEMP"]) / "clawhub-skill-publish-command.sh"
shell_line = " ".join(shlex.quote(part) for part in cmd)
path.write_text("#!/usr/bin/env bash\nset -euo pipefail\n" + shell_line + "\n", encoding="utf-8")
path.chmod(0o755)
print(shell_line)
PY
- name: Run skill sync
run: |
set -euo pipefail
"$RUNNER_TEMP/clawhub-skill-publish-command.sh" | tee "$RUNNER_TEMP/skill-publish.json"
- name: Capture workflow outputs
id: capture
run: |
python3 - <<'PY'
import json
import os
from pathlib import Path
output_path = Path(os.environ["RUNNER_TEMP"]) / "skill-publish.json"
raw = output_path.read_text(encoding="utf-8").strip()
parsed = json.loads(raw)
github_output = Path(os.environ["GITHUB_OUTPUT"])
with github_output.open("a", encoding="utf-8") as fh:
fh.write("publish_json<<__CLAWHUB_JSON__\n")
fh.write(json.dumps(parsed, indent=2))
fh.write("\n__CLAWHUB_JSON__\n")
PY
- name: Upload publish JSON artifact
uses: actions/upload-artifact@v7
with:
name: clawhub-skill-publish-json
path: ${{ runner.temp }}/skill-publish.json
if-no-files-found: error
+2 -14
View File
@@ -3,6 +3,7 @@ node_modules
.bun-build
*.bun-build
.artifacts/
artifacts/
.cache/
.data/
bin/docs-list
@@ -37,23 +38,10 @@ convex/_generated/ai/*
!convex/_generated/ai/guidelines.md
!convex/_generated/ai/ai-files.state.json
skills-lock.json
*/skills/*
!.agents/skills/
!.agents/skills/convex*/
!.agents/skills/convex*/**
!.agents/skills/crabbox/
!.agents/skills/crabbox/**
!.agents/skills/clawhub-ui-proof/
!.agents/skills/clawhub-ui-proof/**
!.agents/skills/clawhub-pr-maintainer/
!.agents/skills/clawhub-pr-maintainer/**
!.agents/skills/clawhub-moderation/
!.agents/skills/clawhub-moderation/**
!.agents/skills/autoreview/
!.agents/skills/autoreview/**
skills/*
.codex/*
!.codex/environments/
!.codex/environments/environment.toml
.crabbox/
/.comux-hooks
/.comux
+9 -2
View File
@@ -51,6 +51,8 @@ Specialized corpus, scanner, security-worker, UI proof, proof publishing, Crabbo
- Tests live in `src/**` and `convex/lib/**`.
- Coverage threshold: 80% global (lines/functions/branches/statements).
- Example: `convex/lib/skills.test.ts`.
- When adding or changing Convex functions, do not rely only on mocked `ctx` tests for behavior that depends on Convex runtime semantics such as pagination, indexes, validators, auth identity, internal/public function boundaries, scheduler/cron behavior, actions calling queries/mutations, HTTP actions, storage, or OCC/transaction behavior. Add or run a real Convex validation path, such as `convex dev --once`, `convex run`, an HTTP action smoke, or a local-auth Playwright flow, covering the changed behavior. Mocked `ctx.db` / `ctx.runQuery` tests are still fine for pure business logic, but they do not count as Convex runtime validation.
- For local UI state testing, prefer creating realistic backend state through seed logic plus a DevPersonaFab entry for the associated test user. Avoid one-off manual DB edits when the state is likely to be reused, such as org membership, official publisher access, moderation holds, or publishing permissions.
## Commit & Pull Request Guidelines
@@ -60,10 +62,10 @@ Specialized corpus, scanner, security-worker, UI proof, proof publishing, Crabbo
- Before commit/PR handoff for non-trivial code changes, use `$autoreview` until no accepted/actionable findings remain, unless equivalent manual review already happened, the change is trivial/docs-only, or the user opts out.
- Before opening a PR for source or test changes, run the targeted tests for the touched behavior and `bun run ci:unit` (`VITE_CONVEX_URL=https://example.invalid bun run coverage`) unless the change is docs/config-only or the user explicitly asks to rely on CI. For runtime, build, or package changes, also run the matching broader gate when it covers the touched surface: `bun run ci:types-build`, `bun run ci:packages`, `bun run ci:e2e-http`, or `bun run ci:playwright-smoke`.
- PRs: include summary + test commands run. Add screenshots for UI changes.
- Screenshot proof MUST come from a real running ClawHub instance in a real browser. Do not use generated HTML mockups, synthetic terminal cards, or manually composed images as proof. For route/status/backend visibility bugs, run ClawHub locally with the relevant Convex code and fixture state, capture the actual browser page, and state the local URL and fixture used.
- Before merging any PR, verify TypeScript cleanly with `bunx tsc -p packages/schema/tsconfig.json --noEmit` and `bunx tsc -p packages/clawhub/tsconfig.json --noEmit`; if Convex code changed, also run the repo typecheck path used by deploy so `bunx convex deploy` will not fail on `tsc`.
- GitHub comments: for multiline `gh` comments/close messages, use `--body-file`, `--input`, or stdin/heredoc with real newlines; never pass literal `\\n` in shell strings.
- Reject PRs that add skills into source code/repo content directly (for example under `skills/` or seed-only additions intended as published skills). Skills must be uploaded/published via CLI.
- Repo-local Convex developer skills under `.agents/skills/convex*/` are allowed when they support working on this codebase; keep top-level `skills/` reserved for installed/published skill content and ignored by git.
- Repo-local developer skills under `.agents/skills/` are allowed only when they are ClawHub-specific, such as Convex, moderation, PR maintainer, or UI proof workflows. Keep generic shared skills such as `crabbox` and `autoreview` in the global `agent-skills` install, not this repo. Keep top-level `skills/` reserved for installed/published skill content and ignored by git.
## Production Release
@@ -101,6 +103,11 @@ Specialized corpus, scanner, security-worker, UI proof, proof publishing, Crabbo
- For non-interactive prod deploys, use `bunx convex deploy -y` to skip confirmation.
- If `bunx convex run --env-file .env.local ...` returns `401 MissingAccessToken` despite `bunx convex login`, workaround: omit `--env-file` and use `--deployment-name <name>` / `--prod`.
## Convex Migrations & Backfills
- Any Convex production data migration, backfill, destructive cleanup, schema narrowing, or table reshaping must start with the `convex-migration-helper` skill. Default to `@convex-dev/migrations` for production data changes because it provides batching, dry runs, resume/progress tracking, and safer operator UX. Exceptions require an explicit note explaining why the component is unnecessary, plus equivalent dry-run support, cursor batching, resume/progress behavior, confirmation for destructive writes, and real Convex runtime validation.
- After a migration or cleanup is verified complete, remove temporary migration functions/code in a follow-up PR unless they are intentionally retained as ongoing maintenance tooling.
## Convex Query & Bandwidth Rules
- **Always use `.withIndex()` instead of `.filter()` for fields that can be indexed.** `.filter()` causes full table scans — every doc is read and billed. Even a single `.filter()` on a 16K-row table reads ~16 MB per call.
+47 -2
View File
@@ -1,6 +1,46 @@
# Changelog
## Unreleased
## 0.21.0 - 2026-06-11
### Changes
- CLI/API: add public `clawhub package trusted-publisher set` and `clawhub package trusted-publisher delete` commands so package managers can configure or remove GitHub Actions OIDC trusted publishing for existing packages.
## 0.20.2 - 2026-06-11
### Changes
- CLI packages now require Node.js 22 or newer, dropping the EOL Node 20 runtime floor.
- CLI: add `clawhub package validate <source>` for local plugin validation with author-facing Plugin Inspector findings, remediation text, and report artifacts.
## 0.20.0 - 2026-06-06
### Changes
- CLI/API: replace local `clawhub scan` uploads with stored submitted-version scan report downloads, including owner-authorized `clawhub scan download <name> --version <version>` support for blocked skill and plugin submissions.
## 0.19.2 - 2026-06-05
### Fixes
- CLI: accept the legacy `clawhub skill verify --json` flag as a hidden compatibility no-op while continuing to print JSON by default.
## 0.19.1 - 2026-06-05
### Fixes
- CLI: install source-backed GitHub skills from the deployed `/api/v1/skills/:slug/install` resolver so `clawhub install` works for skills without hosted ClawHub versions.
## 0.19.0 - 2026-06-03
### Changes
- CLI/API: add authenticated `clawhub scan` submit/poll support for ephemeral local skill bundles and owner-authorized published skill scans, including JSON output and report ZIP downloads (#2479).
### Fixes
- Auth/Ops: keep GitHub account-age lookups on immutable numeric IDs, retry without auth when a configured GitHub token is rejected, and add an operator backfill for missing cached account ages.
- API/CLI: report Skill Card verification with flattened skill/version metadata, ClawScan verdict fields at `security.*`, and supporting scanner evidence under `security.signals`.
## 0.18.0 - 2026-05-25
@@ -12,6 +52,12 @@
### Fixes
- API: fix `GET /api/v1/skills` pagination so `cursor` advances to the next page instead of repeating the first page for supported non-trending sorts (#2275) (thanks @vyctorbrzezowski, @enerj).
- Web: block collaborative membership on personal publishers while allowing the linked owner to clean up stale extra membership rows (thanks @vyctorbrzezowski).
- Security/API: hide owned package/plugin catalog entries, revoke package publish tokens, and restore only matching ban-hidden packages on user unban (thanks @vyctorbrzezowski).
- API: block public raw skill files when moderation already blocks downloads and reject skill tags that point at another skill's version (thanks @vyctorbrzezowski).
- Web: stop stale unban restore batches from reactivating skills after the owner is banned again or deactivated (thanks @vyctorbrzezowski).
- Security/API: reject direct skill owner transfers when the skill is hidden, suspicious, or malicious (thanks @vyctorbrzezowski).
- Security/API: revalidate package publish actor, owner, and owner publisher active state in the final release insert (thanks @vyctorbrzezowski).
## 0.17.0 - 2026-05-19
@@ -71,7 +117,6 @@
### Changes
- Web: add publisher notes and unify ClawScan review pages (#2111).
- Dev: auto-start services for Codex worktrees and add a local dev persona FAB (#2146, #2147).
- Dev: add a local ClawScan dry-run helper script (#2143).
+19 -2
View File
@@ -103,6 +103,24 @@ CLAWHUB_WORKTREE_SOURCE=/path/to/source/worktree bun run setup:worktree
The detached server writes runtime state under `.codex/runtime/`. Stop it with `wt --yes stop` before removing the worktree.
### Local Codex workers
Local dev does not start Codex-backed workers by default, so `dev:worktree` does
not spend Codex quota.
To process local ClawScan or Skill Card jobs, opt in for that shell:
```bash
CLAWHUB_ALLOW_LOCAL_CODEX_SCAN=1 bun run dev:workers -- --workers security-scan --once
CLAWHUB_ALLOW_LOCAL_CODEX_SCAN=1 bun run dev:workers -- --workers skill-card --once
```
Opted-in local runs use an ignored worktree-local `CODEX_HOME` unless you provide
one.
Without those workers, local ClawScan and Skill Card jobs stay pending until you
opt in, seed/mock results, or use the production workflows.
### Seed the database
Populate local QA fixtures and the committed public corpus so the UI isn't empty:
@@ -187,10 +205,9 @@ bun run --cwd packages/clawhub verify
Manual smoke tests are documented in [`specs/manual-testing.md`](specs/manual-testing.md).
## Skill & Soul Publishing
## Skill Publishing
- Skill format reference: [`docs/skill-format.md`](docs/skill-format.md)
- Soul format reference: [`docs/soul-format.md`](docs/soul-format.md)
- End-to-end walkthrough (search, install, publish, sync): [`docs/quickstart.md`](docs/quickstart.md)
Quick publish:
+4 -7
View File
@@ -2,6 +2,8 @@
<img src="public/clawd-logo.png" alt="ClawHub" width="120">
</p>
![ClawHub banner](docs/assets/readme-banner.jpg)
<h1 align="center">ClawHub</h1>
<p align="center">
@@ -28,8 +30,6 @@ It also now exposes a native **OpenClaw package catalog** for code plugins and b
- Publish new skill versions with changelogs + tags (including `latest`).
- Rename an owned skill without breaking old links or installs.
- Merge duplicate owned skills into one canonical slug.
- Browse souls + render their `SOUL.md`.
- Publish new soul versions with changelogs + tags.
- Search via embeddings (vector index) instead of brittle keywords.
- Star + comment; admins/mods can curate and approve skills.
- Pin local skill installs so updates and force reinstalls cannot overwrite frozen copies.
@@ -72,7 +72,7 @@ Docs: [`docs/quickstart.md`](docs/quickstart.md), [`docs/cli.md`](docs/cli.md).
## Telemetry
ClawHub tracks minimal **install telemetry** (to compute install counts) when you run `clawhub sync` while logged in.
ClawHub tracks minimal **install telemetry** (to compute install counts) when you run `clawhub install` while logged in.
Disable via:
```bash
@@ -123,9 +123,6 @@ database seeding), see [CONTRIBUTING.md](CONTRIBUTING.md).
- `VITE_CONVEX_URL`: Convex deployment URL (`https://<deployment>.convex.cloud`).
- `VITE_CONVEX_SITE_URL`: Convex site URL (`https://<deployment>.convex.site`).
- `VITE_SOULHUB_SITE_URL`: SoulHub site URL.
- `VITE_SOULHUB_HOST`: SoulHub host match.
- `VITE_SITE_MODE`: Optional override (`skills` or `souls`) for SSR builds.
- `CONVEX_SITE_URL`: same as `VITE_CONVEX_SITE_URL` (auth + cookies).
- `SITE_URL`: App URL (local: `http://localhost:3000`).
- `AUTH_GITHUB_ID` / `AUTH_GITHUB_SECRET`: GitHub OAuth App.
@@ -201,7 +198,7 @@ metadata: { "clawdbot": { "cliHelp": "padel --help\\nUsage: padel [command]\\n"
## Skill metadata
Skills declare their runtime requirements (env vars, binaries, install specs) in the `SKILL.md` frontmatter. ClawHub's security analysis checks these declarations against actual skill behavior; purpose-aligned ClawScan notes stay as guidance, medium review findings stay visible, and the suspicious filter is reserved for high-impact or malicious concerns.
Skills declare their runtime requirements (env vars, binaries, install specs) in the `SKILL.md` frontmatter. ClawHub's security analysis checks these declarations against actual skill behavior; medium review findings stay visible, and the suspicious filter is reserved for high-impact or malicious concerns.
Full reference: [`docs/skill-format.md`](docs/skill-format.md#frontmatter-metadata)
+32
View File
@@ -0,0 +1,32 @@
# Security Policy
Use GitHub Security Advisories for vulnerabilities in ClawHub itself.
Good ClawHub advisory reports include bugs in:
- the ClawHub website, API, or CLI
- registry publishing, downloads, installs, or artifact integrity
- authentication, authorization, or API tokens
- scanning, moderation, or report handling
Because ClawHub is a hosted cloud application, ClawHub service vulnerabilities
are not publicly disclosed by default. They are publicly disclosed when there is
evidence of real user impact or when users need to take action.
Examples of real user impact include confirmed exploitation, exposure of user
data or secrets, malicious content reaching users because of a platform failure,
or any issue that requires users to rotate credentials, update local software, or
take other protective action.
Vulnerabilities in user-installed software are publicly disclosed, such as
ClawHub CLI packages, binaries, libraries, or other release artifacts that users
need to update locally.
Do not use ClawHub advisories for vulnerabilities in a third-party skill or
plugin's own source code. Report those directly to the publisher or source
repository linked from the ClawHub listing.
Use ClawHub's listing reports for genuinely malicious or deceptive marketplace
content, such as malicious listings, misleading metadata, undeclared
permissions, suspicious install instructions, scam comments, impersonation,
trademark misuse, or policy violations.
+627 -370
View File
File diff suppressed because it is too large Load Diff
+48 -28
View File
@@ -17,16 +17,20 @@ import type * as crons from "../crons.js";
import type * as depRegistryScan from "../depRegistryScan.js";
import type * as devSeed from "../devSeed.js";
import type * as devSeedExtra from "../devSeedExtra.js";
import type * as downloadMetrics from "../downloadMetrics.js";
import type * as downloads from "../downloads.js";
import type * as emailsNode from "../emailsNode.js";
import type * as functions from "../functions.js";
import type * as githubAccountAgeBackfill from "../githubAccountAgeBackfill.js";
import type * as githubBackups from "../githubBackups.js";
import type * as githubBackupsNode from "../githubBackupsNode.js";
import type * as githubApp from "../githubApp.js";
import type * as githubIdentity from "../githubIdentity.js";
import type * as githubImport from "../githubImport.js";
import type * as githubRestore from "../githubRestore.js";
import type * as githubRestoreMutations from "../githubRestoreMutations.js";
import type * as githubSoulBackups from "../githubSoulBackups.js";
import type * as githubSoulBackupsNode from "../githubSoulBackupsNode.js";
import type * as githubSkillSources from "../githubSkillSources.js";
import type * as githubSkillSync from "../githubSkillSync.js";
import type * as http from "../http.js";
import type * as httpApi from "../httpApi.js";
import type * as httpApiV1 from "../httpApiV1.js";
@@ -35,7 +39,6 @@ import type * as httpApiV1_packagesV1 from "../httpApiV1/packagesV1.js";
import type * as httpApiV1_publishersV1 from "../httpApiV1/publishersV1.js";
import type * as httpApiV1_shared from "../httpApiV1/shared.js";
import type * as httpApiV1_skillsV1 from "../httpApiV1/skillsV1.js";
import type * as httpApiV1_soulsV1 from "../httpApiV1/soulsV1.js";
import type * as httpApiV1_starsV1 from "../httpApiV1/starsV1.js";
import type * as httpApiV1_transfersV1 from "../httpApiV1/transfersV1.js";
import type * as httpApiV1_usersV1 from "../httpApiV1/usersV1.js";
@@ -49,31 +52,37 @@ import type * as lib_artifactModeration from "../lib/artifactModeration.js";
import type * as lib_badges from "../lib/badges.js";
import type * as lib_batching from "../lib/batching.js";
import type * as lib_changelog from "../lib/changelog.js";
import type * as lib_clawScanNote from "../lib/clawScanNote.js";
import type * as lib_clawpack from "../lib/clawpack.js";
import type * as lib_commentScamPrompt from "../lib/commentScamPrompt.js";
import type * as lib_contentTypes from "../lib/contentTypes.js";
import type * as lib_depRegistryScan from "../lib/depRegistryScan.js";
import type * as lib_devAuth from "../lib/devAuth.js";
import type * as lib_devSeed from "../lib/devSeed.js";
import type * as lib_emailRendering from "../lib/emailRendering.js";
import type * as lib_emails from "../lib/emails.js";
import type * as lib_embeddingVisibility from "../lib/embeddingVisibility.js";
import type * as lib_embeddings from "../lib/embeddings.js";
import type * as lib_githubAccount from "../lib/githubAccount.js";
import type * as lib_githubAppSync from "../lib/githubAppSync.js";
import type * as lib_githubActionsOidc from "../lib/githubActionsOidc.js";
import type * as lib_githubAuth from "../lib/githubAuth.js";
import type * as lib_githubBackup from "../lib/githubBackup.js";
import type * as lib_githubIdentity from "../lib/githubIdentity.js";
import type * as lib_githubImport from "../lib/githubImport.js";
import type * as lib_githubProfileSync from "../lib/githubProfileSync.js";
import type * as lib_githubRestoreHelpers from "../lib/githubRestoreHelpers.js";
import type * as lib_githubSoulBackup from "../lib/githubSoulBackup.js";
import type * as lib_githubSkillSync from "../lib/githubSkillSync.js";
import type * as lib_globalStats from "../lib/globalStats.js";
import type * as lib_httpHeaders from "../lib/httpHeaders.js";
import type * as lib_httpRateLimit from "../lib/httpRateLimit.js";
import type * as lib_httpUtils from "../lib/httpUtils.js";
import type * as lib_installResolver from "../lib/installResolver.js";
import type * as lib_leaderboards from "../lib/leaderboards.js";
import type * as lib_manualOverrides from "../lib/manualOverrides.js";
import type * as lib_moderation from "../lib/moderation.js";
import type * as lib_moderationEngine from "../lib/moderationEngine.js";
import type * as lib_moderationReasonCodes from "../lib/moderationReasonCodes.js";
import type * as lib_observabilityEvents from "../lib/observabilityEvents.js";
import type * as lib_officialPublishers from "../lib/officialPublishers.js";
import type * as lib_openaiResponse from "../lib/openaiResponse.js";
import type * as lib_packageRegistry from "../lib/packageRegistry.js";
import type * as lib_packageSearchDigest from "../lib/packageSearchDigest.js";
@@ -82,8 +91,11 @@ import type * as lib_parsedEnvSignals from "../lib/parsedEnvSignals.js";
import type * as lib_public from "../lib/public.js";
import type * as lib_publicRouteReservations from "../lib/publicRouteReservations.js";
import type * as lib_publishLimits from "../lib/publishLimits.js";
import type * as lib_publisherAbuseScoring from "../lib/publisherAbuseScoring.js";
import type * as lib_publisherCatalogDisplay from "../lib/publisherCatalogDisplay.js";
import type * as lib_publisherStats from "../lib/publisherStats.js";
import type * as lib_publishers from "../lib/publishers.js";
import type * as lib_recommendationScore from "../lib/recommendationScore.js";
import type * as lib_reporting from "../lib/reporting.js";
import type * as lib_reservedHandles from "../lib/reservedHandles.js";
import type * as lib_reservedSlugs from "../lib/reservedSlugs.js";
@@ -92,6 +104,7 @@ import type * as lib_securityPrompt from "../lib/securityPrompt.js";
import type * as lib_skillBackfill from "../lib/skillBackfill.js";
import type * as lib_skillCapabilityTags from "../lib/skillCapabilityTags.js";
import type * as lib_skillCards from "../lib/skillCards.js";
import type * as lib_skillFileAccess from "../lib/skillFileAccess.js";
import type * as lib_skillIcon from "../lib/skillIcon.js";
import type * as lib_skillPublish from "../lib/skillPublish.js";
import type * as lib_skillQuality from "../lib/skillQuality.js";
@@ -102,8 +115,6 @@ import type * as lib_skillStats from "../lib/skillStats.js";
import type * as lib_skillSummary from "../lib/skillSummary.js";
import type * as lib_skillZip from "../lib/skillZip.js";
import type * as lib_skills from "../lib/skills.js";
import type * as lib_soulChangelog from "../lib/soulChangelog.js";
import type * as lib_soulPublish from "../lib/soulPublish.js";
import type * as lib_staticPublishScan from "../lib/staticPublishScan.js";
import type * as lib_tokens from "../lib/tokens.js";
import type * as lib_userSearch from "../lib/userSearch.js";
@@ -111,24 +122,23 @@ import type * as lib_userSkillStats from "../lib/userSkillStats.js";
import type * as lib_webhooks from "../lib/webhooks.js";
import type * as llmEval from "../llmEval.js";
import type * as maintenance from "../maintenance.js";
import type * as managementDevSeed from "../managementDevSeed.js";
import type * as packageInspectorHttp from "../packageInspectorHttp.js";
import type * as packageInspectorNode from "../packageInspectorNode.js";
import type * as packagePublishTokens from "../packagePublishTokens.js";
import type * as packages from "../packages.js";
import type * as publisherAbuse from "../publisherAbuse.js";
import type * as publisherAbuseDevSeed from "../publisherAbuseDevSeed.js";
import type * as publishers from "../publishers.js";
import type * as rateLimits from "../rateLimits.js";
import type * as search from "../search.js";
import type * as securityDataset from "../securityDataset.js";
import type * as securityDatasetNode from "../securityDatasetNode.js";
import type * as securityScan from "../securityScan.js";
import type * as seed from "../seed.js";
import type * as seedSouls from "../seedSouls.js";
import type * as skillCards from "../skillCards.js";
import type * as skillStatEvents from "../skillStatEvents.js";
import type * as skillTransfers from "../skillTransfers.js";
import type * as skills from "../skills.js";
import type * as soulComments from "../soulComments.js";
import type * as soulDownloads from "../soulDownloads.js";
import type * as soulStars from "../soulStars.js";
import type * as souls from "../souls.js";
import type * as stars from "../stars.js";
import type * as statsMaintenance from "../statsMaintenance.js";
import type * as telemetry from "../telemetry.js";
@@ -154,16 +164,20 @@ declare const fullApi: ApiFromModules<{
depRegistryScan: typeof depRegistryScan;
devSeed: typeof devSeed;
devSeedExtra: typeof devSeedExtra;
downloadMetrics: typeof downloadMetrics;
downloads: typeof downloads;
emailsNode: typeof emailsNode;
functions: typeof functions;
githubAccountAgeBackfill: typeof githubAccountAgeBackfill;
githubBackups: typeof githubBackups;
githubBackupsNode: typeof githubBackupsNode;
githubApp: typeof githubApp;
githubIdentity: typeof githubIdentity;
githubImport: typeof githubImport;
githubRestore: typeof githubRestore;
githubRestoreMutations: typeof githubRestoreMutations;
githubSoulBackups: typeof githubSoulBackups;
githubSoulBackupsNode: typeof githubSoulBackupsNode;
githubSkillSources: typeof githubSkillSources;
githubSkillSync: typeof githubSkillSync;
http: typeof http;
httpApi: typeof httpApi;
httpApiV1: typeof httpApiV1;
@@ -172,7 +186,6 @@ declare const fullApi: ApiFromModules<{
"httpApiV1/publishersV1": typeof httpApiV1_publishersV1;
"httpApiV1/shared": typeof httpApiV1_shared;
"httpApiV1/skillsV1": typeof httpApiV1_skillsV1;
"httpApiV1/soulsV1": typeof httpApiV1_soulsV1;
"httpApiV1/starsV1": typeof httpApiV1_starsV1;
"httpApiV1/transfersV1": typeof httpApiV1_transfersV1;
"httpApiV1/usersV1": typeof httpApiV1_usersV1;
@@ -186,31 +199,37 @@ declare const fullApi: ApiFromModules<{
"lib/badges": typeof lib_badges;
"lib/batching": typeof lib_batching;
"lib/changelog": typeof lib_changelog;
"lib/clawScanNote": typeof lib_clawScanNote;
"lib/clawpack": typeof lib_clawpack;
"lib/commentScamPrompt": typeof lib_commentScamPrompt;
"lib/contentTypes": typeof lib_contentTypes;
"lib/depRegistryScan": typeof lib_depRegistryScan;
"lib/devAuth": typeof lib_devAuth;
"lib/devSeed": typeof lib_devSeed;
"lib/emailRendering": typeof lib_emailRendering;
"lib/emails": typeof lib_emails;
"lib/embeddingVisibility": typeof lib_embeddingVisibility;
"lib/embeddings": typeof lib_embeddings;
"lib/githubAccount": typeof lib_githubAccount;
"lib/githubAppSync": typeof lib_githubAppSync;
"lib/githubActionsOidc": typeof lib_githubActionsOidc;
"lib/githubAuth": typeof lib_githubAuth;
"lib/githubBackup": typeof lib_githubBackup;
"lib/githubIdentity": typeof lib_githubIdentity;
"lib/githubImport": typeof lib_githubImport;
"lib/githubProfileSync": typeof lib_githubProfileSync;
"lib/githubRestoreHelpers": typeof lib_githubRestoreHelpers;
"lib/githubSoulBackup": typeof lib_githubSoulBackup;
"lib/githubSkillSync": typeof lib_githubSkillSync;
"lib/globalStats": typeof lib_globalStats;
"lib/httpHeaders": typeof lib_httpHeaders;
"lib/httpRateLimit": typeof lib_httpRateLimit;
"lib/httpUtils": typeof lib_httpUtils;
"lib/installResolver": typeof lib_installResolver;
"lib/leaderboards": typeof lib_leaderboards;
"lib/manualOverrides": typeof lib_manualOverrides;
"lib/moderation": typeof lib_moderation;
"lib/moderationEngine": typeof lib_moderationEngine;
"lib/moderationReasonCodes": typeof lib_moderationReasonCodes;
"lib/observabilityEvents": typeof lib_observabilityEvents;
"lib/officialPublishers": typeof lib_officialPublishers;
"lib/openaiResponse": typeof lib_openaiResponse;
"lib/packageRegistry": typeof lib_packageRegistry;
"lib/packageSearchDigest": typeof lib_packageSearchDigest;
@@ -219,8 +238,11 @@ declare const fullApi: ApiFromModules<{
"lib/public": typeof lib_public;
"lib/publicRouteReservations": typeof lib_publicRouteReservations;
"lib/publishLimits": typeof lib_publishLimits;
"lib/publisherAbuseScoring": typeof lib_publisherAbuseScoring;
"lib/publisherCatalogDisplay": typeof lib_publisherCatalogDisplay;
"lib/publisherStats": typeof lib_publisherStats;
"lib/publishers": typeof lib_publishers;
"lib/recommendationScore": typeof lib_recommendationScore;
"lib/reporting": typeof lib_reporting;
"lib/reservedHandles": typeof lib_reservedHandles;
"lib/reservedSlugs": typeof lib_reservedSlugs;
@@ -229,6 +251,7 @@ declare const fullApi: ApiFromModules<{
"lib/skillBackfill": typeof lib_skillBackfill;
"lib/skillCapabilityTags": typeof lib_skillCapabilityTags;
"lib/skillCards": typeof lib_skillCards;
"lib/skillFileAccess": typeof lib_skillFileAccess;
"lib/skillIcon": typeof lib_skillIcon;
"lib/skillPublish": typeof lib_skillPublish;
"lib/skillQuality": typeof lib_skillQuality;
@@ -239,8 +262,6 @@ declare const fullApi: ApiFromModules<{
"lib/skillSummary": typeof lib_skillSummary;
"lib/skillZip": typeof lib_skillZip;
"lib/skills": typeof lib_skills;
"lib/soulChangelog": typeof lib_soulChangelog;
"lib/soulPublish": typeof lib_soulPublish;
"lib/staticPublishScan": typeof lib_staticPublishScan;
"lib/tokens": typeof lib_tokens;
"lib/userSearch": typeof lib_userSearch;
@@ -248,24 +269,23 @@ declare const fullApi: ApiFromModules<{
"lib/webhooks": typeof lib_webhooks;
llmEval: typeof llmEval;
maintenance: typeof maintenance;
managementDevSeed: typeof managementDevSeed;
packageInspectorHttp: typeof packageInspectorHttp;
packageInspectorNode: typeof packageInspectorNode;
packagePublishTokens: typeof packagePublishTokens;
packages: typeof packages;
publisherAbuse: typeof publisherAbuse;
publisherAbuseDevSeed: typeof publisherAbuseDevSeed;
publishers: typeof publishers;
rateLimits: typeof rateLimits;
search: typeof search;
securityDataset: typeof securityDataset;
securityDatasetNode: typeof securityDatasetNode;
securityScan: typeof securityScan;
seed: typeof seed;
seedSouls: typeof seedSouls;
skillCards: typeof skillCards;
skillStatEvents: typeof skillStatEvents;
skillTransfers: typeof skillTransfers;
skills: typeof skills;
soulComments: typeof soulComments;
soulDownloads: typeof soulDownloads;
soulStars: typeof soulStars;
souls: typeof souls;
stars: typeof stars;
statsMaintenance: typeof statsMaintenance;
telemetry: typeof telemetry;
-1
View File
@@ -4,4 +4,3 @@ import { internal } from "./_generated/api";
// Public exposure is prevented at runtime by `internalMutation`; this file
// just pins the public references that *should* exist.
void internal.downloads.recordDownloadInternal;
void internal.soulDownloads.incrementInternal;
+47 -2
View File
@@ -3,7 +3,9 @@ import type { Id } from "./_generated/dataModel";
import {
BANNED_REAUTH_MESSAGE,
DELETED_ACCOUNT_REAUTH_MESSAGE,
createGitHubAuthProvider,
handleDeletedUserSignIn,
normalizeGitHubProfileId,
} from "./auth";
function makeCtx({
@@ -128,7 +130,7 @@ describe("handleDeletedUserSignIn", () => {
expect(ctx.db.patch).not.toHaveBeenCalled();
});
it("includes the moderator ban reason in the sign-in error", async () => {
it("does not leak the moderator ban reason in the sign-in error", async () => {
const { ctx } = makeCtx({
user: { deletedAt: 123, banReason: "Chargeback fraud" },
banRecords: [{ action: "user.ban" }],
@@ -136,6 +138,49 @@ describe("handleDeletedUserSignIn", () => {
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(`${BANNED_REAUTH_MESSAGE} Reason: Chargeback fraud`);
).rejects.toThrow(BANNED_REAUTH_MESSAGE);
});
});
describe("GitHub auth provider", () => {
it("does not link ClawHub accounts by GitHub profile email", () => {
const provider = createGitHubAuthProvider() as {
options?: { allowDangerousEmailAccountLinking?: boolean };
};
expect(provider.options?.allowDangerousEmailAccountLinking).toBe(false);
});
it("normalizes numeric GitHub profile ids", () => {
expect(normalizeGitHubProfileId(123456)).toBe("123456");
expect(normalizeGitHubProfileId("789012")).toBe("789012");
});
it("rejects missing or nonnumeric GitHub profile ids", () => {
expect(() => normalizeGitHubProfileId(undefined)).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
expect(() => normalizeGitHubProfileId("undefined")).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
expect(() => normalizeGitHubProfileId("github-user")).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
});
it("fails closed when the GitHub provider receives a malformed profile", () => {
const provider = createGitHubAuthProvider() as {
options?: { profile?: (profile: Record<string, unknown>) => Record<string, unknown> };
};
expect(() => provider.options?.profile?.({ message: "Bad credentials" })).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
expect(provider.options?.profile?.({ id: 123456, login: "fixture-user" })).toEqual({
id: "123456",
name: "fixture-user",
email: undefined,
image: undefined,
});
});
});
+44 -21
View File
@@ -9,19 +9,48 @@ import { isLocalDevAuthEnabled } from "./lib/devAuth";
import { shouldScheduleGitHubProfileSync } from "./lib/githubProfileSync";
export const BANNED_REAUTH_MESSAGE =
"This account has been banned and cannot sign in. If you believe this is a mistake, please contact security@openclaw.ai and we will review it.";
"This account has been banned and cannot sign in. If you believe this is a mistake, appeal this decision: https://appeals.openclaw.ai/.";
export const DELETED_ACCOUNT_REAUTH_MESSAGE =
"This account has been permanently deleted and cannot be restored.";
const REAUTH_BLOCKING_BAN_ACTIONS = new Set(["user.ban", "user.autoban.malware"]);
const DEV_PERSONAS = new Set(["owner", "user", "admin"]);
const DEV_PERSONAS = new Set(["owner", "user", "admin", "officialOrgMember", "abusePublisher"]);
function getBannedReauthMessage(reason: string | undefined) {
const normalizedReason = reason?.trim();
if (!normalizedReason || normalizedReason.toLowerCase() === "malware auto-ban") {
return BANNED_REAUTH_MESSAGE;
export function normalizeGitHubProfileId(profileId: unknown) {
const id =
typeof profileId === "number" && Number.isSafeInteger(profileId)
? String(profileId)
: typeof profileId === "string"
? profileId.trim()
: null;
if (!id || !/^\d+$/.test(id)) {
throw new Error("GitHub OAuth profile is missing a valid numeric id");
}
return `${BANNED_REAUTH_MESSAGE} Reason: ${normalizedReason}`;
return id;
}
export function createGitHubAuthProvider() {
return GitHub({
clientId: process.env.AUTH_GITHUB_ID ?? "",
clientSecret: process.env.AUTH_GITHUB_SECRET ?? "",
// GitHub's OAuth email must not be treated as a ClawHub account key. The
// immutable GitHub provider account id is the only account-linking key.
allowDangerousEmailAccountLinking: false,
profile(profile) {
return {
id: normalizeGitHubProfileId(profile.id),
name: profile.login,
email: profile.email ?? undefined,
image: profile.avatar_url,
};
},
});
}
function getBannedReauthMessage(_reason: string | undefined) {
return BANNED_REAUTH_MESSAGE;
}
export async function handleDeletedUserSignIn(
@@ -75,26 +104,20 @@ export async function handleDeletedUserSignIn(
export const { auth, signIn, signOut, store, isAuthenticated } = convexAuth({
providers: [
GitHub({
clientId: process.env.AUTH_GITHUB_ID ?? "",
clientSecret: process.env.AUTH_GITHUB_SECRET ?? "",
profile(profile) {
return {
id: String(profile.id),
name: profile.login,
email: profile.email ?? undefined,
image: profile.avatar_url,
};
},
}),
createGitHubAuthProvider(),
ConvexCredentials({
id: "dev-persona",
authorize: async (credentials, ctx) => {
if (!isLocalDevAuthEnabled()) throw new Error("Dev auth is disabled");
const devAuthSecret =
typeof credentials.devAuthSecret === "string" ? credentials.devAuthSecret : undefined;
if (!isLocalDevAuthEnabled(process.env, devAuthSecret)) {
throw new Error("Dev auth is disabled");
}
const persona = typeof credentials.persona === "string" ? credentials.persona : "";
if (!DEV_PERSONAS.has(persona)) throw new Error("Unknown dev persona");
const userId: Id<"users"> = await ctx.runMutation(internal.users.upsertDevPersonaInternal, {
persona: persona as "owner" | "user" | "admin",
persona: persona as "owner" | "user" | "admin" | "officialOrgMember" | "abusePublisher",
devAuthSecret,
});
return { userId };
},
File diff suppressed because it is too large Load Diff
-228
View File
@@ -1,228 +0,0 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { requireUser } from "./lib/access";
import { updateLatestClawScanNoteAndRequestRescan as updatePackageClawScanNoteAndRequestRescan } from "./packages";
import { updateLatestClawScanNoteAndRequestRescan as updateSkillClawScanNoteAndRequestRescan } from "./skills";
vi.mock("./lib/access", () => ({
requireUser: vi.fn(),
}));
type WrappedHandler<TArgs, TResult = unknown> = {
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>;
};
const updateSkillClawScanNoteAndRequestRescanHandler = (
updateSkillClawScanNoteAndRequestRescan as unknown as WrappedHandler<{
skillId: string;
clawScanNote?: string;
}>
)._handler;
const updatePackageClawScanNoteAndRequestRescanHandler = (
updatePackageClawScanNoteAndRequestRescan as unknown as WrappedHandler<{
packageId: string;
clawScanNote?: string;
}>
)._handler;
function createDb() {
const auditLogs: Array<Record<string, unknown>> = [];
const skill = {
_id: "skills:1",
slug: "flagged-skill",
ownerUserId: "users:owner",
latestVersionId: "skillVersions:latest",
softDeletedAt: undefined,
};
const version = {
_id: "skillVersions:latest",
skillId: "skills:1",
version: "1.2.3",
clawScanNote: "old skill note",
softDeletedAt: undefined,
};
const pkg = {
_id: "packages:1",
name: "flagged-plugin",
family: "code-plugin",
ownerUserId: "users:owner",
latestReleaseId: "packageReleases:latest",
softDeletedAt: undefined,
};
const release = {
_id: "packageReleases:latest",
packageId: "packages:1",
version: "2.0.0",
clawScanNote: "old plugin note",
softDeletedAt: undefined,
};
const db = {
get: vi.fn(async (tableOrId: string, maybeId?: string) => {
const id = maybeId ?? tableOrId;
if (id === "skills:1") return skill;
if (id === "skillVersions:latest") return version;
if (id === "packages:1") return pkg;
if (id === "packageReleases:latest") return release;
return null;
}),
insert: vi.fn(async (table: string, doc: Record<string, unknown>) => {
if (table !== "auditLogs") throw new Error(`unexpected insert ${table}`);
auditLogs.push(doc);
return `auditLogs:${auditLogs.length}`;
}),
patch: vi.fn(
async (
tableOrId: string,
idOrPatch: string | Record<string, unknown>,
maybePatch?: Record<string, unknown>,
) => {
const id = maybePatch ? (idOrPatch as string) : tableOrId;
const patch = maybePatch ?? (idOrPatch as Record<string, unknown>);
if (id === "skillVersions:latest") Object.assign(version, patch);
if (id === "packageReleases:latest") Object.assign(release, patch);
},
),
query: vi.fn((table: string) => {
throw new Error(`unexpected table ${table}`);
}),
normalizeId: vi.fn((table: string, id: string) => (id.startsWith(`${table}:`) ? id : null)),
system: {},
};
return { db, auditLogs, version, release };
}
beforeEach(() => {
vi.mocked(requireUser).mockReset();
vi.mocked(requireUser).mockResolvedValue({
userId: "users:owner",
user: { _id: "users:owner", role: "user" },
} as never);
});
describe("publisher ClawScan note updates", () => {
it("updates a latest skill publisher note, writes audit metadata, and schedules ClawScan", async () => {
const { db, auditLogs, version } = createDb();
const scheduler = { runAfter: vi.fn(async () => undefined) };
await updateSkillClawScanNoteAndRequestRescanHandler({ db, scheduler } as never, {
skillId: "skills:1",
clawScanNote: "New context for the scanner.",
});
expect(version).toMatchObject({
clawScanNote: "New context for the scanner.",
clawScanNoteUpdatedAt: expect.any(Number),
});
expect(auditLogs[0]).toMatchObject({
action: "skill.clawscan_note.update",
targetType: "skillVersion",
targetId: "skillVersions:latest",
metadata: expect.objectContaining({
hadPreviousNote: true,
hasNextNote: true,
nextLength: 28,
}),
});
expect(scheduler.runAfter).toHaveBeenCalledWith(
0,
expect.anything(),
expect.objectContaining({
versionId: "skillVersions:latest",
}),
);
});
it("clears a latest skill publisher note while preserving the update timestamp", async () => {
const { db, auditLogs, version } = createDb();
const scheduler = { runAfter: vi.fn(async () => undefined) };
await updateSkillClawScanNoteAndRequestRescanHandler({ db, scheduler } as never, {
skillId: "skills:1",
clawScanNote: " ",
});
expect(version).toMatchObject({
clawScanNote: "",
clawScanNoteUpdatedAt: expect.any(Number),
});
expect(auditLogs[0]).toMatchObject({
action: "skill.clawscan_note.update",
metadata: expect.objectContaining({
hadPreviousNote: true,
hasNextNote: false,
nextLength: 0,
}),
});
});
it("updates a latest plugin publisher note, writes audit metadata, and schedules ClawScan", async () => {
const { db, auditLogs, release } = createDb();
const scheduler = { runAfter: vi.fn(async () => undefined) };
await updatePackageClawScanNoteAndRequestRescanHandler({ db, scheduler } as never, {
packageId: "packages:1",
clawScanNote: "Plugin native host is scoped to local files.",
});
expect(release).toMatchObject({
clawScanNote: "Plugin native host is scoped to local files.",
clawScanNoteUpdatedAt: expect.any(Number),
});
expect(auditLogs[0]).toMatchObject({
action: "package.clawscan_note.update",
targetType: "packageRelease",
targetId: "packageReleases:latest",
metadata: expect.objectContaining({
hadPreviousNote: true,
hasNextNote: true,
}),
});
expect(scheduler.runAfter).toHaveBeenCalledWith(
0,
expect.anything(),
expect.objectContaining({
releaseId: "packageReleases:latest",
}),
);
});
it("allows platform moderators to update latest skill publisher notes", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:moderator",
user: { _id: "users:moderator", role: "moderator" },
} as never);
const { db, version } = createDb();
const scheduler = { runAfter: vi.fn(async () => undefined) };
await updateSkillClawScanNoteAndRequestRescanHandler({ db, scheduler } as never, {
skillId: "skills:1",
clawScanNote: "Moderator context.",
});
expect(version).toMatchObject({
clawScanNote: "Moderator context.",
clawScanNoteUpdatedAt: expect.any(Number),
});
});
it("allows platform moderators to update latest plugin publisher notes", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:moderator",
user: { _id: "users:moderator", role: "moderator" },
} as never);
const { db, release } = createDb();
const scheduler = { runAfter: vi.fn(async () => undefined) };
await updatePackageClawScanNoteAndRequestRescanHandler({ db, scheduler } as never, {
packageId: "packages:1",
clawScanNote: "Moderator plugin context.",
});
expect(release).toMatchObject({
clawScanNote: "Moderator plugin context.",
clawScanNoteUpdatedAt: expect.any(Number),
});
});
});
+263
View File
@@ -0,0 +1,263 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("./lib/access", () => ({
requireUser: vi.fn(),
}));
const { requireUser } = await import("./lib/access");
const { approve, deny } = await import("./cliDeviceAuth");
const approveHandler = (approve as unknown as { _handler: Function })._handler;
const denyHandler = (deny as unknown as { _handler: Function })._handler;
function makeCtx(rows: Array<Record<string, unknown>>) {
const take = vi.fn().mockResolvedValue(rows);
const order = vi.fn().mockReturnValue({ take });
const withIndex = vi.fn().mockReturnValue({ order });
const query = vi.fn().mockReturnValue({ withIndex });
const get = vi.fn().mockResolvedValue(null);
const insert = vi.fn().mockResolvedValue("inserted:id");
const patch = vi.fn().mockResolvedValue(undefined);
const replace = vi.fn().mockResolvedValue(undefined);
const delete_ = vi.fn().mockResolvedValue(undefined);
const normalizeId = vi.fn().mockReturnValue(null);
return {
ctx: { db: { get, insert, query, patch, replace, delete: delete_, normalizeId } },
order,
take,
withIndex,
query,
patch,
};
}
describe("cliDeviceAuth approval", () => {
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(new Date("2026-06-10T12:00:00Z"));
vi.mocked(requireUser).mockResolvedValue({
userId: "users:approver",
user: { _id: "users:approver" },
} as never);
});
afterEach(() => {
vi.useRealTimers();
vi.mocked(requireUser).mockReset();
});
it("approves the newest active pending row when duplicate user codes exist", async () => {
const now = Date.now();
const { ctx, order, patch, take } = makeCtx([
{
_id: "cliDeviceCodes:new",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
{
_id: "cliDeviceCodes:old",
_creationTime: now - 10_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 10_000,
expiresAt: now + 60_000,
},
]);
const result = await approveHandler(ctx, { userCode: "q639-nbsx" });
expect(result).toEqual({
ok: true,
userCode: "Q639-NBSX",
expiresAt: now + 60_000,
});
expect(order).toHaveBeenCalledWith("desc");
expect(take).toHaveBeenCalledWith(50);
expect(patch).toHaveBeenCalledTimes(1);
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:new", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
});
it("uses descending index order when duplicate creation timestamps disagree", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:new",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 20_000,
expiresAt: now + 60_000,
},
{
_id: "cliDeviceCodes:old",
_creationTime: now - 10_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 500,
expiresAt: now + 60_000,
},
]);
await approveHandler(ctx, { userCode: "Q639-NBSX" });
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:new", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
});
it("expires stale pending rows before approving an active row", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:expired",
_creationTime: now - 100_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 100_000,
expiresAt: now - 1,
},
{
_id: "cliDeviceCodes:active",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
]);
await approveHandler(ctx, { userCode: "Q639-NBSX" });
expect(patch).toHaveBeenNthCalledWith(1, "cliDeviceCodes:expired", { status: "expired" });
expect(patch).toHaveBeenNthCalledWith(2, "cliDeviceCodes:active", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
});
it("preserves stale terminal rows while expiring stale pending duplicates", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:approved",
_creationTime: now - 100_000,
status: "approved",
userCode: "Q639-NBSX",
createdAt: now - 100_000,
expiresAt: now - 1,
approvedAt: now - 90_000,
approvedByUserId: "users:previous",
},
{
_id: "cliDeviceCodes:denied",
_creationTime: now - 90_000,
status: "denied",
userCode: "Q639-NBSX",
createdAt: now - 90_000,
expiresAt: now - 1,
deniedAt: now - 80_000,
},
{
_id: "cliDeviceCodes:consumed",
_creationTime: now - 80_000,
status: "consumed",
userCode: "Q639-NBSX",
createdAt: now - 80_000,
expiresAt: now - 1,
consumedAt: now - 70_000,
},
{
_id: "cliDeviceCodes:expired",
_creationTime: now - 70_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 70_000,
expiresAt: now - 1,
},
{
_id: "cliDeviceCodes:active",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
]);
await approveHandler(ctx, { userCode: "Q639-NBSX" });
expect(patch).toHaveBeenNthCalledWith(1, "cliDeviceCodes:expired", { status: "expired" });
expect(patch).toHaveBeenNthCalledWith(2, "cliDeviceCodes:active", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:approved", expect.anything());
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:denied", expect.anything());
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:consumed", expect.anything());
});
it("denies the newest active pending row when duplicate user codes exist", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:new",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
{
_id: "cliDeviceCodes:old",
_creationTime: now - 10_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 10_000,
expiresAt: now + 60_000,
},
]);
await expect(denyHandler(ctx, { userCode: "Q639-NBSX" })).resolves.toEqual({ ok: true });
expect(patch).toHaveBeenCalledTimes(1);
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:new", {
status: "denied",
deniedAt: now,
});
});
it("does not deny stale pending rows after expiring them", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:expired",
_creationTime: now - 100_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 100_000,
expiresAt: now - 1,
},
]);
await expect(denyHandler(ctx, { userCode: "Q639-NBSX" })).resolves.toEqual({ ok: true });
expect(patch).toHaveBeenCalledTimes(1);
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:expired", { status: "expired" });
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:expired", {
status: "denied",
deniedAt: now,
});
});
});
+45 -15
View File
@@ -1,10 +1,13 @@
import { v } from "convex/values";
import type { Doc } from "./_generated/dataModel";
import type { MutationCtx } from "./_generated/server";
import { internalMutation, mutation } from "./functions";
import { requireUser } from "./lib/access";
import { generateToken, hashToken } from "./lib/tokens";
const DEVICE_CODE_TTL_MS = 15 * 60_000;
const DEVICE_POLL_INTERVAL_SECONDS = 5;
const MAX_DUPLICATE_USER_CODE_ROWS = 50;
const USER_CODE_ALPHABET = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789";
export const createInternal = internalMutation({
@@ -88,17 +91,13 @@ export const approve = mutation({
if (!normalized) throw new Error("Code required");
const userCodeHash = await hashToken(normalized);
const row = await ctx.db
.query("cliDeviceCodes")
.withIndex("by_user_code_hash", (q) => q.eq("userCodeHash", userCodeHash))
.unique();
if (!row) throw new Error("Device code not found");
const now = Date.now();
if (row.expiresAt <= now) {
if (row.status !== "expired") await ctx.db.patch(row._id, { status: "expired" });
throw new Error("Device code expired");
}
const rows = await expireStaleRows(ctx, await getRowsByUserCodeHash(ctx, userCodeHash), now);
const row =
pickLatestRow(rows, now, "pending") ?? pickLatestRow(rows, now) ?? pickLatestRow(rows);
if (!row) throw new Error("Device code not found");
if (row.expiresAt <= now) throw new Error("Device code expired");
if (row.status === "expired") throw new Error("Device code expired");
if (row.status === "consumed") throw new Error("Device code already used");
if (row.status === "approved") throw new Error("Device code already authorized");
if (row.status === "denied") throw new Error("Device code was denied");
@@ -119,12 +118,11 @@ export const deny = mutation({
const normalized = normalizeUserCode(args.userCode);
if (!normalized) throw new Error("Code required");
const userCodeHash = await hashToken(normalized);
const row = await ctx.db
.query("cliDeviceCodes")
.withIndex("by_user_code_hash", (q) => q.eq("userCodeHash", userCodeHash))
.unique();
if (!row) throw new Error("Device code not found");
const now = Date.now();
const rows = await expireStaleRows(ctx, await getRowsByUserCodeHash(ctx, userCodeHash), now);
const row =
pickLatestRow(rows, now, "pending") ?? pickLatestRow(rows, now) ?? pickLatestRow(rows);
if (!row) throw new Error("Device code not found");
if (row.status === "approved") throw new Error("Device code already authorized");
if (row.status === "pending") {
await ctx.db.patch(row._id, { status: "denied", deniedAt: now });
@@ -140,6 +138,38 @@ function normalizeUserCode(value: string) {
.replace(/[^A-Z0-9]/g, "");
}
async function getRowsByUserCodeHash(ctx: MutationCtx, userCodeHash: string) {
return await ctx.db
.query("cliDeviceCodes")
.withIndex("by_user_code_hash", (q) => q.eq("userCodeHash", userCodeHash))
.order("desc")
.take(MAX_DUPLICATE_USER_CODE_ROWS);
}
async function expireStaleRows(ctx: MutationCtx, rows: Array<Doc<"cliDeviceCodes">>, now: number) {
const nextRows: Array<Doc<"cliDeviceCodes">> = [];
for (const row of rows) {
if (row.status === "pending" && row.expiresAt <= now) {
await ctx.db.patch(row._id, { status: "expired" });
nextRows.push({ ...row, status: "expired" });
} else {
nextRows.push(row);
}
}
return nextRows;
}
function pickLatestRow(
rows: Array<Doc<"cliDeviceCodes">>,
now?: number,
status?: Doc<"cliDeviceCodes">["status"],
) {
return rows.find(
(row) =>
(now === undefined || row.expiresAt > now) && (status === undefined || row.status === status),
);
}
function generateUserCode() {
const bytes = new Uint8Array(8);
crypto.getRandomValues(bytes);
+58
View File
@@ -0,0 +1,58 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from "vitest";
const mocks = vi.hoisted(() => {
const interval = vi.fn();
const githubSkillSyncRef = Symbol("github-skill-source-sync");
return { interval, githubSkillSyncRef };
});
vi.mock("convex/server", () => ({
cronJobs: () => ({
interval: mocks.interval,
}),
}));
vi.mock("./_generated/api", () => ({
internal: {
githubBackupsNode: { syncGitHubBackupsInternal: Symbol("github-backup-sync") },
githubSkillSync: { syncGitHubSkillSourcesInternal: mocks.githubSkillSyncRef },
leaderboards: { rebuildTrendingLeaderboardAction: Symbol("trending-leaderboard") },
statsMaintenance: {
runSkillStatBackfillInternal: Symbol("skill-stats-backfill"),
updateGlobalStatsAction: Symbol("global-stats-update"),
},
skillStatEvents: { processSkillStatEventsAction: Symbol("skill-stat-events") },
packages: {
processPackageStatEventsInternal: Symbol("package-stat-events"),
backfillPackageReleaseScansInternal: Symbol("package-scan-backfill"),
},
publisherAbuse: {
runPublisherAbuseScoreRunInternal: Symbol("publisher-abuse-score-refresh"),
},
vt: {
pollPendingScans: Symbol("vt-pending-scans"),
backfillActiveSkillsVTCache: Symbol("vt-cache-backfill"),
},
securityScan: {
pruneExpiredSkillScanRequestsInternal: Symbol("skill-scan-request-prune"),
},
downloads: { pruneDownloadDedupesInternal: Symbol("download-dedupe-prune") },
downloadMetrics: {
pruneDownloadMetricDedupesInternal: Symbol("download-metric-dedupe-prune"),
},
},
}));
describe("crons", () => {
it("runs GitHub skill source sync every 15 minutes", async () => {
await import("./crons");
expect(mocks.interval).toHaveBeenCalledWith(
"github-skill-source-sync",
{ minutes: 15 },
mocks.githubSkillSyncRef,
{},
);
});
});
+42
View File
@@ -10,6 +10,13 @@ crons.interval(
{ batchSize: 50, maxBatches: 5 },
);
crons.interval(
"github-skill-source-sync",
{ minutes: 15 },
internal.githubSkillSync.syncGitHubSkillSourcesInternal,
{},
);
crons.interval(
"trending-leaderboard",
{ minutes: 60 },
@@ -57,6 +64,27 @@ crons.interval(
{},
);
crons.interval(
"publisher-abuse-score-refresh",
{ hours: 24 },
internal.publisherAbuse.runPublisherAbuseScoreRunInternal,
{ batchSize: 250, maxPages: 5, trigger: "cron" },
);
crons.interval(
"publisher-temporal-abuse-scan",
{ hours: 24 },
internal.publisherAbuse.runTemporalPublisherAbuseScanInternal,
{
mode: "current",
dryRun: false,
candidateLimit: 1000,
batchSize: 50,
maxPages: 20,
trigger: "cron",
},
);
crons.interval("vt-pending-scans", { minutes: 5 }, internal.vt.pollPendingScans, {
batchSize: 100,
});
@@ -72,6 +100,13 @@ crons.interval(
{ batchSize: 100 },
);
crons.interval(
"skill-scan-request-prune",
{ hours: 6 },
internal.securityScan.pruneExpiredSkillScanRequestsInternal,
{ batchSize: 250 },
);
crons.interval(
"download-dedupe-prune",
{ hours: 24 },
@@ -79,4 +114,11 @@ crons.interval(
{},
);
crons.interval(
"download-metric-dedupe-prune",
{ hours: 24 },
internal.downloadMetrics.pruneDownloadMetricDedupesInternal,
{},
);
export default crons;
+10
View File
@@ -0,0 +1,10 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
const { checkDependencyRegistriesHandler } = await import("./depRegistryScan");
describe("dependency registry scan drain", () => {
it("keeps legacy scheduled jobs harmless after the scanner is retired", async () => {
await expect(checkDependencyRegistriesHandler()).resolves.toBeNull();
});
});
+4 -263
View File
@@ -1,270 +1,11 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc, Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { internalAction, internalMutation, internalQuery } from "./functions";
import {
dedupeDeps,
depRegistryUrl,
parseDependencyFile,
SUPPORTED_DEP_REGISTRIES,
summarizeDepRegistryChecks,
type DepEntry,
type DepRegistryResult,
type DepRegistryUnresolved,
type SupportedDepRegistry,
} from "./lib/depRegistryScan";
import { readStorageText } from "./lib/packageRegistry";
import { internalAction } from "./_generated/server";
const REQUEST_TIMEOUT_MS = 8_000;
const MAX_RETRIES = 2;
const BACKOFF_BASE_MS = 750;
const INTER_REQUEST_DELAY_MS = 100;
const MAX_DEPENDENCIES_PER_SCAN = 120;
const CACHE_TTL_EXISTS_MS = 30 * 24 * 60 * 60 * 1_000;
const CACHE_TTL_NOT_EXISTS_MS = 7 * 24 * 60 * 60 * 1_000;
const registryValidator = v.union(v.literal("pypi"), v.literal("npm"), v.literal("cargo"));
type RegistryCheck =
| { kind: "found"; httpStatus: number }
| { kind: "missing"; httpStatus: number }
| { kind: "unresolved"; reason: string };
function isSupportedRegistry(value: string): value is SupportedDepRegistry {
return (SUPPORTED_DEP_REGISTRIES as readonly string[]).includes(value);
}
async function wait(ms: number) {
await new Promise((resolve) => setTimeout(resolve, ms));
}
async function checkRegistry(dep: DepEntry): Promise<RegistryCheck> {
const headers: Record<string, string> = { Accept: "application/json" };
if (dep.registry === "cargo") {
headers["User-Agent"] = "ClawHub-DepRegistryScan/1.0 (https://clawhub.ai)";
}
let lastStatus: number | undefined;
for (let attempt = 0; attempt <= MAX_RETRIES; attempt += 1) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
const response = await fetch(depRegistryUrl(dep.registry, dep.name), {
method: "GET",
headers,
signal: controller.signal,
});
clearTimeout(timeout);
lastStatus = response.status;
if (response.status === 200) return { kind: "found", httpStatus: response.status };
if (response.status === 404) return { kind: "missing", httpStatus: response.status };
if (response.status !== 429 && response.status < 500) {
return {
kind: "unresolved",
reason: `unexpected HTTP ${response.status}`,
};
}
} catch (error) {
clearTimeout(timeout);
if (attempt === MAX_RETRIES) {
return {
kind: "unresolved",
reason: error instanceof Error ? error.message : "network error",
};
}
}
if (attempt < MAX_RETRIES) {
await wait(2 ** attempt * BACKOFF_BASE_MS);
}
}
return {
kind: "unresolved",
reason: lastStatus ? `HTTP ${lastStatus}` : "network error",
};
}
async function extractDependencies(ctx: Pick<ActionCtx, "storage">, version: Doc<"skillVersions">) {
const entries: DepEntry[] = [];
for (const file of version.files) {
const basename = file.path.split("/").pop()?.toLowerCase() ?? "";
if (
basename !== "requirements.txt" &&
basename !== "requirements-dev.txt" &&
basename !== "requirements_dev.txt" &&
basename !== "requirements-test.txt" &&
basename !== "requirements_test.txt" &&
basename !== "package.json" &&
basename !== "cargo.toml" &&
basename !== "pyproject.toml"
) {
continue;
}
const content = await readStorageText(ctx, file.storageId);
entries.push(...parseDependencyFile(file.path, content));
}
return dedupeDeps(entries);
}
export const lookupCacheInternal = internalQuery({
args: {
registry: registryValidator,
name: v.string(),
},
handler: async (ctx, args): Promise<Doc<"depRegistryCache"> | null> => {
return ctx.db
.query("depRegistryCache")
.withIndex("by_registry_name", (q) => q.eq("registry", args.registry).eq("name", args.name))
.unique();
},
});
export const upsertCacheInternal = internalMutation({
args: {
registry: registryValidator,
name: v.string(),
exists: v.boolean(),
httpStatus: v.number(),
checkedAt: v.number(),
},
handler: async (ctx, args) => {
const existing = await ctx.db
.query("depRegistryCache")
.withIndex("by_registry_name", (q) => q.eq("registry", args.registry).eq("name", args.name))
.unique();
const patch = {
registry: args.registry,
name: args.name,
exists: args.exists,
httpStatus: args.httpStatus,
checkedAt: args.checkedAt,
};
if (existing) {
await ctx.db.patch(existing._id, patch);
} else {
await ctx.db.insert("depRegistryCache", patch);
}
},
});
export const getRetryableVersionIdsInternal = internalQuery({
args: {
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const limit = Math.min(Math.max(args.limit ?? 25, 1), 100);
const versions = await ctx.db
.query("skillVersions")
.withIndex("by_dep_registry_scan_status_and_created", (q) =>
q.eq("depRegistryScanStatus", "error"),
)
.order("desc")
.take(limit);
return versions.map((version) => version._id);
},
});
async function checkWithCache(ctx: ActionCtx, dep: DepEntry) {
const now = Date.now();
const cached = (await ctx.runQuery(internal.depRegistryScan.lookupCacheInternal, {
registry: dep.registry,
name: dep.name,
})) as Doc<"depRegistryCache"> | null;
if (cached) {
const ttl = cached.exists ? CACHE_TTL_EXISTS_MS : CACHE_TTL_NOT_EXISTS_MS;
if (now - cached.checkedAt < ttl) {
return cached.exists
? ({ kind: "found", httpStatus: cached.httpStatus } as const)
: ({ kind: "missing", httpStatus: cached.httpStatus } as const);
}
}
const check = await checkRegistry(dep);
if (check.kind !== "unresolved") {
await ctx.runMutation(internal.depRegistryScan.upsertCacheInternal, {
registry: dep.registry,
name: dep.name,
exists: check.kind === "found",
httpStatus: check.httpStatus,
checkedAt: now,
});
}
return check;
export async function checkDependencyRegistriesHandler(): Promise<null> {
return null;
}
export const checkDependencyRegistries = internalAction({
args: { versionId: v.id("skillVersions") },
handler: async (ctx, args) => {
const version = (await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: args.versionId,
})) as Doc<"skillVersions"> | null;
if (!version) return null;
if (version.depRegistryAnalysis && version.depRegistryAnalysis.status !== "error") {
return version.depRegistryAnalysis;
}
const deps = await extractDependencies(ctx, version);
const checkableDeps = deps.slice(0, MAX_DEPENDENCIES_PER_SCAN);
const deferredDeps = deps.slice(MAX_DEPENDENCIES_PER_SCAN);
const results: DepRegistryResult[] = [];
const unresolved: DepRegistryUnresolved[] = deferredDeps.map((dep) => ({
...dep,
reason: "dependency scan limit reached",
}));
for (const dep of checkableDeps) {
if (!isSupportedRegistry(dep.registry)) continue;
const check = await checkWithCache(ctx, dep);
if (check.kind === "unresolved") {
unresolved.push({ ...dep, reason: check.reason });
} else {
results.push({
...dep,
exists: check.kind === "found",
httpStatus: check.httpStatus,
});
}
await wait(INTER_REQUEST_DELAY_MS);
}
const analysis = summarizeDepRegistryChecks({
results,
unresolved,
checkedAt: Date.now(),
});
await ctx.runMutation(internal.skills.updateVersionDepRegistryAnalysisInternal, {
versionId: args.versionId,
depRegistryAnalysis: analysis,
});
return analysis;
},
});
export const rescanErrorDepRegistryVersions = internalAction({
args: {
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const versionIds = (await ctx.runQuery(
internal.depRegistryScan.getRetryableVersionIdsInternal,
{ limit: args.batchSize ?? 25 },
)) as Id<"skillVersions">[];
let scheduled = 0;
for (const versionId of versionIds) {
await ctx.scheduler.runAfter(
scheduled * 2_000,
internal.depRegistryScan.checkDependencyRegistries,
{
versionId,
},
);
scheduled += 1;
}
return { scheduled };
},
handler: checkDependencyRegistriesHandler,
});
+258
View File
@@ -4,7 +4,10 @@ import {
currentUserSeedPackageName,
currentUserSeedSkillSlug,
seedFeaturedPluginPackagesMutation,
seedGitHubBackedSkillSourceMutation,
seedLocalFixtures,
seedLocalModerationFixturesHandler,
seedPublicCorpusBatchMutation,
seedSkillMutation,
} from "./devSeed";
@@ -18,6 +21,15 @@ const seedSkillMutationHandler = (
const seedFeaturedPluginPackagesHandler = (
seedFeaturedPluginPackagesMutation as unknown as WrappedHandler<Record<string, unknown>>
)._handler;
const seedGitHubBackedSkillSourceHandler = (
seedGitHubBackedSkillSourceMutation as unknown as WrappedHandler<Record<string, unknown>>
)._handler;
const seedLocalFixturesHandler = (
seedLocalFixtures as unknown as WrappedHandler<{ reset?: boolean }>
)._handler;
const seedPublicCorpusBatchHandler = (
seedPublicCorpusBatchMutation as unknown as WrappedHandler<Record<string, unknown>>
)._handler;
function chainEq(constraints: Record<string, unknown>) {
return {
@@ -146,6 +158,33 @@ function seedSkillArgs(storageId: string) {
}
describe("devSeed local fixtures", () => {
it("does not preconfigure GitHub-backed source fixtures in the local seed action", async () => {
const mutationCalls: Array<{ args: Record<string, unknown> }> = [];
let storageCounter = 0;
const ctx = {
storage: {
store: async () => `storage:${++storageCounter}`,
},
runMutation: async (_ref: unknown, args: Record<string, unknown>) => {
mutationCalls.push({ args });
return { ok: true, seeded: ["local-moderation-fixtures"], skipped: [] };
},
};
const result = await seedLocalFixturesHandler(ctx as never, { reset: true });
expect(mutationCalls).toHaveLength(1);
expect(mutationCalls[0]?.args).toMatchObject({
reset: true,
});
expect(result).toEqual(
expect.objectContaining({
ok: true,
results: [expect.objectContaining({ slug: "local-moderation-fixtures" })],
}),
);
});
it("seeds core skill fixtures for an explicit local user without creating @local", async () => {
const { db, tables } = createDb();
const userId = (await db.insert("users", {
@@ -190,6 +229,210 @@ describe("devSeed local fixtures", () => {
);
});
it("does not copy publisher ownership onto public corpus skill embeddings", async () => {
const { db, tables } = createDb();
await seedPublicCorpusBatchHandler(
createMutationCtx(db) as never,
{
rows: [
{
kind: "skill",
slug: "corpus-demo",
displayName: "Corpus Demo",
version: "0.1.0",
skillMd: "---\ndescription: Corpus demo\n---\n# Corpus demo",
storageId: "storage:corpus-demo",
embedding: [0, 1, 2],
dummyOwner: {
handle: "corpus-owner",
displayName: "Corpus Owner",
image: "https://example.invalid/avatar.png",
},
},
],
} as never,
);
expect(tables.skills?.[0]).toEqual(
expect.objectContaining({
slug: "corpus-demo",
ownerPublisherId: tables.publishers?.[0]?._id,
}),
);
expect(tables.skillEmbeddings?.[0]).not.toHaveProperty("ownerPublisherId");
});
it("seeds a GitHub-backed source and skills without creating mirrored versions", async () => {
const { db, tables } = createDb();
const userId = (await db.insert("users", {
handle: "nvidia-dev",
displayName: "NVIDIA Dev",
role: "user",
createdAt: 1,
updatedAt: 1,
})) as Id<"users">;
const result = await seedGitHubBackedSkillSourceHandler(
createMutationCtx(db) as never,
{
ownerUserId: userId,
repo: "NVIDIA/skills",
defaultBranch: "main",
displayManifestKind: "skills.sh",
displayManifestHash: "manifest-sha256",
displayManifestCommit: "0".repeat(40),
displayManifestFetchedAt: 123,
displayManifestStatus: "ok",
displayManifest: {
notGrouped: "bottom",
groupings: [
{
title: "Agentic AI",
description: "Agentic AI skills.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
],
},
skills: [
{
slug: "aiq-deploy",
displayName: "AIQ Deploy",
summary: "Deploy AgentIQ workflows.",
githubPath: "skills/aiq-deploy",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy",
githubScanStatus: "clean",
githubCurrentCheckedAt: 456,
},
{
slug: "nemoclaw-user-configure-security",
displayName: "NeMoClaw User Configure Security",
summary: "Configure NeMoClaw user security.",
githubPath: "skills/nemoclaw-user-configure-security",
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: "hash-nemoclaw",
githubScanStatus: "clean",
githubCurrentCheckedAt: 789,
githubRemovedAt: 900,
},
],
} as never,
);
expect(result).toMatchObject({
ok: true,
seeded: ["aiq-deploy", "nemoclaw-user-configure-security"],
skipped: [],
});
expect(tables.githubSkillSources).toHaveLength(1);
expect(tables.githubSkillSources?.[0]).toEqual(
expect.objectContaining({
repo: "NVIDIA/skills",
ownerPublisherId: tables.publishers?.[0]?._id,
defaultBranch: "main",
displayManifestKind: "skills.sh",
displayManifestHash: "manifest-sha256",
displayManifestCommit: "0".repeat(40),
displayManifestFetchedAt: 123,
displayManifestStatus: "ok",
displayManifest: {
notGrouped: "bottom",
groupings: [
{
title: "Agentic AI",
description: "Agentic AI skills.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
],
},
}),
);
expect(tables.skills).toHaveLength(2);
expect(tables.skills).toEqual(
expect.arrayContaining([
expect.objectContaining({
slug: "aiq-deploy",
installKind: "github",
githubSourceId: tables.githubSkillSources?.[0]?._id,
githubPath: "skills/aiq-deploy",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy",
githubScanStatus: "clean",
githubCurrentCheckedAt: 456,
latestVersionId: undefined,
latestVersionSummary: undefined,
tags: {},
stats: expect.objectContaining({ versions: 0 }),
}),
expect.objectContaining({
slug: "nemoclaw-user-configure-security",
installKind: "github",
githubSourceId: tables.githubSkillSources?.[0]?._id,
githubPath: "skills/nemoclaw-user-configure-security",
githubCurrentContentHash: "hash-nemoclaw",
githubRemovedAt: 900,
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
moderationVerdict: undefined,
isSuspicious: false,
latestVersionId: undefined,
tags: {},
}),
]),
);
expect(tables.skillVersions ?? []).toHaveLength(0);
});
it("keeps unscanned GitHub-backed skills hidden from public listings", async () => {
const { db, tables } = createDb();
await seedGitHubBackedSkillSourceHandler(
createMutationCtx(db) as never,
{
repo: "NVIDIA/skills",
displayManifestStatus: "ok",
skills: [
{
slug: "pending-github-skill",
displayName: "Pending GitHub Skill",
githubPath: "skills/pending-github-skill",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-pending",
githubScanStatus: "pending",
},
{
slug: "failed-scan-github-skill",
displayName: "Failed Scan GitHub Skill",
githubPath: "skills/failed-scan-github-skill",
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: "hash-failed-scan",
githubScanStatus: "failed",
},
],
} as never,
);
expect(tables.skills).toEqual(
expect.arrayContaining([
expect.objectContaining({
slug: "pending-github-skill",
moderationStatus: "hidden",
moderationReason: "pending.scan",
moderationVerdict: undefined,
isSuspicious: false,
}),
expect.objectContaining({
slug: "failed-scan-github-skill",
moderationStatus: "hidden",
moderationReason: "scanner.failed",
moderationVerdict: undefined,
isSuspicious: false,
}),
]),
);
});
it("seeds moderation and plugin fixtures for an explicit local user with scoped identifiers", async () => {
const { db, tables } = createDb();
const userId = (await db.insert("users", {
@@ -258,6 +501,21 @@ describe("devSeed local fixtures", () => {
scannedPluginName,
]);
expect(tables.packages?.every((pkg) => pkg.ownerUserId === userId)).toBe(true);
expect(tables.packageInspectorWarnings).toEqual(
expect.arrayContaining([
expect.objectContaining({
packageName: scannedPluginName,
findingKind: "warning",
code: "legacy-before-agent-start",
}),
expect.objectContaining({
packageName: scannedPluginName,
findingKind: "error",
code: "missing-expected-seam",
scanSource: "nightly",
}),
]),
);
});
it("retires legacy @local-owner seed publishers so dev-auth users can claim the handle", async () => {
+1069 -21
View File
File diff suppressed because it is too large Load Diff
+249
View File
@@ -0,0 +1,249 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from "vitest";
import {
__test,
pruneDownloadMetricDedupesInternal,
recordDownloadMetricInternal,
} from "./downloadMetrics";
type WrappedHandler<TArgs, TResult> = {
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>;
};
const recordDownloadMetricHandler = (
recordDownloadMetricInternal as unknown as WrappedHandler<
{
target: { kind: "skill"; id: string } | { kind: "package"; id: string };
identityKind: "user" | "ip";
identityHash: string;
dayStart: number;
occurredAt?: number;
},
void
>
)._handler;
const pruneDownloadMetricDedupesHandler = (
pruneDownloadMetricDedupesInternal as unknown as WrappedHandler<
Record<string, never>,
{ deleted: number; hasMore: boolean }
>
)._handler;
function makeQueryBuilder() {
const builder = {
eq: vi.fn(() => builder),
lt: vi.fn(() => builder),
};
return builder;
}
type QueryBuilder = ReturnType<typeof makeQueryBuilder>;
function makeDb(
existingByTable: Record<string, unknown> = {},
rowsByTable: Record<string, Array<{ _id: string }>> = {},
) {
const indexCalls: Array<{ table: string; indexName: string; builder: QueryBuilder }> = [];
const insert = vi.fn();
const unique = vi.fn(async function uniqueForTable(this: { table: string }) {
return existingByTable[this.table] ?? null;
});
const take = vi.fn(async function takeForTable(this: { table: string }, limit: number) {
return (rowsByTable[this.table] ?? []).slice(0, limit);
});
const query = vi.fn((table: string) => ({
withIndex: vi.fn((indexName: string, buildQuery: (q: unknown) => unknown) => {
const builder = makeQueryBuilder();
buildQuery(builder);
indexCalls.push({ table, indexName, builder });
return {
unique: unique.bind({ table }),
take: take.bind({ table }),
};
}),
}));
const delete_ = vi.fn();
return {
db: {
query,
get: vi.fn(),
insert,
patch: vi.fn(),
replace: vi.fn(),
delete: delete_,
normalizeId: vi.fn(),
system: {
get: vi.fn(),
query: vi.fn(),
},
},
insert,
delete_,
take,
indexCalls,
};
}
describe("download metric helpers", () => {
afterEach(() => {
vi.useRealTimers();
});
it("uses a day bucket for download dedupe", () => {
expect(__test.getDayStart(86_400_000 - 1)).toBe(0);
expect(__test.getDayStart(86_400_000)).toBe(86_400_000);
});
it("prefers user identity and falls back to IP identity", () => {
const request = new Request("https://example.com", {
headers: { "cf-connecting-ip": "203.0.113.10" },
});
expect(__test.getDownloadIdentity(request, "users:one")).toEqual({
identityKind: "user",
identityValue: "users:one",
});
expect(__test.getDownloadIdentity(request, null)).toEqual({
identityKind: "ip",
identityValue: "203.0.113.10",
});
});
it("does not create a metering identity when user and IP are missing", () => {
expect(__test.getDownloadIdentity(new Request("https://example.com"), null)).toBeNull();
});
it("records one authenticated skill download and emits the existing skill stat event", async () => {
const { db, insert, indexCalls } = makeDb();
await recordDownloadMetricHandler(
{ db },
{
target: { kind: "skill", id: "skills:one" },
identityKind: "user",
identityHash: "hash-user",
dayStart: 86_400_000,
occurredAt: 86_500_000,
},
);
expect(indexCalls[0]?.table).toBe("downloadMetricDedupes");
expect(indexCalls[0]?.indexName).toBe("by_target_identity_day");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("targetKind", "skill");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("targetId", "skills:one");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("identityKind", "user");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("identityHash", "hash-user");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("dayStart", 86_400_000);
expect(insert).toHaveBeenCalledWith(
"downloadMetricDedupes",
expect.objectContaining({
targetKind: "skill",
targetId: "skills:one",
identityKind: "user",
identityHash: "hash-user",
dayStart: 86_400_000,
}),
);
expect(insert).toHaveBeenCalledWith(
"skillStatEvents",
expect.objectContaining({
skillId: "skills:one",
kind: "download",
occurredAt: 86_500_000,
}),
);
expect(insert).not.toHaveBeenCalledWith("packageStatEvents", expect.anything());
});
it("records one anonymous package download and emits the existing package stat event", async () => {
const { db, insert } = makeDb();
await recordDownloadMetricHandler(
{ db },
{
target: { kind: "package", id: "packages:one" },
identityKind: "ip",
identityHash: "hash-ip",
dayStart: 86_400_000,
occurredAt: 86_500_000,
},
);
expect(insert).toHaveBeenCalledWith(
"downloadMetricDedupes",
expect.objectContaining({
targetKind: "package",
targetId: "packages:one",
identityKind: "ip",
identityHash: "hash-ip",
dayStart: 86_400_000,
}),
);
expect(insert).toHaveBeenCalledWith(
"packageStatEvents",
expect.objectContaining({
packageId: "packages:one",
kind: "download",
occurredAt: 86_500_000,
}),
);
expect(insert).not.toHaveBeenCalledWith("skillStatEvents", expect.anything());
});
it("ignores duplicate identities in the same target/day bucket", async () => {
const { db, insert } = makeDb({
downloadMetricDedupes: { _id: "downloadMetricDedupes:existing" },
});
await recordDownloadMetricHandler(
{ db },
{
target: { kind: "skill", id: "skills:one" },
identityKind: "ip",
identityHash: "hash-ip",
dayStart: 86_400_000,
},
);
expect(insert).not.toHaveBeenCalled();
});
it("prunes stale dedupe rows by day bucket", async () => {
vi.setSystemTime(30 * 86_400_000);
const { db, delete_, take, indexCalls } = makeDb(
{},
{
downloadMetricDedupes: [
{ _id: "downloadMetricDedupes:one" },
{ _id: "downloadMetricDedupes:two" },
],
},
);
const result = await pruneDownloadMetricDedupesHandler({ db }, {});
expect(result).toEqual({ deleted: 2, hasMore: false });
expect(indexCalls[0]?.table).toBe("downloadMetricDedupes");
expect(indexCalls[0]?.indexName).toBe("by_day");
expect(take).toHaveBeenCalledWith(200);
expect(delete_).toHaveBeenCalledWith("downloadMetricDedupes:one");
expect(delete_).toHaveBeenCalledWith("downloadMetricDedupes:two");
});
it("reschedules stale dedupe pruning when one bounded batch fills", async () => {
vi.setSystemTime(30 * 86_400_000);
const rows = Array.from({ length: 200 }, (_, index) => ({
_id: `downloadMetricDedupes:${index}`,
}));
const { db, delete_ } = makeDb({}, { downloadMetricDedupes: rows });
const runAfter = vi.fn();
const result = await pruneDownloadMetricDedupesHandler({ db, scheduler: { runAfter } }, {});
expect(result).toEqual({ deleted: 200, hasMore: true });
expect(delete_).toHaveBeenCalledTimes(200);
expect(runAfter).toHaveBeenCalledWith(0, expect.anything(), {});
});
});
+137
View File
@@ -0,0 +1,137 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import { internalMutation } from "./functions";
import { getClientIp } from "./lib/httpRateLimit";
import { hashToken } from "./lib/tokens";
import { insertStatEvent } from "./skillStatEvents";
const DAY_MS = 86_400_000;
const DEDUPE_RETENTION_MS = 14 * DAY_MS;
const PRUNE_BATCH_SIZE = 200;
const identityKindValidator = v.union(v.literal("user"), v.literal("ip"));
const targetValidator = v.union(
v.object({ kind: v.literal("skill"), id: v.id("skills") }),
v.object({ kind: v.literal("package"), id: v.id("packages") }),
);
type DownloadIdentityKind = "user" | "ip";
type DownloadIdentity = {
identityKind: DownloadIdentityKind;
identityValue: string;
};
export function getDownloadIdentity(
request: Request,
userId: string | null,
): DownloadIdentity | null {
if (userId) return { identityKind: "user", identityValue: userId };
const ip = getClientIp(request);
if (!ip) return null;
return { identityKind: "ip", identityValue: ip };
}
export async function buildDownloadMetricArgs(params: {
target: { kind: "skill"; id: Id<"skills"> } | { kind: "package"; id: Id<"packages"> };
identity: DownloadIdentity;
now: number;
}) {
return {
target: params.target,
identityKind: params.identity.identityKind,
identityHash: await hashToken(
`${params.identity.identityKind}:${params.identity.identityValue}`,
),
dayStart: getDayStart(params.now),
occurredAt: params.now,
};
}
export const recordDownloadMetricInternal = internalMutation({
args: {
target: targetValidator,
identityKind: identityKindValidator,
identityHash: v.string(),
dayStart: v.number(),
occurredAt: v.optional(v.number()),
},
handler: async (ctx, args) => {
const targetId = args.target.id;
const existing = await ctx.db
.query("downloadMetricDedupes")
.withIndex("by_target_identity_day", (q) =>
q
.eq("targetKind", args.target.kind)
.eq("targetId", targetId)
.eq("identityKind", args.identityKind)
.eq("identityHash", args.identityHash)
.eq("dayStart", args.dayStart),
)
.unique();
if (existing) return;
const now = Date.now();
await ctx.db.insert("downloadMetricDedupes", {
targetKind: args.target.kind,
targetId,
identityKind: args.identityKind,
identityHash: args.identityHash,
dayStart: args.dayStart,
createdAt: now,
});
if (args.target.kind === "skill") {
await insertStatEvent(ctx, {
skillId: args.target.id,
kind: "download",
occurredAt: args.occurredAt,
});
return;
}
await ctx.db.insert("packageStatEvents", {
packageId: args.target.id,
kind: "download",
occurredAt: args.occurredAt ?? now,
processedAt: undefined,
});
},
});
export const pruneDownloadMetricDedupesInternal = internalMutation({
args: {},
handler: async (ctx) => {
const cutoffDayStart = getDayStart(Date.now() - DEDUPE_RETENTION_MS);
const stale = await ctx.db
.query("downloadMetricDedupes")
.withIndex("by_day", (q) => q.lt("dayStart", cutoffDayStart))
.take(PRUNE_BATCH_SIZE);
for (const entry of stale) {
await ctx.db.delete(entry._id);
}
const hasMore = stale.length === PRUNE_BATCH_SIZE;
if (hasMore) {
await ctx.scheduler.runAfter(
0,
internal.downloadMetrics.pruneDownloadMetricDedupesInternal,
{},
);
}
return { deleted: stale.length, hasMore };
},
});
function getDayStart(timestamp: number) {
return Math.floor(timestamp / DAY_MS) * DAY_MS;
}
export const __test = {
getDayStart,
getDownloadIdentity,
};
+290 -14
View File
@@ -21,6 +21,19 @@ const okRate = () => ({
resetAt: Date.now() + 60_000,
});
function stubZipResponse() {
class MockResponse {
status: number;
headers: Headers;
constructor(_body?: BodyInit | null, init?: ResponseInit) {
this.status = init?.status ?? 200;
this.headers = new Headers(init?.headers);
}
}
vi.stubGlobal("Response", MockResponse as unknown as typeof Response);
}
describe("downloads helpers", () => {
afterEach(() => {
vi.unstubAllEnvs();
@@ -63,16 +76,7 @@ describe("downloads helpers", () => {
});
it("schedules zip download stats outside the response path", async () => {
class MockResponse {
status: number;
headers: Headers;
constructor(_body?: BodyInit | null, init?: ResponseInit) {
this.status = init?.status ?? 200;
this.headers = new Headers(init?.headers);
}
}
vi.stubGlobal("Response", MockResponse as unknown as typeof Response);
stubZipResponse();
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
@@ -91,6 +95,7 @@ describe("downloads helpers", () => {
if ("versionId" in args) {
return {
_id: "skillVersions:1",
skillId: "skills:1",
version: "1.0.0",
createdAt: 3,
files: [{ path: "SKILL.md", storageId: "_storage:1" }],
@@ -126,9 +131,10 @@ describe("downloads helpers", () => {
if (!args || typeof args !== "object") return false;
const value = args as Record<string, unknown>;
return (
value.skillId === "skills:1" &&
typeof value.target === "object" &&
typeof value.identityHash === "string" &&
typeof value.hourStart === "number"
value.identityKind === "ip" &&
typeof value.dayStart === "number"
);
});
expect(recordCalls).toHaveLength(1);
@@ -136,9 +142,279 @@ describe("downloads helpers", () => {
expect(recordCalls[0]?.[0]).toBeGreaterThanOrEqual(0);
expect(recordCalls[0]?.[0]).toBeLessThan(60_000);
expect(recordCalls[0]?.[2]).toEqual({
skillId: "skills:1",
target: { kind: "skill", id: "skills:1" },
identityKind: "ip",
identityHash: expect.any(String),
hourStart: expect.any(Number),
dayStart: expect.any(Number),
occurredAt: expect.any(Number),
});
});
it("does not serve a tag that points at another skill's version", async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
if ("slug" in args) {
return {
skill: {
_id: "skills:1",
ownerUserId: "users:1",
slug: "demo",
tags: { old: "skillVersions:other" },
latestVersionId: "skillVersions:1",
},
moderationInfo: null,
};
}
if (args.versionId === "skillVersions:1") {
return {
_id: "skillVersions:1",
skillId: "skills:1",
version: "1.0.0",
createdAt: 3,
files: [],
softDeletedAt: undefined,
};
}
if (args.versionId === "skillVersions:other") {
return {
_id: "skillVersions:other",
skillId: "skills:other",
version: "9.9.9",
createdAt: 4,
files: [{ path: "SKILL.md", storageId: "_storage:other" }],
softDeletedAt: undefined,
};
}
return null;
});
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
return null;
});
const storageGet = vi.fn();
const response = await downloadZipHandler(
{
runQuery,
runMutation,
scheduler: { runAfter: vi.fn() },
storage: { get: storageGet },
} as unknown as ActionCtx,
new Request("https://example.com/api/v1/download?slug=demo&tag=old", {
headers: { "cf-connecting-ip": "1.2.3.4" },
}),
);
expect(response.status).toBe(404);
expect(await response.text()).toBe("Version not found");
expect(storageGet).not.toHaveBeenCalled();
});
it("blocks explicit downloads of a malicious historical version even when the skill is staff-cleared", async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
if ("slug" in args) {
return {
skill: {
_id: "skills:1",
ownerUserId: "users:1",
slug: "demo",
tags: {},
latestVersionId: "skillVersions:2",
},
moderationInfo: {
isMalwareBlocked: false,
isPendingScan: false,
isHiddenByMod: false,
isRemoved: false,
overrideActive: true,
verdict: "clean",
},
};
}
if ("skillId" in args && "version" in args) {
return {
_id: "skillVersions:1",
skillId: "skills:1",
version: "1.0.0",
createdAt: 3,
files: [{ path: "SKILL.md", storageId: "_storage:bad" }],
softDeletedAt: undefined,
llmAnalysis: {
status: "completed",
verdict: "malicious",
checkedAt: 4,
},
};
}
if (args.versionId === "skillVersions:2") {
return {
_id: "skillVersions:2",
skillId: "skills:1",
version: "1.0.1",
createdAt: 5,
files: [],
softDeletedAt: undefined,
llmAnalysis: {
status: "completed",
verdict: "clean",
checkedAt: 6,
},
};
}
return null;
});
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
return null;
});
const storageGet = vi.fn();
const response = await downloadZipHandler(
{
runQuery,
runMutation,
scheduler: { runAfter: vi.fn() },
storage: { get: storageGet },
} as unknown as ActionCtx,
new Request("https://example.com/api/v1/download?slug=demo&version=1.0.0", {
headers: { "cf-connecting-ip": "1.2.3.4" },
}),
);
expect(response.status).toBe(403);
expect(await response.text()).toBe(
"Blocked: this skill version has been flagged as malicious by ClawScan and cannot be downloaded.",
);
expect(storageGet).not.toHaveBeenCalled();
});
it("uses API token user identity for zip download stats when present", async () => {
stubZipResponse();
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
if ("tokenHash" in args) {
return { _id: "apiTokens:1", revokedAt: undefined };
}
if ("tokenId" in args) {
return { _id: "users:token", deletedAt: undefined, deactivatedAt: undefined };
}
if ("slug" in args) {
return {
skill: {
_id: "skills:1",
ownerUserId: "users:1",
slug: "demo",
tags: {},
latestVersionId: "skillVersions:1",
},
moderationInfo: null,
};
}
if ("versionId" in args) {
return {
_id: "skillVersions:1",
skillId: "skills:1",
version: "1.0.0",
createdAt: 3,
files: [{ path: "SKILL.md", storageId: "_storage:1" }],
softDeletedAt: undefined,
};
}
return null;
});
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
return { tokenTouched: "tokenId" in args };
});
const runAfter = vi.fn();
const storageGet = vi.fn().mockResolvedValue(new Blob(["hello"], { type: "text/markdown" }));
const response = await downloadZipHandler(
{
runQuery,
runMutation,
scheduler: { runAfter },
storage: { get: storageGet },
} as unknown as ActionCtx,
new Request("https://example.com/api/v1/download?slug=demo", {
headers: {
authorization: "Bearer clh_test",
"cf-connecting-ip": "1.2.3.4",
},
}),
);
expect(response.status).toBe(200);
expect(runAfter).toHaveBeenCalledWith(
expect.any(Number),
expect.anything(),
expect.objectContaining({
target: { kind: "skill", id: "skills:1" },
identityKind: "user",
identityHash: expect.stringMatching(/^[a-f0-9]{64}$/),
}),
);
});
it("returns zip downloads when download metering is scheduled", async () => {
stubZipResponse();
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
if ("slug" in args) {
return {
skill: {
_id: "skills:1",
ownerUserId: "users:1",
slug: "demo",
tags: {},
latestVersionId: "skillVersions:1",
},
moderationInfo: null,
};
}
if ("versionId" in args) {
return {
_id: "skillVersions:1",
skillId: "skills:1",
version: "1.0.0",
createdAt: 3,
files: [{ path: "SKILL.md", storageId: "_storage:1" }],
softDeletedAt: undefined,
};
}
return null;
});
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
return { mutationRecorded: true };
});
const runAfter = vi.fn();
const storageGet = vi.fn().mockResolvedValue(new Blob(["hello"], { type: "text/markdown" }));
const response = await downloadZipHandler(
{
runQuery,
runMutation,
scheduler: { runAfter },
storage: { get: storageGet },
} as unknown as ActionCtx,
new Request("https://example.com/api/v1/download?slug=demo", {
headers: { "cf-connecting-ip": "1.2.3.4" },
}),
);
expect(response.status).toBe(200);
expect(runAfter).toHaveBeenCalledWith(
expect.any(Number),
expect.anything(),
expect.objectContaining({
target: { kind: "skill", id: "skills:1" },
identityKind: "ip",
identityHash: expect.stringMatching(/^[a-f0-9]{64}$/),
}),
);
});
});
+30 -46
View File
@@ -1,11 +1,14 @@
import { v } from "convex/values";
import { api, internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import { buildDownloadMetricArgs, getDownloadIdentity } from "./downloadMetrics";
import { httpAction, internalMutation } from "./functions";
import { getOptionalActiveAuthUserIdFromAction } from "./lib/access";
import { getOptionalApiTokenUserId } from "./lib/apiTokenAuth";
import { corsHeaders, mergeHeaders } from "./lib/httpHeaders";
import { applyRateLimit, getClientIp } from "./lib/httpRateLimit";
import { getPublicSkillVersionDownloadBlock, isSkillVersionForSkill } from "./lib/skillFileAccess";
import { buildDeterministicZip } from "./lib/skillZip";
import { hashToken } from "./lib/tokens";
import { insertStatEvent } from "./skillStatEvents";
const HOUR_MS = 3_600_000;
@@ -41,39 +44,6 @@ export async function downloadZipHandler(
});
}
// Block downloads based on moderation status.
const mod = skillResult.moderationInfo;
if (mod?.isMalwareBlocked) {
return new Response(
"Blocked: this skill has been flagged as malicious by ClawScan and cannot be downloaded.",
{
status: 403,
headers: mergeHeaders(rate.headers, corsHeaders()),
},
);
}
if (mod?.isPendingScan) {
return new Response(
"This skill is pending a ClawScan security review. Please try again in a few minutes.",
{
status: 423,
headers: mergeHeaders(rate.headers, corsHeaders()),
},
);
}
if (mod?.isRemoved) {
return new Response("This skill has been removed by a moderator.", {
status: 410,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
if (mod?.isHiddenByMod) {
return new Response("This skill is currently unavailable.", {
status: 403,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
const skill = skillResult.skill;
let version = skill.latestVersionId
? await ctx.runQuery(internal.skills.getVersionByIdInternal, {
@@ -93,15 +63,20 @@ export async function downloadZipHandler(
}
}
if (!version) {
if (!version || !isSkillVersionForSkill(version, skill._id)) {
return new Response("Version not found", {
status: 404,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
if (version.softDeletedAt) {
return new Response("Version not available", {
status: 410,
const moderationBlock = getPublicSkillVersionDownloadBlock(
skillResult.moderationInfo,
version,
skill.latestVersionId ?? skill.tags.latest,
);
if (moderationBlock) {
return new Response(moderationBlock.message, {
status: moderationBlock.status,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
@@ -122,17 +97,17 @@ export async function downloadZipHandler(
const zipBlob = new Blob([zipArray], { type: "application/zip" });
try {
const userId = await getOptionalApiTokenUserId(ctx, request);
const identity = getDownloadIdentityValue(request, userId ? String(userId) : null);
const userId = await getOptionalDownloadUserId(ctx, request);
const identity = getDownloadIdentity(request, userId ? String(userId) : null);
if (identity) {
await ctx.scheduler.runAfter(
Math.floor(Math.random() * DOWNLOAD_STAT_JITTER_MS),
internal.downloads.recordDownloadInternal,
{
skillId: skill._id,
identityHash: await hashToken(identity),
hourStart: getHourStart(Date.now()),
},
internal.downloadMetrics.recordDownloadMetricInternal,
await buildDownloadMetricArgs({
target: { kind: "skill", id: skill._id },
identity,
now: Date.now(),
}),
);
}
} catch {
@@ -220,6 +195,15 @@ export function getDownloadIdentityValue(request: Request, userId: string | null
return `ip:${ip}`;
}
async function getOptionalDownloadUserId(
ctx: Parameters<Parameters<typeof httpAction>[0]>[0],
request: Request,
): Promise<Id<"users"> | null> {
const apiTokenUserId = await getOptionalApiTokenUserId(ctx, request);
if (apiTokenUserId) return apiTokenUserId;
return (await getOptionalActiveAuthUserIdFromAction(ctx)) ?? null;
}
export const __test = {
getHourStart,
getDownloadIdentityValue,
+168
View File
@@ -0,0 +1,168 @@
"use node";
import { mkdir, appendFile } from "node:fs/promises";
import { dirname } from "node:path";
import { v } from "convex/values";
import { Resend } from "resend";
import { internalAction } from "./functions";
import {
buildBanNotificationEmail,
buildMaliciousArtifactEmail,
buildRestoredAccountEmail,
type NotificationArtifact,
} from "./lib/emails";
const DEFAULT_FROM = "ClawHub Security <noreply@notifications.openclaw.ai>";
const DEFAULT_REPLY_TO = "security@notifications.openclaw.ai";
const notificationArtifactValidator = v.object({
kind: v.union(v.literal("skill"), v.literal("plugin")),
name: v.string(),
});
type SendEmailArgs = {
idempotencyKey: string;
to: string;
subject: string;
text: string;
html: string;
};
function getEmailConfig() {
return {
apiKey: process.env.RESEND_API_KEY,
from: process.env.CLAWHUB_SECURITY_EMAIL_FROM || DEFAULT_FROM,
replyTo: process.env.CLAWHUB_SECURITY_EMAIL || DEFAULT_REPLY_TO,
};
}
async function sendTransactionalEmail(args: SendEmailArgs) {
const captureFile = process.env.CLAWHUB_EMAIL_CAPTURE_FILE?.trim();
if (captureFile) {
await mkdir(dirname(captureFile), { recursive: true });
await appendFile(
captureFile,
`${JSON.stringify({ ...args, capturedAt: Date.now() })}\n`,
"utf8",
);
return { ok: true as const, id: "local-capture" };
}
const config = getEmailConfig();
if (!config.apiKey) {
console.warn(`[emails] RESEND_API_KEY is not configured; skipped ${args.idempotencyKey}`);
return { ok: false as const, reason: "missing_api_key" as const };
}
try {
const resend = new Resend(config.apiKey);
const result = await resend.emails.send(
{
from: config.from,
to: args.to,
replyTo: config.replyTo,
subject: args.subject,
text: args.text,
html: args.html,
},
{ idempotencyKey: args.idempotencyKey },
);
if (result.error) {
console.error("[emails] Resend error", result.error);
return { ok: false as const, reason: "resend_error" as const };
}
return { ok: true as const, id: result.data?.id ?? null };
} catch (error) {
console.error("[emails] Send failed", error);
return { ok: false as const, reason: "send_error" as const };
}
}
export const sendBanNotificationInternal = internalAction({
args: {
userId: v.id("users"),
bannedAt: v.number(),
to: v.string(),
handle: v.optional(v.string()),
source: v.union(v.literal("manual"), v.literal("autoban")),
reason: v.optional(v.string()),
trigger: v.optional(v.string()),
artifact: v.optional(notificationArtifactValidator),
hiddenArtifacts: v.optional(v.number()),
},
handler: async (_ctx, args) => {
const email = await buildBanNotificationEmail({
handle: args.handle,
source: args.source,
reason: args.reason,
trigger: args.trigger,
artifact: args.artifact as NotificationArtifact | undefined,
bannedAt: args.bannedAt,
hiddenArtifacts: args.hiddenArtifacts,
});
return await sendTransactionalEmail({
idempotencyKey: `ban:${args.userId}:${args.bannedAt}`,
to: args.to,
subject: email.subject,
text: email.text,
html: email.html,
});
},
});
export const sendRestoredAccountNotificationInternal = internalAction({
args: {
userId: v.id("users"),
restoredAt: v.number(),
to: v.string(),
handle: v.optional(v.string()),
restoredListings: v.optional(v.array(notificationArtifactValidator)),
skillsRestored: v.optional(v.number()),
packagesRestored: v.optional(v.number()),
},
handler: async (_ctx, args) => {
const email = await buildRestoredAccountEmail({
handle: args.handle,
restoredListings: args.restoredListings as NotificationArtifact[] | undefined,
restoredAt: args.restoredAt,
skillsRestored: args.skillsRestored,
packagesRestored: args.packagesRestored,
});
return await sendTransactionalEmail({
idempotencyKey: `account-restored:${args.userId}:${args.restoredAt}`,
to: args.to,
subject: email.subject,
text: email.text,
html: email.html,
});
},
});
export const sendMaliciousArtifactNotificationInternal = internalAction({
args: {
userId: v.id("users"),
findingAt: v.number(),
to: v.string(),
handle: v.optional(v.string()),
artifact: notificationArtifactValidator,
version: v.optional(v.string()),
trigger: v.optional(v.string()),
findingSummary: v.optional(v.string()),
},
handler: async (_ctx, args) => {
const email = await buildMaliciousArtifactEmail({
handle: args.handle,
artifact: args.artifact as NotificationArtifact,
version: args.version,
trigger: args.trigger,
findingSummary: args.findingSummary,
});
return await sendTransactionalEmail({
idempotencyKey: `malicious-artifact:${args.userId}:${args.findingAt}:${args.artifact.kind}:${args.artifact.name}:${args.version ?? ""}`,
to: args.to,
subject: email.subject,
text: email.text,
html: email.html,
});
},
});
+289
View File
@@ -2,16 +2,50 @@
import { describe, expect, it, vi } from "vitest";
import { internal } from "./_generated/api";
import type { Doc, Id, TableNames } from "./_generated/dataModel";
import {
internalMutation,
isGitHubMirrorEligibleSkillDoc,
repointPackageLatestRelease,
scheduleGitHubBackupDeletionForSkill,
scheduleOwnerPublisherDigestSync,
shouldScheduleOwnerPublisherDigestSyncForPublisherChange,
shouldScheduleOwnerUserPackageDigestSyncForUserChange,
syncPackageSearchDigestForPackageId,
syncPackageSearchDigestsForOwnerPublisherId,
syncPackageSearchDigestsForOwnerUserId,
syncSkillSearchDigestsForOwnerPublisherId,
} from "./functions";
import { computeRecommendationScore } from "./lib/recommendationScore";
type WrappedHandler = {
_handler: (ctx: unknown, args: Record<string, never>) => Promise<unknown>;
};
function hasWrappedHandler(value: unknown): value is WrappedHandler {
return typeof value === "function" && "_handler" in value && typeof value._handler === "function";
}
function getWrappedHandler(value: unknown): WrappedHandler["_handler"] {
if (!hasWrappedHandler(value)) {
throw new Error("Expected a Convex function with a test-callable _handler");
}
return value._handler;
}
function testId<TableName extends TableNames>(
tableName: TableName,
value: `${TableName}:${string}`,
): Id<TableName> {
if (!value.startsWith(`${tableName}:`)) {
throw new Error(`Expected ${value} to be a ${tableName} id`);
}
return value as Id<TableName>;
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
describe("package digest sync", () => {
it("identifies GitHub mirror eligibility from skill visibility fields", () => {
@@ -22,6 +56,20 @@ describe("package digest sync", () => {
moderationStatus: "active",
}),
).toBe(true);
expect(
isGitHubMirrorEligibleSkillDoc({
softDeletedAt: undefined,
moderationStatus: "active",
moderationVerdict: "malicious",
}),
).toBe(false);
expect(
isGitHubMirrorEligibleSkillDoc({
softDeletedAt: undefined,
moderationStatus: "active",
moderationFlags: ["blocked.malware"],
}),
).toBe(false);
expect(
isGitHubMirrorEligibleSkillDoc({
softDeletedAt: undefined,
@@ -583,7 +631,92 @@ describe("package digest sync", () => {
});
});
describe("user package digest scheduling", () => {
const user = {
_id: "users:owner",
handle: "owner",
deletedAt: undefined,
deactivatedAt: undefined,
};
it("schedules package digest sync when an active user's handle changes", () => {
expect(
shouldScheduleOwnerUserPackageDigestSyncForUserChange({
id: "users:owner",
operation: "update",
oldDoc: user,
newDoc: { ...user, handle: "renamed" },
} as never),
).toBe(true);
});
it("skips redundant package digest sync when a user becomes deactivated", () => {
expect(
shouldScheduleOwnerUserPackageDigestSyncForUserChange({
id: "users:owner",
operation: "update",
oldDoc: user,
newDoc: { ...user, handle: null, deactivatedAt: 1_700_000_000_000 },
} as never),
).toBe(false);
});
it("skips unchanged user updates", () => {
expect(
shouldScheduleOwnerUserPackageDigestSyncForUserChange({
id: "users:owner",
operation: "update",
oldDoc: user,
newDoc: { ...user },
} as never),
).toBe(false);
});
});
describe("publisher digest scheduling", () => {
const publisherChangeDoc = {
_id: "publishers:demo",
kind: "org",
handle: "demo",
displayName: "Demo",
image: null,
deletedAt: undefined,
deactivatedAt: undefined,
};
it("schedules digest sync when an active publisher's profile changes", () => {
expect(
shouldScheduleOwnerPublisherDigestSyncForPublisherChange({
id: "publishers:demo",
operation: "update",
oldDoc: publisherChangeDoc,
newDoc: { ...publisherChangeDoc, displayName: "Renamed Demo" },
} as never),
).toBe(true);
});
it("skips redundant digest sync when a publisher becomes deactivated", () => {
expect(
shouldScheduleOwnerPublisherDigestSyncForPublisherChange({
id: "publishers:demo",
operation: "update",
oldDoc: publisherChangeDoc,
newDoc: { ...publisherChangeDoc, handle: null, deactivatedAt: 1_700_000_000_000 },
} as never),
).toBe(false);
});
it("skips unchanged publisher updates", () => {
expect(
shouldScheduleOwnerPublisherDigestSyncForPublisherChange({
id: "publishers:demo",
operation: "update",
oldDoc: publisherChangeDoc,
newDoc: { ...publisherChangeDoc },
} as never),
).toBe(false);
});
it("schedules package and skill digest sync in separate background mutations", async () => {
const ctx = {
scheduler: {
@@ -677,4 +810,160 @@ describe("publisher digest scheduling", () => {
{ ownerPublisherId: "publishers:demo", cursor: "next-skills" },
);
});
it("syncs recommended rank stats into the skill search digest after wrapped skill patches", async () => {
const skillId = testId("skills", "skills:demo");
const ownerUserId = testId("users", "users:owner");
const publisherId = testId("publishers", "publishers:owner");
const digestId = testId("skillSearchDigest", "skillSearchDigest:demo");
const skill = {
_id: skillId,
_creationTime: 1,
slug: "demo-skill",
displayName: "Demo Skill",
summary: "Demo summary",
ownerUserId,
ownerPublisherId: publisherId,
tags: {},
statsDownloads: 3,
statsStars: 2,
statsInstallsCurrent: 4,
statsInstallsAllTime: 5,
stats: {
downloads: 3,
stars: 2,
installsCurrent: 4,
installsAllTime: 5,
versions: 1,
comments: 0,
},
createdAt: 10,
updatedAt: 20,
} satisfies Doc<"skills">;
const publisher = {
_id: publisherId,
_creationTime: 2,
kind: "user",
handle: "owner",
displayName: "Owner",
linkedUserId: ownerUserId,
publishedSkills: 1,
publishedPackages: 0,
totalInstalls: 5,
totalDownloads: 3,
totalStars: 2,
skillTotalInstalls: 5,
skillTotalDownloads: 3,
skillTotalStars: 2,
createdAt: 10,
updatedAt: 20,
} satisfies Doc<"publishers">;
const digest = {
_id: digestId,
_creationTime: 3,
skillId,
slug: "demo-skill",
displayName: "Demo Skill",
summary: "Demo summary",
ownerUserId,
ownerPublisherId: publisherId,
ownerHandle: "owner",
ownerKind: "user",
ownerDisplayName: "Owner",
tags: {},
statsDownloads: 3,
statsStars: 2,
statsInstallsCurrent: 4,
statsInstallsAllTime: 5,
stats: {
downloads: 3,
stars: 2,
installsCurrent: 4,
installsAllTime: 5,
versions: 1,
comments: 0,
},
createdAt: 10,
updatedAt: 20,
} satisfies Doc<"skillSearchDigest">;
const docs = new Map<string, unknown>([
[skillId, skill],
[publisherId, publisher],
[digestId, digest],
]);
const patchSkillRankStats = internalMutation({
args: {},
handler: async (ctx) => {
await ctx.db.patch(skillId, {
statsDownloads: 13,
statsStars: 7,
statsInstallsAllTime: 11,
stats: {
downloads: 13,
stars: 7,
installsCurrent: 4,
installsAllTime: 11,
versions: 1,
comments: 0,
},
});
},
});
const handler = getWrappedHandler(patchSkillRankStats);
const db = {
system: {},
normalizeId: vi.fn((tableName: string, id: string) =>
id.startsWith(`${tableName}:`) ? id : null,
),
get: vi.fn(async (first: string, second?: string) => docs.get(second ?? first) ?? null),
insert: vi.fn(async (tableName: string, value: unknown) => {
if (!isRecord(value))
throw new Error(`Expected inserted ${tableName} value to be an object`);
const insertedId = `${tableName}:inserted`;
docs.set(insertedId, { ...value, _id: insertedId, _creationTime: 0 });
return insertedId;
}),
patch: vi.fn(
async (first: string, second: string | Record<string, unknown>, third?: unknown) => {
const id = typeof second === "string" ? second : first;
const patch = typeof second === "string" ? third : second;
if (!isRecord(patch)) throw new Error(`Expected patch for ${id} to be an object`);
const existing = docs.get(id);
if (!isRecord(existing)) throw new Error(`Missing test doc ${id}`);
docs.set(id, { ...existing, ...patch });
},
),
delete: vi.fn(async (first: string, second?: string) => {
docs.delete(second ?? first);
}),
query: vi.fn((tableName: string) => ({
withIndex: vi.fn(() => ({
unique: vi.fn(async () => {
if (tableName === "skillSearchDigest") return docs.get(digestId) ?? null;
return null;
}),
collect: vi.fn(async () => []),
paginate: vi.fn(async () => ({ page: [], isDone: true, continueCursor: "" })),
take: vi.fn(async () => []),
})),
})),
};
await expect(handler({ db }, {})).resolves.toBeUndefined();
expect(docs.get(digestId)).toEqual(
expect.objectContaining({
statsDownloads: 13,
statsStars: 7,
statsInstallsAllTime: 11,
recommendedScore: computeRecommendationScore({ downloads: 13, installs: 11, stars: 7 }),
stats: expect.objectContaining({
downloads: 13,
stars: 7,
installsAllTime: 11,
}),
}),
);
});
});
+104 -43
View File
@@ -1,5 +1,5 @@
import { customCtx, customMutation } from "convex-helpers/server/customFunctions";
import { Triggers } from "convex-helpers/server/triggers";
import { Triggers, type Change } from "convex-helpers/server/triggers";
import { v } from "convex/values";
import semver from "semver";
import { internal } from "./_generated/api";
@@ -14,6 +14,7 @@ import {
httpAction,
} from "./_generated/server";
import type { MutationCtx } from "./_generated/server";
import { isPublicSkillDoc } from "./lib/globalStats";
import {
deletePackageSearchDigests,
extractPackageDigestFields,
@@ -24,7 +25,7 @@ import {
adjustPublisherStatsForPackageChange,
adjustPublisherStatsForSkillChange,
} from "./lib/publisherStats";
import { extractDigestFields, upsertSkillSearchDigest } from "./lib/skillSearchDigest";
import { extractValidatedDigestFields, upsertSkillSearchDigest } from "./lib/skillSearchDigest";
const triggers = new Triggers<DataModel>();
@@ -152,22 +153,28 @@ export async function syncPackageSearchDigestForPackageId(
}
export async function syncPackageSearchDigestsForOwnerUserId(
ctx: PackageDigestSyncCtx,
ctx: PackageDigestSyncCtx & OwnerPublisherDigestScheduleCtx,
ownerUserId: Id<"users"> | null | undefined,
cursor: string | null = null,
) {
if (!ownerUserId) return;
let cursor: string | null = null;
try {
while (true) {
const page = await ctx.db
.query("packages")
.withIndex("by_owner", (q) => q.eq("ownerUserId", ownerUserId))
.paginate({ cursor, numItems: 100 });
for (const pkg of page.page) {
await syncPackageSearchDigest(ctx, pkg);
}
if (page.isDone) break;
cursor = page.continueCursor;
const page = await ctx.db
.query("packages")
.withIndex("by_owner", (q) => q.eq("ownerUserId", ownerUserId))
.paginate({ cursor, numItems: OWNER_PUBLISHER_DIGEST_PAGE_SIZE });
for (const pkg of page.page) {
await syncPackageSearchDigest(ctx, pkg);
}
if (!page.isDone && ctx.scheduler && page.continueCursor) {
await ctx.scheduler.runAfter(
0,
internal.functions.syncPackageSearchDigestsForOwnerUserIdInternal,
{
ownerUserId,
cursor: page.continueCursor,
},
);
}
} catch (error) {
if (isMissingTableError(error, "packages")) return;
@@ -207,7 +214,7 @@ async function syncSkillSearchDigestForSkill(
skill: Doc<"skills"> | null | undefined,
) {
if (!skill) return;
const fields = extractDigestFields(skill);
const fields = await extractValidatedDigestFields(ctx, skill);
const owner = await getOwnerPublisher(ctx, {
ownerPublisherId: skill.ownerPublisherId,
ownerUserId: skill.ownerUserId,
@@ -223,21 +230,28 @@ async function syncSkillSearchDigestForSkill(
}
export function isGitHubMirrorEligibleSkillDoc(
skill: Pick<Doc<"skills">, "softDeletedAt" | "moderationStatus"> | null | undefined,
skill:
| Pick<
Doc<"skills">,
"softDeletedAt" | "moderationStatus" | "moderationFlags" | "moderationVerdict"
>
| null
| undefined,
) {
if (!skill || skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
return isPublicSkillDoc(skill);
}
export async function scheduleGitHubBackupDeletionForSkill(
ctx: GitHubBackupDeletionCtx,
skill: Pick<
Doc<"skills">,
"slug" | "ownerPublisherId" | "ownerUserId" | "softDeletedAt" | "moderationStatus"
| "slug"
| "ownerPublisherId"
| "ownerUserId"
| "softDeletedAt"
| "moderationStatus"
| "moderationFlags"
| "moderationVerdict"
>,
) {
const owner = await getOwnerPublisher(ctx, {
@@ -295,6 +309,69 @@ export async function scheduleOwnerPublisherDigestSync(
);
}
export async function scheduleOwnerUserPackageDigestSync(
ctx: OwnerPublisherDigestScheduleCtx,
ownerUserId: Id<"users"> | null | undefined,
) {
if (!ownerUserId || !ctx.scheduler) return;
await ctx.scheduler.runAfter(
0,
internal.functions.syncPackageSearchDigestsForOwnerUserIdInternal,
{
ownerUserId,
},
);
}
export function shouldScheduleOwnerUserPackageDigestSyncForUserChange(
change: Change<DataModel, "users">,
) {
if (change.operation === "delete") return true;
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return false;
}
if (change.operation === "update" && (change.newDoc.deletedAt || change.newDoc.deactivatedAt)) {
return false;
}
return true;
}
export function shouldScheduleOwnerPublisherDigestSyncForPublisherChange(
change: Change<DataModel, "publishers">,
) {
if (change.operation === "delete") return true;
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.kind === change.newDoc.kind &&
change.oldDoc.displayName === change.newDoc.displayName &&
change.oldDoc.image === change.newDoc.image &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return false;
}
if (change.operation === "update" && (change.newDoc.deletedAt || change.newDoc.deactivatedAt)) {
return false;
}
return true;
}
export const syncPackageSearchDigestsForOwnerUserIdInternal = rawInternalMutation({
args: {
ownerUserId: v.id("users"),
cursor: v.optional(v.union(v.string(), v.null())),
},
handler: async (ctx, args) => {
await syncPackageSearchDigestsForOwnerUserId(ctx, args.ownerUserId, args.cursor ?? null);
},
});
export const syncPackageSearchDigestsForOwnerPublisherIdInternal = rawInternalMutation({
args: {
ownerPublisherId: v.id("publishers"),
@@ -331,6 +408,7 @@ export async function repointPackageLatestRelease(
if (!packageId || !affectedReleaseId) return;
const pkg = await ctx.db.get(packageId);
if (!pkg) return;
if (pkg.softDeletedAt) return;
const nextTags = Object.fromEntries(
Object.entries(pkg.tags).filter(([, releaseId]) => releaseId !== affectedReleaseId),
@@ -431,30 +509,13 @@ triggers.register("packageReleases", async (ctx, change) => {
});
triggers.register("users", async (ctx, change) => {
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return;
}
if (!shouldScheduleOwnerUserPackageDigestSyncForUserChange(change)) return;
const ownerUserId = change.operation === "delete" ? change.id : change.newDoc._id;
await syncPackageSearchDigestsForOwnerUserId(ctx, ownerUserId);
await scheduleOwnerUserPackageDigestSync(ctx, ownerUserId);
});
triggers.register("publishers", async (ctx, change) => {
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.kind === change.newDoc.kind &&
change.oldDoc.displayName === change.newDoc.displayName &&
change.oldDoc.image === change.newDoc.image &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return;
}
if (!shouldScheduleOwnerPublisherDigestSyncForPublisherChange(change)) return;
const ownerPublisherId = change.operation === "delete" ? change.id : change.newDoc._id;
await scheduleOwnerPublisherDigestSync(ctx, ownerPublisherId);
});
+278
View File
@@ -0,0 +1,278 @@
import { ConvexError, v } from "convex/values";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { internalAction, internalMutation, internalQuery } from "./functions";
import { fetchGitHubCreatedAtByProviderAccountId } from "./lib/githubAccount";
import { getGitHubProviderAccountId } from "./lib/githubIdentity";
import { getUserByHandleOrPersonalPublisher } from "./lib/publishers";
const DEFAULT_BATCH_SIZE = 25;
const MAX_ACTION_BATCH_SIZE = 50;
const MAX_LIST_BATCH_SIZE = 500;
const DEFAULT_MAX_PAGES = 1;
const MAX_MAX_PAGES = 20;
type BackfillCandidate = {
userId: Id<"users">;
providerAccountId: string;
handle: string | null;
};
type BackfillStats = {
scanned: number;
candidates: number;
fetched: number;
patched: number;
failed: number;
missingHandles: string[];
errors: Array<{ userId: string; handle: string | null; message: string }>;
};
type BackfillPageResult = {
candidates: BackfillCandidate[];
scanned: number;
cursor: string | null;
isDone: boolean;
};
type BackfillHandlesResult = {
candidates: BackfillCandidate[];
missingHandles: string[];
};
type BackfillResult =
| { ok: true; stats: BackfillStats; cursor: string | null; isDone: boolean }
| { ok: false; rateLimited: true; stats: BackfillStats; cursor: string | null; isDone: false };
function clampPositiveInteger(value: number | undefined, fallback: number, max: number) {
if (!value || !Number.isFinite(value)) return fallback;
return Math.max(1, Math.min(max, Math.floor(value)));
}
async function candidateForUser(
ctx: Parameters<typeof getGitHubProviderAccountId>[0],
userId: Id<"users">,
): Promise<BackfillCandidate | null> {
const user = await ctx.db.get(userId);
if (!user || user.deletedAt || user.deactivatedAt || user.githubCreatedAt) return null;
const providerAccountId = await getGitHubProviderAccountId(ctx, userId);
if (!providerAccountId || !/^\d+$/.test(providerAccountId)) return null;
return { userId, providerAccountId, handle: user.handle ?? null };
}
export const listGitHubCreatedAtBackfillPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampPositiveInteger(args.batchSize, DEFAULT_BATCH_SIZE, MAX_LIST_BATCH_SIZE);
const page = await ctx.db
.query("authAccounts")
.withIndex("providerAndAccountId", (q) => q.eq("provider", "github"))
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
const candidates: BackfillCandidate[] = [];
for (const account of page.page) {
if (!/^\d+$/.test(account.providerAccountId)) continue;
const user = await ctx.db.get(account.userId);
if (!user || user.deletedAt || user.deactivatedAt || user.githubCreatedAt) continue;
candidates.push({
userId: account.userId,
providerAccountId: account.providerAccountId,
handle: user.handle ?? null,
});
}
return {
candidates,
scanned: page.page.length,
cursor: page.continueCursor,
isDone: page.isDone,
};
},
});
export const listGitHubCreatedAtBackfillHandlesInternal = internalQuery({
args: { handles: v.array(v.string()) },
handler: async (ctx, args) => {
const seen = new Set<string>();
const candidates: BackfillCandidate[] = [];
const missingHandles: string[] = [];
for (const handle of args.handles) {
const user = await getUserByHandleOrPersonalPublisher(ctx, handle);
if (!user) {
missingHandles.push(handle);
continue;
}
if (seen.has(user._id)) continue;
seen.add(user._id);
const candidate = await candidateForUser(ctx, user._id);
if (candidate) candidates.push(candidate);
}
return { candidates, missingHandles };
},
});
export const applyGitHubCreatedAtBackfillInternal = internalMutation({
args: {
userId: v.id("users"),
githubCreatedAt: v.number(),
fetchedAt: v.number(),
dryRun: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const user = await ctx.db.get(args.userId);
if (!user || user.deletedAt || user.deactivatedAt || user.githubCreatedAt) {
return { patched: false };
}
if (args.dryRun) return { patched: false };
await ctx.db.patch(args.userId, {
githubCreatedAt: args.githubCreatedAt,
githubFetchedAt: args.fetchedAt,
updatedAt: Date.now(),
});
return { patched: true };
},
});
export const applyGitHubCreatedAtBackfillBatchInternal = internalMutation({
args: {
items: v.array(
v.object({
userId: v.id("users"),
githubCreatedAt: v.number(),
}),
),
fetchedAt: v.number(),
dryRun: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
let patched = 0;
let skipped = 0;
for (const item of args.items) {
const user = await ctx.db.get(item.userId);
if (!user || user.deletedAt || user.deactivatedAt || user.githubCreatedAt) {
skipped += 1;
continue;
}
if (!args.dryRun) {
await ctx.db.patch(item.userId, {
githubCreatedAt: item.githubCreatedAt,
githubFetchedAt: args.fetchedAt,
updatedAt: Date.now(),
});
}
patched += 1;
}
return { patched, skipped };
},
});
export const backfillGitHubCreatedAtInternal = internalAction({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
maxPages: v.optional(v.number()),
dryRun: v.optional(v.boolean()),
handles: v.optional(v.array(v.string())),
},
handler: async (ctx: ActionCtx, args): Promise<BackfillResult> => {
const batchSize = clampPositiveInteger(
args.batchSize,
DEFAULT_BATCH_SIZE,
MAX_ACTION_BATCH_SIZE,
);
const maxPages = clampPositiveInteger(args.maxPages, DEFAULT_MAX_PAGES, MAX_MAX_PAGES);
const dryRun = args.dryRun ?? false;
const fetchedAt = Date.now();
const stats = {
scanned: 0,
candidates: 0,
fetched: 0,
patched: 0,
failed: 0,
missingHandles: [] as string[],
errors: [] as Array<{ userId: string; handle: string | null; message: string }>,
};
let cursor = args.cursor ?? null;
let isDone = true;
let pages = 0;
while (pages < maxPages) {
pages += 1;
const page: BackfillPageResult | BackfillHandlesResult = args.handles
? ((await ctx.runQuery(
internal.githubAccountAgeBackfill.listGitHubCreatedAtBackfillHandlesInternal,
{
handles: args.handles,
},
)) as BackfillHandlesResult)
: ((await ctx.runQuery(
internal.githubAccountAgeBackfill.listGitHubCreatedAtBackfillPageInternal,
{
cursor: cursor ?? undefined,
batchSize,
},
)) as BackfillPageResult);
const candidates = page.candidates;
stats.scanned += "scanned" in page ? page.scanned : (args.handles?.length ?? 0);
if ("missingHandles" in page) stats.missingHandles.push(...page.missingHandles);
stats.candidates += candidates.length;
for (const candidate of candidates) {
try {
const githubCreatedAt = await fetchGitHubCreatedAtByProviderAccountId(
candidate.providerAccountId,
);
stats.fetched += 1;
const result: { patched: boolean } = await ctx.runMutation(
internal.githubAccountAgeBackfill.applyGitHubCreatedAtBackfillInternal,
{
userId: candidate.userId,
githubCreatedAt,
fetchedAt,
dryRun,
},
);
if (result.patched) stats.patched += 1;
} catch (error) {
stats.failed += 1;
const message = error instanceof ConvexError ? String(error.data) : String(error);
if (stats.errors.length < 10) {
stats.errors.push({
userId: candidate.userId,
handle: candidate.handle,
message,
});
}
if (/rate limit/i.test(message)) {
return { ok: false as const, rateLimited: true as const, stats, cursor, isDone: false };
}
}
}
if (args.handles) return { ok: true as const, stats, cursor: null, isDone: true };
cursor = "cursor" in page ? page.cursor : null;
isDone = "isDone" in page ? page.isDone : true;
if (isDone) break;
}
if (!dryRun && !isDone && cursor) {
await ctx.scheduler.runAfter(
0,
internal.githubAccountAgeBackfill.backfillGitHubCreatedAtInternal,
{
cursor,
batchSize,
maxPages,
},
);
}
return { ok: true as const, stats, cursor, isDone };
},
});
+2175
View File
File diff suppressed because it is too large Load Diff
+9 -11
View File
@@ -1,8 +1,9 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { Doc, Id } from "./_generated/dataModel";
import { action, internalMutation, internalQuery } from "./functions";
import { assertRole, requireUserFromAction } from "./lib/access";
import { isPublicSkillDoc } from "./lib/globalStats";
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
@@ -104,16 +105,13 @@ export const getGitHubBackupPageInternal = internalQuery({
},
});
function isPubliclyAvailableSkill(skill: {
softDeletedAt?: number;
moderationStatus?: string | null;
}) {
if (skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
function isPubliclyAvailableSkill(
skill: Pick<
Doc<"skillSearchDigest">,
"softDeletedAt" | "moderationStatus" | "moderationFlags" | "moderationVerdict"
>,
) {
return isPublicSkillDoc(skill);
}
function isStaleCursorError(error: unknown) {
+2 -6
View File
@@ -14,6 +14,7 @@ import {
listGitHubSkillBackupEntries,
normalizeOwner,
} from "./lib/githubBackup";
import { isPublicSkillDoc } from "./lib/globalStats";
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
@@ -273,12 +274,7 @@ async function pruneDeletedSkillBackups(
}
function isMirrorEligibleSkill(skill: Doc<"skills"> | null): skill is Doc<"skills"> {
if (!skill || skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
return isPublicSkillDoc(skill);
}
async function deleteBackupIfNeeded(
+241
View File
@@ -0,0 +1,241 @@
import { ConvexError } from "convex/values";
import { beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("./lib/access", () => ({
requireUser: vi.fn(),
}));
vi.mock("./lib/publishers", async () => {
const actual = await vi.importActual<typeof import("./lib/publishers")>("./lib/publishers");
return {
...actual,
requirePublisherRole: vi.fn(),
};
});
const { requireUser } = await import("./lib/access");
const { requirePublisherRole } = await import("./lib/publishers");
const { deleteForPublisherHandler } = await import("./githubSkillSources");
const { buildSkillInstallResolution } = await import("./lib/installResolver");
type Row = Record<string, unknown> & { _id: string };
function chainEq(constraints: Record<string, unknown>) {
return {
eq(field: string, value: unknown) {
constraints[field] = value;
return chainEq(constraints);
},
};
}
function matches(doc: Row, constraints: Record<string, unknown>) {
return Object.entries(constraints).every(([key, value]) => doc[key] === value);
}
function createDb(initial: Record<string, Row[]> = {}) {
const tables: Record<string, Row[]> = Object.fromEntries(
Object.entries(initial).map(([table, rows]) => [table, [...rows]]),
);
const list = (table: string) => {
tables[table] ??= [];
return tables[table];
};
const db = {
get: async (id: string) => {
const table = id.split(":")[0] ?? "";
return list(table).find((row) => row._id === id) ?? null;
},
patch: async (id: string, patch: Record<string, unknown>) => {
const table = id.split(":")[0] ?? "";
const row = list(table).find((candidate) => candidate._id === id);
if (!row) return;
for (const [key, value] of Object.entries(patch)) {
if (value === undefined) delete row[key];
else row[key] = value;
}
},
insert: async (table: string, doc: Record<string, unknown>) => {
const id = `${table}:${list(table).length + 1}`;
list(table).push({ _id: id, ...doc });
return id;
},
delete: async (id: string) => {
const table = id.split(":")[0] ?? "";
const rows = list(table);
const index = rows.findIndex((row) => row._id === id);
if (index >= 0) rows.splice(index, 1);
},
query: (table: string) => ({
withIndex: (_indexName: string, build: (q: ReturnType<typeof chainEq>) => unknown) => {
const constraints: Record<string, unknown> = {};
build(chainEq(constraints));
const matched = () => list(table).filter((row) => matches(row, constraints));
return {
collect: async () => matched(),
unique: async () => matched()[0] ?? null,
};
},
}),
};
return { db, tables };
}
describe("githubSkillSources.deleteForPublisherHandler", () => {
beforeEach(() => {
vi.mocked(requireUser).mockResolvedValue({ userId: "users:owner" } as never);
vi.mocked(requirePublisherRole).mockResolvedValue(undefined as never);
});
it("deletes a source and removes only GitHub-backed skills from that source", async () => {
const { db, tables } = createDb({
githubSkillSources: [
{
_id: "githubSkillSources:matt",
repo: "mattpocock/skills",
ownerPublisherId: "publishers:openclaw",
createdAt: 1,
updatedAt: 2,
},
],
githubSkillContents: [
{
_id: "githubSkillContents:one",
skillId: "skills:github",
githubSourceId: "githubSkillSources:matt",
},
],
skills: [
{
_id: "skills:github",
slug: "source-backed",
displayName: "Source Backed",
installKind: "github",
githubSourceId: "githubSkillSources:matt",
githubPath: "skills/source-backed",
githubCurrentCommit: "a".repeat(40),
githubCurrentContentHash: "hash-source-backed",
githubCurrentStatus: "present",
githubScanStatus: "clean",
ownerUserId: "users:owner",
ownerPublisherId: "publishers:openclaw",
forkOf: undefined,
tags: {},
capabilityTags: undefined,
badges: {},
stats: {
comments: 0,
downloads: 0,
installsAllTime: 0,
installsCurrent: 0,
stars: 0,
versions: 0,
},
moderationStatus: "active",
moderationFlags: [],
isSuspicious: false,
createdAt: 1,
updatedAt: 2,
softDeletedAt: undefined,
},
{
_id: "skills:direct",
slug: "direct-upload",
displayName: "Direct Upload",
ownerPublisherId: "publishers:openclaw",
softDeletedAt: undefined,
},
{
_id: "skills:other-source",
slug: "other-source",
displayName: "Other Source",
installKind: "github",
githubSourceId: "githubSkillSources:other",
githubPath: "skills/other-source",
githubCurrentCommit: "b".repeat(40),
githubCurrentContentHash: "hash-other-source",
githubCurrentStatus: "present",
githubScanStatus: "clean",
ownerPublisherId: "publishers:openclaw",
softDeletedAt: undefined,
},
],
});
await expect(
deleteForPublisherHandler({ db } as never, {
ownerPublisherId: "publishers:openclaw" as never,
sourceId: "githubSkillSources:matt" as never,
now: 123,
}),
).resolves.toEqual({ ok: true, deletedSkills: 1 });
expect(requirePublisherRole).toHaveBeenCalledWith(
expect.anything(),
expect.objectContaining({
publisherId: "publishers:openclaw",
userId: "users:owner",
allowed: ["admin"],
}),
);
expect(tables.githubSkillSources).toHaveLength(0);
expect(tables.githubSkillContents).toHaveLength(0);
const deletedSkill = tables.skills.find((skill) => skill._id === "skills:github");
expect(deletedSkill).toMatchObject({
softDeletedAt: 123,
githubRemovedAt: 123,
githubCurrentStatus: "missing",
updatedAt: 123,
});
expect(tables.skillSearchDigest).toEqual([
expect.objectContaining({
skillId: "skills:github",
githubCurrentStatus: "missing",
githubScanStatus: "clean",
softDeletedAt: 123,
}),
]);
expect(
buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: deletedSkill as never,
source: null,
}),
).toMatchObject({
ok: false,
reason: "github_upstream_removed",
status: 410,
});
expect(tables.skills.find((skill) => skill._id === "skills:direct")).toMatchObject({
softDeletedAt: undefined,
});
expect(tables.skills.find((skill) => skill._id === "skills:other-source")).toMatchObject({
githubCurrentStatus: "present",
softDeletedAt: undefined,
});
});
it("rejects deleting a source from another publisher", async () => {
const { db } = createDb({
githubSkillSources: [
{
_id: "githubSkillSources:matt",
repo: "mattpocock/skills",
ownerPublisherId: "publishers:other",
createdAt: 1,
updatedAt: 2,
},
],
});
await expect(
deleteForPublisherHandler({ db } as never, {
ownerPublisherId: "publishers:openclaw" as never,
sourceId: "githubSkillSources:matt" as never,
now: 123,
}),
).rejects.toBeInstanceOf(ConvexError);
});
});
+206
View File
@@ -0,0 +1,206 @@
import { ConvexError, v } from "convex/values";
import type { Doc, Id } from "./_generated/dataModel";
import type { MutationCtx, QueryCtx } from "./_generated/server";
import { internalQuery, mutation, query } from "./functions";
import { requireUser } from "./lib/access";
import { adjustGlobalPublicSkillsCount, getPublicSkillVisibilityDelta } from "./lib/globalStats";
import { isOfficialPublisher } from "./lib/officialPublishers";
import { isPublisherActive, isPublisherRoleAllowed, requirePublisherRole } from "./lib/publishers";
import { syncSkillSearchDigestForSkill } from "./lib/skillSearchDigest";
type PublicGitHubSkillSource = Pick<
Doc<"githubSkillSources">,
| "_id"
| "repo"
| "defaultBranch"
| "lastSyncStatus"
| "lastSyncError"
| "lastSyncErrorAt"
| "displayManifestStatus"
| "displayManifestFetchedAt"
| "displayManifestCommit"
| "lastSyncIssues"
| "lastSyncInvalidSkills"
| "createdAt"
| "updatedAt"
> & {
ownerPublisher: Pick<Doc<"publishers">, "_id" | "handle" | "displayName"> | null;
skills: Array<
Pick<Doc<"skills">, "_id" | "slug" | "displayName" | "githubPath" | "githubCurrentStatus">
>;
};
export const getByIdInternal = internalQuery({
args: { sourceId: v.id("githubSkillSources") },
handler: async (ctx, args) => ctx.db.get(args.sourceId),
});
async function toPublicGitHubSkillSource(
ctx: Pick<QueryCtx, "db">,
source: Doc<"githubSkillSources">,
): Promise<PublicGitHubSkillSource> {
const skills = await ctx.db
.query("skills")
.withIndex("by_github_source", (q) => q.eq("githubSourceId", source._id))
.collect();
const visibleGitHubSkills = skills
.filter((skill) => skill.installKind === "github" && !skill.softDeletedAt)
.sort((a, b) => a.displayName.localeCompare(b.displayName))
.map((skill) => ({
_id: skill._id,
slug: skill.slug,
displayName: skill.displayName,
githubPath: skill.githubPath,
githubCurrentStatus: skill.githubCurrentStatus,
}));
const ownerPublisher = source.ownerPublisherId ? await ctx.db.get(source.ownerPublisherId) : null;
return {
_id: source._id as Id<"githubSkillSources">,
repo: source.repo,
ownerPublisher: ownerPublisher
? {
_id: ownerPublisher._id,
handle: ownerPublisher.handle,
displayName: ownerPublisher.displayName,
}
: null,
defaultBranch: source.defaultBranch,
lastSyncStatus: source.lastSyncStatus,
lastSyncError: source.lastSyncError,
lastSyncErrorAt: source.lastSyncErrorAt,
displayManifestStatus: source.displayManifestStatus,
displayManifestFetchedAt: source.displayManifestFetchedAt,
displayManifestCommit: source.displayManifestCommit,
lastSyncIssues: source.lastSyncIssues,
lastSyncInvalidSkills: source.lastSyncInvalidSkills,
createdAt: source.createdAt,
updatedAt: source.updatedAt,
skills: visibleGitHubSkills,
};
}
export const listForPublisher = query({
args: { ownerPublisherId: v.id("publishers") },
handler: async (ctx, args): Promise<PublicGitHubSkillSource[]> => {
const { userId } = await requireUser(ctx);
await requirePublisherRole(ctx, {
publisherId: args.ownerPublisherId,
userId,
allowed: ["admin"],
});
const sources = await ctx.db
.query("githubSkillSources")
.withIndex("by_owner_publisher", (q) => q.eq("ownerPublisherId", args.ownerPublisherId))
.collect();
const sortedSources = sources.sort((a, b) => b.updatedAt - a.updatedAt);
return await Promise.all(sortedSources.map((source) => toPublicGitHubSkillSource(ctx, source)));
},
});
export const listForManageableOfficialPublishers = query({
args: {},
handler: async (ctx): Promise<PublicGitHubSkillSource[]> => {
const { userId } = await requireUser(ctx);
const memberships = await ctx.db
.query("publisherMembers")
.withIndex("by_user", (q) => q.eq("userId", userId))
.collect();
const ownerPublisherIds: Id<"publishers">[] = [];
for (const membership of memberships) {
if (!isPublisherRoleAllowed(membership.role, ["admin"])) continue;
const publisher = await ctx.db.get(membership.publisherId);
if (
!publisher ||
publisher.kind !== "org" ||
!isPublisherActive(publisher) ||
!(await isOfficialPublisher(ctx, publisher))
) {
continue;
}
ownerPublisherIds.push(publisher._id);
}
const sourceGroups = await Promise.all(
ownerPublisherIds.map((ownerPublisherId) =>
ctx.db
.query("githubSkillSources")
.withIndex("by_owner_publisher", (q) => q.eq("ownerPublisherId", ownerPublisherId))
.collect(),
),
);
const sortedSources = sourceGroups.flat().sort((a, b) => b.updatedAt - a.updatedAt);
return await Promise.all(sortedSources.map((source) => toPublicGitHubSkillSource(ctx, source)));
},
});
export async function deleteForPublisherHandler(
ctx: MutationCtx,
args: {
ownerPublisherId: Id<"publishers">;
sourceId: Id<"githubSkillSources">;
now?: number;
},
) {
const { userId } = await requireUser(ctx);
await requirePublisherRole(ctx, {
publisherId: args.ownerPublisherId,
userId,
allowed: ["admin"],
});
const source = await ctx.db.get(args.sourceId);
if (!source || source.ownerPublisherId !== args.ownerPublisherId) {
throw new ConvexError("GitHub source not found.");
}
const now = args.now ?? Date.now();
const contents = await ctx.db
.query("githubSkillContents")
.withIndex("by_github_source", (q) => q.eq("githubSourceId", args.sourceId))
.collect();
for (const content of contents) {
await ctx.db.delete(content._id);
}
const skills = await ctx.db
.query("skills")
.withIndex("by_github_source", (q) => q.eq("githubSourceId", args.sourceId))
.collect();
let deletedSkills = 0;
let publicSkillDelta = 0;
for (const skill of skills) {
if (skill.installKind !== "github") continue;
const nextSkill: Doc<"skills"> = {
...skill,
softDeletedAt: skill.softDeletedAt ?? now,
githubCurrentStatus: "missing",
githubRemovedAt: skill.githubRemovedAt ?? now,
updatedAt: now,
};
publicSkillDelta += getPublicSkillVisibilityDelta(skill, nextSkill);
await ctx.db.patch(skill._id, {
softDeletedAt: nextSkill.softDeletedAt,
githubCurrentStatus: nextSkill.githubCurrentStatus,
githubRemovedAt: nextSkill.githubRemovedAt,
updatedAt: now,
});
await syncSkillSearchDigestForSkill(ctx, nextSkill);
deletedSkills += 1;
}
if (publicSkillDelta !== 0) {
await adjustGlobalPublicSkillsCount(ctx, publicSkillDelta, now);
}
await ctx.db.delete(args.sourceId);
return { ok: true as const, deletedSkills };
}
export const deleteForPublisher: ReturnType<typeof mutation> = mutation({
args: {
ownerPublisherId: v.id("publishers"),
sourceId: v.id("githubSkillSources"),
},
handler: async (ctx, args) => deleteForPublisherHandler(ctx, args),
});
+249
View File
@@ -0,0 +1,249 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
import {
applyGitHubSkillSourceSyncHandler,
applyGitHubSkillVerificationResultHandler,
configurePublicGitHubSkillSourceHandler,
upsertGitHubSkillContentHandler,
verifyGitHubSkillHandler,
} from "./githubSkillSync";
import { buildSkillInstallResolution } from "./lib/installResolver";
type Row = Record<string, unknown> & { _id: string };
function chainEq(constraints: Record<string, unknown>) {
return {
eq(field: string, value: unknown) {
constraints[field] = value;
return chainEq(constraints);
},
};
}
function matches(doc: Row, constraints: Record<string, unknown>) {
return Object.entries(constraints).every(([key, value]) => doc[key] === value);
}
function createDb(initial: Record<string, Row[]> = {}) {
const tables: Record<string, Row[]> = Object.fromEntries(
Object.entries(initial).map(([table, rows]) => [table, [...rows]]),
);
const counters: Record<string, number> = {};
const list = (table: string) => {
tables[table] ??= [];
return tables[table];
};
const db = {
get: async (id: string) => {
const table = id.split(":")[0] ?? "";
return list(table).find((row) => row._id === id) ?? null;
},
insert: async (table: string, doc: Record<string, unknown>) => {
counters[table] = (counters[table] ?? 0) + 1;
const inserted = {
_id: `${table}:new-${counters[table]}`,
_creationTime: counters[table],
...doc,
};
list(table).push(inserted);
return inserted._id;
},
patch: async (id: string, patch: Record<string, unknown>) => {
const table = id.split(":")[0] ?? "";
const row = list(table).find((candidate) => candidate._id === id);
if (!row) return;
for (const [key, value] of Object.entries(patch)) {
if (value === undefined) delete row[key];
else row[key] = value;
}
},
query: (table: string) => ({
withIndex: (_indexName: string, build?: (q: ReturnType<typeof chainEq>) => unknown) => {
const constraints: Record<string, unknown> = {};
build?.(chainEq(constraints));
const matched = () => list(table).filter((row) => matches(row, constraints));
return {
collect: async () => matched(),
unique: async () => matched()[0] ?? null,
};
},
}),
};
return { db, tables };
}
function getSkill(tables: Record<string, Row[]>, slug: string) {
const skill = tables.skills?.find((row) => row.slug === slug);
if (!skill) throw new Error(`Live GitHub canary did not discover skill: ${slug}`);
return skill;
}
function resolveInstallFromTables(tables: Record<string, Row[]>, slug: string) {
const skill = getSkill(tables, slug);
const source =
typeof skill.githubSourceId === "string"
? (tables.githubSkillSources?.find((row) => row._id === skill.githubSourceId) ?? null)
: null;
return buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: skill as never,
source: source as never,
});
}
const liveCanaryEnabled = process.env.CLAWHUB_LIVE_GITHUB_CANARY === "1";
const itIfLive = liveCanaryEnabled ? it : it.skip;
describe("GitHub-backed skills live canary", () => {
itIfLive(
"discovers and verifies an installable skill from a real GitHub repo",
{ timeout: 45_000 },
async () => {
const repo = process.env.CLAWHUB_LIVE_GITHUB_REPO?.trim() || "openclaw/agent-skills";
const skillSlug = process.env.CLAWHUB_LIVE_GITHUB_SKILL?.trim() || "handoff";
const { db, tables } = createDb({
globalStats: [
{
_id: "globalStats:default",
key: "default",
activeSkillsCount: 0,
updatedAt: 1,
},
],
});
const scheduler = { runAfter: async () => undefined };
let now = Date.now();
const actionCtx = {
runQuery: async (_query: unknown, args: Record<string, unknown>) => {
if ("ownerPublisherId" in args && "actorUserId" in args) {
return {
ownerUserId: "users:live-owner",
existingSource:
tables.githubSkillSources?.find((source) => source.repo === repo) ?? null,
official: true,
};
}
if ("skillId" in args) {
const skill = tables.skills?.find((row) => row._id === args.skillId);
const source =
skill && typeof skill.githubSourceId === "string"
? tables.githubSkillSources?.find((row) => row._id === skill.githubSourceId)
: null;
return skill && source ? { skill, source } : null;
}
if ("sourceId" in args) {
return (tables.skills ?? []).flatMap((skill) => {
if (
skill.githubSourceId !== args.sourceId ||
skill.installKind !== "github" ||
skill.githubCurrentStatus !== "present" ||
typeof skill.githubPath !== "string" ||
typeof skill.githubCurrentContentHash !== "string"
) {
return [];
}
return [
{
skillId: skill._id,
githubPath: skill.githubPath,
githubCurrentContentHash: skill.githubCurrentContentHash,
},
];
});
}
throw new Error(`unexpected live canary query args: ${JSON.stringify(args)}`);
},
runMutation: async (_mutation: unknown, args: Record<string, unknown>) => {
if ("snapshot" in args) {
return await applyGitHubSkillSourceSyncHandler(
{ db, scheduler } as never,
{
...args,
now,
} as never,
);
}
if ("scanStatus" in args && "contentHash" in args) {
return await applyGitHubSkillVerificationResultHandler(
{ db } as never,
{
...args,
now,
} as never,
);
}
if ("discovered" in args && "commit" in args) {
return await upsertGitHubSkillContentHandler(
{ db } as never,
{
...args,
now,
} as never,
);
}
throw new Error(`unexpected live canary mutation args: ${JSON.stringify(args)}`);
},
auth: { getUserIdentity: async () => null },
};
const configured = await configurePublicGitHubSkillSourceHandler(
actionCtx as never,
{
ownerPublisherId: "publishers:live" as never,
repo,
},
fetch,
{ userId: "users:live-owner" as never },
);
expect(configured.stats.discovered).toBeGreaterThan(0);
expect(configured.manifestStatus === "missing" || configured.manifestStatus === "ok").toBe(
true,
);
expect(configured.commit).toMatch(/^[a-f0-9]{40}$/);
let skill = getSkill(tables, skillSlug);
expect(skill).toMatchObject({
installKind: "github",
githubPath: `skills/${skillSlug}`,
githubCurrentCommit: configured.commit,
githubCurrentStatus: "present",
githubScanStatus: "pending",
});
expect(resolveInstallFromTables(tables, skillSlug)).toMatchObject({
ok: false,
reason: "github_verification_pending",
});
now = Date.now();
const verified = await verifyGitHubSkillHandler(
actionCtx as never,
{
skillId: skill._id as never,
contentHash: skill.githubCurrentContentHash as string,
},
fetch,
);
expect(verified).toMatchObject({ ok: true, scanStatus: "clean" });
skill = getSkill(tables, skillSlug);
expect(skill).toMatchObject({
githubCurrentCommit: configured.commit,
githubScanStatus: "clean",
moderationStatus: "active",
});
expect(resolveInstallFromTables(tables, skillSlug)).toMatchObject({
ok: true,
installKind: "github",
github: {
repo,
path: `skills/${skillSlug}`,
commit: configured.commit,
contentHash: skill.githubCurrentContentHash,
},
});
},
);
});
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
-170
View File
@@ -1,170 +0,0 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc, Id } from "./_generated/dataModel";
import { action, internalMutation, internalQuery } from "./functions";
import { assertRole, requireUserFromAction } from "./lib/access";
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
const SYNC_STATE_KEY = "souls";
type BackupPageItem =
| {
kind: "ok";
soulId: Id<"souls">;
versionId: Id<"soulVersions">;
slug: string;
displayName: string;
version: string;
ownerHandle: string;
files: Doc<"soulVersions">["files"];
publishedAt: number;
}
| { kind: "missingLatestVersion"; soulId: Id<"souls"> }
| { kind: "missingVersionDoc"; soulId: Id<"souls">; versionId: Id<"soulVersions"> }
| { kind: "missingOwner"; soulId: Id<"souls">; ownerUserId: Id<"users"> };
type BackupPageResult = {
items: BackupPageItem[];
cursor: string | null;
isDone: boolean;
};
type BackupSyncState = {
cursor: string | null;
};
export type SyncGitHubSoulBackupsResult = {
stats: {
soulsScanned: number;
soulsSkipped: number;
soulsBackedUp: number;
soulsMissingVersion: number;
soulsMissingOwner: number;
errors: number;
};
cursor: string | null;
isDone: boolean;
};
export const getGitHubSoulBackupPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BackupPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const { page, isDone, continueCursor } = await ctx.db
.query("souls")
.order("asc")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
const items: BackupPageItem[] = [];
for (const soul of page) {
if (soul.softDeletedAt) continue;
if (!soul.latestVersionId) {
items.push({ kind: "missingLatestVersion", soulId: soul._id });
continue;
}
const version = await ctx.db.get(soul.latestVersionId);
if (!version) {
items.push({
kind: "missingVersionDoc",
soulId: soul._id,
versionId: soul.latestVersionId,
});
continue;
}
const owner = await ctx.db.get(soul.ownerUserId);
if (!owner || owner.deletedAt || owner.deactivatedAt) {
items.push({ kind: "missingOwner", soulId: soul._id, ownerUserId: soul.ownerUserId });
continue;
}
items.push({
kind: "ok",
soulId: soul._id,
versionId: version._id,
slug: soul.slug,
displayName: soul.displayName,
version: version.version,
ownerHandle: owner.handle ?? owner._id,
files: version.files,
publishedAt: version.createdAt,
});
}
return { items, cursor: continueCursor, isDone };
},
});
export const getGitHubSoulBackupSyncStateInternal = internalQuery({
args: {},
handler: async (ctx): Promise<BackupSyncState> => {
const state = await ctx.db
.query("githubBackupSyncState")
.withIndex("by_key", (q) => q.eq("key", SYNC_STATE_KEY))
.unique();
return { cursor: state?.cursor ?? null };
},
});
export const setGitHubSoulBackupSyncStateInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
},
handler: async (ctx, args) => {
const now = Date.now();
const state = await ctx.db
.query("githubBackupSyncState")
.withIndex("by_key", (q) => q.eq("key", SYNC_STATE_KEY))
.unique();
if (!state) {
await ctx.db.insert("githubBackupSyncState", {
key: SYNC_STATE_KEY,
cursor: args.cursor,
updatedAt: now,
});
return { ok: true as const };
}
await ctx.db.patch(state._id, {
cursor: args.cursor,
updatedAt: now,
});
return { ok: true as const };
},
});
export const syncGitHubSoulBackups: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
resetCursor: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SyncGitHubSoulBackupsResult> => {
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin"]);
if (args.resetCursor && !args.dryRun) {
await ctx.runMutation(internal.githubSoulBackups.setGitHubSoulBackupSyncStateInternal, {
cursor: undefined,
});
}
return ctx.runAction(internal.githubSoulBackupsNode.syncGitHubSoulBackupsInternal, {
dryRun: args.dryRun,
batchSize: args.batchSize,
maxBatches: args.maxBatches,
}) as Promise<SyncGitHubSoulBackupsResult>;
},
});
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)));
}
-186
View File
@@ -1,186 +0,0 @@
"use node";
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { internalAction } from "./functions";
import {
backupSoulToGitHub,
fetchGitHubSoulMeta,
getGitHubSoulBackupContext,
isGitHubSoulBackupConfigured,
} from "./lib/githubSoulBackup";
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
const DEFAULT_MAX_BATCHES = 5;
const MAX_MAX_BATCHES = 200;
type BackupPageItem =
| {
kind: "ok";
slug: string;
version: string;
displayName: string;
ownerHandle: string;
files: Doc<"soulVersions">["files"];
publishedAt: number;
}
| { kind: "missingLatestVersion" }
| { kind: "missingVersionDoc" }
| { kind: "missingOwner" };
export type GitHubSoulBackupSyncStats = {
soulsScanned: number;
soulsSkipped: number;
soulsBackedUp: number;
soulsMissingVersion: number;
soulsMissingOwner: number;
errors: number;
};
export type SyncGitHubSoulBackupsInternalArgs = {
dryRun?: boolean;
batchSize?: number;
maxBatches?: number;
};
export type SyncGitHubSoulBackupsInternalResult = {
stats: GitHubSoulBackupSyncStats;
cursor: string | null;
isDone: boolean;
};
export const backupSoulForPublishInternal = internalAction({
args: {
slug: v.string(),
version: v.string(),
displayName: v.string(),
ownerHandle: v.string(),
files: v.array(
v.object({
path: v.string(),
size: v.number(),
storageId: v.id("_storage"),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
),
publishedAt: v.number(),
},
handler: async (ctx, args) => {
if (!isGitHubSoulBackupConfigured()) {
return { skipped: true as const };
}
await backupSoulToGitHub(ctx, args);
return { skipped: false as const };
},
});
export async function syncGitHubSoulBackupsInternalHandler(
ctx: ActionCtx,
args: SyncGitHubSoulBackupsInternalArgs,
): Promise<SyncGitHubSoulBackupsInternalResult> {
const dryRun = Boolean(args.dryRun);
const stats: GitHubSoulBackupSyncStats = {
soulsScanned: 0,
soulsSkipped: 0,
soulsBackedUp: 0,
soulsMissingVersion: 0,
soulsMissingOwner: 0,
errors: 0,
};
if (!isGitHubSoulBackupConfigured()) {
return { stats, cursor: null, isDone: true };
}
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES);
const context = await getGitHubSoulBackupContext();
const state = dryRun
? { cursor: null as string | null }
: ((await ctx.runQuery(
internal.githubSoulBackups.getGitHubSoulBackupSyncStateInternal,
{},
)) as {
cursor: string | null;
});
let cursor: string | null = state.cursor;
let isDone = false;
for (let batch = 0; batch < maxBatches; batch++) {
const page = (await ctx.runQuery(internal.githubSoulBackups.getGitHubSoulBackupPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as { items: BackupPageItem[]; cursor: string | null; isDone: boolean };
cursor = page.cursor;
isDone = page.isDone;
for (const item of page.items) {
if (item.kind !== "ok") {
if (item.kind === "missingLatestVersion" || item.kind === "missingVersionDoc") {
stats.soulsMissingVersion += 1;
} else if (item.kind === "missingOwner") {
stats.soulsMissingOwner += 1;
}
continue;
}
stats.soulsScanned += 1;
try {
const meta = await fetchGitHubSoulMeta(context, item.ownerHandle, item.slug);
if (meta?.latest?.version === item.version) {
stats.soulsSkipped += 1;
continue;
}
if (!dryRun) {
await backupSoulToGitHub(
ctx,
{
slug: item.slug,
version: item.version,
displayName: item.displayName,
ownerHandle: item.ownerHandle,
files: item.files,
publishedAt: item.publishedAt,
},
context,
);
stats.soulsBackedUp += 1;
}
} catch (error) {
console.error("GitHub soul backup sync failed", error);
stats.errors += 1;
}
}
if (!dryRun) {
await ctx.runMutation(internal.githubSoulBackups.setGitHubSoulBackupSyncStateInternal, {
cursor: isDone ? undefined : (cursor ?? undefined),
});
}
if (isDone) break;
}
return { stats, cursor, isDone };
}
export const syncGitHubSoulBackupsInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: syncGitHubSoulBackupsInternalHandler,
});
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)));
}
+89 -38
View File
@@ -2,13 +2,14 @@ import { ApiRoutes, LegacyApiRoutes } from "clawhub-schema";
import { httpRouter } from "convex/server";
import { auth } from "./auth";
import { downloadZip } from "./downloads";
import { githubWebhookHttp } from "./githubApp";
import {
cliPublishHttp,
cliDeviceCodeHttp,
cliDeviceTokenHttp,
cliSkillDeleteHttp,
cliSkillUndeleteHttp,
cliTelemetrySyncHttp,
cliTelemetryInstallHttp,
cliUploadUrlHttp,
cliWhoamiHttp,
getSkillHttp,
@@ -17,12 +18,12 @@ import {
} from "./httpApi";
import {
exportSkillsV1Http,
exportPluginsV1Http,
listBundlePluginsV1Http,
listCodePluginsV1Http,
listPackagesV1Http,
listPluginsV1Http,
listSkillsV1Http,
listSoulsV1Http,
mintPublishTokenV1Http,
npmMirrorGetHttp,
packagesDeleteRouterV1Http,
@@ -32,24 +33,32 @@ import {
createPublisherV1Http,
publishPackageV1Http,
publishSkillV1Http,
publishSoulV1Http,
resolveSkillVersionV1Http,
searchSkillsV1Http,
skillScanBatchStatusV1Http,
skillScanBatchSubmitV1Http,
skillScanGetRouterV1Http,
skillScanSubmitV1Http,
skillSecurityVerdictsV1Http,
skillsDeleteRouterV1Http,
skillsGetRouterV1Http,
skillsPostRouterV1Http,
soulsDeleteRouterV1Http,
soulsGetRouterV1Http,
soulsPostRouterV1Http,
starsDeleteRouterV1Http,
starsPostRouterV1Http,
transfersGetRouterV1Http,
banAppealContextV1Http,
usersGetRouterV1Http,
usersListV1Http,
usersPostRouterV1Http,
verifyDocsSessionV1Http,
whoamiV1Http,
} from "./httpApiV1";
import { preflightHandler } from "./httpPreflight";
import {
packageInspectorArtifactHttp,
packageInspectorClaimHttp,
packageInspectorResultsHttp,
} from "./packageInspectorHttp";
const http = httpRouter();
@@ -85,6 +94,12 @@ http.route({
handler: listSkillsV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.skillScans}/`,
method: "GET",
handler: skillScanGetRouterV1Http,
});
http.route({
path: ApiRoutes.packages,
method: "GET",
@@ -97,6 +112,12 @@ http.route({
handler: listPluginsV1Http,
});
http.route({
path: ApiRoutes.pluginsExport,
method: "GET",
handler: exportPluginsV1Http,
});
http.route({
path: ApiRoutes.codePlugins,
method: "GET",
@@ -139,6 +160,24 @@ http.route({
handler: publishSkillV1Http,
});
http.route({
path: ApiRoutes.skillScans,
method: "POST",
handler: skillScanSubmitV1Http,
});
http.route({
path: `${ApiRoutes.skillScans}/batch`,
method: "POST",
handler: skillScanBatchSubmitV1Http,
});
http.route({
path: `${ApiRoutes.skillScans}/batch/status`,
method: "POST",
handler: skillScanBatchStatusV1Http,
});
http.route({
path: ApiRoutes.packages,
method: "POST",
@@ -151,6 +190,24 @@ http.route({
handler: mintPublishTokenV1Http,
});
http.route({
path: "/api/v1/package-inspector/claim",
method: "POST",
handler: packageInspectorClaimHttp,
});
http.route({
path: "/api/v1/package-inspector/artifact",
method: "GET",
handler: packageInspectorArtifactHttp,
});
http.route({
path: "/api/v1/package-inspector/results",
method: "POST",
handler: packageInspectorResultsHttp,
});
http.route({
pathPrefix: `${ApiRoutes.packages}/`,
method: "POST",
@@ -163,6 +220,12 @@ http.route({
handler: packagesDeleteRouterV1Http,
});
http.route({
path: `${ApiRoutes.skills}/-/security-verdicts`,
method: "POST",
handler: skillSecurityVerdictsV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.skills}/`,
method: "POST",
@@ -229,48 +292,36 @@ http.route({
handler: usersPostRouterV1Http,
});
http.route({
path: "/api/v1/users/ban-appeal-context",
method: "GET",
handler: banAppealContextV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.users}/`,
method: "GET",
handler: usersGetRouterV1Http,
});
http.route({
path: ApiRoutes.users,
method: "GET",
handler: usersListV1Http,
});
http.route({
path: ApiRoutes.souls,
method: "GET",
handler: listSoulsV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: "GET",
handler: soulsGetRouterV1Http,
});
http.route({
path: ApiRoutes.souls,
method: "POST",
handler: publishSoulV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: "POST",
handler: soulsPostRouterV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: "DELETE",
handler: soulsDeleteRouterV1Http,
});
http.route({
pathPrefix: "/api/",
method: "OPTIONS",
handler: preflightHandler,
});
http.route({
path: "/api/webhooks/github-app",
method: "POST",
handler: githubWebhookHttp,
});
// TODO: remove legacy /api routes after deprecation window.
http.route({
path: LegacyApiRoutes.download,
@@ -314,9 +365,9 @@ http.route({
});
http.route({
path: LegacyApiRoutes.cliTelemetrySync,
path: LegacyApiRoutes.cliTelemetryInstall,
method: "POST",
handler: cliTelemetrySyncHttp,
handler: cliTelemetryInstallHttp,
});
http.route({
+61 -56
View File
@@ -4,13 +4,15 @@ import { afterEach, describe, expect, it, vi } from "vitest";
vi.mock("./lib/apiTokenAuth", () => ({
getOptionalApiTokenUser: vi.fn(),
requireApiTokenUser: vi.fn(),
requirePackagePublishAuth: vi.fn(),
}));
vi.mock("./skills", () => ({
publishVersionForUser: vi.fn(),
}));
const { getOptionalApiTokenUser, requireApiTokenUser } = await import("./lib/apiTokenAuth");
const { getOptionalApiTokenUser, requireApiTokenUser, requirePackagePublishAuth } =
await import("./lib/apiTokenAuth");
const { publishVersionForUser } = await import("./skills");
const { __handlers } = await import("./httpApi");
const { hashSkillFiles } = await import("./lib/skills");
@@ -23,6 +25,7 @@ describe("httpApi handlers", () => {
afterEach(() => {
vi.mocked(getOptionalApiTokenUser).mockReset();
vi.mocked(requireApiTokenUser).mockReset();
vi.mocked(requirePackagePublishAuth).mockReset();
vi.mocked(publishVersionForUser).mockReset();
});
@@ -240,7 +243,7 @@ describe("httpApi handlers", () => {
it("cliWhoamiHttp returns 401 on auth failure", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(
new Error(
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, contact security@openclaw.ai.",
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, open a GitHub issue: https://github.com/openclaw/clawhub/issues/new.",
),
);
const response = await __handlers.cliWhoamiHandler(
@@ -264,49 +267,40 @@ describe("httpApi handlers", () => {
expect(json.user.handle).toBe("p");
});
it("cliTelemetrySyncHttp forwards roots and returns ok", async () => {
it("cliTelemetryInstallHttp forwards one install event and returns ok", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const runMutation = vi.fn().mockResolvedValue(null);
const response = await __handlers.cliTelemetrySyncHandler(
const response = await __handlers.cliTelemetryInstallHandler(
makeCtx({ runMutation }),
new Request("https://x/api/cli/telemetry/sync", {
new Request("https://x/api/cli/telemetry/install", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
roots: [
{
rootId: "abc",
label: "~/skills",
skills: [{ slug: "weather", version: null }],
},
],
event: "install",
slug: "weather",
version: "1.0.0",
rootId: "abc",
rootLabel: "~/skills",
}),
}),
);
expect(response.status).toBe(200);
expect(await response.json()).toEqual({ ok: true });
expect(runMutation).toHaveBeenCalledTimes(1);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: "users:1",
slug: "weather",
version: "1.0.0",
rootId: "abc",
rootLabel: "~/skills",
});
});
it("cliTelemetrySyncHttp returns 400 on invalid payload", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation: vi.fn() }),
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ roots: "nope" }),
}),
);
expect(response.status).toBe(400);
});
it("cliTelemetrySyncHttp forwards skill versions when provided", async () => {
it("cliTelemetryInstallHttp rejects sync-shaped roots snapshots", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const runMutation = vi.fn().mockResolvedValue(null);
await __handlers.cliTelemetrySyncHandler(
const response = await __handlers.cliTelemetryInstallHandler(
makeCtx({ runMutation }),
new Request("https://x/api/cli/telemetry/sync", {
new Request("https://x/api/cli/telemetry/install", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
@@ -320,31 +314,9 @@ describe("httpApi handlers", () => {
}),
}),
);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: "users:1",
roots: [
{ rootId: "abc", label: "~/skills", skills: [{ slug: "weather", version: "1.0.0" }] },
],
});
});
it("cliTelemetrySyncHttp returns 400 on invalid json", async () => {
const request = new Request("https://x/api/cli/telemetry/sync", { method: "POST", body: "{" });
const response = await __handlers.cliTelemetrySyncHandler(makeCtx({}), request);
expect(response.status).toBe(400);
});
it("cliTelemetrySyncHttp returns 401 when unauthorized", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({}),
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ roots: [] }),
}),
);
expect(response.status).toBe(401);
expect(runMutation).not.toHaveBeenCalled();
});
it("cliDeviceCodeHttp rate limits and creates a device code", async () => {
@@ -444,18 +416,51 @@ describe("httpApi handlers", () => {
});
it("cliUploadUrlHttp returns uploadUrl", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const runMutation = vi.fn().mockResolvedValue("https://upload.local");
vi.mocked(requirePackagePublishAuth).mockResolvedValueOnce({
kind: "user",
userId: "user1",
} as never);
const runMutation = vi.fn().mockResolvedValue({
uploadUrl: "https://upload.local",
uploadTicket: "packagePublishUploadTickets:1",
});
const response = await __handlers.cliUploadUrlHandler(
makeCtx({ runMutation }),
new Request("https://x/api/cli/upload-url", { method: "POST" }),
);
expect(response.status).toBe(200);
expect(await response.json()).toEqual({ uploadUrl: "https://upload.local" });
expect(await response.json()).toEqual({
uploadUrl: "https://upload.local",
uploadTicket: "packagePublishUploadTickets:1",
});
});
it("cliUploadUrlHttp accepts package publish tokens", async () => {
vi.mocked(requirePackagePublishAuth).mockResolvedValueOnce({
kind: "github-actions",
publishToken: { _id: "packagePublishTokens:1" },
} as never);
const runMutation = vi.fn().mockResolvedValue({
uploadUrl: "https://upload.local/package",
uploadTicket: "packagePublishUploadTickets:2",
});
const response = await __handlers.cliUploadUrlHandler(
makeCtx({ runMutation }),
new Request("https://x/api/cli/upload-url", { method: "POST" }),
);
expect(response.status).toBe(200);
expect(await response.json()).toEqual({
uploadUrl: "https://upload.local/package",
uploadTicket: "packagePublishUploadTickets:2",
});
expect(runMutation).toHaveBeenCalledWith(
expect.anything(),
expect.objectContaining({ publishTokenId: "packagePublishTokens:1" }),
);
});
it("cliUploadUrlHttp returns 401 when unauthorized", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
vi.mocked(requirePackagePublishAuth).mockRejectedValueOnce(new Error("Unauthorized"));
const response = await __handlers.cliUploadUrlHandler(
makeCtx({}),
new Request("https://x/api/cli/upload-url", { method: "POST" }),
+23 -23
View File
@@ -1,16 +1,16 @@
import {
ApiCliSkillDeleteResponseSchema,
ApiCliTelemetrySyncResponseSchema,
ApiCliTelemetryInstallResponseSchema,
CliTelemetryInstallRequestSchema,
CliPublishRequestSchema,
CliSkillDeleteRequestSchema,
CliTelemetrySyncRequestSchema,
parseArk,
} from "clawhub-schema";
import { api, internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { httpAction } from "./functions";
import { requireApiTokenUser } from "./lib/apiTokenAuth";
import { requireApiTokenUser, requirePackagePublishAuth } from "./lib/apiTokenAuth";
import { corsHeaders, mergeHeaders } from "./lib/httpHeaders";
import { applyRateLimit } from "./lib/httpRateLimit";
import { parseBooleanQueryParam, resolveBooleanQueryParam } from "./lib/httpUtils";
@@ -148,11 +148,16 @@ export const cliWhoamiHttp = httpAction(cliWhoamiHandler);
async function cliUploadUrlHandler(ctx: ActionCtx, request: Request) {
try {
const { userId } = await requireApiTokenUser(ctx, request);
const uploadUrl = await ctx.runMutation(internal.uploads.generateUploadUrlForUserInternal, {
userId,
});
return json({ uploadUrl });
const auth = await requirePackagePublishAuth(ctx, request);
const upload =
auth.kind === "user"
? await ctx.runMutation(internal.uploads.createPackagePublishUploadForUserInternal, {
userId: auth.userId,
})
: await ctx.runMutation(internal.uploads.createPackagePublishUploadForTokenInternal, {
publishTokenId: auth.publishToken._id,
});
return json(upload);
} catch (error) {
return text(formatAuthFailure(error), 401);
}
@@ -222,7 +227,7 @@ export const cliSkillUndeleteHttp = httpAction((ctx, request) =>
cliSkillDeleteHandler(ctx, request, false),
);
async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
async function cliTelemetryInstallHandler(ctx: ActionCtx, request: Request) {
let body: unknown;
try {
body = await request.json();
@@ -232,19 +237,15 @@ async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
try {
const { userId } = await requireApiTokenUser(ctx, request);
const args = parseArk(CliTelemetrySyncRequestSchema, body, "Telemetry payload");
await ctx.runMutation(internal.telemetry.reportCliSyncInternal, {
const args = parseArk(CliTelemetryInstallRequestSchema, body, "Install telemetry payload");
await ctx.runMutation(internal.telemetry.reportCliInstallInternal, {
userId,
roots: args.roots.map((root) => ({
rootId: root.rootId,
label: root.label,
skills: root.skills.map((skill) => ({
slug: skill.slug,
version: skill.version ?? undefined,
})),
})),
slug: args.slug,
version: args.version,
rootId: args.rootId,
rootLabel: args.rootLabel,
});
const ok = parseArk(ApiCliTelemetrySyncResponseSchema, { ok: true }, "Telemetry response");
const ok = parseArk(ApiCliTelemetryInstallResponseSchema, { ok: true }, "Telemetry response");
return json(ok);
} catch (error) {
const message = error instanceof Error ? error.message : "Telemetry failed";
@@ -253,7 +254,7 @@ async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
}
}
export const cliTelemetrySyncHttp = httpAction(cliTelemetrySyncHandler);
export const cliTelemetryInstallHttp = httpAction(cliTelemetryInstallHandler);
async function cliDeviceCodeHandler(ctx: ActionCtx, request: Request) {
if (request.method !== "POST") return text("Method not allowed", 405);
@@ -358,7 +359,6 @@ function parsePublishBody(body: unknown) {
displayName: parsed.displayName,
version: parsed.version,
changelog: parsed.changelog,
clawScanNote: parsed.clawScanNote?.trim() || undefined,
acceptLicenseTerms: parsed.acceptLicenseTerms,
tags,
source: parsed.source ?? undefined,
@@ -388,7 +388,7 @@ export const __handlers = {
cliUploadUrlHandler,
cliPublishHandler,
cliSkillDeleteHandler,
cliTelemetrySyncHandler,
cliTelemetryInstallHandler,
cliDeviceCodeHandler,
cliDeviceTokenHandler,
};
File diff suppressed because it is too large Load Diff
+114 -9
View File
@@ -1,8 +1,14 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from "vitest";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { formatUserFacingErrorMessage, resolveVersionTagsBatch } from "./httpApiV1/shared";
import {
formatUserFacingErrorMessage,
parseMultipartSkillScan,
resolveTagsBatch,
softDeleteErrorToResponse,
} from "./httpApiV1/shared";
function makeCtx() {
return {
@@ -28,13 +34,65 @@ describe("http API v1 shared helpers", () => {
).toBe("Publisher not found");
});
it("maps soft-delete validation failures to 400 with cleaned messages", async () => {
const response = softDeleteErrorToResponse(
"package",
new Error(
"[CONVEX M] [Request ID: abc] Server Error Called by client Uncaught ConvexError: Package name must be lowercase and npm-safe (example: @scope/name or plugin-name)",
),
{},
);
expect(response.status).toBe(400);
await expect(response.text()).resolves.toBe(
"Package name must be lowercase and npm-safe (example: @scope/name or plugin-name)",
);
});
it("maps reserved package route validation failures to 400 with cleaned messages", async () => {
const response = softDeleteErrorToResponse(
"package",
new Error(
'[CONVEX M] [Request ID: abc] Server Error Called by client Uncaught ConvexError: Package name "publish" is reserved for ClawHub routes. Use a scoped name or choose a different package name.',
),
{},
);
expect(response.status).toBe(400);
await expect(response.text()).resolves.toBe(
'Package name "publish" is reserved for ClawHub routes. Use a scoped name or choose a different package name.',
);
});
it("keeps unknown soft-delete failures generic 500s", async () => {
const response = softDeleteErrorToResponse("skill", new Error("boom"), {});
expect(response.status).toBe(500);
await expect(response.text()).resolves.toBe("Internal Server Error");
});
it("keeps unrelated reserved-word failures generic 500s", async () => {
const response = softDeleteErrorToResponse(
"package",
new Error("database reserved capacity exceeded"),
{},
);
expect(response.status).toBe(500);
await expect(response.text()).resolves.toBe("Internal Server Error");
});
it("resolves latest tags without reading version documents", async () => {
const ctx = makeCtx();
const versionId = "skillVersions:latest" as Id<"skillVersions">;
const skillId = "skills:demo" as Id<"skills">;
const result = await resolveVersionTagsBatch(ctx, [{ latest: versionId }], {} as never, [
{ _id: versionId, version: "2.0.0" },
]);
const result = await resolveTagsBatch(
ctx,
[{ latest: versionId }],
[{ _id: versionId, skillId, version: "2.0.0" }],
[skillId],
);
expect(result).toEqual([{ latest: "2.0.0" }]);
expect(ctx.runQuery).not.toHaveBeenCalled();
@@ -44,16 +102,63 @@ describe("http API v1 shared helpers", () => {
const ctx = makeCtx();
const latestId = "skillVersions:latest" as Id<"skillVersions">;
const stableId = "skillVersions:stable" as Id<"skillVersions">;
ctx.runQuery.mockResolvedValueOnce([{ _id: stableId, version: "1.5.0" }]);
const skillId = "skills:demo" as Id<"skills">;
ctx.runQuery.mockResolvedValueOnce([{ _id: stableId, skillId, version: "1.5.0" }]);
const result = await resolveVersionTagsBatch(
const result = await resolveTagsBatch(
ctx,
[{ latest: latestId, stable: stableId }],
{} as never,
[{ _id: latestId, version: "2.0.0" }],
[{ _id: latestId, skillId, version: "2.0.0" }],
[skillId],
);
expect(ctx.runQuery).toHaveBeenCalledWith({}, { versionIds: [stableId] });
expect(ctx.runQuery).toHaveBeenCalledWith(internal.skills.getVersionsByIdsInternal, {
versionIds: [stableId],
});
expect(result).toEqual([{ latest: "2.0.0", stable: "1.5.0" }]);
});
it("filters resolved skill tags by owning skill", async () => {
const ctx = makeCtx();
const otherId = "skillVersions:other" as Id<"skillVersions">;
const stableId = "skillVersions:stable" as Id<"skillVersions">;
const skillId = "skills:1" as Id<"skills">;
ctx.runQuery.mockResolvedValueOnce([
{ _id: otherId, skillId: "skills:other", version: "9.9.9" },
{ _id: stableId, skillId, version: "1.5.0" },
]);
const result = await resolveTagsBatch(
ctx,
[{ latest: otherId, stable: stableId }],
[{ _id: otherId, skillId: "skills:other" as Id<"skills">, version: "9.9.9" }],
[skillId],
);
expect(result).toEqual([{ stable: "1.5.0" }]);
});
it("validates skill scan multipart payloads before storing uploaded files", async () => {
const form = new FormData();
form.set("payload", JSON.stringify({ source: { kind: "upload" }, update: true }));
form.append("files", new Blob(["# Demo"], { type: "text/markdown" }), "SKILL.md");
const request = new Request("https://clawhub.ai/api/v1/skills/-/scan", {
method: "POST",
body: form,
});
const store = vi.fn();
const ctx = {
storage: {
store,
delete: vi.fn(),
},
} as unknown as ActionCtx;
await expect(
parseMultipartSkillScan(ctx, request, () => {
throw new Error("update is not valid for uploaded scans");
}),
).rejects.toThrow("update is not valid for uploaded scans");
expect(store).not.toHaveBeenCalled();
});
});
+28 -19
View File
@@ -1,6 +1,7 @@
import { httpAction } from "./functions";
import { verifyDocsSessionV1Handler } from "./httpApiV1/docsSessionV1";
import {
exportPluginsV1Handler,
listBundlePluginsV1Handler,
listCodePluginsV1Handler,
listPackagesV1Handler,
@@ -20,24 +21,28 @@ import {
publishSkillV1Handler,
resolveSkillVersionV1Handler,
searchSkillsV1Handler,
skillScanBatchStatusV1Handler,
skillScanBatchSubmitV1Handler,
skillScanGetRouterV1Handler,
skillScanSubmitV1Handler,
skillSecurityVerdictsV1Handler,
skillsDeleteRouterV1Handler,
skillsGetRouterV1Handler,
skillsPostRouterV1Handler,
} from "./httpApiV1/skillsV1";
import {
listSoulsV1Handler,
publishSoulV1Handler,
soulsDeleteRouterV1Handler,
soulsGetRouterV1Handler,
soulsPostRouterV1Handler,
} from "./httpApiV1/soulsV1";
import { starsDeleteRouterV1Handler, starsPostRouterV1Handler } from "./httpApiV1/starsV1";
import { transfersGetRouterV1Handler } from "./httpApiV1/transfersV1";
import { usersListV1Handler, usersPostRouterV1Handler } from "./httpApiV1/usersV1";
import {
banAppealContextV1Handler,
usersGetRouterV1Handler,
usersListV1Handler,
usersPostRouterV1Handler,
} from "./httpApiV1/usersV1";
import { whoamiV1Handler } from "./httpApiV1/whoamiV1";
export const listPackagesV1Http = httpAction(listPackagesV1Handler);
export const listPluginsV1Http = httpAction(listPluginsV1Handler);
export const exportPluginsV1Http = httpAction(exportPluginsV1Handler);
export const packagesGetRouterV1Http = httpAction(packagesGetRouterV1Handler);
export const packagesPostRouterV1Http = httpAction(packagesPostRouterV1Handler);
export const packagesDeleteRouterV1Http = httpAction(packagesDeleteRouterV1Handler);
@@ -55,27 +60,29 @@ export const resolveSkillVersionV1Http = httpAction(resolveSkillVersionV1Handler
export const listSkillsV1Http = httpAction(listSkillsV1Handler);
export const skillsGetRouterV1Http = httpAction(skillsGetRouterV1Handler);
export const publishSkillV1Http = httpAction(publishSkillV1Handler);
export const skillSecurityVerdictsV1Http = httpAction(skillSecurityVerdictsV1Handler);
export const skillScanSubmitV1Http = httpAction(skillScanSubmitV1Handler);
export const skillScanGetRouterV1Http = httpAction(skillScanGetRouterV1Handler);
export const skillScanBatchSubmitV1Http = httpAction(skillScanBatchSubmitV1Handler);
export const skillScanBatchStatusV1Http = httpAction(skillScanBatchStatusV1Handler);
export const skillsPostRouterV1Http = httpAction(skillsPostRouterV1Handler);
export const skillsDeleteRouterV1Http = httpAction(skillsDeleteRouterV1Handler);
export const exportSkillsV1Http = httpAction(exportSkillsV1Handler);
export const listSoulsV1Http = httpAction(listSoulsV1Handler);
export const soulsGetRouterV1Http = httpAction(soulsGetRouterV1Handler);
export const publishSoulV1Http = httpAction(publishSoulV1Handler);
export const soulsPostRouterV1Http = httpAction(soulsPostRouterV1Handler);
export const soulsDeleteRouterV1Http = httpAction(soulsDeleteRouterV1Handler);
export const starsPostRouterV1Http = httpAction(starsPostRouterV1Handler);
export const starsDeleteRouterV1Http = httpAction(starsDeleteRouterV1Handler);
export const transfersGetRouterV1Http = httpAction(transfersGetRouterV1Handler);
export const whoamiV1Http = httpAction(whoamiV1Handler);
export const usersGetRouterV1Http = httpAction(usersGetRouterV1Handler);
export const usersPostRouterV1Http = httpAction(usersPostRouterV1Handler);
export const usersListV1Http = httpAction(usersListV1Handler);
export const banAppealContextV1Http = httpAction(banAppealContextV1Handler);
export const __handlers = {
listPackagesV1Handler,
listPluginsV1Handler,
exportPluginsV1Handler,
packagesGetRouterV1Handler,
packagesPostRouterV1Handler,
packagesDeleteRouterV1Handler,
@@ -92,18 +99,20 @@ export const __handlers = {
listSkillsV1Handler,
skillsGetRouterV1Handler,
publishSkillV1Handler,
skillSecurityVerdictsV1Handler,
skillScanSubmitV1Handler,
skillScanGetRouterV1Handler,
skillScanBatchSubmitV1Handler,
skillScanBatchStatusV1Handler,
skillsPostRouterV1Handler,
skillsDeleteRouterV1Handler,
exportSkillsV1Handler,
listSoulsV1Handler,
soulsGetRouterV1Handler,
publishSoulV1Handler,
soulsPostRouterV1Handler,
soulsDeleteRouterV1Handler,
starsPostRouterV1Handler,
starsDeleteRouterV1Handler,
transfersGetRouterV1Handler,
whoamiV1Handler,
usersGetRouterV1Handler,
usersPostRouterV1Handler,
usersListV1Handler,
banAppealContextV1Handler,
};
File diff suppressed because it is too large Load Diff
+177 -56
View File
@@ -9,6 +9,7 @@ import { getPublishFileSizeError, MAX_PUBLISH_FILE_BYTES } from "../lib/publishL
import { isMacJunkPath } from "../lib/skills";
export const MAX_RAW_FILE_BYTES = 200 * 1024;
const DEFAULT_PUBLIC_SITE_URL = "https://clawhub.ai";
const SAFE_TEXT_FILE_CSP =
"default-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'";
@@ -88,6 +89,65 @@ export async function parseJsonPayload(request: Request, headers: HeadersInit) {
}
}
function normalizeOrigin(value: string | null | undefined) {
const trimmed = value?.trim();
if (!trimmed) return null;
try {
return new URL(trimmed).origin;
} catch {
return null;
}
}
function firstForwardedValue(value: string | null) {
return value?.split(",")[0]?.trim() || null;
}
function isProductionDeployment() {
const deployment = process.env.CONVEX_DEPLOYMENT?.trim() ?? "";
return deployment.startsWith("prod:") || deployment.includes("production");
}
function isTrustedForwardedHost(value: string) {
try {
const hostname = new URL(`https://${value}`).hostname.toLowerCase();
return (
hostname === "clawhub.ai" ||
hostname === "www.clawhub.ai" ||
hostname === "localhost" ||
hostname === "127.0.0.1" ||
hostname === "0.0.0.0"
);
} catch {
return false;
}
}
export function publicApiOrigin(request: Request) {
const configured = normalizeOrigin(process.env.SITE_URL ?? process.env.VITE_SITE_URL);
if (configured) return configured;
const forwardedHost = firstForwardedValue(request.headers.get("x-forwarded-host"));
if (
forwardedHost &&
!forwardedHost.endsWith(".convex.site") &&
isTrustedForwardedHost(forwardedHost)
) {
const forwardedProto =
firstForwardedValue(request.headers.get("x-forwarded-proto")) ??
firstForwardedValue(request.headers.get("x-forwarded-protocol")) ??
"https";
const proto = forwardedProto === "http" ? "http" : "https";
return `${proto}://${forwardedHost}`;
}
const requestUrl = new URL(request.url);
if (isProductionDeployment() && requestUrl.hostname.endsWith(".convex.site")) {
return DEFAULT_PUBLIC_SITE_URL;
}
return requestUrl.origin;
}
export async function requireApiTokenUserOrResponse(
ctx: ActionCtx,
request: Request,
@@ -139,42 +199,12 @@ export function toOptionalNumber(value: string | null) {
return Number.isFinite(parsed) ? parsed : undefined;
}
/**
* Batch resolve soul version tags to version strings.
* Collects all version IDs, fetches them in a single query, then maps back.
* Reduces N sequential queries to 1 batch query.
*/
export async function resolveSoulTagsBatch(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<"soulVersions">>>,
latestVersions?: Array<LatestVersionTag<"soulVersions">>,
): Promise<Array<Record<string, string>>> {
return resolveVersionTagsBatch(
ctx,
tagsList,
internal.souls.getVersionsByIdsInternal,
latestVersions,
);
}
export async function resolveTagsBatch(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<"skillVersions">>>,
latestVersions?: Array<LatestVersionTag<"skillVersions">>,
): Promise<Array<Record<string, string>>> {
return resolveVersionTagsBatch(
ctx,
tagsList,
internal.skills.getVersionsByIdsInternal,
latestVersions,
);
}
type LatestVersionTag<TTable extends "skillVersions" | "soulVersions"> =
type LatestVersionTag =
| {
_id: Id<TTable>;
_id: Id<"skillVersions">;
version?: string;
softDeletedAt?: unknown;
skillId?: Id<"skills">;
}
| null
| undefined;
@@ -187,18 +217,24 @@ type LatestVersionTag<TTable extends "skillVersions" | "soulVersions"> =
* - Uses `internal.*` queries to avoid expanding the public Convex API surface.
* - Sorts ids for stable query args (helps caching/log diffs).
*/
export async function resolveVersionTagsBatch<TTable extends "skillVersions" | "soulVersions">(
export async function resolveTagsBatch(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<TTable>>>,
getVersionsByIdsQuery: unknown,
latestVersions?: Array<LatestVersionTag<TTable>>,
tagsList: Array<Record<string, Id<"skillVersions">>>,
latestVersions: Array<LatestVersionTag>,
skillIds: Array<Id<"skills">>,
): Promise<Array<Record<string, string>>> {
const allVersionIds = new Set<Id<TTable>>();
const allVersionIds = new Set<Id<"skillVersions">>();
const preResolvedTags = tagsList.map((tags, idx) => {
const resolved: Record<string, string> = {};
const latest = latestVersions?.[idx];
const latest = latestVersions[idx];
const skillId = skillIds[idx];
for (const [tag, versionId] of Object.entries(tags)) {
if (latest?._id === versionId && latest.version && !latest.softDeletedAt) {
if (
latest?._id === versionId &&
latest.version &&
!latest.softDeletedAt &&
latest.skillId === skillId
) {
resolved[tag] = latest.version;
} else {
allVersionIds.add(versionId);
@@ -211,25 +247,28 @@ export async function resolveVersionTagsBatch<TTable extends "skillVersions" | "
return preResolvedTags;
}
const versionIds = [...allVersionIds].sort() as Array<Id<TTable>>;
const versionIds = [...allVersionIds].sort();
const versions =
((await ctx.runQuery(getVersionsByIdsQuery as never, { versionIds } as never)) as Array<{
_id: Id<TTable>;
version: string;
softDeletedAt?: unknown;
}> | null) ?? [];
(await ctx.runQuery(internal.skills.getVersionsByIdsInternal, { versionIds })) ?? [];
const versionMap = new Map<Id<TTable>, string>();
const versionMap = new Map<
Id<"skillVersions">,
{
version: string;
skillId?: Id<"skills">;
}
>();
for (const v of versions) {
if (!v?.softDeletedAt) versionMap.set(v._id, v.version);
if (!v?.softDeletedAt) versionMap.set(v._id, { version: v.version, skillId: v.skillId });
}
return tagsList.map((tags, idx) => {
const resolved = { ...preResolvedTags[idx] };
const skillId = skillIds[idx];
for (const [tag, versionId] of Object.entries(tags)) {
if (resolved[tag]) continue;
const version = versionMap.get(versionId);
if (version) resolved[tag] = version;
if (version?.skillId === skillId) resolved[tag] = version.version;
}
return resolved;
});
@@ -314,7 +353,6 @@ export async function parseMultipartPublish(
...(typeof payload.migrateOwner === "boolean" ? { migrateOwner: payload.migrateOwner } : {}),
version: payload.version,
changelog: typeof payload.changelog === "string" ? payload.changelog : "",
...(typeof payload.clawScanNote === "string" ? { clawScanNote: payload.clawScanNote } : {}),
...(hasAcceptLicenseTerms ? { acceptLicenseTerms: payload.acceptLicenseTerms } : {}),
tags: Array.isArray(payload.tags) ? payload.tags : undefined,
...(payload.source ? { source: payload.source } : {}),
@@ -325,6 +363,71 @@ export async function parseMultipartPublish(
return parsePublishBody(body);
}
export async function parseMultipartSkillScan(
ctx: ActionCtx,
request: Request,
validatePayload?: (payload: Record<string, unknown>) => Record<string, unknown>,
): Promise<{
payload: Record<string, unknown>;
files: Array<{
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
}>;
}> {
const form = await request.formData();
const payloadRaw = form.get("payload");
if (!payloadRaw || typeof payloadRaw !== "string") {
throw new Error("Missing payload");
}
let payload: Record<string, unknown>;
try {
payload = JSON.parse(payloadRaw) as Record<string, unknown>;
} catch {
throw new Error("Invalid JSON payload");
}
const validatedPayload = validatePayload ? validatePayload(payload) : payload;
const fileEntries = form
.getAll("files")
.map((entry) => toFileLike(entry))
.filter((file): file is FileLikeEntry => Boolean(file))
.filter((file) => !isMacJunkPath(file.name));
if (fileEntries.length === 0) throw new Error("files required");
if (!fileEntries.some((file) => file.name.trim().toLowerCase() === "skill.md")) {
throw new Error("SKILL.md required");
}
const oversized = fileEntries.find((file) => file.size > MAX_PUBLISH_FILE_BYTES);
if (oversized) throw new Error(getPublishFileSizeError(oversized.name));
const files: Array<{
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
}> = [];
try {
for (const file of fileEntries) {
const path = file.name;
const size = file.size;
const contentType = file.type || undefined;
const buffer = new Uint8Array(await file.arrayBuffer());
const sha256 = await sha256Hex(buffer);
const storageId = await ctx.storage.store(file as Blob);
files.push({ path, size, storageId, sha256, contentType });
}
} catch (error) {
await Promise.allSettled(files.map((file) => ctx.storage.delete(file.storageId)));
throw error;
}
return { payload: validatedPayload, files };
}
export function parsePublishBody(body: unknown) {
const parsed = parseArk(CliPublishRequestSchema, body, "Publish payload");
if (parsed.files.length === 0) throw new Error("files required");
@@ -352,22 +455,40 @@ export function parsePublishBody(body: unknown) {
};
}
// Substrings that indicate user-input validation failures from the underlying
// mutations (e.g. `normalizePackageName` ConvexErrors). These are surfaced as
// 400s with the cleaned message so CLI/API clients can see the actual reason
// instead of an opaque 500.
const SOFT_DELETE_BAD_REQUEST_HINTS = [
"slug required",
"package name required",
"package name must be",
"must be lowercase",
"npm-safe",
"reserved for clawhub routes",
"version required",
] as const;
export function softDeleteErrorToResponse(
entity: "skill" | "soul" | "package",
entity: "skill" | "package",
error: unknown,
headers: HeadersInit,
) {
const message = error instanceof Error ? error.message : `${entity} delete failed`;
const lower = message.toLowerCase();
const rawMessage = error instanceof Error ? error.message : `${entity} delete failed`;
const cleaned = cleanUserFacingErrorMessage(rawMessage) || rawMessage;
const lower = cleaned.toLowerCase();
if (lower.includes("unauthorized"))
return text(formatAuthzMessage(error, "Unauthorized"), 401, headers);
if (lower.includes("forbidden"))
return text(formatAuthzMessage(error, "Forbidden"), 403, headers);
if (lower.includes("not found")) return text(message, 404, headers);
if (lower.includes("slug required")) return text("Slug required", 400, headers);
if (lower.includes("not found")) return text(cleaned, 404, headers);
if (SOFT_DELETE_BAD_REQUEST_HINTS.some((hint) => lower.includes(hint))) {
return text(cleaned, 400, headers);
}
// Unknown: server-side failure. Keep body generic.
// Unknown: server-side failure. Keep the body generic; only known
// user-input validation failures above surface the cleaned mutation message.
return text("Internal Server Error", 500, headers);
}
+1251 -143
View File
File diff suppressed because it is too large Load Diff
-352
View File
@@ -1,352 +0,0 @@
import { api, internal } from "../_generated/api";
import type { Doc, Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { requireApiTokenUser } from "../lib/apiTokenAuth";
import { applyRateLimit } from "../lib/httpRateLimit";
import { publishSoulVersionForUser } from "../souls";
import {
MAX_RAW_FILE_BYTES,
getPathSegments,
json,
parseMultipartPublish,
parsePublishBody,
requireApiTokenUserOrResponse,
resolveSoulTagsBatch,
safeTextFileResponse,
softDeleteErrorToResponse,
text,
toOptionalNumber,
} from "./shared";
type ListSoulsResult = {
items: Array<{
soul: {
_id: Id<"souls">;
slug: string;
displayName: string;
summary?: string;
tags: Record<string, Id<"soulVersions">>;
stats: unknown;
createdAt: number;
updatedAt: number;
latestVersionId?: Id<"soulVersions">;
};
latestVersion: { version: string; createdAt: number; changelog: string } | null;
}>;
nextCursor: string | null;
};
type GetSoulBySlugResult = {
soul: {
_id: Id<"souls">;
slug: string;
displayName: string;
summary?: string;
tags: Record<string, Id<"soulVersions">>;
stats: unknown;
createdAt: number;
updatedAt: number;
} | null;
latestVersion: PublicSoulVersion | null;
owner: { handle?: string; displayName?: string; image?: string } | null;
} | null;
type ListSoulVersionsResult = {
items: PublicSoulVersion[];
nextCursor: string | null;
};
type PublicSoulVersion = Pick<
Doc<"soulVersions">,
| "_id"
| "_creationTime"
| "soulId"
| "version"
| "fingerprint"
| "changelog"
| "changelogSource"
| "createdBy"
| "createdAt"
| "softDeletedAt"
> & {
files: Array<{
path: string;
size: number;
sha256: string;
contentType?: string;
}>;
parsed?: {
clawdis?: Doc<"soulVersions">["parsed"]["clawdis"];
};
};
type SoulFile = PublicSoulVersion["files"][number];
export async function listSoulsV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const url = new URL(request.url);
const limit = toOptionalNumber(url.searchParams.get("limit"));
const cursor = url.searchParams.get("cursor")?.trim() || undefined;
const result = (await ctx.runQuery(api.souls.listPublicPage, {
limit,
cursor,
})) as ListSoulsResult;
// Batch resolve all tags in a single query instead of N queries
const resolvedTagsList = await resolveSoulTagsBatch(
ctx,
result.items.map((item) => item.soul.tags),
);
const items = result.items.map((item, idx) => ({
slug: item.soul.slug,
displayName: item.soul.displayName,
summary: item.soul.summary ?? null,
tags: resolvedTagsList[idx],
stats: item.soul.stats,
createdAt: item.soul.createdAt,
updatedAt: item.soul.updatedAt,
latestVersion: item.latestVersion
? {
version: item.latestVersion.version,
createdAt: item.latestVersion.createdAt,
changelog: item.latestVersion.changelog,
}
: null,
}));
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers);
}
export async function soulsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/souls/");
if (segments.length === 0) return text("Missing slug", 400, rate.headers);
const slug = segments[0]?.trim().toLowerCase() ?? "";
const second = segments[1];
const third = segments[2];
if (segments.length === 1) {
const result = (await ctx.runQuery(api.souls.getBySlug, { slug })) as GetSoulBySlugResult;
if (!result?.soul) return text("Soul not found", 404, rate.headers);
const [tags] = await resolveSoulTagsBatch(ctx, [result.soul.tags]);
return json(
{
soul: {
slug: result.soul.slug,
displayName: result.soul.displayName,
summary: result.soul.summary ?? null,
tags,
stats: result.soul.stats,
createdAt: result.soul.createdAt,
updatedAt: result.soul.updatedAt,
},
latestVersion: result.latestVersion
? {
version: result.latestVersion.version,
createdAt: result.latestVersion.createdAt,
changelog: result.latestVersion.changelog,
}
: null,
owner: result.owner
? {
handle: result.owner.handle ?? null,
displayName: result.owner.displayName ?? null,
image: result.owner.image ?? null,
}
: null,
},
200,
rate.headers,
);
}
if (second === "versions" && segments.length === 2) {
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug });
if (!soul || soul.softDeletedAt) return text("Soul not found", 404, rate.headers);
const url = new URL(request.url);
const limit = toOptionalNumber(url.searchParams.get("limit"));
const cursor = url.searchParams.get("cursor")?.trim() || undefined;
const result = (await ctx.runQuery(api.souls.listVersionsPage, {
soulId: soul._id,
limit,
cursor,
})) as ListSoulVersionsResult;
const items = result.items
.filter((version) => !version.softDeletedAt)
.map((version) => ({
version: version.version,
createdAt: version.createdAt,
changelog: version.changelog,
changelogSource: version.changelogSource ?? null,
}));
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers);
}
if (second === "versions" && third && segments.length === 3) {
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug });
if (!soul || soul.softDeletedAt) return text("Soul not found", 404, rate.headers);
const version = await ctx.runQuery(api.souls.getVersionBySoulAndVersion, {
soulId: soul._id,
version: third,
});
if (!version) return text("Version not found", 404, rate.headers);
if (version.softDeletedAt) return text("Version not available", 410, rate.headers);
return json(
{
soul: { slug: soul.slug, displayName: soul.displayName },
version: {
version: version.version,
createdAt: version.createdAt,
changelog: version.changelog,
changelogSource: version.changelogSource ?? null,
files: version.files.map((file: SoulFile) => ({
path: file.path,
size: file.size,
sha256: file.sha256,
contentType: file.contentType ?? null,
})),
},
},
200,
rate.headers,
);
}
if (second === "file" && segments.length === 2) {
const url = new URL(request.url);
const path = url.searchParams.get("path")?.trim();
if (!path) return text("Missing path", 400, rate.headers);
const versionParam = url.searchParams.get("version")?.trim();
const tagParam = url.searchParams.get("tag")?.trim();
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug });
if (!soul || soul.softDeletedAt) return text("Soul not found", 404, rate.headers);
let version = soul.latestVersionId
? await ctx.runQuery(internal.souls.getVersionByIdInternal, {
versionId: soul.latestVersionId,
})
: null;
if (versionParam) {
version = await ctx.runQuery(internal.souls.getVersionBySoulAndVersionInternal, {
soulId: soul._id,
version: versionParam,
});
} else if (tagParam) {
const versionId = soul.tags[tagParam];
if (versionId) {
version = await ctx.runQuery(internal.souls.getVersionByIdInternal, { versionId });
}
}
if (!version) return text("Version not found", 404, rate.headers);
if (version.softDeletedAt) return text("Version not available", 410, rate.headers);
const normalized = path.trim();
const normalizedLower = normalized.toLowerCase();
const file =
version.files.find((entry) => entry.path === normalized) ??
version.files.find((entry) => entry.path.toLowerCase() === normalizedLower);
if (!file) return text("File not found", 404, rate.headers);
if (file.size > MAX_RAW_FILE_BYTES) return text("File exceeds 200KB limit", 413, rate.headers);
const blob = await ctx.storage.get(file.storageId);
if (!blob) return text("File missing in storage", 410, rate.headers);
const textContent = await blob.text();
void ctx.runMutation(internal.soulDownloads.incrementInternal, { soulId: soul._id });
return safeTextFileResponse({
textContent,
path: file.path,
contentType: file.contentType ?? undefined,
sha256: file.sha256,
size: file.size,
headers: rate.headers,
});
}
return text("Not found", 404, rate.headers);
}
export async function publishSoulV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const auth = await requireApiTokenUserOrResponse(ctx, request, rate.headers);
if (!auth.ok) return auth.response;
const contentType = request.headers.get("content-type") ?? "";
try {
if (contentType.includes("application/json")) {
const body = await request.json();
const payload = parsePublishBody(body);
const result = await publishSoulVersionForUser(ctx, auth.userId, payload);
return json({ ok: true, ...result }, 200, rate.headers);
}
if (contentType.includes("multipart/form-data")) {
const payload = await parseMultipartPublish(ctx, request);
const result = await publishSoulVersionForUser(ctx, auth.userId, payload);
return json({ ok: true, ...result }, 200, rate.headers);
}
} catch (error) {
const message = error instanceof Error ? error.message : "Publish failed";
return text(message, 400, rate.headers);
}
return text("Unsupported content type", 415, rate.headers);
}
export async function soulsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/souls/");
if (segments.length !== 2 || segments[1] !== "undelete") {
return text("Not found", 404, rate.headers);
}
const slug = segments[0]?.trim().toLowerCase() ?? "";
try {
const { userId } = await requireApiTokenUser(ctx, request);
await ctx.runMutation(internal.souls.setSoulSoftDeletedInternal, {
userId,
slug,
deleted: false,
});
return json({ ok: true }, 200, rate.headers);
} catch (error) {
return softDeleteErrorToResponse("soul", error, rate.headers);
}
}
export async function soulsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/souls/");
if (segments.length !== 1) return text("Not found", 404, rate.headers);
const slug = segments[0]?.trim().toLowerCase() ?? "";
try {
const { userId } = await requireApiTokenUser(ctx, request);
await ctx.runMutation(internal.souls.setSoulSoftDeletedInternal, {
userId,
slug,
deleted: true,
});
return json({ ok: true }, 200, rate.headers);
} catch (error) {
return softDeleteErrorToResponse("soul", error, rate.headers);
}
}
+425 -57
View File
@@ -1,6 +1,7 @@
import { api, internal } from "../_generated/api";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { ADMIN_ONE_OFF_TEMPLATE, buildAdminOneOffEmail } from "../lib/emails";
import { applyRateLimit } from "../lib/httpRateLimit";
import {
getPathSegments,
@@ -12,11 +13,23 @@ import {
toOptionalNumber,
} from "./shared";
const DEFAULT_CLAWHUB_NOREPLY_FROM = "ClawHub <noreply@notifications.openclaw.ai>";
const usersV1InternalRefs = internal as unknown as {
publishers: {
addOfficialPublisherInternal: unknown;
deleteEmptyOrgPublisherInternal: unknown;
listOfficialPublishersInternal: unknown;
removeOrgPublisherMemberInternal: unknown;
removeOfficialPublisherInternal: unknown;
};
users: {
getBanAppealContextByGitHubProviderAccountIdInternal: unknown;
getByHandleInternal: unknown;
remediateAutobansInternal: unknown;
recordStaffEmailAttemptAuditInternal: unknown;
recordStaffEmailSentAuditInternal: unknown;
reclassifyBanInternal: unknown;
unbanUserForBanAppealServiceInternal: unknown;
};
};
@@ -36,6 +49,29 @@ async function runUsersV1MutationRef<T>(
return (await ctx.runMutation(ref as never, args as never)) as T;
}
function getBanAppealsServiceToken() {
return process.env.CLAWHUB_BAN_APPEALS_TOKEN?.trim() || "";
}
function readBearerToken(request: Request) {
return (
request.headers
.get("authorization")
?.match(/^Bearer\s+(.+)$/i)?.[1]
?.trim() ?? ""
);
}
function requireBanAppealsServiceOrResponse(request: Request, headers: HeadersInit) {
const expected = getBanAppealsServiceToken();
if (!expected)
return { ok: false as const, response: text("Ban appeals service unavailable", 503, headers) };
if (readBearerToken(request) !== expected) {
return { ok: false as const, response: text("Unauthorized", 401, headers) };
}
return { ok: true as const };
}
export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
@@ -50,11 +86,15 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
action !== "unban" &&
action !== "role" &&
action !== "restore" &&
action !== "remediate-autobans" &&
action !== "reclassify-ban" &&
action !== "ban-appeal-unban" &&
action !== "reclaim" &&
action !== "reserve" &&
action !== "publisher"
action !== "email" &&
action !== "publisher" &&
action !== "publisher-delete" &&
action !== "publisher-official" &&
action !== "publisher-member"
) {
return text("Not found", 404, rate.headers);
}
@@ -63,6 +103,10 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
if (!payloadResult.ok) return payloadResult.response;
const payload = payloadResult.payload;
if (action === "ban-appeal-unban") {
return handleBanAppealUnban(ctx, request, payload, rate.headers);
}
const authResult = await requireApiTokenUserOrResponse(ctx, request, rate.headers);
if (!authResult.ok) return authResult.response;
const actorUserId = authResult.userId;
@@ -75,12 +119,6 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
return handleAdminRestore(ctx, request, payload, actorUserId, rate.headers);
}
if (action === "remediate-autobans") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminRemediateAutobans(ctx, payload, actorUserId, rate.headers);
}
if (action === "reclassify-ban") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
@@ -99,12 +137,36 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
return handleAdminReserve(ctx, payload, actorUserId, rate.headers);
}
if (action === "email") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminStaffEmail(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminEnsurePublisher(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher-delete") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminDeletePublisher(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher-official") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminOfficialPublisherPost(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher-member") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminRemovePublisherMember(ctx, payload, actorUserId, rate.headers);
}
const handleRaw = typeof payload.handle === "string" ? payload.handle.trim() : "";
const userIdRaw = typeof payload.userId === "string" ? payload.userId.trim() : "";
const reasonRaw = typeof payload.reason === "string" ? payload.reason.trim() : "";
@@ -201,6 +263,206 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
}
}
async function handleAdminDeletePublisher(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
const reason = typeof payload.reason === "string" ? payload.reason.trim() : "";
const dryRun = payload.dryRun !== false;
if (!handle) return text("Missing handle", 400, headers);
if (!reason) return text("Missing reason", 400, headers);
if (reason.length > 500) return text("Reason too long (max 500 chars)", 400, headers);
try {
const result = await runUsersV1MutationRef(
ctx,
usersV1InternalRefs.publishers.deleteEmptyOrgPublisherInternal,
{
actorUserId,
handle,
reason,
dryRun,
},
);
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Publisher delete failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, headers);
}
if (message.toLowerCase().includes("not found")) {
return text(message, 404, headers);
}
return text(message, 400, headers);
}
}
async function handleAdminStaffEmail(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const toEmail = typeof payload.toEmail === "string" ? payload.toEmail.trim().toLowerCase() : "";
const userHandle =
typeof payload.userHandle === "string"
? payload.userHandle.trim().replace(/^@+/, "").toLowerCase()
: "";
const suppliedRecipientHandle =
typeof payload.recipientHandle === "string"
? payload.recipientHandle.trim().replace(/^@+/, "").toLowerCase()
: "";
const subject = typeof payload.subject === "string" ? payload.subject.trim() : "";
const title = typeof payload.title === "string" ? payload.title.trim() : "";
const body = typeof payload.body === "string" ? payload.body.trim() : "";
const template =
typeof payload.template === "string" && payload.template.trim()
? payload.template.trim()
: ADMIN_ONE_OFF_TEMPLATE;
const primaryActionLabel =
typeof payload.primaryActionLabel === "string" ? payload.primaryActionLabel.trim() : "";
const primaryActionUrl =
typeof payload.primaryActionUrl === "string" ? payload.primaryActionUrl.trim() : "";
const confirmUserRequest = payload.confirmUserRequest === true;
const confirmUserSignoff = payload.confirmUserSignoff === true;
if (template !== ADMIN_ONE_OFF_TEMPLATE) {
return text(`Unsupported staff email template: ${template || "missing"}`, 400, headers);
}
if (toEmail && userHandle) return text("Pass toEmail or userHandle, not both", 400, headers);
if (!toEmail && !userHandle) return text("Missing toEmail or userHandle", 400, headers);
if (toEmail && !/^[^@\s]+@[^@\s]+\.[^@\s]+$/.test(toEmail)) {
return text("Invalid toEmail", 400, headers);
}
if (!subject) return text("Missing subject", 400, headers);
if (subject.length > 200) return text("Subject too long (max 200 chars)", 400, headers);
if (title.length > 160) return text("Title too long (max 160 chars)", 400, headers);
if (suppliedRecipientHandle.length > 80) {
return text("Recipient handle too long (max 80 chars)", 400, headers);
}
if (!body) return text("Missing body", 400, headers);
if (body.length > 20_000) return text("Body too long (max 20000 chars)", 400, headers);
if ((primaryActionLabel && !primaryActionUrl) || (!primaryActionLabel && primaryActionUrl)) {
return text("Pass primaryActionLabel and primaryActionUrl together", 400, headers);
}
if (primaryActionLabel.length > 80) {
return text("Primary action label too long (max 80 chars)", 400, headers);
}
if (primaryActionUrl.length > 2_000) {
return text("Primary action URL too long (max 2000 chars)", 400, headers);
}
if (primaryActionUrl) {
let parsedUrl: URL;
try {
parsedUrl = new URL(primaryActionUrl);
} catch {
return text("Primary action URL must be an http(s) URL", 400, headers);
}
if (parsedUrl.protocol !== "http:" && parsedUrl.protocol !== "https:") {
return text("Primary action URL must be an http(s) URL", 400, headers);
}
}
if (!confirmUserRequest || !confirmUserSignoff) {
return text(
"Staff email requires explicit user request and user sign-off on final recipient, subject, and body.",
400,
headers,
);
}
let recipientEmail = toEmail;
let recipientUserId: Id<"users"> | undefined;
let recipientHandle: string | null | undefined = suppliedRecipientHandle || undefined;
if (userHandle) {
const user = await runUsersV1QueryRef<{
_id?: Id<"users">;
handle?: string | null;
email?: string | null;
} | null>(ctx, usersV1InternalRefs.users.getByHandleInternal, { handle: userHandle });
if (!user?._id) return text("User not found", 404, headers);
if (!user.email?.trim()) return text("User has no email address", 400, headers);
recipientEmail = user.email.trim().toLowerCase();
recipientUserId = user._id;
recipientHandle = user.handle ?? userHandle;
}
const apiKey = process.env.RESEND_API_KEY?.trim();
if (!apiKey) return text("RESEND_API_KEY is not configured", 500, headers);
const from = process.env.CLAWHUB_NOREPLY_FROM?.trim() || DEFAULT_CLAWHUB_NOREPLY_FROM;
const emailAudit = await runUsersV1MutationRef<{ auditLogId: Id<"auditLogs"> }>(
ctx,
usersV1InternalRefs.users.recordStaffEmailAttemptAuditInternal,
{
actorUserId,
toEmail: recipientEmail,
...(recipientUserId ? { recipientUserId } : {}),
...(recipientHandle ? { recipientHandle } : {}),
subject,
template,
},
);
const email = await buildAdminOneOffEmail({
recipientHandle: recipientHandle ?? (userHandle || undefined),
subject,
...(title ? { title } : {}),
body,
...(primaryActionLabel && primaryActionUrl ? { primaryActionLabel, primaryActionUrl } : {}),
});
const response = await fetch("https://api.resend.com/emails", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
from,
to: [recipientEmail],
subject: email.subject,
text: email.text,
html: email.html,
}),
});
if (!response.ok) {
const errorText = await response.text().catch(() => "");
return text(
`Email provider failed (${response.status})${errorText ? `: ${errorText}` : ""}`,
502,
headers,
);
}
const providerJson = (await response.json().catch(() => null)) as { id?: unknown } | null;
const providerId = typeof providerJson?.id === "string" ? providerJson.id : null;
await runUsersV1MutationRef(ctx, usersV1InternalRefs.users.recordStaffEmailSentAuditInternal, {
actorUserId,
auditLogId: emailAudit.auditLogId,
providerId,
});
return json(
{
ok: true,
sent: true,
recipient: {
email: recipientEmail,
...(recipientUserId ? { userId: recipientUserId } : {}),
...(recipientHandle ? { handle: recipientHandle } : {}),
},
subject,
template,
providerId,
},
200,
headers,
);
}
async function handleAdminReclassifyBan(
ctx: ActionCtx,
payload: unknown,
@@ -253,62 +515,34 @@ async function handleAdminReclassifyBan(
}
}
async function handleAdminRemediateAutobans(
ctx: ActionCtx,
payload: unknown,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const body = payload && typeof payload === "object" ? (payload as Record<string, unknown>) : {};
const handle = typeof body.handle === "string" ? body.handle.trim() : "";
const userId = typeof body.userId === "string" ? body.userId.trim() : "";
const reason = typeof body.reason === "string" ? body.reason.trim() : "";
const since = typeof body.since === "string" ? body.since.trim() : "";
const cursor = typeof body.cursor === "string" ? body.cursor.trim() : "";
const dryRun = body.dryRun !== false;
const limit =
typeof body.limit === "number"
? body.limit
: typeof body.limit === "string" || body.limit === null
? toOptionalNumber(body.limit)
: undefined;
export async function usersGetRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
if (handle && userId) return text("Pass handle or userId, not both", 400, headers);
if (reason && reason.length > 500) {
return text("Reason too long (max 500 chars)", 400, headers);
}
if (since && Number.isNaN(Date.parse(since))) {
return text("Invalid since date", 400, headers);
}
if (limit !== undefined && (!Number.isFinite(limit) || limit < 1)) {
return text("Invalid limit", 400, headers);
const segments = getPathSegments(request, "/api/v1/users/");
if (segments.length !== 1 || segments[0] !== "publisher-official") {
return text("Not found", 404, rate.headers);
}
const authResult = await requireApiTokenUserOrResponse(ctx, request, rate.headers);
if (!authResult.ok) return authResult.response;
const admin = requireAdminOrResponse(authResult.user, rate.headers);
if (!admin.ok) return admin.response;
try {
const result = await runUsersV1MutationRef(
const result = await runUsersV1QueryRef(
ctx,
usersV1InternalRefs.users.remediateAutobansInternal,
{
actorUserId,
...(userId ? { targetUserId: userId as Id<"users"> } : {}),
...(handle ? { handle } : {}),
dryRun,
...(reason ? { reason } : {}),
...(since ? { since } : {}),
...(cursor ? { cursor } : {}),
...(limit !== undefined ? { limit } : {}),
},
usersV1InternalRefs.publishers.listOfficialPublishersInternal,
{ actorUserId: authResult.userId },
);
return json(result, 200, headers);
return json(result, 200, rate.headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Autoban remediation failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, headers);
const message = error instanceof Error ? error.message : "Official publisher list failed";
if (message.toLowerCase().includes("forbidden")) return text("Forbidden", 403, rate.headers);
if (message.toLowerCase().includes("unauthorized")) {
return text("Unauthorized", 401, rate.headers);
}
if (message.toLowerCase().includes("not found")) {
return text(message, 404, headers);
}
return text(message, 400, headers);
return text(message, 400, rate.headers);
}
}
@@ -492,6 +726,42 @@ async function handleAdminReserve(
return json({ ok: true, results, succeeded, failed }, 200, headers);
}
async function handleAdminOfficialPublisherPost(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const action = typeof payload.action === "string" ? payload.action.trim().toLowerCase() : "";
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
const reason = typeof payload.reason === "string" ? payload.reason.trim() : "";
if (action !== "add" && action !== "remove") return text("Invalid action", 400, headers);
if (!handle) return text("Missing handle", 400, headers);
if (!reason) return text("Missing reason", 400, headers);
if (reason.length > 500) return text("Reason too long (max 500 chars)", 400, headers);
try {
const result = await runUsersV1MutationRef(
ctx,
action === "add"
? usersV1InternalRefs.publishers.addOfficialPublisherInternal
: usersV1InternalRefs.publishers.removeOfficialPublisherInternal,
{
actorUserId,
handle,
reason,
},
);
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Official publisher update failed";
if (message.toLowerCase().includes("forbidden")) return text("Forbidden", 403, headers);
if (message.toLowerCase().includes("unauthorized")) return text("Unauthorized", 401, headers);
if (message.toLowerCase().includes("not found")) return text(message, 404, headers);
return text(message, 400, headers);
}
}
async function handleAdminEnsurePublisher(
ctx: ActionCtx,
payload: Record<string, unknown>,
@@ -538,6 +808,104 @@ async function handleAdminEnsurePublisher(
}
}
async function handleBanAppealUnban(
ctx: ActionCtx,
request: Request,
payload: Record<string, unknown>,
headers: HeadersInit,
) {
const service = requireBanAppealsServiceOrResponse(request, headers);
if (!service.ok) return service.response;
const targetUserIdRaw = typeof payload.userId === "string" ? payload.userId.trim() : "";
if (!targetUserIdRaw) return text("Missing userId", 400, headers);
const reasonRaw = typeof payload.reason === "string" ? payload.reason.trim() : "";
const reviewerDiscordId =
typeof payload.reviewerDiscordId === "string" ? payload.reviewerDiscordId.trim() : "";
const reason = reasonRaw || "Ban appeal accepted";
if (reason.length > 500) return text("Reason too long (max 500 chars)", 400, headers);
if (!reviewerDiscordId) return text("Missing reviewerDiscordId", 400, headers);
try {
const result = await runUsersV1MutationRef(
ctx,
usersV1InternalRefs.users.unbanUserForBanAppealServiceInternal,
{
targetUserId: targetUserIdRaw as Id<"users">,
reason,
reviewerDiscordId,
},
);
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Ban appeal unban failed";
if (message.toLowerCase().includes("forbidden")) return text("Forbidden", 403, headers);
if (message.toLowerCase().includes("not found")) return text(message, 404, headers);
return text(message, 400, headers);
}
}
export async function banAppealContextV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const service = requireBanAppealsServiceOrResponse(request, rate.headers);
if (!service.ok) return service.response;
const providerAccountId = new URL(request.url).searchParams
.get("githubProviderAccountId")
?.trim();
if (!providerAccountId) return text("Missing githubProviderAccountId", 400, rate.headers);
try {
const result = await runUsersV1QueryRef(
ctx,
usersV1InternalRefs.users.getBanAppealContextByGitHubProviderAccountIdInternal,
{ providerAccountId },
);
return json(result, 200, rate.headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Ban appeal context failed";
return text(message, 400, rate.headers);
}
}
async function handleAdminRemovePublisherMember(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
const memberHandle =
typeof payload.memberHandle === "string" ? payload.memberHandle.trim().toLowerCase() : "";
if (!handle) return text("Missing handle", 400, headers);
if (!memberHandle) return text("Missing memberHandle", 400, headers);
try {
const result = await runUsersV1MutationRef(
ctx,
usersV1InternalRefs.publishers.removeOrgPublisherMemberInternal,
{
actorUserId,
handle,
memberHandle,
},
);
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Publisher member removal failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, headers);
}
if (message.toLowerCase().includes("not found")) {
return text(message, 404, headers);
}
return text(message, 400, headers);
}
}
export async function usersListV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
+97 -1
View File
@@ -1,4 +1,4 @@
import { describe, expect, it, vi } from "vitest";
import { beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("@convex-dev/auth/server", () => ({
getAuthUserId: vi.fn(),
@@ -9,6 +9,10 @@ const { getAuthUserId } = await import("@convex-dev/auth/server");
const { assertAdmin, assertModerator, assertRole, requireUser, requireUserFromAction } =
await import("./access");
beforeEach(() => {
vi.mocked(getAuthUserId).mockReset();
});
describe("access.requireUser", () => {
it("throws when auth is missing", async () => {
vi.mocked(getAuthUserId).mockResolvedValue(null);
@@ -59,6 +63,67 @@ describe("access.requireUser", () => {
expect(dbGet).toHaveBeenCalledWith("users:2");
expect(result).toEqual({ userId: "users:2", user });
});
it("uses the local dev impersonation user before browser auth", async () => {
const previousHandle = process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
const previousEnabled = process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
try {
process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = "local";
process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = "1";
vi.mocked(getAuthUserId).mockResolvedValue("users:browser" as never);
const user = { _id: "users:local", handle: "local", role: "admin" };
const unique = vi.fn().mockResolvedValue(user as never);
const withIndex = vi.fn().mockReturnValue({ unique });
const query = vi.fn().mockReturnValue({ withIndex });
const dbGet = vi.fn().mockResolvedValue(user as never);
const result = await requireUser({
db: { get: dbGet, query },
} as never);
expect(query).toHaveBeenCalledWith("users");
expect(withIndex).toHaveBeenCalledWith("handle", expect.any(Function));
expect(dbGet).toHaveBeenCalledWith("users:local");
expect(getAuthUserId).not.toHaveBeenCalled();
expect(result).toEqual({ userId: "users:local", user });
} finally {
if (previousHandle === undefined) delete process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
else process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = previousHandle;
if (previousEnabled === undefined) delete process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
else process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = previousEnabled;
}
});
it("does not use local dev impersonation in production deployments", async () => {
const previousHandle = process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
const previousEnabled = process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
const previousDeployment = process.env.CONVEX_DEPLOYMENT;
try {
process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = "local";
process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = "1";
process.env.CONVEX_DEPLOYMENT = "prod:wry-manatee-359";
vi.mocked(getAuthUserId).mockResolvedValue("users:browser" as never);
const user = { _id: "users:browser", handle: "browser", role: "user" };
const query = vi.fn();
const dbGet = vi.fn().mockResolvedValue(user as never);
const result = await requireUser({
db: { get: dbGet, query },
} as never);
expect(query).not.toHaveBeenCalled();
expect(getAuthUserId).toHaveBeenCalled();
expect(dbGet).toHaveBeenCalledWith("users:browser");
expect(result).toEqual({ userId: "users:browser", user });
} finally {
if (previousHandle === undefined) delete process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
else process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = previousHandle;
if (previousEnabled === undefined) delete process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
else process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = previousEnabled;
if (previousDeployment === undefined) delete process.env.CONVEX_DEPLOYMENT;
else process.env.CONVEX_DEPLOYMENT = previousDeployment;
}
});
});
describe("access.requireUserFromAction", () => {
@@ -111,6 +176,37 @@ describe("access.requireUserFromAction", () => {
expect(runQuery).toHaveBeenCalledTimes(1);
expect(result).toEqual({ userId: "users:9", user });
});
it("uses the local dev impersonation user before action auth", async () => {
const previousHandle = process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
const previousEnabled = process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
try {
process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = "local";
process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = "1";
vi.mocked(getAuthUserId).mockResolvedValue("users:browser" as never);
const user = { _id: "users:local", handle: "local", role: "admin" };
const runQuery = vi.fn(async (_query, args: { handle?: string; userId?: string }) => {
if (args.handle === "local") return user;
if (args.userId === "users:local") return user;
return null;
});
const result = await requireUserFromAction({ runQuery } as never);
expect(runQuery).toHaveBeenCalledTimes(2);
expect(runQuery).toHaveBeenNthCalledWith(1, expect.anything(), { handle: "local" });
expect(runQuery).toHaveBeenNthCalledWith(2, expect.anything(), {
userId: "users:local",
});
expect(getAuthUserId).not.toHaveBeenCalled();
expect(result).toEqual({ userId: "users:local", user });
} finally {
if (previousHandle === undefined) delete process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
else process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = previousHandle;
if (previousEnabled === undefined) delete process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
else process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = previousEnabled;
}
});
});
describe("access role assertions", () => {
+36 -7
View File
@@ -8,10 +8,23 @@ export type Role = "admin" | "moderator" | "user" | "mirror";
const DEV_IMPERSONATE_LOCAL_HANDLE = "local";
function readEnv(name: string) {
const value = process.env[name]?.trim();
const value = readKnownEnv(name)?.trim();
return value ? value : undefined;
}
function readKnownEnv(name: string) {
if (name === "CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE") {
return process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
}
if (name === "CLAW_HUB_ENABLE_DEV_IMPERSONATION") {
return process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
}
if (name === "CONVEX_DEPLOYMENT") {
return process.env.CONVEX_DEPLOYMENT;
}
return undefined;
}
function isDevImpersonationAllowed() {
const requestedHandle = readEnv("CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE");
if (requestedHandle !== DEV_IMPERSONATE_LOCAL_HANDLE) return false;
@@ -52,39 +65,49 @@ async function getDevImpersonatedUserIdFromAction(
export async function getOptionalActiveAuthUserId(
ctx: MutationCtx | QueryCtx,
): Promise<Id<"users"> | undefined> {
const devUserId = await getDevImpersonatedUserId(ctx);
if (devUserId) return devUserId;
try {
const userId = await getAuthUserId(ctx);
if (!userId) return await getDevImpersonatedUserId(ctx);
if (!userId) return undefined;
const user = await ctx.db.get(userId);
if (!user || user.deletedAt || user.deactivatedAt) return undefined;
return userId;
} catch {
return await getDevImpersonatedUserId(ctx);
return undefined;
}
}
export async function getOptionalActiveAuthUserIdFromAction(
ctx: ActionCtx,
): Promise<Id<"users"> | undefined> {
const devUserId = await getDevImpersonatedUserIdFromAction(ctx);
if (devUserId) return devUserId;
try {
const userId = await getAuthUserId(ctx);
if (!userId) return await getDevImpersonatedUserIdFromAction(ctx);
if (!userId) return undefined;
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId });
if (!user || user.deletedAt || user.deactivatedAt) return undefined;
return userId;
} catch {
return await getDevImpersonatedUserIdFromAction(ctx);
return undefined;
}
}
export async function requireUser(ctx: MutationCtx | QueryCtx) {
const devUserId = await getDevImpersonatedUserId(ctx);
if (devUserId) {
const devUser = await ctx.db.get(devUserId);
if (!devUser || devUser.deletedAt || devUser.deactivatedAt) throw new Error("User not found");
return { userId: devUserId, user: devUser };
}
let userId: Id<"users"> | null | undefined = null;
try {
userId = await getAuthUserId(ctx);
} catch {
userId = null;
}
userId ??= await getDevImpersonatedUserId(ctx);
if (!userId) throw new Error("Unauthorized");
let user: Doc<"users"> | null;
try {
@@ -99,13 +122,19 @@ export async function requireUser(ctx: MutationCtx | QueryCtx) {
export async function requireUserFromAction(
ctx: ActionCtx,
): Promise<{ userId: Id<"users">; user: Doc<"users"> }> {
const devUserId = await getDevImpersonatedUserIdFromAction(ctx);
if (devUserId) {
const devUser = await ctx.runQuery(internal.users.getByIdInternal, { userId: devUserId });
if (!devUser || devUser.deletedAt || devUser.deactivatedAt) throw new Error("User not found");
return { userId: devUserId, user: devUser as Doc<"users"> };
}
let userId: Id<"users"> | null | undefined = null;
try {
userId = await getAuthUserId(ctx);
} catch {
userId = null;
}
userId ??= await getDevImpersonatedUserIdFromAction(ctx);
if (!userId) throw new Error("Unauthorized");
let user: Doc<"users"> | null;
try {
+1 -1
View File
@@ -34,7 +34,7 @@ export const MISSING_API_TOKEN_MESSAGE =
export const INVALID_API_TOKEN_MESSAGE =
"Unauthorized: API token is invalid or revoked. Run `clawhub login` again.";
export const BLOCKED_API_TOKEN_ACCOUNT_MESSAGE =
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, contact security@openclaw.ai.";
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, open a GitHub issue: https://github.com/openclaw/clawhub/issues/new.";
export async function requireApiTokenUser(
ctx: ActionCtx,
-12
View File
@@ -1,12 +0,0 @@
import { MAX_CLAWSCAN_NOTE_CHARS, normalizeClawScanNote } from "clawhub-schema";
import { ConvexError } from "convex/values";
export { MAX_CLAWSCAN_NOTE_CHARS };
export function normalizeClawScanNoteForWrite(value: string | null | undefined) {
try {
return normalizeClawScanNote(value);
} catch (error) {
throw new ConvexError(error instanceof Error ? error.message : "Invalid ClawScan note.");
}
}
+18 -2
View File
@@ -37,9 +37,9 @@ function tarFile(path: string, content: string) {
return [header, body];
}
function npmPackFixture(files: Record<string, string>) {
function npmPackFixtureEntries(files: Array<[string, string]>) {
const parts: Uint8Array[] = [];
for (const [path, content] of Object.entries(files)) {
for (const [path, content] of files) {
parts.push(...tarFile(path, content));
}
parts.push(new Uint8Array(BLOCK_SIZE), new Uint8Array(BLOCK_SIZE));
@@ -53,6 +53,10 @@ function npmPackFixture(files: Record<string, string>) {
return gzipSync(tar);
}
function npmPackFixture(files: Record<string, string>) {
return npmPackFixtureEntries(Object.entries(files));
}
describe("clawpack", () => {
it("parses npm pack tarballs and computes npm integrity fields", async () => {
const pack = npmPackFixture({
@@ -95,6 +99,18 @@ describe("clawpack", () => {
await expect(parseClawPack(pack)).rejects.toThrow("rooted under package");
});
it("rejects duplicate normalized archive paths", async () => {
const pack = npmPackFixtureEntries([
["package/package.json", JSON.stringify({ name: "demo", version: "1.0.0" })],
["package/openclaw.plugin.json", JSON.stringify({ id: "demo" })],
["package/package.json", JSON.stringify({ name: "other", version: "9.9.9" })],
]);
await expect(parseClawPack(pack)).rejects.toThrow(
"ClawPack contains duplicate path: package.json",
);
});
it("uses npm-style tarball names", () => {
expect(npmTarballName("demo", "1.0.0")).toBe("demo-1.0.0.tgz");
expect(npmTarballName("@scope/demo", "1.0.0")).toBe("scope-demo-1.0.0.tgz");
+5
View File
@@ -67,6 +67,7 @@ function isZeroBlock(block: Uint8Array) {
function parseTarEntries(bytes: Uint8Array): ClawPackEntry[] {
const entries: ClawPackEntry[] = [];
const paths = new Set<string>();
let offset = 0;
while (offset + TAR_BLOCK_SIZE <= bytes.byteLength) {
@@ -93,6 +94,10 @@ function parseTarEntries(bytes: Uint8Array): ClawPackEntry[] {
offset = nextTarOffset(payloadOffset, size);
continue;
}
if (paths.has(relPath)) {
throw new Error(`ClawPack contains duplicate path: ${relPath}`);
}
paths.add(relPath);
entries.push({
path: relPath,
bytes: Uint8Array.from(tarEntryPayload(bytes, payloadOffset, size)),
-100
View File
@@ -1,100 +0,0 @@
import { describe, expect, it } from "vitest";
import {
depRegistryUrl,
mergeDepRegistryFinding,
parseDependencyFile,
summarizeDepRegistryChecks,
} from "./depRegistryScan";
import { summarizeReasonCodes, verdictFromCodes } from "./moderationReasonCodes";
describe("depRegistryScan", () => {
it("parses registry dependency manifests and skips vendored or non-registry specs", () => {
expect(
parseDependencyFile(
"package.json",
JSON.stringify({
dependencies: {
"@types/node": "^24.0.0",
local: "file:../local",
remote: "github:owner/repo",
},
optionalDependencies: {
undici: "^7.0.0",
},
}),
),
).toEqual([
{ name: "@types/node", registry: "npm", source: "package.json" },
{ name: "undici", registry: "npm", source: "package.json" },
]);
expect(
parseDependencyFile("vendor/package.json", '{"dependencies":{"phantom":"1.0.0"}}'),
).toEqual([]);
expect(
parseDependencyFile(
"requirements.txt",
["requests>=2", "demo @ git+https://example.test/demo.git", "-r dev.txt"].join("\n"),
),
).toEqual([{ name: "requests", registry: "pypi", source: "requirements.txt" }]);
});
it("keeps npm scope names compatible with registry URL lookup", () => {
expect(depRegistryUrl("npm", "@types/node")).toBe("https://registry.npmjs.org/@types%2Fnode");
});
it("does not produce clean status when registry lookups are unresolved", () => {
const analysis = summarizeDepRegistryChecks({
checkedAt: 123,
results: [{ name: "requests", registry: "pypi", source: "requirements.txt", exists: true }],
unresolved: [
{
name: "maybe-real",
registry: "npm",
source: "package.json",
reason: "network error",
},
],
});
expect(analysis.status).toBe("error");
expect(analysis.notFoundPackages).toEqual([]);
expect(analysis.unresolvedPackages).toEqual(["maybe-real (npm)"]);
});
it("injects a static finding only for confirmed missing packages", () => {
const suspicious = summarizeDepRegistryChecks({
checkedAt: 456,
results: [
{
name: "phantom-package-xyz",
registry: "npm",
source: "package.json",
exists: false,
httpStatus: 404,
},
],
unresolved: [],
});
const merged = mergeDepRegistryFinding({
staticScan: undefined,
analysis: suspicious,
statusFromCodes: verdictFromCodes,
summarizeCodes: summarizeReasonCodes,
});
expect(merged.status).toBe("suspicious");
expect(merged.reasonCodes).toEqual(["suspicious.dep_not_found_on_registry"]);
expect(merged.findings[0]?.file).toBe("Dependency manifests");
const cleanAgain = mergeDepRegistryFinding({
staticScan: merged,
analysis: summarizeDepRegistryChecks({ checkedAt: 789, results: [], unresolved: [] }),
statusFromCodes: verdictFromCodes,
summarizeCodes: summarizeReasonCodes,
});
expect(cleanAgain.status).toBe("clean");
expect(cleanAgain.findings).toEqual([]);
});
});
-321
View File
@@ -1,321 +0,0 @@
import {
MODERATION_ENGINE_VERSION,
REASON_CODES,
type ModerationFinding,
type ModerationVerdict,
} from "./moderationReasonCodes";
export const SUPPORTED_DEP_REGISTRIES = ["pypi", "npm", "cargo"] as const;
export type SupportedDepRegistry = (typeof SUPPORTED_DEP_REGISTRIES)[number];
export type DepRegistryStatus = "clean" | "suspicious" | "error";
export type DepEntry = {
name: string;
registry: SupportedDepRegistry;
source: string;
};
export type DepRegistryResult = DepEntry & {
exists: boolean;
httpStatus?: number;
};
export type DepRegistryUnresolved = DepEntry & {
reason: string;
};
export type DepRegistryAnalysis = {
status: DepRegistryStatus;
results: DepRegistryResult[];
notFoundPackages: string[];
unresolvedPackages: string[];
summary: string;
checkedAt: number;
};
const DEP_FILE_PARSERS: Record<string, (content: string, path: string) => DepEntry[]> = {
"requirements.txt": parseRequirementsTxt,
"requirements-dev.txt": parseRequirementsTxt,
"requirements_dev.txt": parseRequirementsTxt,
"requirements-test.txt": parseRequirementsTxt,
"requirements_test.txt": parseRequirementsTxt,
"package.json": parsePackageJson,
"cargo.toml": parseCargoToml,
"pyproject.toml": parsePyprojectToml,
};
const NON_REGISTRY_NPM_SPEC_PREFIXES = [
"file:",
"link:",
"git+",
"git://",
"github:",
"bitbucket:",
"gist:",
"http:",
"https:",
"workspace:",
"npm:",
];
const VENDORED_PATH_PATTERNS = [
/(^|\/)node_modules\//,
/(^|\/)vendor\//,
/(^|\/)__pycache__\//,
/(^|\/)\.venv\//,
/(^|\/)venv\//,
/(^|\/)target\//,
/(^|\/)\.cargo\//,
/(^|\/)dist\//,
/(^|\/)build\//,
];
function normalizeName(name: string, registry: SupportedDepRegistry) {
const normalized = name.trim().toLowerCase();
return registry === "cargo" ? normalized.replaceAll("_", "-") : normalized;
}
export function isVendoredDependencyPath(path: string) {
return VENDORED_PATH_PATTERNS.some((pattern) => pattern.test(path));
}
export function parseDependencyFile(path: string, content: string): DepEntry[] {
if (isVendoredDependencyPath(path)) return [];
const basename = path.split("/").pop()?.toLowerCase() ?? "";
const parser = DEP_FILE_PARSERS[basename];
return parser ? dedupeDeps(parser(content, path)) : [];
}
export function dedupeDeps(entries: DepEntry[]) {
const seen = new Set<string>();
return entries.filter((entry) => {
const key = `${entry.registry}:${entry.name}`;
if (seen.has(key)) return false;
seen.add(key);
return true;
});
}
function stripInlineComment(line: string) {
return line.replace(/\s+#.*$/, "").trim();
}
function parseRequirementsTxt(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
for (const rawLine of content.split("\n")) {
const line = stripInlineComment(rawLine);
if (!line || line.startsWith("-")) continue;
if (/^(?:git\+|https?:|file:|\.{0,2}\/)/i.test(line)) continue;
if (/\s@\s/.test(line)) continue;
const match = line.match(/^([a-zA-Z0-9_][a-zA-Z0-9._-]*)/);
if (!match) continue;
entries.push({ name: normalizeName(match[1], "pypi"), registry: "pypi", source: path });
}
return entries;
}
function parsePackageJson(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
let pkg: Record<string, unknown>;
try {
pkg = JSON.parse(content) as Record<string, unknown>;
} catch {
return entries;
}
for (const field of ["dependencies", "devDependencies", "optionalDependencies"]) {
const deps = pkg[field];
if (!deps || typeof deps !== "object" || Array.isArray(deps)) continue;
for (const [rawName, rawSpec] of Object.entries(deps as Record<string, unknown>)) {
const spec = typeof rawSpec === "string" ? rawSpec.trim().toLowerCase() : "";
if (NON_REGISTRY_NPM_SPEC_PREFIXES.some((prefix) => spec.startsWith(prefix))) continue;
entries.push({ name: normalizeName(rawName, "npm"), registry: "npm", source: path });
}
}
return entries;
}
function parseCargoToml(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
let inDepSection = false;
for (const rawLine of content.split("\n")) {
const line = stripInlineComment(rawLine);
if (/^\[.*\]$/.test(line)) {
const section = line.replace(/[[\]\s]/g, "").toLowerCase();
inDepSection =
section === "dependencies" ||
section === "dev-dependencies" ||
section === "build-dependencies";
continue;
}
if (!inDepSection || !line) continue;
const match = line.match(/^([a-zA-Z0-9_][a-zA-Z0-9_-]*)\s*=/);
if (!match) continue;
entries.push({ name: normalizeName(match[1], "cargo"), registry: "cargo", source: path });
}
return entries;
}
function parsePyprojectToml(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
let inDepArray = false;
let inPoetryDepTable = false;
for (const rawLine of content.split("\n")) {
const line = stripInlineComment(rawLine);
if (/^\[.*\]$/.test(line)) {
inDepArray = false;
const section = line.replace(/[[\]\s]/g, "").toLowerCase();
inPoetryDepTable =
section === "tool.poetry.dependencies" ||
section === "tool.poetry.dev-dependencies" ||
section === "tool.poetry.group.dev.dependencies";
continue;
}
if (/^dependencies\s*=\s*\[/.test(line)) {
inDepArray = true;
const inline = line.match(/\[\s*(.*)\s*\]/);
if (inline) {
for (const item of extractQuotedStrings(inline[1])) addPyPiDependency(entries, item, path);
inDepArray = false;
}
continue;
}
if (inDepArray) {
if (line === "]") {
inDepArray = false;
continue;
}
const quoted = line.match(/^["']([^"']+)["']/);
if (quoted) addPyPiDependency(entries, quoted[1], path);
continue;
}
if (!inPoetryDepTable || !line) continue;
const match = line.match(/^([a-zA-Z0-9_][a-zA-Z0-9._-]*)\s*=/);
if (!match || match[1].toLowerCase() === "python") continue;
entries.push({ name: normalizeName(match[1], "pypi"), registry: "pypi", source: path });
}
return entries;
}
function addPyPiDependency(entries: DepEntry[], spec: string, path: string) {
if (/\s@\s/.test(spec)) return;
const match = spec.match(/^([a-zA-Z0-9_][a-zA-Z0-9._-]*)/);
if (!match) return;
entries.push({ name: normalizeName(match[1], "pypi"), registry: "pypi", source: path });
}
function extractQuotedStrings(s: string) {
return [...s.matchAll(/["']([^"']+)["']/g)].map((match) => match[1]);
}
export function depRegistryUrl(registry: SupportedDepRegistry, packageName: string) {
const encoded =
registry === "npm" && packageName.startsWith("@")
? `@${encodeURIComponent(packageName.slice(1))}`
: encodeURIComponent(packageName);
if (registry === "pypi") return `https://pypi.org/pypi/${encoded}/json`;
if (registry === "npm") return `https://registry.npmjs.org/${encoded}`;
return `https://crates.io/api/v1/crates/${encoded}`;
}
export function summarizeDepRegistryChecks(params: {
results: DepRegistryResult[];
unresolved: DepRegistryUnresolved[];
checkedAt?: number;
}): DepRegistryAnalysis {
const notFound = params.results.filter((result) => !result.exists);
const notFoundPackages = notFound.map((result) => `${result.name} (${result.registry})`);
const unresolvedPackages = params.unresolved.map(
(result) => `${result.name} (${result.registry})`,
);
const checkedAt = params.checkedAt ?? Date.now();
if (notFoundPackages.length > 0) {
const partial =
unresolvedPackages.length > 0
? ` ${unresolvedPackages.length} package(s) could not be checked and will be retried.`
: "";
return {
status: "suspicious",
results: params.results,
notFoundPackages,
unresolvedPackages,
summary: `${notFoundPackages.length} declared dependency package(s) were not found on their public registry: ${notFoundPackages.join(", ")}.${partial}`,
checkedAt,
};
}
if (unresolvedPackages.length > 0) {
return {
status: "error",
results: params.results,
notFoundPackages: [],
unresolvedPackages,
summary: `${unresolvedPackages.length} dependency package(s) could not be verified due to registry lookup errors. The scan will be retried.`,
checkedAt,
};
}
return {
status: "clean",
results: params.results,
notFoundPackages: [],
unresolvedPackages: [],
summary: `All ${params.results.length} declared dependency package(s) verified as present on their public registries.`,
checkedAt,
};
}
export function buildDepRegistryFinding(analysis: DepRegistryAnalysis): ModerationFinding | null {
if (analysis.status !== "suspicious" || analysis.notFoundPackages.length === 0) return null;
return {
code: REASON_CODES.DEP_NOT_FOUND,
severity: "critical",
file: "Dependency manifests",
line: 1,
message: `${analysis.notFoundPackages.length} package(s) referenced in dependency files do not exist on their public registries: ${analysis.notFoundPackages.join(", ")}`,
evidence:
"An attacker could register these phantom package names and inject malicious install-time code through dependency confusion.",
};
}
export function mergeDepRegistryFinding(params: {
staticScan:
| {
status: ModerationVerdict;
reasonCodes: string[];
findings: ModerationFinding[];
summary: string;
engineVersion: string;
checkedAt: number;
}
| undefined;
analysis: DepRegistryAnalysis;
statusFromCodes: (codes: string[]) => ModerationVerdict;
summarizeCodes: (codes: string[]) => string;
}) {
const base = params.staticScan ?? {
status: "clean" as ModerationVerdict,
reasonCodes: [],
findings: [],
summary: "No suspicious patterns detected.",
engineVersion: MODERATION_ENGINE_VERSION,
checkedAt: params.analysis.checkedAt,
};
const findings = base.findings.filter((finding) => finding.code !== REASON_CODES.DEP_NOT_FOUND);
const depFinding = buildDepRegistryFinding(params.analysis);
if (depFinding) findings.push(depFinding);
const reasonCodes = Array.from(new Set(findings.map((finding) => finding.code))).sort((a, b) =>
a.localeCompare(b),
);
return {
...base,
status: params.statusFromCodes(reasonCodes),
reasonCodes,
findings,
summary: params.summarizeCodes(reasonCodes),
checkedAt: params.analysis.checkedAt,
};
}
+67 -3
View File
@@ -1,6 +1,8 @@
import { describe, expect, it } from "vitest";
import { isLocalDevAuthEnabled } from "./devAuth";
const CLOUD_DEV_AUTH_SECRET = "dev-auth-secret-with-enough-entropy-123";
describe("isLocalDevAuthEnabled", () => {
it("requires the explicit dev auth flag", () => {
expect(
@@ -41,16 +43,78 @@ describe("isLocalDevAuthEnabled", () => {
).toBe(true);
});
it("rejects cloud dev deployments even when the dev auth flag is set", () => {
it("allows cloud dev deployments with an explicit localhost site and matching secret", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
},
CLOUD_DEV_AUTH_SECRET,
),
).toBe(true);
});
it("allows cloud dev deployments from the fallback marker when Convex deployment is blank", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_DEPLOYMENT: "",
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
},
CLOUD_DEV_AUTH_SECRET,
),
).toBe(true);
});
it("rejects cloud dev deployments when the secret is missing", () => {
expect(
isLocalDevAuthEnabled({
CONVEX_SITE_URL: "http://127.0.0.1:3211",
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
}),
).toBe(false);
});
it("rejects cloud dev deployments when the configured secret is too short", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: "short",
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
},
"short",
),
).toBe(false);
});
it("rejects cloud dev deployments without an explicit localhost dev auth site", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_SITE_URL: "http://127.0.0.1:3211",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
},
CLOUD_DEV_AUTH_SECRET,
),
).toBe(false);
});
it("rejects localhost site URLs without a local deployment marker", () => {
expect(
isLocalDevAuthEnabled({
+33 -3
View File
@@ -3,18 +3,48 @@ type DevAuthEnv = {
CONVEX_SITE_URL?: string;
DEV_AUTH_CONVEX_DEPLOYMENT?: string;
DEV_AUTH_ENABLED?: string;
DEV_AUTH_SECRET?: string;
DEV_AUTH_SITE_URL?: string;
};
export function isLocalDevAuthEnabled(env: DevAuthEnv = process.env) {
const MIN_CLOUD_DEV_AUTH_SECRET_LENGTH = 32;
export function isLocalDevAuthEnabled(env: DevAuthEnv = process.env, suppliedSecret?: string) {
if (env.DEV_AUTH_ENABLED !== "1") return false;
const deployment = env.CONVEX_DEPLOYMENT?.trim() || env.DEV_AUTH_CONVEX_DEPLOYMENT?.trim() || "";
return isLocalConvexDeployment(deployment) && isLocalhostUrl(env.CONVEX_SITE_URL);
const convexDeployment = env.CONVEX_DEPLOYMENT?.trim();
const devAuthDeployment = env.DEV_AUTH_CONVEX_DEPLOYMENT?.trim();
const deployment = convexDeployment || devAuthDeployment || "";
if (isLocalConvexDeployment(deployment)) {
return isLocalhostUrl(env.CONVEX_SITE_URL);
}
if (isDevConvexDeployment(deployment)) {
return isLocalhostUrl(env.DEV_AUTH_SITE_URL) && hasValidCloudDevAuthSecret(env, suppliedSecret);
}
return false;
}
function isLocalConvexDeployment(deployment: string) {
return deployment.startsWith("local:") || deployment.startsWith("anonymous:");
}
function isDevConvexDeployment(deployment: string) {
return deployment.startsWith("dev:");
}
function hasValidCloudDevAuthSecret(env: DevAuthEnv, suppliedSecret: string | undefined) {
const expected = env.DEV_AUTH_SECRET?.trim();
const actual = suppliedSecret?.trim();
return Boolean(
expected &&
actual &&
expected.length >= MIN_CLOUD_DEV_AUTH_SECRET_LENGTH &&
actual === expected,
);
}
function isLocalhostUrl(value: string | undefined) {
if (!value) return false;
try {
+18
View File
@@ -0,0 +1,18 @@
export function assertLocalDevSeedAllowed(seedName: string): void {
const deployment =
process.env.CONVEX_DEPLOYMENT?.trim() || process.env.DEV_AUTH_CONVEX_DEPLOYMENT?.trim() || "";
if (
deployment.startsWith("dev:") ||
deployment.startsWith("local:") ||
deployment.startsWith("anonymous:")
) {
return;
}
if (
!deployment &&
(process.env.DEV_AUTH_ENABLED === "1" || process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION === "1")
) {
return;
}
throw new Error(`${seedName} dev seed is disabled outside local/dev deployments`);
}
+41
View File
@@ -0,0 +1,41 @@
import { render, toPlainText } from "@react-email/render";
import type { ReactElement } from "react";
import AccountReinstatedEmail, {
type AccountReinstatedEmailProps,
} from "../../emails/account-reinstated";
import AccountSuspendedEmail, {
type AccountSuspendedEmailProps,
} from "../../emails/account-suspended";
import AdminOneOffEmail, { type AdminOneOffEmailProps } from "../../emails/admin-one-off";
import BlockedVersionEmail, { type BlockedVersionEmailProps } from "../../emails/blocked-version";
import PluginInspectorFindingsEmail, {
type PluginInspectorFindingsEmailProps,
} from "../../emails/plugin-inspector-findings";
export async function renderAccountSuspendedEmail(props: AccountSuspendedEmailProps) {
return await renderEmail(<AccountSuspendedEmail {...props} />);
}
export async function renderAccountReinstatedEmail(props: AccountReinstatedEmailProps) {
return await renderEmail(<AccountReinstatedEmail {...props} />);
}
export async function renderBlockedVersionEmail(props: BlockedVersionEmailProps) {
return await renderEmail(<BlockedVersionEmail {...props} />);
}
export async function renderPluginInspectorFindingsEmail(props: PluginInspectorFindingsEmailProps) {
return await renderEmail(<PluginInspectorFindingsEmail {...props} />);
}
export async function renderAdminOneOffEmail(props: AdminOneOffEmailProps) {
return await renderEmail(<AdminOneOffEmail {...props} />);
}
async function renderEmail(element: ReactElement) {
const html = await render(element);
return {
html,
text: toPlainText(html),
};
}
+376
View File
@@ -0,0 +1,376 @@
import { describe, expect, it } from "vitest";
import {
APPEALS_URL,
buildAdminOneOffEmail,
buildMaliciousArtifactEmail,
buildBanNotificationEmail,
buildPackageInspectorFindingsEmail,
buildRestoredAccountEmail,
} from "./emails";
function expectFooterLinksUnderlined(html: string) {
expect(html).toMatch(
/href="https:\/\/clawhub\.ai"[^>]*style="[^"]*color:#8a8a8e[^"]*text-decoration[^"]*underline[^"]*"[^>]*>ClawHub<\/a>/,
);
expect(html).toMatch(
/href="https:\/\/(?:clawhub\.ai\/docs|docs\.openclaw\.ai)"[^>]*style="[^"]*color:#8a8a8e[^"]*text-decoration[^"]*underline[^"]*"[^>]*>Docs<\/a>/,
);
}
describe("moderation notification email copy", () => {
it("builds public-safe malicious skill context with appeal but no local scan guidance", async () => {
const email = await buildBanNotificationEmail({
handle: "gingiris",
source: "autoban",
reason: "malicious.llm_malicious",
artifact: { kind: "skill", name: "gingiris-launch" },
trigger: "scanner.llm.malicious",
});
expect(email.subject).toBe("Your ClawHub account has been suspended");
expect(email.context).toMatchObject({
appealUrl: APPEALS_URL,
artifact: { kind: "skill", name: "gingiris-launch" },
scannerLabel: "ClawScan",
findingSummary: "ClawScan classified the uploaded skill as malicious.",
});
expect(email.text).toContain("Skill: gingiris-launch");
expect(email.text).not.toContain("Scanner:");
expect(email.html).toContain("background-color:#0a0a0b");
expect(email.html).toContain("ClawHub");
expect(email.html).toMatch(
/<p[^>]*>ClawScan classified the uploaded skill as malicious\.<\/p>/,
);
expect(email.html).toMatch(/<li[^>]*>Your ClawHub account cannot sign in\.<\/li>/);
expect(email.html).toMatch(
/<li[^>]*>Existing API tokens for the account have been revoked\.<\/li>/,
);
expect(email.html).toMatch(
/<li[^>]*>Published listings owned by the account may be hidden from public view\.<\/li>/,
);
expect(email.html).not.toContain("<strong>Scanner:</strong>");
expect(email.text).not.toContain("republishing");
expect(email.html).not.toContain("republishing");
expect(email.text).not.toContain("To support your appeal, include scan results");
expect(email.html).not.toContain("Include scan results with your appeal");
expect(email.text).toContain("Appeal: https://appeals.openclaw.ai/");
expect(email.html).not.toContain("If you already appealed");
expect(email.html).not.toContain("separate support email");
expect(email.html).not.toContain("You received this email because");
expect(email.html).toContain("https://clawhub.ai/docs");
expectFooterLinksUnderlined(email.html);
expect(email.text).not.toContain("clawhub scan ./my-skill --output clawhub-scan.zip");
expect(email.text).not.toContain("https://docs.openclaw.ai/clawhub/cli#scan-path");
});
it("does not leak raw manual moderator notes into outbound email", async () => {
const email = await buildBanNotificationEmail({
handle: "target",
source: "manual",
reason: "internal reviewer note: reporter=user_123 secret finding id=abc",
});
expect(email.context.findingSummary).toBe(
"ClawHub staff disabled the account after a security review.",
);
expect(email.text).not.toContain("internal reviewer note");
expect(email.text).not.toContain("reporter=user_123");
expect(email.html).not.toContain("secret finding id");
});
it("uses rate-limit copy without scan remediation guidance", async () => {
const email = await buildBanNotificationEmail({
handle: "publish-loop",
source: "manual",
reason: "rate limit triggered by automated CLI publishing",
});
expect(email.context).toMatchObject({
scannerLabel: null,
findingSummary: "Publishing automation triggered ClawHub rate-limit abuse controls.",
});
expect(email.text).toContain("Publishing automation");
expect(email.text).not.toContain("clawhub scan");
expect(email.text).not.toContain("Include scan results");
expect(email.html).not.toContain("Include scan results");
expect(email.html).not.toContain("fixed local copy");
});
it("builds restored-account copy that explains tokens stay revoked", async () => {
const email = await buildRestoredAccountEmail({
handle: "restored",
restoredListings: [
{ kind: "skill", name: "safe-one" },
{ kind: "plugin", name: "@scope/demo" },
],
});
expect(email.subject).toBe("Your ClawHub account has been reinstated");
expect(email.text).toContain("Your ClawHub account can sign in again.");
expect(email.text).toContain("Skill: safe-one");
expect(email.text).toContain("Plugin: @scope/demo");
expect(email.text).toContain("Previously revoked API tokens stay revoked.");
expect(email.html).toContain("ACCOUNT REINSTATED");
expect(email.html).toContain("API tokens issued before the suspension");
expect(email.html).not.toContain("You received this email because");
expect(email.html).toContain("https://clawhub.ai/docs");
expectFooterLinksUnderlined(email.html);
});
it("omits restored count rows when batch totals are not complete", async () => {
const email = await buildRestoredAccountEmail({
handle: "restored",
restoredAt: 1_700_000_000_000,
skillsRestored: 5,
packagesRestored: undefined,
});
expect(email.html).not.toContain("Skills restored");
expect(email.html).not.toContain("Packages restored");
expect(email.html).not.toContain(">12<");
expect(email.html).not.toContain(">3<");
expect(email.html).toContain("Your account is active again.");
});
it("builds malicious artifact copy without account appeal language", async () => {
const email = await buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "skill", name: "demo-skill" },
version: "1.2.3",
trigger: "malicious.llm_malicious",
findingSummary: "Attempts to exfiltrate credentials.",
});
expect(email.subject).toBe("ClawHub blocked a skill version");
expect(email.text).toContain("Reason: Attempts to exfiltrate credentials.");
expect(email.html).toContain("Attempts to exfiltrate credentials.");
expect(email.text).toContain("Skill: demo-skill");
expect(email.text).toContain("Version: 1.2.3");
expect(email.text).toContain("clawhub scan download demo-skill --version 1.2.3");
expect(email.text).toContain("Increment the version number before uploading the fixed skill.");
expect(email.text).toContain("https://docs.openclaw.ai/clawhub/moderation");
expect(email.text).not.toContain("clawhub scan ./my-skill --output clawhub-scan.zip");
expect(email.text).not.toContain("fixed local copy");
expect(email.text).toContain("Repeated malicious rejections may lead to account disablement");
expect(email.html).toContain("Repeated malicious rejections may lead to account disablement");
expect(email.html).toContain("Skill Review");
expect(email.html).not.toContain("Plugin Review");
expect(email.html).toContain("ClawHub blocked a skill version");
expect(email.html).not.toContain("Open ClawHub");
expect(email.html).not.toContain('href="https://clawhub.ai" style="display:inline-block');
expect(email.text).not.toContain(APPEALS_URL);
expect(email.html).not.toContain(APPEALS_URL);
expect(email.html).not.toContain("appeal this decision");
});
it("falls back to generic malicious artifact copy when no ClawScan summary is available", async () => {
const email = await buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "skill", name: "demo-skill" },
version: "1.2.3",
trigger: "malicious.llm_malicious",
});
expect(email.text).toContain("Reason: ClawScan classified the uploaded artifact as malicious.");
});
it("keeps supplied ClawScan summaries to one email-safe line", async () => {
const email = await buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "skill", name: "demo-skill" },
version: "1.2.3",
findingSummary: ` ${"credential exfiltration ".repeat(30)}\nwith hidden tooling `,
});
const reasonLine = email.text.split("\n").find((line) => line.startsWith("Reason: "));
expect(reasonLine).toBeDefined();
expect(reasonLine).not.toContain("\n");
expect(reasonLine?.length).toBeLessThanOrEqual("Reason: ".length + 280);
expect(reasonLine).toContain("...");
});
it("builds plugin scan download copy with an explicit artifact kind", async () => {
const email = await buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "plugin", name: "@scope/demo" },
version: "2.0.0",
trigger: "malicious.static",
});
expect(email.text).toContain("Plugin: @scope/demo");
expect(email.text).toContain("clawhub scan download @scope/demo --version 2.0.0 --kind plugin");
expect(email.text).toContain("Increment the version number before uploading the fixed plugin.");
});
it("builds plugin inspector warning copy with local validation guidance", async () => {
const email = await buildPackageInspectorFindingsEmail({
handle: "octocat",
packageName: "demo-plugin",
version: "1.0.0",
findings: [
{
findingKind: "warning",
code: "legacy-before-agent-start",
issueClass: "deprecation-warning",
severity: "P2",
message: "legacy before_agent_start hook is deprecated",
inspectorVersion: "0.4.0",
targetOpenClawVersion: "0.9.0",
scanSource: "publish",
authorRemediation: {
summary: "Replace the legacy before_agent_start hook with current prompt hooks.",
docsUrl:
"https://docs.openclaw.ai/clawhub/plugin-validation-fixes#legacy-before-agent-start",
},
},
],
});
expect(email.subject).toBe("Plugin Inspector findings for demo-plugin@1.0.0");
expect(email.text).toContain("Hi octocat,");
expect(email.text).toContain("We found 1 issue with version 1.0.0 of demo-plugin.");
expect(email.text).toContain("OpenClaw Version: 0.9.0");
expect(email.text).toContain("Address the findings below in your plugin package.");
expect(email.text).toContain("Run the validation command locally against your changes.");
expect(email.text).toContain("clawhub package validate <path-to-plugin>");
expect(email.text).toContain(
"- **WARNING** `legacy-before-agent-start` (deprecation-warning, P2)",
);
expect(email.text).toContain(" legacy before_agent_start hook is deprecated");
expect(email.text).toContain(" Fix:");
expect(email.text).toContain(
" Replace the legacy before_agent_start hook with current prompt hooks.",
);
expect(email.text).toContain(" Docs:");
expect(email.text).toContain(
" https://docs.openclaw.ai/clawhub/plugin-validation-fixes#legacy-before-agent-start",
);
expect(email.text).not.toContain("ClawHub Security");
expect(email.html).toContain("Validate a local fix");
expect(email.html).toContain("Plugin Review");
expect(email.html).not.toContain("Open ClawHub");
expect(email.html).not.toContain('href="https://clawhub.ai" style="display:inline-block');
expect(email.html).not.toContain("You&#39;re receiving this because");
expect(email.html).not.toContain("You're receiving this because");
expect(email.html).toContain("https://docs.openclaw.ai");
expectFooterLinksUnderlined(email.html);
expect(email.html).toContain("OpenClaw Version");
expect(email.html).toContain("0.9.0");
expect(email.html).toContain("clawhub package validate &lt;path-to-plugin&gt;");
expect(email.html).toContain("legacy-before-agent-start");
expect(email.html).toContain("legacy-before-agent-start · deprecation-warning · P2");
expect(email.html).toContain("Fix");
expect(email.html).toContain("Replace the legacy before_agent_start hook");
expect(email.html).toContain("Docs →");
expect(email.html).toContain("plugin-validation-fixes#legacy-before-agent-start");
expect(email.html).not.toContain("plugin validation fix docs");
expect(email.html).not.toContain("ClawHub Security");
expect(email.text).not.toContain("Plugin Inspector: 0.4.0");
expect(email.text).not.toContain("Target OpenClaw:");
expect(email.html).not.toContain("<strong>Plugin Inspector:</strong>");
expect(email.html).not.toContain("<strong>Target OpenClaw:</strong>");
expect(email.html).not.toContain("Review:");
expect(email.html).not.toContain("plugin validation findings");
expect(email.html).not.toContain("https://clawhub.ai/plugins/demo-plugin#validation");
expect(email.html).not.toContain("Your plugin was published");
expect(email.html).not.toContain("published successfully");
});
it("builds plugin inspector error copy without publish-time wording", async () => {
const email = await buildPackageInspectorFindingsEmail({
packageName: "demo-plugin",
version: "1.0.1",
findings: [
{
findingKind: "error",
code: "missing-expected-seam",
issueClass: "compatibility-error",
severity: "P0",
level: "breakage",
message: "registerTool is no longer available",
inspectorVersion: "0.5.0",
targetOpenClawVersion: "0.10.0",
scanSource: "nightly",
},
],
});
expect(email.text).toContain("We found 1 issue with version 1.0.1 of demo-plugin.");
expect(email.text).toContain("Address the findings below in your plugin package.");
expect(email.text).toContain("Run the validation command locally against your changes.");
expect(email.text).toContain("clawhub package validate <path-to-plugin>");
expect(email.text).toContain("- **ERROR** `missing-expected-seam` (compatibility-error, P0)");
expect(email.text).not.toContain("Your plugin was published");
expect(email.text).not.toContain("was published, but");
expect(email.text).not.toContain("Some findings are errors");
expect(email.text).not.toContain("nightly");
expect(email.html).toContain("missing-expected-seam");
expect(email.html).toContain("compatibility-error · P0");
});
it("does not rewrite inserted package names, versions, or issue counts", async () => {
const findings = Array.from({ length: 11 }, (_, index) => ({
findingKind: "warning" as const,
code: `finding-${index + 1}`,
issueClass: "compatibility-warning",
severity: "P2",
message: "review finding",
}));
const email = await buildPackageInspectorFindingsEmail({
packageName: "my-demo-plugin",
version: "1.0.0-beta",
findings,
});
expect(email.text).toContain("We found 11 issues with version 1.0.0-beta of my-demo-plugin.");
expect(email.html).toContain("11 issues found");
expect(email.html).toContain("my-demo-plugin@1.0.0-beta");
expect(email.html).not.toContain("my-my-demo-plugin");
expect(email.html).not.toContain("1.0.0-beta-beta");
expect(email.html).not.toContain("11 issueses");
});
it("builds a templated admin one-off email with escaped staff-authored content", async () => {
const email = await buildAdminOneOffEmail({
recipientHandle: "octocat",
subject: "Content rights report",
title: "Action required: content rights report",
body: "We received a report about <package>. Please reply with context.",
primaryActionLabel: "Open appeal",
primaryActionUrl: "https://appeals.openclaw.ai/case-123",
});
expect(email.subject).toBe("Content rights report");
expect(email.text).toContain("Hi octocat,");
expect(email.text).toContain("Action required: content rights report");
expect(email.text).toContain("Open appeal: https://appeals.openclaw.ai/case-123");
expect(email.html).toContain("font-size:18px");
expect(email.html).toContain("ClawHub");
expect(email.html).toContain("Action required: content rights report");
expect(email.html).toContain("We received a report about &lt;package&gt;.");
expect(email.html).toContain("Open appeal");
expect(email.html).not.toContain("<package>");
expect(email.html).not.toContain("You received this email because");
expect(email.html).toContain("https://clawhub.ai/docs");
expectFooterLinksUnderlined(email.html);
});
it("omits the admin one-off button when no action is provided", async () => {
const email = await buildAdminOneOffEmail({
recipientHandle: "octocat",
subject: "Content rights report",
title: "Action required: content rights report",
body: "We received a report about <package>. Please reply with context.",
});
expect(email.text).not.toContain("Open ClawHub:");
expect(email.html).toContain("Action required: content rights report");
expect(email.html).not.toContain("Open ClawHub");
expect(email.html).not.toContain("{{primary_action_label}}");
expect(email.html).not.toContain("{{primary_action_url}}");
expect(email.html).not.toContain('href="https://clawhub.ai" style="display:inline-block');
expect(email.html).not.toContain("You received this email because");
expect(email.html).toContain("https://clawhub.ai/docs");
expectFooterLinksUnderlined(email.html);
});
});
+491
View File
@@ -0,0 +1,491 @@
export const APPEALS_URL = "https://appeals.openclaw.ai/";
export const MODERATION_GUIDELINES_URL = "https://docs.openclaw.ai/clawhub/moderation";
export const MALICIOUS_REJECTION_ACCOUNT_WARNING =
"Repeated malicious rejections may lead to account disablement.";
const MAX_EMAIL_FINDING_SUMMARY_LENGTH = 280;
export const ADMIN_ONE_OFF_TEMPLATE = "generic-one-off";
export type NotificationArtifact = {
kind: "skill" | "plugin";
name: string;
};
export type BanNotificationSource = "manual" | "autoban";
export type BanNotificationEmailArgs = {
handle?: string;
source: BanNotificationSource;
reason?: string;
trigger?: string;
artifact?: NotificationArtifact;
bannedAt?: number;
hiddenArtifacts?: number;
};
export type BanNotificationEmailContext = {
appealUrl: typeof APPEALS_URL;
artifact: NotificationArtifact | null;
scannerLabel: string | null;
findingSummary: string;
};
export type TransactionalEmail = {
subject: string;
context: BanNotificationEmailContext;
text: string;
html: string;
};
export type RestoredAccountEmailArgs = {
handle?: string;
restoredListings?: NotificationArtifact[];
restoredAt?: number;
skillsRestored?: number;
packagesRestored?: number;
};
export type MaliciousArtifactEmailArgs = {
handle?: string;
artifact: NotificationArtifact;
version?: string;
trigger?: string;
findingSummary?: string;
};
export type PackageInspectorEmailFinding = {
findingKind: "warning" | "error";
code: string;
issueClass?: string;
level?: string;
severity?: string;
message: string;
authorRemediation?: {
summary: string;
docsUrl?: string;
};
inspectorVersion?: string;
targetOpenClawVersion?: string;
scanSource?: "publish" | "nightly";
};
export type PackageInspectorFindingsEmailArgs = {
handle?: string;
packageName: string;
version: string;
findings: PackageInspectorEmailFinding[];
};
export type AdminOneOffEmailArgs = {
recipientHandle?: string;
subject: string;
title?: string;
body: string;
primaryActionLabel?: string;
primaryActionUrl?: string;
};
type BanReasonSummary = {
scannerLabel: string | null;
findingSummary: string;
};
function normalizeReasonInput(args: Pick<BanNotificationEmailArgs, "reason" | "trigger">) {
return `${args.reason ?? ""} ${args.trigger ?? ""}`.trim().toLowerCase();
}
function summarizeBanReason(args: BanNotificationEmailArgs): BanReasonSummary {
const normalized = normalizeReasonInput(args);
if (args.source === "autoban") {
if (normalized.includes("virustotal") || normalized.includes("virus_total")) {
return {
scannerLabel: "VirusTotal",
findingSummary: "VirusTotal telemetry contributed to a malicious upload finding.",
};
}
if (normalized.includes("static")) {
return {
scannerLabel: "Static analysis",
findingSummary: "Static analysis flagged malicious upload patterns.",
};
}
if (
normalized.includes("clawscan") ||
normalized.includes("llm") ||
normalized.includes("malicious")
) {
return {
scannerLabel: "ClawScan",
findingSummary: "ClawScan classified the uploaded skill as malicious.",
};
}
return {
scannerLabel: "ClawHub security checks",
findingSummary: "ClawHub security checks classified the uploaded skill as malicious.",
};
}
if (/rate[-\s]?limit|publishing automation|automated(?: cli)? publishing/.test(normalized)) {
return {
scannerLabel: null,
findingSummary: "Publishing automation triggered ClawHub rate-limit abuse controls.",
};
}
return {
scannerLabel: null,
findingSummary: "ClawHub staff disabled the account after a security review.",
};
}
function artifactLabel(artifact: NotificationArtifact) {
return `${artifact.kind === "skill" ? "Skill" : "Plugin"}: ${artifact.name}`;
}
function greeting(handle: string | undefined) {
return handle?.trim() ? `Hi ${handle.trim()},` : "Hi,";
}
function handleLabel(handle: string | undefined) {
const normalized = handle?.trim().replace(/^@+/, "");
return normalized ? `@${normalized}` : "your account";
}
function formatUtcTimestamp(value: number | undefined, fallback: string) {
if (!Number.isFinite(value)) return fallback;
return new Date(value as number)
.toISOString()
.replace("T", " ")
.replace(/\.\d{3}Z$/, " UTC");
}
async function renderAccountSuspendedTemplate(args: {
handle?: string;
suspendedAt?: number;
hiddenArtifacts?: number;
findingSummary: string;
preheader: string;
}) {
const { renderAccountSuspendedEmail } = await import("./emailRendering");
const hiddenArtifacts =
typeof args.hiddenArtifacts === "number" && Number.isFinite(args.hiddenArtifacts)
? Math.max(0, Math.trunc(args.hiddenArtifacts))
: undefined;
const rendered = await renderAccountSuspendedEmail({
handle: handleLabel(args.handle),
suspendedAt: formatUtcTimestamp(args.suspendedAt, "moderation review"),
...(hiddenArtifacts === undefined ? {} : { hiddenArtifacts }),
findingSummary: args.findingSummary,
preheader: args.preheader,
});
return rendered.html;
}
async function renderAccountReinstatedTemplate(args: {
handle?: string;
restoredAt?: number;
skillsRestored?: number;
packagesRestored?: number;
}) {
const { renderAccountReinstatedEmail } = await import("./emailRendering");
const hasRestoredCounts =
typeof args.skillsRestored === "number" && typeof args.packagesRestored === "number";
const preheader = hasRestoredCounts
? `Your account is active again - ${args.skillsRestored} skills and ${args.packagesRestored} packages restored. Note: previous API tokens remain revoked.`
: "Your account is active again. Note: previous API tokens remain revoked.";
const rendered = await renderAccountReinstatedEmail({
handle: handleLabel(args.handle),
restoredAt: formatUtcTimestamp(args.restoredAt, "account review"),
...(hasRestoredCounts
? { skillsRestored: args.skillsRestored, packagesRestored: args.packagesRestored }
: {}),
preheader,
});
return rendered.html;
}
async function renderGenericOneOffTemplate(args: AdminOneOffEmailArgs) {
const { renderAdminOneOffEmail } = await import("./emailRendering");
const subject = args.subject.trim();
const title = args.title?.trim() || subject;
const actionLabel = args.primaryActionLabel?.trim();
const actionUrl = args.primaryActionUrl?.trim();
const rendered = await renderAdminOneOffEmail({
recipientHandle: args.recipientHandle?.trim() || "there",
subject,
title,
body: args.body.trim(),
...(actionLabel && actionUrl ? { primaryAction: { label: actionLabel, url: actionUrl } } : {}),
});
return rendered.html;
}
function buildScanDownloadCommand(args: MaliciousArtifactEmailArgs) {
const version = args.version?.trim() || "<version>";
const kindFlag = args.artifact.kind === "plugin" ? " --kind plugin" : "";
return `clawhub scan download ${args.artifact.name} --version ${version}${kindFlag}`;
}
function buildPluginValidateCommand() {
return "clawhub package validate <path-to-plugin>";
}
function normalizeEmailFindingSummary(value: string | undefined) {
const normalized = value?.replace(/\s+/g, " ").trim();
if (!normalized) return undefined;
if (normalized.length <= MAX_EMAIL_FINDING_SUMMARY_LENGTH) return normalized;
return `${normalized.slice(0, MAX_EMAIL_FINDING_SUMMARY_LENGTH - 3).trimEnd()}...`;
}
export async function buildBanNotificationEmail(
args: BanNotificationEmailArgs,
): Promise<TransactionalEmail> {
const summary = summarizeBanReason(args);
const artifact = args.artifact ?? null;
const context: BanNotificationEmailContext = {
appealUrl: APPEALS_URL,
artifact,
scannerLabel: summary.scannerLabel,
findingSummary: summary.findingSummary,
};
const lines = [
greeting(args.handle),
"",
"Your ClawHub account has been suspended.",
`Reason: ${context.findingSummary}`,
];
if (artifact) lines.push(artifactLabel(artifact));
lines.push(
"",
"What changed:",
"- Your ClawHub account cannot sign in.",
"- Existing API tokens for the account have been revoked.",
"- Published listings owned by the account may be hidden from public view.",
"",
`Appeal: ${APPEALS_URL}`,
);
lines.push("", "ClawHub Security");
const impactItems = [
"Your ClawHub account cannot sign in.",
"Existing API tokens for the account have been revoked.",
"Published listings owned by the account may be hidden from public view.",
];
const detailLines = [
context.findingSummary,
...(artifact ? [artifact.name] : []),
...impactItems,
];
const hiddenArtifacts =
typeof args.hiddenArtifacts === "number" && Number.isFinite(args.hiddenArtifacts)
? args.hiddenArtifacts
: artifact
? 1
: undefined;
const html = await renderAccountSuspendedTemplate({
handle: args.handle,
suspendedAt: args.bannedAt,
hiddenArtifacts,
findingSummary: context.findingSummary,
preheader: detailLines.join(" "),
});
return {
subject: "Your ClawHub account has been suspended",
context,
text: lines.join("\n"),
html,
};
}
export async function buildRestoredAccountEmail(args: RestoredAccountEmailArgs) {
const restoredListings = args.restoredListings ?? [];
const listingLines = restoredListings.map(artifactLabel);
const lines = [
greeting(args.handle),
"",
"Your ClawHub account can sign in again.",
"Previously revoked API tokens stay revoked. Create a new token before using the CLI or API again.",
];
if (listingLines.length > 0) {
lines.push("", "Restored listings:", ...listingLines);
}
lines.push("", "ClawHub Security");
const skillsRestored = Object.hasOwn(args, "skillsRestored")
? args.skillsRestored
: restoredListings.filter((listing) => listing.kind === "skill").length;
const packagesRestored = Object.hasOwn(args, "packagesRestored")
? args.packagesRestored
: restoredListings.filter((listing) => listing.kind === "plugin").length;
const html = await renderAccountReinstatedTemplate({
handle: args.handle,
restoredAt: args.restoredAt,
skillsRestored,
packagesRestored,
});
return {
subject: "Your ClawHub account has been reinstated",
text: lines.join("\n"),
html,
};
}
export async function buildMaliciousArtifactEmail(args: MaliciousArtifactEmailArgs) {
const artifactKind = args.artifact.kind === "skill" ? "skill" : "plugin";
const artifactLabelText = artifactLabel(args.artifact);
const scanDownloadCommand = buildScanDownloadCommand(args);
const findingSummary =
normalizeEmailFindingSummary(args.findingSummary) ??
(args.trigger?.includes("static") === true
? "Static analysis flagged malicious upload patterns."
: args.trigger?.includes("virustotal") === true || args.trigger?.includes("vt_") === true
? "VirusTotal telemetry contributed to a malicious upload finding."
: "ClawScan classified the uploaded artifact as malicious.");
const subject = `ClawHub blocked a ${artifactKind} version`;
const lines = [
greeting(args.handle),
"",
`ClawHub blocked a ${artifactKind} version after a security scan.`,
`Reason: ${findingSummary}`,
artifactLabelText,
];
if (args.version?.trim()) lines.push(`Version: ${args.version.trim()}`);
lines.push(
"",
"What changed:",
"- This version was not made public.",
"- Your account can still sign in.",
`- You can upload a fixed version of this ${artifactKind}.`,
`- ${MALICIOUS_REJECTION_ACCOUNT_WARNING}`,
"",
"Download the scan results for the blocked submitted version:",
scanDownloadCommand,
`Docs: ${MODERATION_GUIDELINES_URL}`,
`Increment the version number before uploading the fixed ${artifactKind}.`,
"",
"ClawHub Security",
);
const { renderBlockedVersionEmail } = await import("./emailRendering");
const rendered = await renderBlockedVersionEmail({
artifactKind,
artifactName: args.artifact.name,
version: args.version?.trim() || "<version>",
findingSummary,
validateCommand: scanDownloadCommand,
docsUrl: MODERATION_GUIDELINES_URL,
preheader: `${artifactLabelText} was blocked by ClawHub security scans.`,
});
return {
subject,
text: lines.join("\n"),
html: rendered.html,
};
}
export async function buildPackageInspectorFindingsEmail(args: PackageInspectorFindingsEmailArgs) {
const targetOpenClawVersion = args.findings.find(
(finding) => finding.targetOpenClawVersion,
)?.targetOpenClawVersion;
const validateCommand = buildPluginValidateCommand();
const subject = `Plugin Inspector findings for ${args.packageName}@${args.version}`;
const findingCount = args.findings.length;
const intro = `We found ${findingCount} ${findingCount === 1 ? "issue" : "issues"} with version ${args.version} of ${args.packageName}.`;
const nextSteps = [
"Address the findings below in your plugin package.",
"Run the validation command locally against your changes.",
"When validation passes, upload a new version.",
];
const findingLines = formatPackageInspectorFindingsText(args.findings);
const metadataLines = [
`Plugin: ${args.packageName}@${args.version}`,
targetOpenClawVersion ? `OpenClaw Version: ${targetOpenClawVersion}` : null,
].filter((line): line is string => line !== null);
const lines = [
greeting(args.handle),
"",
intro,
"",
...metadataLines,
"",
"Next steps:",
...nextSteps.map((item) => `- ${item}`),
"",
"Findings:",
...findingLines,
"",
"Validate a local fix:",
validateCommand,
];
const { renderPluginInspectorFindingsEmail } = await import("./emailRendering");
const rendered = await renderPluginInspectorFindingsEmail({
packageName: args.packageName,
version: args.version,
...(targetOpenClawVersion ? { openClawVersion: targetOpenClawVersion } : {}),
findings: args.findings.map((finding) => ({
code: finding.code,
kind: finding.findingKind,
meta: [finding.code, finding.issueClass, finding.severity].filter(Boolean).join(" · "),
message: finding.message,
...(finding.authorRemediation?.summary ? { fix: finding.authorRemediation.summary } : {}),
...(finding.authorRemediation?.docsUrl ? { docsUrl: finding.authorRemediation.docsUrl } : {}),
})),
validateCommand,
preheader: intro,
});
return {
subject,
text: lines.join("\n"),
html: rendered.html,
};
}
export async function buildAdminOneOffEmail(args: AdminOneOffEmailArgs) {
const title = args.title?.trim() || args.subject.trim();
const lines = [greeting(args.recipientHandle), "", title, "", args.body.trim()];
if (args.primaryActionLabel?.trim() && args.primaryActionUrl?.trim()) {
lines.push("", `${args.primaryActionLabel.trim()}: ${args.primaryActionUrl.trim()}`);
}
lines.push("", "ClawHub Team");
const html = await renderGenericOneOffTemplate(args);
return {
subject: args.subject.trim(),
text: lines.join("\n"),
html,
};
}
function formatPackageInspectorFindingsText(findings: PackageInspectorEmailFinding[]) {
if (findings.length === 0) return ["- No findings were included."];
return findings.flatMap((finding) => {
const lines = [
`- **${finding.findingKind.toUpperCase()}** \`${finding.code}\`${formatFindingMetaText(finding)}`,
` ${finding.message}`,
];
if (finding.authorRemediation?.summary) {
lines.push(" Fix:");
lines.push(` ${finding.authorRemediation.summary}`);
if (finding.authorRemediation.docsUrl) {
lines.push(" Docs:");
lines.push(` ${finding.authorRemediation.docsUrl}`);
}
}
return lines;
});
}
function formatFindingMetaText(finding: PackageInspectorEmailFinding) {
const meta = [finding.issueClass, finding.severity].filter(Boolean).join(", ");
return meta ? ` (${meta})` : "";
}
+140 -2
View File
@@ -55,6 +55,29 @@ describe("requireGitHubAccountAge", () => {
);
});
it("allows admins without GitHub account age lookup", async () => {
const runQuery = vi.fn().mockResolvedValue({
_id: "users:admin",
role: "admin",
githubCreatedAt: undefined,
});
const runMutation = vi.fn();
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
await requireGitHubAccountAge({ runQuery, runMutation } as never, "users:admin" as never);
expect(fetchMock).not.toHaveBeenCalled();
expect(runMutation).not.toHaveBeenCalled();
expect(runQuery).toHaveBeenCalledWith(internal.users.getByIdInternal, {
userId: "users:admin",
});
expect(runQuery).not.toHaveBeenCalledWith(
internal.githubIdentity.getGitHubProviderAccountIdInternal,
{ userId: "users:admin" },
);
});
it("rejects deactivated users", async () => {
const runQuery = vi.fn().mockResolvedValue({
_id: "users:1",
@@ -259,13 +282,128 @@ describe("requireGitHubAccountAge", () => {
expect(fetchMock).toHaveBeenCalledWith(
"https://api.github.com/user/12345",
expect.objectContaining({
headers: {
headers: expect.objectContaining({
"User-Agent": "clawhub",
Authorization: "Bearer ghp_test123",
},
}),
}),
);
});
it("omits Authorization header when GITHUB_TOKEN is blank", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
vi.stubEnv("GITHUB_TOKEN", " ");
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
created_at: "2020-01-01T00:00:00Z",
}),
});
vi.stubGlobal("fetch", fetchMock);
await requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never);
expect(fetchMock).toHaveBeenCalledWith(
"https://api.github.com/user/12345",
expect.objectContaining({
headers: expect.objectContaining({ "User-Agent": "clawhub" }),
}),
);
expect(fetchMock.mock.calls[0]?.[1]?.headers).not.toHaveProperty("Authorization");
});
it("retries without Authorization when GITHUB_TOKEN is rejected", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
vi.stubEnv("GITHUB_TOKEN", "ghp_expired");
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi
.fn()
.mockResolvedValueOnce({ ok: false, status: 401 })
.mockResolvedValueOnce({
ok: true,
json: async () => ({
created_at: "2020-01-01T00:00:00Z",
}),
});
vi.stubGlobal("fetch", fetchMock);
await requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never);
expect(fetchMock).toHaveBeenNthCalledWith(
1,
"https://api.github.com/user/12345",
expect.objectContaining({
headers: expect.objectContaining({
"User-Agent": "clawhub",
Authorization: "Bearer ghp_expired",
}),
}),
);
expect(fetchMock).toHaveBeenNthCalledWith(
2,
"https://api.github.com/user/12345",
expect.objectContaining({
headers: { "User-Agent": "clawhub" },
}),
);
expect(runMutation).toHaveBeenCalledWith(internal.users.setGitHubCreatedAtInternal, {
userId: "users:1",
githubCreatedAt: Date.parse("2020-01-01T00:00:00Z"),
});
expect(warnSpy).toHaveBeenCalledWith(
"[githubAccount] GitHub API auth was rejected; retrying lookup without auth",
);
});
it("does not retry unauthenticated 401 responses", async () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 401 });
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/GitHub account lookup failed/i);
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(fetchMock).toHaveBeenCalledWith(
"https://api.github.com/user/12345",
expect.objectContaining({
headers: expect.objectContaining({ "User-Agent": "clawhub" }),
}),
);
expect(fetchMock.mock.calls[0]?.[1]?.headers).not.toHaveProperty("Authorization");
});
});
describe("syncGitHubProfile", () => {
+31 -26
View File
@@ -2,6 +2,7 @@ import { ConvexError } from "convex/values";
import { internal } from "../_generated/api";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { buildGitHubApiHeaders } from "./githubAuth";
import { GITHUB_PROFILE_SYNC_WINDOW_MS } from "./githubProfileSync";
const GITHUB_API = "https://api.github.com";
@@ -22,18 +23,40 @@ function assertGitHubNumericId(providerAccountId: string) {
}
}
function buildGitHubHeaders() {
const headers: Record<string, string> = { "User-Agent": "clawhub" };
const token = process.env.GITHUB_TOKEN;
if (token) {
headers.Authorization = `Bearer ${token}`;
async function fetchGitHubUserByNumericId(providerAccountId: string) {
assertGitHubNumericId(providerAccountId);
const url = `${GITHUB_API}/user/${providerAccountId}`;
const headers = await buildGitHubApiHeaders({ userAgent: "clawhub" });
const response = await fetch(url, {
headers,
});
if (response.status !== 401 || !headers.Authorization) return response;
console.warn("[githubAccount] GitHub API auth was rejected; retrying lookup without auth");
return await fetch(url, {
headers: { "User-Agent": "clawhub" },
});
}
export async function fetchGitHubCreatedAtByProviderAccountId(providerAccountId: string) {
const response = await fetchGitHubUserByNumericId(providerAccountId);
if (!response.ok) {
if (response.status === 403 || response.status === 429) {
throw new ConvexError("GitHub API rate limit exceeded — please try again in a few minutes");
}
throw new ConvexError("GitHub account lookup failed");
}
return headers;
const payload = (await response.json()) as GitHubUser;
const parsed = payload.created_at ? Date.parse(payload.created_at) : Number.NaN;
if (!Number.isFinite(parsed)) throw new ConvexError("GitHub account lookup failed");
return parsed;
}
export async function requireGitHubAccountAge(ctx: GitHubAccountGateCtx, userId: Id<"users">) {
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId });
if (!user || user.deletedAt || user.deactivatedAt) throw new ConvexError("User not found");
if (user.role === "admin") return;
const now = Date.now();
let createdAt = user.githubCreatedAt ?? null;
@@ -47,24 +70,8 @@ export async function requireGitHubAccountAge(ctx: GitHubAccountGateCtx, userId:
// Invariant: GitHub is our only auth provider, so this should never happen.
throw new ConvexError("GitHub account required");
}
assertGitHubNumericId(providerAccountId);
// Fetch by immutable GitHub numeric ID to avoid username swap attacks entirely.
const response = await fetch(`${GITHUB_API}/user/${providerAccountId}`, {
headers: buildGitHubHeaders(),
});
if (!response.ok) {
if (response.status === 403 || response.status === 429) {
throw new ConvexError("GitHub API rate limit exceeded — please try again in a few minutes");
}
throw new ConvexError("GitHub account lookup failed");
}
const payload = (await response.json()) as GitHubUser;
const parsed = payload.created_at ? Date.parse(payload.created_at) : Number.NaN;
if (!Number.isFinite(parsed)) throw new ConvexError("GitHub account lookup failed");
createdAt = parsed;
createdAt = await fetchGitHubCreatedAtByProviderAccountId(providerAccountId);
await ctx.runMutation(internal.users.setGitHubCreatedAtInternal, {
userId,
githubCreatedAt: createdAt,
@@ -106,9 +113,7 @@ export async function syncGitHubProfile(ctx: ActionCtx, userId: Id<"users">) {
assertGitHubNumericId(providerAccountId);
const response = await fetch(`${GITHUB_API}/user/${providerAccountId}`, {
headers: buildGitHubHeaders(),
});
const response = await fetchGitHubUserByNumericId(providerAccountId);
if (!response.ok) {
// Silently fail - this is a best-effort sync, not critical path
console.warn(`[syncGitHubProfile] GitHub API error for user ${userId}: ${response.status}`);
+28
View File
@@ -76,6 +76,34 @@ describe("fetchGitHubRepositoryIdentity", () => {
);
});
it("does not use GitHub App auth for arbitrary repository lookup", async () => {
vi.stubEnv("GITHUB_APP_ID", "123");
vi.stubEnv("GITHUB_APP_INSTALLATION_ID", "456");
vi.stubEnv("GITHUB_APP_PRIVATE_KEY", "not-needed-for-this-test");
vi.stubEnv("GITHUB_TOKEN", "ghs_test_token");
const fetchMock = vi.fn(async () =>
Response.json({
id: 123,
full_name: "openclaw/clawhub",
owner: { login: "openclaw", id: 456 },
}),
);
await fetchGitHubRepositoryIdentity("openclaw/clawhub", fetchMock);
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(fetchMock).toHaveBeenCalledWith(
"https://api.github.com/repos/openclaw/clawhub",
expect.objectContaining({
headers: expect.objectContaining({
Accept: "application/vnd.github+json",
Authorization: "Bearer ghs_test_token",
"User-Agent": "clawhub/package-trusted-publisher",
}),
}),
);
});
it("omits Authorization for repository lookup when GITHUB_TOKEN is blank", async () => {
vi.stubEnv("GITHUB_TOKEN", " ");
const fetchMock = vi.fn(async () =>
+13 -11
View File
@@ -1,3 +1,5 @@
import { buildGitHubApiHeaders } from "./githubAuth";
type JwtHeader = {
alg?: unknown;
kid?: unknown;
@@ -217,7 +219,7 @@ export async function fetchGitHubRepositoryIdentity(
throw new Error(`Invalid GitHub repository: ${repository}`);
}
const response = await fetchImpl(`https://api.github.com/repos/${normalizedRepository}`, {
headers: buildGitHubRepositoryLookupHeaders(),
headers: await buildGitHubRepositoryLookupHeaders(fetchImpl),
});
if (!response.ok) {
throw new Error(
@@ -239,16 +241,16 @@ export async function fetchGitHubRepositoryIdentity(
};
}
function buildGitHubRepositoryLookupHeaders() {
const headers: Record<string, string> = {
Accept: "application/vnd.github+json",
"User-Agent": "clawhub/package-trusted-publisher",
};
const token = process.env.GITHUB_TOKEN?.trim();
if (token) {
headers.Authorization = `Bearer ${token}`;
}
return headers;
async function buildGitHubRepositoryLookupHeaders(fetchImpl: typeof fetch) {
return await buildGitHubApiHeaders({
accept: "application/vnd.github+json",
fetchImpl,
userAgent: "clawhub/package-trusted-publisher",
// This lookup accepts arbitrary public repositories. GitHub App installation
// tokens only see repositories where the App is installed, so prefer PAT or
// anonymous auth here.
useGitHubApp: false,
});
}
export function normalizeGitHubRepository(repository: string) {
+180
View File
@@ -0,0 +1,180 @@
import { generateKeyPairSync } from "node:crypto";
import { describe, expect, it } from "vitest";
import {
buildGitHubAppInstallUrl,
createGitHubAppJwt,
deriveSlugFromCandidatePath,
hashGitHubAppState,
isPathUnderAnyRoot,
normalizeGitHubRepoFullName,
normalizeGitHubSyncRoots,
signGitHubAppState,
sourceLinkMatchesProvenance,
verifyGitHubAppState,
verifyGitHubWebhookSignature,
} from "./githubAppSync";
describe("github app sync helpers", () => {
it("normalizes repository identity and sync roots", () => {
expect(normalizeGitHubRepoFullName("https://github.com/OpenClaw/Skills.git")).toBe(
"OpenClaw/Skills",
);
expect(normalizeGitHubRepoFullName("git+https://github.com/OpenClaw/Skills.git")).toBe(
"OpenClaw/Skills",
);
expect(normalizeGitHubRepoFullName("git@github.com:OpenClaw/Skills.git")).toBe(
"OpenClaw/Skills",
);
expect(normalizeGitHubRepoFullName("https://www.github.com/OpenClaw/Skills/tree/main")).toBe(
"OpenClaw/Skills",
);
expect(normalizeGitHubRepoFullName("not a repo")).toBeNull();
expect(isPathUnderAnyRoot("skills/demo/SKILL.md", ["skills"])).toBe(true);
expect(isPathUnderAnyRoot("packages/demo/package.json", ["skills"])).toBe(false);
expect(normalizeGitHubSyncRoots(["", "skills/demo"])).toEqual(["", "skills/demo"]);
expect(() => normalizeGitHubSyncRoots(["../skills"])).toThrow(/Invalid sync root/);
});
it("derives stable skill slugs from candidate paths", () => {
expect(deriveSlugFromCandidatePath("skills/Demo Skill", "OpenClaw/catalog")).toBe("demo-skill");
expect(deriveSlugFromCandidatePath("", "OpenClaw/Catalog Repo")).toBe("catalog-repo");
});
it("signs setup state, verifies it, and rejects tampering", async () => {
const secret = "state-secret";
const state = await signGitHubAppState(
{
publisherId: "publishers:org",
requestedByUserId: "users:admin",
nonce: "nonce",
targetAccountId: "12345",
exp: 2_000,
},
secret,
1_000,
);
await expect(hashGitHubAppState(state)).resolves.toMatch(/^[a-f0-9]{64}$/);
await expect(verifyGitHubAppState(state, secret, 1_500)).resolves.toEqual({
publisherId: "publishers:org",
requestedByUserId: "users:admin",
nonce: "nonce",
targetAccountId: "12345",
exp: 2_000,
});
await expect(verifyGitHubAppState(`${state}x`, secret, 1_500)).rejects.toThrow(
/Invalid GitHub setup state/,
);
await expect(verifyGitHubAppState(state, secret, 2_001)).rejects.toThrow(
/GitHub setup state expired/,
);
});
it("builds the app install URL with signed state", () => {
const url = buildGitHubAppInstallUrl({
appSlug: "clawhub-test",
state: "signed-state",
targetId: "123",
});
expect(url).toBe(
"https://github.com/apps/clawhub-test/installations/new?state=signed-state&target_id=123",
);
});
it("creates app JWTs from PKCS#8 and GitHub-style PKCS#1 RSA private keys", async () => {
const { privateKey } = generateKeyPairSync("rsa", { modulusLength: 2048 });
const pkcs8Pem = privateKey.export({ type: "pkcs8", format: "pem" });
const pkcs1Pem = privateKey.export({ type: "pkcs1", format: "pem" });
await expect(
createGitHubAppJwt({ appId: "12345", privateKeyPem: pkcs8Pem, now: 1_700_000_000_000 }),
).resolves.toMatch(/^[^.]+\.[^.]+\.[^.]+$/);
await expect(
createGitHubAppJwt({ appId: "12345", privateKeyPem: pkcs1Pem, now: 1_700_000_000_000 }),
).resolves.toMatch(/^[^.]+\.[^.]+\.[^.]+$/);
});
it("verifies webhook signatures and rejects bad signatures", async () => {
const body = new TextEncoder().encode(JSON.stringify({ zen: "Keep it logically awesome." }));
const signature = await buildGitHubWebhookSignature(body, "webhook-secret");
await expect(
verifyGitHubWebhookSignature({
body: body.buffer as ArrayBuffer,
signatureHeader: signature,
secret: "webhook-secret",
}),
).resolves.toEqual({ ok: true });
await expect(
verifyGitHubWebhookSignature({
body: body.buffer as ArrayBuffer,
signatureHeader: signature,
secret: "wrong-secret",
}),
).resolves.toEqual({ ok: false, reason: "bad-signature" });
});
it("requires exact source sync context for source-managed publishes", () => {
const link = {
repoFullName: "OpenClaw/catalog",
path: "skills/demo",
status: "active",
};
const sourceProvenance = {
kind: "github" as const,
repo: "openclaw/catalog",
path: "skills/demo",
};
expect(
sourceLinkMatchesProvenance({
link,
sourceProvenance,
sourceSync: { sourceLinkId: "skillSourceLinks:1" },
expectedSourceLinkId: "skillSourceLinks:1",
}),
).toBe(true);
expect(
sourceLinkMatchesProvenance({
link: { ...link, status: "conflict" },
sourceProvenance,
sourceSync: { sourceLinkId: "skillSourceLinks:1" },
expectedSourceLinkId: "skillSourceLinks:1",
}),
).toBe(true);
expect(
sourceLinkMatchesProvenance({
link: { ...link, status: "disabled" },
sourceProvenance,
sourceSync: { sourceLinkId: "skillSourceLinks:1" },
expectedSourceLinkId: "skillSourceLinks:1",
}),
).toBe(false);
expect(
sourceLinkMatchesProvenance({
link,
sourceProvenance,
sourceSync: undefined,
expectedSourceLinkId: "skillSourceLinks:1",
}),
).toBe(false);
expect(
sourceLinkMatchesProvenance({
link,
sourceProvenance: { ...sourceProvenance, path: "skills/other" },
sourceSync: { sourceLinkId: "skillSourceLinks:1" },
expectedSourceLinkId: "skillSourceLinks:1",
}),
).toBe(false);
});
});
async function buildGitHubWebhookSignature(body: Uint8Array, secret: string) {
const key = await crypto.subtle.importKey(
"raw",
new TextEncoder().encode(secret),
{ name: "HMAC", hash: "SHA-256" },
false,
["sign"],
);
const signature = await crypto.subtle.sign("HMAC", key, body.buffer as ArrayBuffer);
const hex = Array.from(new Uint8Array(signature), (byte) => byte.toString(16).padStart(2, "0"));
return `sha256=${hex.join("")}`;
}
+183
View File
@@ -0,0 +1,183 @@
import { ConvexError } from "convex/values";
import {
base64UrlDecode,
base64UrlEncode,
hmacSha256Base64Url,
hmacSha256Hex,
isRepoPathUnderRoot,
normalizeGitHubRepo,
sha256Hex,
timingSafeEqual,
} from "./githubCommon";
import { normalizeRepoPath } from "./githubImport";
import { normalizeSkillSlug } from "./skillSlugValidator";
export { createGitHubAppJwt } from "./githubCommon";
export type GitHubWebhookVerificationResult =
| { ok: true }
| {
ok: false;
reason: "missing-secret" | "missing-signature" | "malformed-signature" | "bad-signature";
};
export type GitHubAppStatePayload = {
publisherId: string;
requestedByUserId: string;
nonce: string;
targetAccountId?: string;
exp: number;
};
const DEFAULT_SETUP_STATE_TTL_MS = 10 * 60 * 1000;
const MAX_SYNC_ROOTS = 25;
export function normalizeGitHubRepoFullName(value: string) {
return normalizeGitHubRepo(value);
}
export function normalizeGitHubSyncRef(value: string | undefined | null, defaultBranch: string) {
const raw = value?.trim() || defaultBranch.trim();
if (!raw) throw new ConvexError("Sync ref is required");
return raw.replace(/^refs\/heads\//, "");
}
export function normalizeGitHubSyncRoots(roots: string[] | undefined | null) {
const normalized = new Set<string>();
for (const root of roots?.length ? roots : [""]) {
const trimmed = root.trim();
if (!trimmed) {
normalized.add("");
continue;
}
const value = normalizeRepoPath(root);
if (!value) throw new ConvexError("Invalid sync root");
normalized.add(value);
if (normalized.size > MAX_SYNC_ROOTS) throw new ConvexError("Too many sync roots");
}
return Array.from(normalized).sort((a, b) => a.localeCompare(b));
}
export function isPathUnderAnyRoot(path: string, roots: string[]) {
const normalizedPath = normalizeRepoPath(path);
const normalizedRoots = normalizeGitHubSyncRoots(roots);
if (normalizedRoots.includes("")) return true;
return normalizedRoots.some((root) => isRepoPathUnderRoot(normalizedPath, root));
}
export function deriveSlugFromCandidatePath(candidatePath: string, repoFullName: string) {
const repoName = repoFullName.split("/").at(1) ?? repoFullName;
const base = candidatePath ? (candidatePath.split("/").at(-1) ?? candidatePath) : repoName;
return normalizeSkillSlug(
base
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+/, "")
.replace(/-+$/, "")
.replace(/--+/g, "-"),
);
}
export function sourceLinkMatchesProvenance(params: {
link: { repoFullName: string; path: string; status: string };
sourceProvenance?: { kind: "github"; repo: string; path?: string } | null;
sourceSync?: { sourceLinkId: string } | null;
expectedSourceLinkId: string;
}) {
if (params.link.status === "disabled") return false;
if (params.sourceSync?.sourceLinkId !== params.expectedSourceLinkId) return false;
const provenance = params.sourceProvenance;
if (!provenance || provenance.kind !== "github") return false;
return (
normalizeGitHubRepoFullName(provenance.repo)?.toLowerCase() ===
normalizeGitHubRepoFullName(params.link.repoFullName)?.toLowerCase() &&
normalizeRepoPath(provenance.path ?? "") === normalizeRepoPath(params.link.path)
);
}
export async function signGitHubAppState(
payload: Omit<GitHubAppStatePayload, "exp"> & { exp?: number },
secret: string,
now = Date.now(),
) {
const exp = payload.exp ?? now + DEFAULT_SETUP_STATE_TTL_MS;
const body = base64UrlEncode(
new TextEncoder().encode(
JSON.stringify({
publisherId: payload.publisherId,
requestedByUserId: payload.requestedByUserId,
nonce: payload.nonce,
targetAccountId: payload.targetAccountId,
exp,
} satisfies GitHubAppStatePayload),
),
);
const signature = await hmacSha256Base64Url(secret, body);
return `${body}.${signature}`;
}
export async function verifyGitHubAppState(
state: string,
secret: string,
now = Date.now(),
): Promise<GitHubAppStatePayload> {
const [body, signature, extra] = state.split(".");
if (!body || !signature || extra) throw new ConvexError("Invalid GitHub setup state");
const expected = await hmacSha256Base64Url(secret, body);
if (!timingSafeEqual(signature, expected)) throw new ConvexError("Invalid GitHub setup state");
let parsed: unknown;
try {
parsed = JSON.parse(new TextDecoder().decode(base64UrlDecode(body)));
} catch {
throw new ConvexError("Invalid GitHub setup state");
}
const payload = parsed as Partial<GitHubAppStatePayload>;
if (
typeof payload.publisherId !== "string" ||
typeof payload.requestedByUserId !== "string" ||
typeof payload.nonce !== "string" ||
(payload.targetAccountId !== undefined && typeof payload.targetAccountId !== "string") ||
typeof payload.exp !== "number"
) {
throw new ConvexError("Invalid GitHub setup state");
}
if (payload.exp < now) throw new ConvexError("GitHub setup state expired");
return {
publisherId: payload.publisherId,
requestedByUserId: payload.requestedByUserId,
nonce: payload.nonce,
targetAccountId: payload.targetAccountId,
exp: payload.exp,
};
}
export async function hashGitHubAppState(state: string) {
return sha256Hex(state);
}
export async function verifyGitHubWebhookSignature(params: {
body: ArrayBuffer;
signatureHeader: string | null;
secret: string | undefined;
}): Promise<GitHubWebhookVerificationResult> {
const secret = params.secret?.trim();
if (!secret) return { ok: false, reason: "missing-secret" };
const signature = params.signatureHeader?.trim();
if (!signature) return { ok: false, reason: "missing-signature" };
if (!signature.startsWith("sha256=")) return { ok: false, reason: "malformed-signature" };
const expected = `sha256=${await hmacSha256Hex(secret, params.body)}`;
if (!timingSafeEqual(signature, expected)) return { ok: false, reason: "bad-signature" };
return { ok: true };
}
export function buildGitHubAppInstallUrl(params: {
appSlug: string;
state: string;
targetId?: string;
}) {
const url = new URL(`https://github.com/apps/${params.appSlug}/installations/new`);
url.searchParams.set("state", params.state);
if (params.targetId) url.searchParams.set("target_id", params.targetId);
return url.toString();
}
+100
View File
@@ -0,0 +1,100 @@
/* @vitest-environment node */
import { generateKeyPairSync } from "node:crypto";
import { afterEach, describe, expect, it, vi } from "vitest";
import { buildGitHubApiHeaders, createGitHubAppInstallationToken } from "./githubAuth";
function stubGitHubAppEnv() {
const { privateKey } = generateKeyPairSync("rsa", {
modulusLength: 2048,
privateKeyEncoding: { type: "pkcs1", format: "pem" },
publicKeyEncoding: { type: "spki", format: "pem" },
});
vi.stubEnv("GITHUB_APP_ID", "3536245");
vi.stubEnv("GITHUB_APP_INSTALLATION_ID", "987654");
vi.stubEnv("GITHUB_APP_PRIVATE_KEY", privateKey);
}
describe("githubAuth", () => {
afterEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
});
it("mints a GitHub App installation token from app credentials", async () => {
stubGitHubAppEnv();
const fetchMock = vi.fn(async () =>
Response.json({
token: "ghs_app_token",
expires_at: "2026-02-02T13:00:00Z",
}),
);
await expect(
createGitHubAppInstallationToken({ fetchImpl: fetchMock, userAgent: "clawhub/test" }),
).resolves.toEqual({
token: "ghs_app_token",
expiresAt: Date.parse("2026-02-02T13:00:00Z"),
});
expect(fetchMock).toHaveBeenCalledWith(
"https://api.github.com/app/installations/987654/access_tokens",
expect.objectContaining({
method: "POST",
headers: expect.objectContaining({
Accept: "application/vnd.github+json",
Authorization: expect.stringMatching(/^Bearer [^.]+\.[^.]+\.[^.]+$/),
"User-Agent": "clawhub/test",
}),
}),
);
});
it("builds API headers with GitHub App auth before PAT fallback", async () => {
stubGitHubAppEnv();
vi.stubEnv("GITHUB_TOKEN", "ghp_pat_token");
const fetchMock = vi.fn(async () =>
Response.json({
token: "ghs_app_token",
expires_at: "2026-02-02T13:00:00Z",
}),
);
await expect(
buildGitHubApiHeaders({ fetchImpl: fetchMock, userAgent: "clawhub/test" }),
).resolves.toEqual({
Accept: "application/vnd.github+json",
Authorization: "Bearer ghs_app_token",
"User-Agent": "clawhub/test",
});
});
it("falls back to GITHUB_TOKEN when GitHub App credentials are absent", async () => {
vi.stubEnv("GITHUB_TOKEN", "ghp_pat_token");
await expect(buildGitHubApiHeaders({ userAgent: "clawhub/test" })).resolves.toEqual({
Accept: "application/vnd.github+json",
Authorization: "Bearer ghp_pat_token",
"User-Agent": "clawhub/test",
});
});
it("can skip GitHub App auth for arbitrary public resources", async () => {
stubGitHubAppEnv();
vi.stubEnv("GITHUB_TOKEN", "ghp_pat_token");
const fetchMock = vi.fn();
await expect(
buildGitHubApiHeaders({
fetchImpl: fetchMock,
userAgent: "clawhub/test",
useGitHubApp: false,
}),
).resolves.toEqual({
Accept: "application/vnd.github+json",
Authorization: "Bearer ghp_pat_token",
"User-Agent": "clawhub/test",
});
expect(fetchMock).not.toHaveBeenCalled();
});
});
+272
View File
@@ -0,0 +1,272 @@
const GITHUB_API = "https://api.github.com";
const DEFAULT_ACCEPT = "application/vnd.github+json";
const DEFAULT_USER_AGENT = "clawhub/github-api";
const APP_TOKEN_CACHE_BUFFER_MS = 60 * 1000;
type FetchImpl = typeof fetch;
type GitHubAppConfig = {
appId: string;
installationId: string;
privateKey: string;
};
type InstallationToken = {
token: string;
expiresAt: number;
};
type CachedInstallationToken = InstallationToken & {
cacheKey: string;
};
let cachedInstallationToken: CachedInstallationToken | null = null;
export function isGitHubAppConfigured(env: NodeJS.ProcessEnv = process.env) {
return Boolean(readGitHubAppConfig(env));
}
export async function buildGitHubApiHeaders(options: {
userAgent: string;
accept?: string;
fetchImpl?: FetchImpl;
allowAnonymous?: boolean;
useGitHubApp?: boolean;
}): Promise<Record<string, string>> {
const headers = buildGitHubHeaders({
userAgent: options.userAgent,
accept: options.accept,
});
if (options.useGitHubApp !== false) {
const appToken = await getCachedGitHubAppInstallationToken({
fetchImpl: options.fetchImpl,
userAgent: options.userAgent,
});
if (appToken) {
headers.Authorization = `Bearer ${appToken}`;
return headers;
}
}
const token = process.env.GITHUB_TOKEN?.trim();
if (token) {
headers.Authorization = `Bearer ${token}`;
return headers;
}
if (options.allowAnonymous === false) {
throw new Error("GitHub API authentication is not configured");
}
return headers;
}
export function buildGitHubHeaders(options: {
userAgent: string;
accept?: string;
token?: string;
isAppJwt?: boolean;
}) {
const headers: Record<string, string> = {
Accept: options.accept ?? DEFAULT_ACCEPT,
"User-Agent": options.userAgent,
};
if (options.token) {
headers.Authorization = `Bearer ${options.token}`;
}
return headers;
}
export async function createGitHubAppInstallationToken(
options: {
fetchImpl?: FetchImpl;
userAgent?: string;
env?: NodeJS.ProcessEnv;
now?: number;
} = {},
): Promise<InstallationToken> {
const env = options.env ?? process.env;
const config = readGitHubAppConfig(env);
if (!config) throw new Error("GitHub App credentials missing");
const jwt = await createGitHubAppJwt(config.appId, config.privateKey, options.now ?? Date.now());
const response = await (options.fetchImpl ?? fetch)(
`${GITHUB_API}/app/installations/${config.installationId}/access_tokens`,
{
method: "POST",
headers: buildGitHubHeaders({
userAgent: options.userAgent ?? DEFAULT_USER_AGENT,
token: jwt,
isAppJwt: true,
}),
},
);
if (!response.ok) {
const message = await response.text();
throw new Error(`GitHub App token failed: ${message}`);
}
const payload = (await response.json()) as { token?: string; expires_at?: string };
const token = payload.token?.trim();
if (!token) throw new Error("GitHub App token missing");
const expiresAt = payload.expires_at ? Date.parse(payload.expires_at) : Number.NaN;
if (!Number.isFinite(expiresAt)) throw new Error("GitHub App token expiry missing");
return { token, expiresAt };
}
async function getCachedGitHubAppInstallationToken(options: {
fetchImpl?: FetchImpl;
userAgent: string;
}) {
const config = readGitHubAppConfig(process.env);
if (!config) return null;
const now = Date.now();
const cacheKey = `${config.appId}:${config.installationId}:${hashCacheKey(config.privateKey)}`;
if (
cachedInstallationToken?.cacheKey === cacheKey &&
cachedInstallationToken.expiresAt - APP_TOKEN_CACHE_BUFFER_MS > now
) {
return cachedInstallationToken.token;
}
try {
const next = await createGitHubAppInstallationToken({
fetchImpl: options.fetchImpl,
userAgent: options.userAgent,
now,
});
cachedInstallationToken = { ...next, cacheKey };
return next.token;
} catch (error) {
console.warn(`[githubAuth] GitHub App token unavailable: ${errorMessage(error)}`);
return null;
}
}
function readGitHubAppConfig(env: NodeJS.ProcessEnv): GitHubAppConfig | null {
const appId = env.GITHUB_APP_ID?.trim();
const installationId = env.GITHUB_APP_INSTALLATION_ID?.trim();
const privateKey = env.GITHUB_APP_PRIVATE_KEY?.trim();
if (!appId || !installationId || !privateKey) return null;
return { appId, installationId, privateKey };
}
async function createGitHubAppJwt(appId: string, rawPrivateKey: string, nowMs: number) {
const now = Math.floor(nowMs / 1000);
const header = { alg: "RS256", typ: "JWT" };
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId };
const signingInput = `${base64UrlString(JSON.stringify(header))}.${base64UrlString(
JSON.stringify(payload),
)}`;
const key = await importPrivateKey(rawPrivateKey);
const signature = await crypto.subtle.sign(
"RSASSA-PKCS1-v1_5",
key,
new TextEncoder().encode(signingInput),
);
return `${signingInput}.${base64UrlBytes(new Uint8Array(signature))}`;
}
async function importPrivateKey(rawPrivateKey: string) {
const { label, der } = parsePem(rawPrivateKey);
const pkcs8 = label === "RSA PRIVATE KEY" ? wrapPkcs1PrivateKeyAsPkcs8(der) : der;
return await crypto.subtle.importKey(
"pkcs8",
pkcs8,
{ name: "RSASSA-PKCS1-v1_5", hash: "SHA-256" },
false,
["sign"],
);
}
function parsePem(raw: string) {
const normalized = raw.replace(/\\n/g, "\n").trim();
const match = /^-----BEGIN ([A-Z0-9 ]+)-----\s*([A-Za-z0-9+/=\s]+)\s*-----END \1-----$/m.exec(
normalized,
);
if (!match) throw new Error("Invalid GitHub App private key");
const label = match[1];
if (label !== "PRIVATE KEY" && label !== "RSA PRIVATE KEY") {
throw new Error(`Unsupported GitHub App private key type: ${label}`);
}
return { label, der: base64ToBytes(match[2]) };
}
function wrapPkcs1PrivateKeyAsPkcs8(pkcs1: Uint8Array) {
const version = derInteger(0);
const rsaEncryptionAlgorithm = derSequence(
new Uint8Array([0x06, 0x09, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x01]),
new Uint8Array([0x05, 0x00]),
);
return derSequence(version, rsaEncryptionAlgorithm, derOctetString(pkcs1));
}
function derSequence(...parts: Uint8Array[]) {
return derTagged(0x30, concatBytes(parts));
}
function derInteger(value: number) {
return derTagged(0x02, new Uint8Array([value]));
}
function derOctetString(value: Uint8Array) {
return derTagged(0x04, value);
}
function derTagged(tag: number, value: Uint8Array) {
return concatBytes([new Uint8Array([tag]), derLength(value.length), value]);
}
function derLength(length: number) {
if (length < 0x80) return new Uint8Array([length]);
const bytes: number[] = [];
let remaining = length;
while (remaining > 0) {
bytes.unshift(remaining & 0xff);
remaining >>= 8;
}
return new Uint8Array([0x80 | bytes.length, ...bytes]);
}
function concatBytes(parts: Uint8Array[]) {
const total = parts.reduce((sum, part) => sum + part.length, 0);
const out = new Uint8Array(total);
let offset = 0;
for (const part of parts) {
out.set(part, offset);
offset += part.length;
}
return out;
}
function base64UrlString(value: string) {
return base64UrlBytes(new TextEncoder().encode(value));
}
function base64UrlBytes(value: Uint8Array) {
let binary = "";
for (const byte of value) binary += String.fromCharCode(byte);
return btoa(binary).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/g, "");
}
function base64ToBytes(value: string) {
const binary = atob(value.replace(/\s/g, ""));
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i += 1) {
bytes[i] = binary.charCodeAt(i);
}
return bytes;
}
function hashCacheKey(value: string) {
let hash = 0;
for (let i = 0; i < value.length; i += 1) {
hash = (hash * 31 + value.charCodeAt(i)) | 0;
}
return String(hash);
}
function errorMessage(error: unknown) {
return error instanceof Error ? error.message : String(error);
}
+3 -54
View File
@@ -1,8 +1,8 @@
"use node";
import { createPrivateKey, createSign } from "node:crypto";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { buildGitHubHeaders, createGitHubAppInstallationToken } from "./githubAuth";
const GITHUB_API = "https://api.github.com";
const DEFAULT_REPO = "clawdbot/skills";
@@ -93,7 +93,7 @@ export async function getGitHubBackupContext(): Promise<GitHubBackupContext> {
const repo = process.env.GITHUB_SKILLS_REPO ?? DEFAULT_REPO;
const root = process.env.GITHUB_SKILLS_ROOT ?? DEFAULT_ROOT;
const [repoOwner, repoName] = parseRepo(repo);
const token = await createInstallationToken();
const { token } = await createGitHubAppInstallationToken({ userAgent: USER_AGENT });
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`);
const branch = repoInfo.default_branch ?? "main";
@@ -439,48 +439,6 @@ async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<"_storage">) {
return buffer.toString("base64");
}
async function createInstallationToken() {
const appId = process.env.GITHUB_APP_ID;
const installationId = process.env.GITHUB_APP_INSTALLATION_ID;
if (!appId || !installationId) {
throw new Error("GitHub App credentials missing");
}
const jwt = createAppJwt(appId);
const response = await fetch(`${GITHUB_API}/app/installations/${installationId}/access_tokens`, {
method: "POST",
headers: buildHeaders(jwt, true),
});
if (!response.ok) {
const message = await response.text();
throw new Error(`GitHub App token failed: ${message}`);
}
const payload = (await response.json()) as { token?: string };
if (!payload.token) throw new Error("GitHub App token missing");
return payload.token;
}
function createAppJwt(appId: string) {
const privateKey = loadPrivateKey();
const now = Math.floor(Date.now() / 1000);
const header = { alg: "RS256", typ: "JWT" };
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId };
const encodedHeader = base64Url(JSON.stringify(header));
const encodedPayload = base64Url(JSON.stringify(payload));
const signingInput = `${encodedHeader}.${encodedPayload}`;
const sign = createSign("RSA-SHA256");
sign.update(signingInput);
sign.end();
const signature = sign.sign(privateKey);
return `${signingInput}.${base64Url(signature)}`;
}
function loadPrivateKey() {
const raw = process.env.GITHUB_APP_PRIVATE_KEY;
if (!raw) throw new Error("GITHUB_APP_PRIVATE_KEY is not configured");
const normalized = raw.replace(/\\n/g, "\n");
return createPrivateKey(normalized);
}
async function createBlob(token: string, repoOwner: string, repoName: string, content: string) {
const result = await githubPost<{ sha: string }>(
token,
@@ -531,11 +489,7 @@ async function githubPatch(token: string, path: string, body: unknown) {
}
function buildHeaders(token: string, isAppJwt = false) {
return {
Authorization: `${isAppJwt ? "Bearer" : "token"} ${token}`,
Accept: "application/vnd.github+json",
"User-Agent": USER_AGENT,
};
return buildGitHubHeaders({ token, isAppJwt, userAgent: USER_AGENT });
}
function parseRepo(repo: string) {
@@ -570,11 +524,6 @@ function encodePath(path: string) {
.join("/");
}
function base64Url(value: string | Uint8Array) {
const buffer = typeof value === "string" ? Buffer.from(value) : Buffer.from(value);
return buffer.toString("base64").replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/g, "");
}
function toBase64(value: string) {
return Buffer.from(value).toString("base64");
}
+204
View File
@@ -0,0 +1,204 @@
const GITHUB_HOSTS = new Set(["github.com", "www.github.com"]);
export function normalizeGitHubRepo(value: string) {
const trimmed = value
.trim()
.replace(/^git\+/, "")
.replace(/\.git$/i, "")
.replace(/^git@github\.com:/i, "https://github.com/");
if (!trimmed) return null;
const shorthand = /^([A-Za-z0-9_.-]+)\/([A-Za-z0-9_.-]+)$/.exec(trimmed);
if (shorthand) return `${shorthand[1]}/${shorthand[2]}`;
try {
const url = new URL(trimmed);
if (!GITHUB_HOSTS.has(url.hostname)) return null;
const segments = decodePathSegments(url.pathname);
const owner = segments[0] ?? "";
const repo = (segments[1] ?? "").replace(/\.git$/i, "");
if (!owner || !repo) return null;
return `${owner}/${repo}`;
} catch {
return null;
}
}
export function isRepoPathUnderRoot(path: string, root: string) {
if (!root) return true;
return path === root || path.startsWith(`${root}/`);
}
export async function createGitHubAppJwt(params: {
appId: string;
privateKeyPem: string;
now?: number;
}) {
const nowSeconds = Math.floor((params.now ?? Date.now()) / 1000);
const header = base64UrlEncode(
new TextEncoder().encode(JSON.stringify({ alg: "RS256", typ: "JWT" })),
);
const payload = base64UrlEncode(
new TextEncoder().encode(
JSON.stringify({
iat: nowSeconds - 60,
exp: nowSeconds + 9 * 60,
iss: params.appId,
}),
),
);
const signingInput = `${header}.${payload}`;
const key = await importPrivateKey(params.privateKeyPem);
const signature = await crypto.subtle.sign(
"RSASSA-PKCS1-v1_5",
key,
new TextEncoder().encode(signingInput),
);
return `${signingInput}.${base64UrlEncode(new Uint8Array(signature))}`;
}
export async function sha256Hex(value: string) {
return toHex(
new Uint8Array(await crypto.subtle.digest("SHA-256", new TextEncoder().encode(value))),
);
}
export async function hmacSha256Base64Url(secret: string, value: string) {
const digest = await hmacSha256(secret, new TextEncoder().encode(value));
return base64UrlEncode(new Uint8Array(digest));
}
export async function hmacSha256Hex(secret: string, value: ArrayBuffer) {
const digest = await hmacSha256(secret, value);
return toHex(new Uint8Array(digest));
}
export function timingSafeEqual(a: string, b: string) {
const aBytes = new TextEncoder().encode(a);
const bBytes = new TextEncoder().encode(b);
if (aBytes.length !== bBytes.length) return false;
let diff = 0;
for (let i = 0; i < aBytes.length; i += 1) {
diff |= (aBytes[i] ?? 0) ^ (bBytes[i] ?? 0);
}
return diff === 0;
}
export function base64UrlEncode(bytes: Uint8Array) {
return bytesToBase64(bytes).replaceAll("+", "-").replaceAll("/", "_").replace(/=+$/, "");
}
export function base64UrlDecode(value: string) {
const padded = value
.replaceAll("-", "+")
.replaceAll("_", "/")
.padEnd(Math.ceil(value.length / 4) * 4, "=");
return base64Decode(padded);
}
function decodePathSegments(pathname: string) {
return pathname
.split("/")
.map((segment) => segment.trim())
.filter(Boolean)
.map((segment) => {
try {
return decodeURIComponent(segment);
} catch {
return "";
}
})
.filter(Boolean);
}
async function importPrivateKey(privateKeyPem: string) {
const normalized = privateKeyPem.replace(/\\n/g, "\n").trim();
const pkcs8Match = /-----BEGIN PRIVATE KEY-----([\s\S]+?)-----END PRIVATE KEY-----/.exec(
normalized,
);
const pkcs1Match = /-----BEGIN RSA PRIVATE KEY-----([\s\S]+?)-----END RSA PRIVATE KEY-----/.exec(
normalized,
);
const der = pkcs8Match
? base64Decode(pkcs8Match[1]?.replace(/\s+/g, "") ?? "")
: pkcs1Match
? wrapPkcs1RsaPrivateKeyAsPkcs8(base64Decode(pkcs1Match[1]?.replace(/\s+/g, "") ?? ""))
: base64Decode(normalized.replace(/\s+/g, ""));
return await crypto.subtle.importKey(
"pkcs8",
toArrayBuffer(der),
{ name: "RSASSA-PKCS1-v1_5", hash: "SHA-256" },
false,
["sign"],
);
}
function wrapPkcs1RsaPrivateKeyAsPkcs8(pkcs1Der: Uint8Array) {
const rsaEncryptionOid = new Uint8Array([
0x30, 0x0d, 0x06, 0x09, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x01, 0x05, 0x00,
]);
const version = new Uint8Array([0x02, 0x01, 0x00]);
const privateKey = derEncode(0x04, pkcs1Der);
return derEncode(0x30, concatBytes([version, rsaEncryptionOid, privateKey]));
}
function derEncode(tag: number, value: Uint8Array) {
return concatBytes([new Uint8Array([tag]), derLength(value.byteLength), value]);
}
function derLength(length: number) {
if (length < 0x80) return new Uint8Array([length]);
const bytes: number[] = [];
let remaining = length;
while (remaining > 0) {
bytes.unshift(remaining & 0xff);
remaining >>= 8;
}
return new Uint8Array([0x80 | bytes.length, ...bytes]);
}
function concatBytes(parts: Uint8Array[]) {
const total = parts.reduce((sum, part) => sum + part.byteLength, 0);
const out = new Uint8Array(total);
let offset = 0;
for (const part of parts) {
out.set(part, offset);
offset += part.byteLength;
}
return out;
}
async function hmacSha256(secret: string, value: ArrayBuffer | Uint8Array) {
const key = await crypto.subtle.importKey(
"raw",
new TextEncoder().encode(secret),
{ name: "HMAC", hash: "SHA-256" },
false,
["sign"],
);
return await crypto.subtle.sign("HMAC", key, toArrayBuffer(value));
}
function base64Decode(value: string) {
const binary = atob(value);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i += 1) bytes[i] = binary.charCodeAt(i);
return bytes;
}
function bytesToBase64(bytes: Uint8Array) {
let binary = "";
for (const byte of bytes) binary += String.fromCharCode(byte);
return btoa(binary);
}
function toHex(bytes: Uint8Array) {
let out = "";
for (const byte of bytes) out += byte.toString(16).padStart(2, "0");
return out;
}
function toArrayBuffer(value: ArrayBuffer | Uint8Array) {
if (value instanceof ArrayBuffer) return value;
return new Uint8Array(value).buffer as ArrayBuffer;
}
+82 -1
View File
@@ -1,5 +1,10 @@
import { describe, expect, it } from "vitest";
import { canHealSkillOwnershipByGitHubProviderAccountId } from "./githubIdentity";
import type { Doc, Id } from "../_generated/dataModel";
import type { QueryCtx } from "../_generated/server";
import {
canHealSkillOwnershipByGitHubProviderAccountId,
getGitHubProviderAccountId,
} from "./githubIdentity";
describe("canHealSkillOwnershipByGitHubProviderAccountId", () => {
it("denies when either providerAccountId is missing", () => {
@@ -17,3 +22,79 @@ describe("canHealSkillOwnershipByGitHubProviderAccountId", () => {
expect(canHealSkillOwnershipByGitHubProviderAccountId("123", "123")).toBe(true);
});
});
describe("getGitHubProviderAccountId", () => {
const userId = "users:github-user" as Id<"users">;
it("returns null when the user has no GitHub auth account", async () => {
const ctx = createQueryCtx([]);
await expect(getGitHubProviderAccountId(ctx, userId)).resolves.toBeNull();
});
it("returns the providerAccountId for duplicate rows with the same GitHub identity", async () => {
const ctx = createQueryCtx([
createAuthAccount("authAccounts:first", "123"),
createAuthAccount("authAccounts:second", "123"),
]);
await expect(getGitHubProviderAccountId(ctx, userId)).resolves.toBe("123");
});
it("fails closed when duplicate rows disagree on the GitHub identity", async () => {
const ctx = createQueryCtx([
createAuthAccount("authAccounts:first", "123"),
createAuthAccount("authAccounts:second", "456"),
]);
await expect(getGitHubProviderAccountId(ctx, userId)).rejects.toThrow(
"Conflicting GitHub auth accounts for user users:github-user: [authAccounts:first, authAccounts:second]",
);
});
it("fails closed when duplicate rows exceed the bounded reconciliation window", async () => {
const ctx = createQueryCtx(
Array.from({ length: 11 }, (_, index) =>
createAuthAccount(`authAccounts:${index + 1}`, "123"),
),
);
await expect(getGitHubProviderAccountId(ctx, userId)).rejects.toThrow(
"Too many GitHub auth accounts for user users:github-user; manual reconciliation required: [authAccounts:1, authAccounts:2, authAccounts:3, authAccounts:4, authAccounts:5, authAccounts:6, authAccounts:7, authAccounts:8, authAccounts:9, authAccounts:10, authAccounts:11]",
);
});
function createAuthAccount(id: string, providerAccountId: string): Doc<"authAccounts"> {
return {
_id: id,
_creationTime: 1,
userId,
provider: "github",
providerAccountId,
} as unknown as Doc<"authAccounts">;
}
function createQueryCtx(accounts: Array<Doc<"authAccounts">>): Pick<QueryCtx, "db"> {
const builder = {
eq: () => builder,
};
const query = {
withIndex: (name: string, configure: (q: typeof builder) => typeof builder) => {
expect(name).toBe("userIdAndProvider");
configure(builder);
return {
take: async (limit: number) => accounts.slice(0, limit),
};
},
};
return {
db: {
query: (table: string) => {
expect(table).toBe("authAccounts");
return query;
},
},
} as unknown as Pick<QueryCtx, "db">;
}
});
+35 -4
View File
@@ -1,6 +1,8 @@
import type { Id } from "../_generated/dataModel";
import type { Doc, Id } from "../_generated/dataModel";
import type { QueryCtx } from "../_generated/server";
const MAX_GITHUB_AUTH_ACCOUNTS_PER_USER = 10;
export function canHealSkillOwnershipByGitHubProviderAccountId(
ownerProviderAccountId: string | null | undefined,
callerProviderAccountId: string | null | undefined,
@@ -14,9 +16,38 @@ export async function getGitHubProviderAccountId(
ctx: Pick<QueryCtx, "db">,
userId: Id<"users">,
): Promise<string | null> {
const account = await ctx.db
const accounts = await ctx.db
.query("authAccounts")
.withIndex("userIdAndProvider", (q) => q.eq("userId", userId).eq("provider", "github"))
.unique();
return account?.providerAccountId ?? null;
.take(MAX_GITHUB_AUTH_ACCOUNTS_PER_USER + 1);
if (accounts.length === 0) return null;
if (accounts.length > MAX_GITHUB_AUTH_ACCOUNTS_PER_USER) {
throw new Error(formatTooManyGitHubAuthAccountsError(userId, accounts));
}
const providerAccountId = accounts[0]?.providerAccountId;
if (
typeof providerAccountId !== "string" ||
accounts.some((account) => account.providerAccountId !== providerAccountId)
) {
throw new Error(formatConflictingGitHubAuthAccountsError(userId, accounts));
}
return providerAccountId;
}
function formatConflictingGitHubAuthAccountsError(
userId: Id<"users">,
accounts: Array<Doc<"authAccounts">>,
) {
const accountIds = accounts.map((account) => account._id).join(", ");
return `Conflicting GitHub auth accounts for user ${userId}: [${accountIds}]`;
}
function formatTooManyGitHubAuthAccountsError(
userId: Id<"users">,
accounts: Array<Doc<"authAccounts">>,
) {
const accountIds = accounts.map((account) => account._id).join(", ");
return `Too many GitHub auth accounts for user ${userId}; manual reconciliation required: [${accountIds}]`;
}
+1 -6
View File
@@ -91,12 +91,7 @@ export function parseGitHubImportUrl(input: string): GitHubImportUrl {
}
function canonicalGitHubImportUrl(url: URL) {
const canonical = new URL(url.toString());
canonical.username = "";
canonical.password = "";
canonical.search = "";
canonical.hash = "";
return `${canonical.origin}${canonical.pathname}`;
return `https://${url.hostname}${url.pathname}`;
}
export async function resolveGitHubCommit(
+566
View File
@@ -0,0 +1,566 @@
import { describe, expect, it } from "vitest";
import {
buildGitHubSkillSourceSnapshot,
buildGitHubSkillSyncPlan,
parseSkillsShDisplayManifest,
} from "./githubSkillSync";
const encoder = new TextEncoder();
function bytes(text: string) {
return encoder.encode(text);
}
function repoEntries(entries: Record<string, string>) {
return Object.fromEntries(Object.entries(entries).map(([path, text]) => [path, bytes(text)]));
}
describe("parseSkillsShDisplayManifest", () => {
it("keeps the supported skills.sh rendering fields and drops invalid groups", () => {
const result = parseSkillsShDisplayManifest(
JSON.stringify({
notGrouped: "top",
groupings: [
{
title: "Agentic AI",
description: "Agentic workflows.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
{ title: "Broken", skills: [123] },
{ description: "Missing title", skills: ["ignored"] },
],
}),
);
expect(result).toEqual({
status: "ok",
manifest: {
notGrouped: "top",
groupings: [
{
title: "Agentic AI",
description: "Agentic workflows.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
],
},
});
});
it("marks missing and invalid manifests so the UI can fall back", () => {
expect(parseSkillsShDisplayManifest(undefined)).toEqual({
status: "missing",
manifest: undefined,
});
expect(parseSkillsShDisplayManifest("{nope")).toEqual({
status: "invalid",
manifest: undefined,
});
expect(parseSkillsShDisplayManifest(JSON.stringify({ groupings: [] }))).toEqual({
status: "invalid",
manifest: undefined,
});
});
});
describe("buildGitHubSkillSourceSnapshot", () => {
it("discovers skill folders, parses SKILL.md metadata, and hashes exact folder bytes", async () => {
const baseEntries = repoEntries({
"skills/aiq-deploy/SKILL.md":
"---\nname: AIQ Deploy\nversion: 0.2.0\ndescription: Deploy AgentIQ workflows.\n---\n# AIQ Deploy\n",
"skills/aiq-deploy/skill-card.md": "# Card\n",
"skills/vision-helper/SKILL.md": "# Vision Helper\n",
"skills.sh.json": JSON.stringify({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
}),
});
const changedEntries = {
...baseEntries,
"skills/aiq-deploy/skill-card.md": bytes("# Card changed\n"),
};
const base = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: baseEntries,
});
const changed = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: changedEntries,
});
expect(base.manifestStatus).toBe("ok");
expect(base.manifest).toEqual({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
});
expect(base.skills).toEqual(
expect.arrayContaining([
expect.objectContaining({
slug: "aiq-deploy",
displayName: "AIQ Deploy",
summary: "Deploy AgentIQ workflows.",
upstreamVersion: "0.2.0",
path: "skills/aiq-deploy",
skillMarkdownPath: "skills/aiq-deploy/SKILL.md",
skillMarkdown:
"---\nname: AIQ Deploy\nversion: 0.2.0\ndescription: Deploy AgentIQ workflows.\n---\n# AIQ Deploy\n",
skillCardMarkdownPath: "skills/aiq-deploy/skill-card.md",
skillCardMarkdown: "# Card\n",
}),
expect.objectContaining({
slug: "vision-helper",
displayName: "Vision Helper",
path: "skills/vision-helper",
skillMarkdownPath: "skills/vision-helper/SKILL.md",
skillMarkdown: "# Vision Helper\n",
}),
]),
);
expect(changed.skills.find((skill) => skill.slug === "aiq-deploy")?.contentHash).not.toBe(
base.skills.find((skill) => skill.slug === "aiq-deploy")?.contentHash,
);
expect(changed.skills.find((skill) => skill.slug === "vision-helper")?.contentHash).toBe(
base.skills.find((skill) => skill.slug === "vision-helper")?.contentHash,
);
});
it("includes valid filenames containing dot-dot text in folder hashes", async () => {
const base = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
"skills/aiq-deploy/payload..sh": "echo safe\n",
}),
});
const changed = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
"skills/aiq-deploy/payload..sh": "echo changed\n",
}),
});
expect(changed.skills[0]?.contentHash).not.toBe(base.skills[0]?.contentHash);
});
it("rejects duplicate normalized skill slugs before syncing content", async () => {
await expect(
buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq_deploy/SKILL.md": "# AIQ Deploy A\n",
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy B\n",
}),
}),
).rejects.toThrow(/duplicate normalized slug/i);
});
it("prefers the top-level skills catalog folder over duplicate plugin copies", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
"plugins/nvidia-skills/skills/aiq-deploy/SKILL.md": "# Plugin Copy\n",
"skills.sh.json": JSON.stringify({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
}),
}),
});
expect(snapshot.skills.map((skill) => skill.path)).toEqual(["skills/aiq-deploy"]);
expect(snapshot.skills[0]?.displayName).toBe("AIQ Deploy");
});
it("rejects oversized cached markdown before writing Convex content docs", async () => {
await expect(
buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": `# AIQ Deploy\n${"x".repeat(513 * 1024)}`,
}),
}),
).rejects.toThrow(/too large to cache/i);
});
});
describe("buildGitHubSkillSyncPlan", () => {
it("marks changed upstream content pending", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy v2\n",
"skills.sh.json": JSON.stringify({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
}),
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: "old-hash",
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches).toEqual([
expect.objectContaining({
skillId: "skills:aiq-deploy",
slug: "aiq-deploy",
patch: expect.objectContaining({
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: snapshot.skills[0]?.contentHash,
githubScanStatus: "pending",
moderationStatus: "active",
moderationReason: "pending.scan",
}),
}),
]);
expect(plan.skillInserts).toEqual([]);
expect(plan.stats.changed).toBe(1);
});
it("keeps clean scan status when only the repo commit changes", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "---\nversion: 0.2.0\n---\n# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
latestVersionSummary: {
version: "0.2.0",
createdAt: 7,
},
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: contentHash,
githubScanStatus: "clean",
moderationStatus: "active",
moderationVerdict: "clean",
});
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("updatedAt");
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("latestVersionSummary");
expect(plan.stats.unchanged).toBe(1);
});
it("updates existing skill ownership when a source is reassigned", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:new-owner",
ownerPublisherId: "publishers:new-owner",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: snapshot.skills[0]?.contentHash ?? "",
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
ownerUserId: "users:new-owner",
ownerPublisherId: "publishers:new-owner",
});
});
it("preserves pending scan status for unchanged pending content", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "pending",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentCommit: "3".repeat(40),
githubCurrentContentHash: contentHash,
githubScanStatus: "pending",
moderationStatus: "active",
moderationReason: "pending.scan",
});
expect(plan.stats.unchanged).toBe(1);
});
it("preserves terminal scan status for unchanged current content", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
moderationStatus: "hidden",
moderationReason: "scanner.llm.malicious",
});
});
it("preserves terminal scan status for unchanged current bytes", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
moderationStatus: "hidden",
moderationReason: "scanner.llm.malicious",
});
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("updatedAt");
expect(plan.stats.unchanged).toBe(1);
});
it("revives soft-deleted skills when a configured repo is synced again", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "mattpocock/skills",
defaultBranch: "main",
commit: "4".repeat(40),
entries: repoEntries({
"skills/engineering/tdd/SKILL.md": "# TDD\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:matt",
ownerUserId: "users:matt",
ownerPublisherId: "publishers:matt",
existingSkills: [
{
_id: "skills:tdd",
slug: "tdd",
displayName: "TDD",
githubPath: "skills/engineering/tdd",
githubCurrentStatus: "missing",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentStatus: "present",
githubRemovedAt: undefined,
softDeletedAt: undefined,
});
});
it("tombstones upstream removals instead of leaving stale installs active", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/vision-helper/SKILL.md": "# Vision Helper\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches).toEqual([
expect.objectContaining({
skillId: "skills:aiq-deploy",
slug: "aiq-deploy",
patch: expect.objectContaining({
githubCurrentCommit: "2".repeat(40),
githubCurrentStatus: "missing",
githubRemovedAt: 123,
softDeletedAt: 123,
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
}),
}),
]);
expect(plan.skillInserts).toHaveLength(1);
expect(plan.stats.removed).toBe(1);
});
it("preserves first upstream removal time on later syncs", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/vision-helper/SKILL.md": "# Vision Helper\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "missing",
githubRemovedAt: 77,
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentCommit: "3".repeat(40),
githubCurrentStatus: "missing",
githubCurrentCheckedAt: 123,
githubRemovedAt: 77,
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
});
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("updatedAt");
});
});
+670
View File
@@ -0,0 +1,670 @@
import { getFrontmatterValue, parseFrontmatter } from "./skills";
export type GitHubSkillScanStatus = "clean" | "suspicious" | "malicious" | "pending" | "failed";
export type GitHubCurrentStatus = "present" | "missing" | "unknown";
export type DisplayManifestStatus = "ok" | "missing" | "invalid" | "failed";
export type DisplayManifest = {
notGrouped?: "top" | "bottom";
groupings: Array<{
title: string;
description?: string;
skills: string[];
}>;
};
export type GitHubSkillSourceSnapshot = {
repo: string;
defaultBranch: string;
commit: string;
manifestStatus: DisplayManifestStatus;
manifestHash?: string;
manifest?: DisplayManifest;
skills: DiscoveredGitHubSkill[];
};
export type GitHubSkillSourceMetadataSnapshot = Omit<GitHubSkillSourceSnapshot, "skills"> & {
skills: DiscoveredGitHubSkillMetadata[];
};
export type DiscoveredGitHubSkill = {
slug: string;
displayName: string;
summary?: string;
upstreamVersion?: string;
path: string;
skillMarkdownPath: string;
skillMarkdown: string;
skillCardMarkdownPath?: string;
skillCardMarkdown?: string;
contentHash: string;
};
export type DiscoveredGitHubSkillMetadata = Omit<
DiscoveredGitHubSkill,
"skillMarkdown" | "skillCardMarkdown"
>;
export type ExistingGitHubSkillForSync = {
_id: string;
slug: string;
displayName: string;
summary?: string;
latestVersionSummary?: {
version: string;
createdAt: number;
};
githubPath?: string;
githubCurrentCommit?: string;
githubCurrentContentHash?: string;
githubCurrentStatus?: GitHubCurrentStatus;
githubScanStatus?: GitHubSkillScanStatus;
githubRemovedAt?: number;
softDeletedAt?: number;
};
export type GitHubBackedSkillModeration = {
moderationStatus: "active" | "hidden";
moderationReason?: string;
moderationVerdict?: "clean" | "suspicious" | "malicious";
moderationFlags: string[];
isSuspicious: boolean;
};
export type GitHubSkillPatchForSync = {
skillId: string;
slug: string;
patch: Record<string, unknown>;
};
export type GitHubSkillInsertForSync = {
slug: string;
doc: Record<string, unknown>;
};
export type GitHubSkillSyncPlan = {
sourcePatch: Record<string, unknown>;
skillPatches: GitHubSkillPatchForSync[];
skillInserts: GitHubSkillInsertForSync[];
stats: {
discovered: number;
inserted: number;
changed: number;
unchanged: number;
removed: number;
};
};
const SKILL_MARKDOWN_BASENAME = "skill.md";
const SKILL_CARD_MARKDOWN_BASENAME = "skill-card.md";
const MAX_STORED_MARKDOWN_BYTES = 512 * 1024;
const MAX_STORED_SKILL_CONTENT_BYTES = 768 * 1024;
export function parseSkillsShDisplayManifest(raw: string | undefined | null): {
status: DisplayManifestStatus;
manifest?: DisplayManifest;
} {
if (raw === undefined || raw === null) return { status: "missing", manifest: undefined };
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
return { status: "invalid", manifest: undefined };
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
return { status: "invalid", manifest: undefined };
}
const record = parsed as Record<string, unknown>;
const rawGroups = record.groupings;
if (!Array.isArray(rawGroups)) return { status: "invalid", manifest: undefined };
const groupings = rawGroups.flatMap((group): DisplayManifest["groupings"] => {
if (!group || typeof group !== "object" || Array.isArray(group)) return [];
const groupRecord = group as Record<string, unknown>;
const title = typeof groupRecord.title === "string" ? groupRecord.title.trim() : "";
const description =
typeof groupRecord.description === "string" ? groupRecord.description.trim() : "";
const skills = Array.isArray(groupRecord.skills)
? groupRecord.skills
.filter((value): value is string => typeof value === "string")
.map((value) => value.trim())
.filter(Boolean)
: [];
if (!title || skills.length === 0) return [];
return [
{
title,
...(description ? { description } : {}),
skills,
},
];
});
if (groupings.length === 0) return { status: "invalid", manifest: undefined };
const notGrouped =
record.notGrouped === "top" || record.notGrouped === "bottom" ? record.notGrouped : undefined;
return {
status: "ok",
manifest: {
...(notGrouped ? { notGrouped } : {}),
groupings,
},
};
}
export async function buildGitHubSkillSourceSnapshot({
repo,
defaultBranch,
commit,
entries,
}: {
repo: string;
defaultBranch: string;
commit: string;
entries: Record<string, Uint8Array>;
}): Promise<GitHubSkillSourceSnapshot> {
const normalizedEntries = normalizeEntryMap(entries);
const manifestBytes = normalizedEntries["skills.sh.json"];
const manifestText = manifestBytes ? decodeUtf8(manifestBytes) : undefined;
const parsedManifest = parseSkillsShDisplayManifest(manifestText);
const manifestHash = manifestBytes ? await sha256Hex(manifestBytes) : undefined;
const skillPaths = discoverSkillPaths(normalizedEntries);
const skills: DiscoveredGitHubSkill[] = [];
for (const skillMdPath of skillPaths) {
const path = parentPath(skillMdPath);
const markdownBytes = normalizedEntries[skillMdPath] ?? new Uint8Array();
assertStoredMarkdownSize(skillMdPath, markdownBytes);
const markdown = decodeUtf8(markdownBytes);
const frontmatter = parseFrontmatter(markdown);
const folderName = path.split("/").filter(Boolean).at(-1) ?? "";
const slug = slugFromPathSegment(folderName);
if (!slug) continue;
const frontmatterName = getFrontmatterValue(frontmatter, "name")?.trim();
const frontmatterDescription = getFrontmatterValue(frontmatter, "description")?.trim();
const frontmatterVersion = getFrontmatterValue(frontmatter, "version")?.trim();
const heading = firstMarkdownHeading(markdown);
const skillCardMarkdownPath = findFolderFilePath(
normalizedEntries,
path,
SKILL_CARD_MARKDOWN_BASENAME,
);
const skillCardBytes = skillCardMarkdownPath
? normalizedEntries[skillCardMarkdownPath]
: undefined;
if (skillCardMarkdownPath && skillCardBytes) {
assertStoredMarkdownSize(skillCardMarkdownPath, skillCardBytes);
assertStoredSkillContentSize(markdownBytes.byteLength + skillCardBytes.byteLength);
} else {
assertStoredSkillContentSize(markdownBytes.byteLength);
}
const skillCardMarkdown = skillCardBytes ? decodeUtf8(skillCardBytes) : undefined;
skills.push({
slug,
displayName: frontmatterName || heading || titleizeSlug(slug),
...(frontmatterDescription ? { summary: frontmatterDescription } : {}),
...(frontmatterVersion ? { upstreamVersion: frontmatterVersion } : {}),
path,
skillMarkdownPath: skillMdPath,
skillMarkdown: markdown,
...(skillCardMarkdownPath ? { skillCardMarkdownPath } : {}),
...(skillCardMarkdown !== undefined ? { skillCardMarkdown } : {}),
contentHash: await computeGitHubSkillFolderContentHash(normalizedEntries, path),
});
}
const sortedSkills = skills.sort((a, b) => a.path.localeCompare(b.path));
assertUniqueDiscoveredSlugs(sortedSkills);
return {
repo,
defaultBranch,
commit,
manifestStatus: parsedManifest.status,
...(manifestHash ? { manifestHash } : {}),
...(parsedManifest.manifest ? { manifest: parsedManifest.manifest } : {}),
skills: sortedSkills,
};
}
export async function computeGitHubSkillFolderContentHash(
entries: Record<string, Uint8Array>,
folderPath: string,
) {
const normalizedEntries = normalizeEntryMap(entries);
const root = folderPath ? `${folderPath}/` : "";
const lines: string[] = [];
for (const [path, content] of Object.entries(normalizedEntries).sort(([a], [b]) =>
a.localeCompare(b),
)) {
if (root && path !== folderPath && !path.startsWith(root)) continue;
if (!root && path.includes("/")) continue;
const relativePath = root ? path.slice(root.length) : path;
if (!relativePath) continue;
const fileHash = await sha256Hex(content);
lines.push(`${relativePath}\0${content.byteLength}\0${fileHash}`);
}
return sha256Hex(new TextEncoder().encode(lines.join("\n")));
}
export function buildGitHubSkillSyncPlan({
sourceId,
ownerUserId,
ownerPublisherId,
existingSkills,
snapshot,
now,
}: {
sourceId: string;
ownerUserId: string;
ownerPublisherId?: string;
existingSkills: ExistingGitHubSkillForSync[];
snapshot: GitHubSkillSourceSnapshot | GitHubSkillSourceMetadataSnapshot;
now: number;
}): GitHubSkillSyncPlan {
const sourcePatch = {
repo: snapshot.repo,
defaultBranch: snapshot.defaultBranch,
lastSyncStatus: "ok",
lastSyncError: undefined,
lastSyncErrorAt: undefined,
displayManifestKind: "skills.sh",
displayManifestHash: snapshot.manifestHash,
displayManifestCommit: snapshot.commit,
displayManifestFetchedAt: now,
displayManifestStatus: snapshot.manifestStatus,
displayManifest: snapshot.manifest,
...(ownerPublisherId ? { ownerPublisherId } : {}),
updatedAt: now,
};
const existingByPath = new Map(
existingSkills
.filter((skill) => skill.githubPath)
.map((skill) => [skill.githubPath as string, skill]),
);
const existingBySlug = new Map(existingSkills.map((skill) => [skill.slug, skill]));
const matchedSkillIds = new Set<string>();
const skillPatches: GitHubSkillPatchForSync[] = [];
const skillInserts: GitHubSkillInsertForSync[] = [];
const stats = {
discovered: snapshot.skills.length,
inserted: 0,
changed: 0,
unchanged: 0,
removed: 0,
};
for (const discovered of snapshot.skills) {
const existing = existingByPath.get(discovered.path) ?? existingBySlug.get(discovered.slug);
if (!existing) {
const scanStatus: GitHubSkillScanStatus = "pending";
const moderation = githubBackedSkillModeration(scanStatus);
skillInserts.push({
slug: discovered.slug,
doc: {
slug: discovered.slug,
displayName: discovered.displayName,
summary: discovered.summary,
ownerUserId,
ownerPublisherId,
installKind: "github",
githubSourceId: sourceId,
githubPath: discovered.path,
githubHasSkillCard: Boolean(discovered.skillCardMarkdownPath),
githubCurrentCommit: snapshot.commit,
githubCurrentContentHash: discovered.contentHash,
githubCurrentStatus: "present",
githubCurrentCheckedAt: now,
githubScanStatus: scanStatus,
githubRemovedAt: undefined,
latestVersionId: undefined,
latestVersionSummary: latestVersionSummary(discovered.upstreamVersion, now),
tags: {},
capabilityTags: [],
softDeletedAt: undefined,
badges: undefined,
statsDownloads: 0,
statsStars: 0,
statsInstallsCurrent: 0,
statsInstallsAllTime: 0,
stats: {
downloads: 0,
stars: 0,
installsCurrent: 0,
installsAllTime: 0,
versions: 0,
comments: 0,
},
...moderation,
createdAt: now,
updatedAt: now,
},
});
stats.inserted += 1;
continue;
}
matchedSkillIds.add(existing._id);
const currentContentUnchanged =
existing.githubCurrentStatus === "present" &&
existing.githubCurrentContentHash === discovered.contentHash;
const scanStatus: GitHubSkillScanStatus = currentContentUnchanged
? githubScanStatusForUnchangedContent(existing.githubScanStatus)
: "pending";
const moderation = githubBackedSkillModeration(scanStatus);
const nextLatestVersionSummary = latestVersionSummary(
discovered.upstreamVersion,
existing.latestVersionSummary?.createdAt ?? now,
);
const materialChanged =
!currentContentUnchanged ||
existing.displayName !== discovered.displayName ||
(existing.summary ?? undefined) !== (discovered.summary ?? undefined) ||
(existing.githubPath ?? undefined) !== discovered.path ||
!sameLatestVersionSummary(existing.latestVersionSummary, nextLatestVersionSummary);
const patch = {
displayName: discovered.displayName,
summary: discovered.summary,
ownerUserId,
...(ownerPublisherId ? { ownerPublisherId } : {}),
githubSourceId: sourceId,
githubPath: discovered.path,
githubHasSkillCard: Boolean(discovered.skillCardMarkdownPath),
githubCurrentCommit: snapshot.commit,
githubCurrentContentHash: discovered.contentHash,
githubCurrentStatus: "present",
githubCurrentCheckedAt: now,
githubScanStatus: scanStatus,
githubRemovedAt: undefined,
softDeletedAt: undefined,
...(materialChanged
? {
latestVersionSummary: latestVersionSummary(discovered.upstreamVersion, now),
updatedAt: now,
}
: {}),
...moderation,
};
skillPatches.push({ skillId: existing._id, slug: existing.slug, patch });
if (materialChanged) stats.changed += 1;
else stats.unchanged += 1;
}
for (const existing of existingSkills) {
if (matchedSkillIds.has(existing._id)) continue;
const removedAt = existing.githubRemovedAt ?? now;
const moderation = githubBackedSkillModeration(
existing.githubScanStatus ?? "pending",
removedAt,
);
const wasAlreadyRemoved =
existing.githubCurrentStatus === "missing" && existing.githubRemovedAt !== undefined;
skillPatches.push({
skillId: existing._id,
slug: existing.slug,
patch: {
githubCurrentCommit: snapshot.commit,
githubCurrentStatus: "missing",
githubCurrentCheckedAt: now,
githubRemovedAt: removedAt,
softDeletedAt: existing.softDeletedAt ?? removedAt,
...(wasAlreadyRemoved ? {} : { updatedAt: now }),
...moderation,
},
});
stats.removed += 1;
}
return { sourcePatch, skillPatches, skillInserts, stats };
}
function githubScanStatusForUnchangedContent(
status: GitHubSkillScanStatus | undefined,
): GitHubSkillScanStatus {
if (
status === "clean" ||
status === "failed" ||
status === "malicious" ||
status === "suspicious"
) {
return status;
}
return "pending";
}
export function githubBackedSkillModeration(
scanStatus: GitHubSkillScanStatus,
removedAt?: number,
): GitHubBackedSkillModeration {
if (typeof removedAt === "number") {
return {
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
moderationVerdict: undefined,
moderationFlags: [],
isSuspicious: false,
};
}
if (scanStatus === "pending") {
return {
moderationStatus: "active",
moderationReason: "pending.scan",
moderationVerdict: undefined,
moderationFlags: [],
isSuspicious: false,
};
}
if (scanStatus === "failed") {
return {
moderationStatus: "hidden",
moderationReason: "scanner.failed",
moderationVerdict: undefined,
moderationFlags: [],
isSuspicious: false,
};
}
if (scanStatus === "malicious") {
return {
moderationStatus: "hidden",
moderationReason: "scanner.llm.malicious",
moderationVerdict: "malicious",
moderationFlags: ["blocked.malware"],
isSuspicious: true,
};
}
if (scanStatus === "suspicious") {
return {
moderationStatus: "active",
moderationReason: "scanner.llm.suspicious",
moderationVerdict: "suspicious",
moderationFlags: ["flagged.suspicious"],
isSuspicious: true,
};
}
return {
moderationStatus: "active",
moderationReason: undefined,
moderationVerdict: "clean",
moderationFlags: [],
isSuspicious: false,
};
}
function latestVersionSummary(version: string | undefined, now: number) {
if (!version) return undefined;
return {
version,
createdAt: now,
changelog: "Synced from GitHub source.",
changelogSource: "auto" as const,
};
}
function sameLatestVersionSummary(
a: ExistingGitHubSkillForSync["latestVersionSummary"] | undefined,
b: ReturnType<typeof latestVersionSummary>,
) {
if (!a && !b) return true;
if (!a || !b) return false;
return a.version === b.version;
}
function assertStoredMarkdownSize(path: string, bytes: Uint8Array) {
if (bytes.byteLength > MAX_STORED_MARKDOWN_BYTES) {
throw new Error(`GitHub skill markdown file is too large to cache: ${path}`);
}
}
function assertStoredSkillContentSize(totalBytes: number) {
if (totalBytes > MAX_STORED_SKILL_CONTENT_BYTES) {
throw new Error("GitHub skill cached markdown is too large");
}
}
function assertUniqueDiscoveredSlugs(skills: DiscoveredGitHubSkill[]) {
const firstPathBySlug = new Map<string, string>();
for (const skill of skills) {
const firstPath = firstPathBySlug.get(skill.slug);
if (firstPath) {
throw duplicateSkillSlugError(skill.slug, firstPath, skill.path);
}
firstPathBySlug.set(skill.slug, skill.path);
}
}
function normalizeEntryMap(entries: Record<string, Uint8Array>) {
const out: Record<string, Uint8Array> = {};
for (const [rawPath, bytes] of Object.entries(entries)) {
const normalized = normalizeRepoPath(rawPath);
if (!normalized) continue;
out[normalized] = new Uint8Array(bytes);
}
return out;
}
function discoverSkillPaths(entries: Record<string, Uint8Array>) {
const candidates = Object.keys(entries)
.filter((path) => path.split("/").at(-1)?.toLowerCase() === SKILL_MARKDOWN_BASENAME)
.sort((a, b) => a.localeCompare(b));
const pathsBySlug = new Map<string, string[]>();
for (const skillMdPath of candidates) {
const path = parentPath(skillMdPath);
const folderName = path.split("/").filter(Boolean).at(-1) ?? "";
const slug = slugFromPathSegment(folderName);
if (!slug) continue;
const paths = pathsBySlug.get(slug) ?? [];
paths.push(skillMdPath);
pathsBySlug.set(slug, paths);
}
const selected: string[] = [];
for (const [slug, paths] of pathsBySlug) {
if (paths.length === 1) {
selected.push(paths[0] as string);
continue;
}
const canonicalPath = `skills/${slug}/${SKILL_MARKDOWN_BASENAME}`;
const exactTopLevelMatches = paths.filter((path) => path.toLowerCase() === canonicalPath);
const topLevelSkillMatches = paths.filter((path) => path.toLowerCase().startsWith("skills/"));
if (exactTopLevelMatches.length === 1 && topLevelSkillMatches.length === 1) {
selected.push(exactTopLevelMatches[0] as string);
continue;
}
throw duplicateSkillSlugError(
slug,
parentPath(paths[0] as string),
parentPath(paths[1] as string),
);
}
return selected.sort((a, b) => a.localeCompare(b));
}
function duplicateSkillSlugError(slug: string, firstPath: string, secondPath: string) {
return new Error(
`GitHub skill source has duplicate normalized slug "${slug}" at ${firstPath} and ${secondPath}`,
);
}
function findFolderFilePath(
entries: Record<string, Uint8Array>,
folderPath: string,
basename: string,
) {
const prefix = folderPath ? `${folderPath}/` : "";
return Object.keys(entries).find((entryPath) => {
if (prefix) {
if (!entryPath.startsWith(prefix)) return false;
const relativePath = entryPath.slice(prefix.length);
return !relativePath.includes("/") && relativePath.toLowerCase() === basename;
}
return !entryPath.includes("/") && entryPath.toLowerCase() === basename;
});
}
function normalizeRepoPath(path: string) {
if (path.includes("\u0000")) return "";
const normalized = path
.replaceAll("\\", "/")
.trim()
.replace(/^\.\/+/, "")
.replace(/^\/+/, "");
if (!normalized) return "";
const segments = normalized.split("/").filter(Boolean);
if (segments.some((segment) => segment === "." || segment === "..")) return "";
return segments.join("/");
}
function parentPath(path: string) {
return path.split("/").slice(0, -1).join("/");
}
function decodeUtf8(bytes: Uint8Array) {
return new TextDecoder().decode(bytes);
}
async function sha256Hex(bytes: Uint8Array) {
const safe = new Uint8Array(bytes);
const digest = await crypto.subtle.digest("SHA-256", safe);
return toHex(new Uint8Array(digest));
}
function toHex(bytes: Uint8Array) {
let out = "";
for (const byte of bytes) out += byte.toString(16).padStart(2, "0");
return out;
}
function slugFromPathSegment(value: string) {
return value
.trim()
.toLowerCase()
.replace(/[\s_]+/g, "-")
.replace(/[^a-z0-9-]+/g, "-")
.replace(/-+/g, "-")
.replace(/^-|-$/g, "");
}
function titleizeSlug(slug: string) {
return slug
.split("-")
.filter(Boolean)
.map((part) => part.charAt(0).toUpperCase() + part.slice(1))
.join(" ");
}
function firstMarkdownHeading(markdown: string) {
for (const line of markdown.split(/\r?\n/)) {
const match = /^#\s+(.+)$/.exec(line.trim());
if (match?.[1]) return match[1].trim();
}
return undefined;
}
-448
View File
@@ -1,448 +0,0 @@
"use node";
import { createPrivateKey, createSign } from "node:crypto";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
const GITHUB_API = "https://api.github.com";
const DEFAULT_REPO = "clawdbot/souls";
const DEFAULT_ROOT = "souls";
const META_FILENAME = "_meta.json";
const USER_AGENT = "clawhub/souls-backup";
type BackupFile = {
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
};
type BackupParams = {
slug: string;
version: string;
displayName: string;
ownerHandle: string;
files: BackupFile[];
publishedAt: number;
};
type RepoInfo = {
default_branch?: string;
};
type GitRef = {
object: { sha: string };
};
type GitCommit = {
sha: string;
tree: { sha: string };
};
type GitTreeEntry = {
path?: string;
type?: string;
};
type GitTree = {
tree?: GitTreeEntry[];
};
type MetaFile = {
owner: string;
slug: string;
displayName: string;
latest: {
version: string;
publishedAt: number;
commit: string | null;
};
history: Array<{
version: string;
publishedAt: number;
commit: string;
}>;
};
export type GitHubBackupContext = {
token: string;
repo: string;
repoOwner: string;
repoName: string;
branch: string;
root: string;
};
export function isGitHubSoulBackupConfigured() {
return Boolean(
process.env.GITHUB_APP_ID &&
process.env.GITHUB_APP_PRIVATE_KEY &&
process.env.GITHUB_APP_INSTALLATION_ID,
);
}
export async function getGitHubSoulBackupContext(): Promise<GitHubBackupContext> {
const repo = process.env.GITHUB_SOULS_REPO ?? DEFAULT_REPO;
const root = process.env.GITHUB_SOULS_ROOT ?? DEFAULT_ROOT;
const [repoOwner, repoName] = parseRepo(repo);
const token = await createInstallationToken();
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`);
const branch = repoInfo.default_branch ?? "main";
return { token, repo, repoOwner, repoName, branch, root };
}
export async function fetchGitHubSoulMeta(
context: GitHubBackupContext,
ownerHandle: string,
slug: string,
): Promise<MetaFile | null> {
const soulRoot = buildSoulRoot(context.root, ownerHandle, slug);
return fetchMetaFile(
context.token,
context.repoOwner,
context.repoName,
`${soulRoot}/${META_FILENAME}`,
context.branch,
);
}
export async function backupSoulToGitHub(
ctx: ActionCtx,
params: BackupParams,
context?: GitHubBackupContext,
) {
if (!isGitHubSoulBackupConfigured()) return;
const resolved = context ?? (await getGitHubSoulBackupContext());
const soulRoot = buildSoulRoot(resolved.root, params.ownerHandle, params.slug);
const ref = await githubGet<GitRef>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/ref/heads/${resolved.branch}`,
);
const baseCommitSha = ref.object.sha;
const baseCommit = await githubGet<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits/${baseCommitSha}`,
);
const baseTreeSha = baseCommit.tree.sha;
const existingTree = await githubGet<GitTree>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees/${baseTreeSha}?recursive=1`,
);
const prefix = `${soulRoot}/`;
const existingPaths = new Set(
(existingTree.tree ?? [])
.filter((entry) => entry.type === "blob" && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? ""),
);
const newPaths = new Set<string>();
const treeEntries: Array<{
path: string;
mode: "100644";
type: "blob";
sha: string | null;
}> = [];
for (const file of params.files) {
const content = await fetchStorageBase64(ctx, file.storageId);
const blobSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
content,
);
const path = `${soulRoot}/${file.path}`;
newPaths.add(path);
treeEntries.push({ path, mode: "100644", type: "blob", sha: blobSha });
}
const existingMeta = await fetchMetaFile(
resolved.token,
resolved.repoOwner,
resolved.repoName,
`${soulRoot}/${META_FILENAME}`,
resolved.branch,
);
const metaPath = `${soulRoot}/${META_FILENAME}`;
const metaDraft = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, null);
const metaDraftContent = `${JSON.stringify(metaDraft, null, 2)}\n`;
const metaDraftSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaDraftContent),
);
newPaths.add(metaPath);
treeEntries.push({ path: metaPath, mode: "100644", type: "blob", sha: metaDraftSha });
for (const path of existingPaths) {
if (newPaths.has(path)) continue;
treeEntries.push({ path, mode: "100644", type: "blob", sha: null });
}
const newTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: baseTreeSha,
tree: treeEntries,
},
);
const commit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `soul: ${params.slug} v${params.version}`,
tree: newTree.sha,
parents: [baseCommitSha],
},
);
const metaFinal = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, commit.sha);
const metaFinalContent = `${JSON.stringify(metaFinal, null, 2)}\n`;
const metaFinalSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaFinalContent),
);
const metaTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: commit.tree.sha,
tree: [{ path: metaPath, mode: "100644", type: "blob", sha: metaFinalSha }],
},
);
const metaCommit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `meta: ${params.slug} v${params.version}`,
tree: metaTree.sha,
parents: [commit.sha],
},
);
await githubPatch(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/refs/heads/${resolved.branch}`,
{
sha: metaCommit.sha,
},
);
}
function buildMetaFile(
params: BackupParams,
existing: MetaFile | null,
repo: string,
baseCommitSha: string,
latestCommitSha: string | null,
): MetaFile {
let history = [...(existing?.history ?? [])];
if (existing?.latest?.version) {
const previousCommit = existing.latest.commit ?? commitUrl(repo, baseCommitSha);
const previous = {
version: existing.latest.version,
publishedAt: existing.latest.publishedAt,
commit: previousCommit,
};
history = [previous, ...history.filter((entry) => entry.version !== previous.version)];
}
return {
owner: normalizeOwner(params.ownerHandle),
slug: params.slug,
displayName: params.displayName,
latest: {
version: params.version,
publishedAt: params.publishedAt,
commit: latestCommitSha ? commitUrl(repo, latestCommitSha) : null,
},
history: history.slice(0, 200),
};
}
async function fetchMetaFile(
token: string,
repoOwner: string,
repoName: string,
path: string,
branch: string,
): Promise<MetaFile | null> {
try {
const response = await githubGet<{ content?: string }>(
token,
`/repos/${repoOwner}/${repoName}/contents/${encodePath(path)}?ref=${branch}`,
);
if (!response.content) return null;
const raw = fromBase64(response.content);
return JSON.parse(raw) as MetaFile;
} catch (error) {
if (isNotFoundError(error)) return null;
throw error;
}
}
async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<"_storage">) {
const blob = await ctx.storage.get(storageId);
if (!blob) throw new Error("File missing in storage");
const buffer = Buffer.from(await blob.arrayBuffer());
return buffer.toString("base64");
}
async function createInstallationToken() {
const appId = process.env.GITHUB_APP_ID;
const installationId = process.env.GITHUB_APP_INSTALLATION_ID;
if (!appId || !installationId) {
throw new Error("GitHub App credentials missing");
}
const jwt = createAppJwt(appId);
const response = await fetch(`${GITHUB_API}/app/installations/${installationId}/access_tokens`, {
method: "POST",
headers: buildHeaders(jwt, true),
});
if (!response.ok) {
const message = await response.text();
throw new Error(`GitHub App token failed: ${message}`);
}
const payload = (await response.json()) as { token?: string };
if (!payload.token) throw new Error("GitHub App token missing");
return payload.token;
}
function createAppJwt(appId: string) {
const privateKey = loadPrivateKey();
const now = Math.floor(Date.now() / 1000);
const header = { alg: "RS256", typ: "JWT" };
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId };
const encodedHeader = base64Url(JSON.stringify(header));
const encodedPayload = base64Url(JSON.stringify(payload));
const signingInput = `${encodedHeader}.${encodedPayload}`;
const sign = createSign("RSA-SHA256");
sign.update(signingInput);
sign.end();
const signature = sign.sign(privateKey);
return `${signingInput}.${base64Url(signature)}`;
}
function loadPrivateKey() {
const raw = process.env.GITHUB_APP_PRIVATE_KEY;
if (!raw) throw new Error("GITHUB_APP_PRIVATE_KEY is not configured");
const normalized = raw.replace(/\\n/g, "\n");
return createPrivateKey(normalized);
}
async function createBlob(token: string, repoOwner: string, repoName: string, content: string) {
const result = await githubPost<{ sha: string }>(
token,
`/repos/${repoOwner}/${repoName}/git/blobs`,
{
content,
encoding: "base64",
},
);
if (!result.sha) throw new Error("GitHub blob missing sha");
return result.sha;
}
async function githubGet<T>(token: string, path: string): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
headers: buildHeaders(token),
});
if (!response.ok) {
const message = await response.text();
throw new Error(`GitHub GET ${path} failed: ${message}`);
}
return (await response.json()) as T;
}
async function githubPost<T>(token: string, path: string, body: unknown): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
method: "POST",
headers: buildHeaders(token),
body: JSON.stringify(body),
});
if (!response.ok) {
const message = await response.text();
throw new Error(`GitHub POST ${path} failed: ${message}`);
}
return (await response.json()) as T;
}
async function githubPatch(token: string, path: string, body: unknown) {
const response = await fetch(`${GITHUB_API}${path}`, {
method: "PATCH",
headers: buildHeaders(token),
body: JSON.stringify(body),
});
if (!response.ok) {
const message = await response.text();
throw new Error(`GitHub PATCH ${path} failed: ${message}`);
}
}
function buildHeaders(token: string, isAppJwt = false) {
return {
Authorization: `${isAppJwt ? "Bearer" : "token"} ${token}`,
Accept: "application/vnd.github+json",
"User-Agent": USER_AGENT,
};
}
function parseRepo(repo: string) {
const [owner, name] = repo.split("/");
if (!owner || !name) throw new Error("GITHUB_SOULS_REPO must be owner/repo");
return [owner, name] as const;
}
function normalizeOwner(value: string) {
const normalized = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]/g, "-")
.replace(/-+/g, "-")
.replace(/^-+|-+$/g, "");
return normalized || "unknown";
}
function commitUrl(repo: string, sha: string) {
return `https://github.com/${repo}/commit/${sha}`;
}
function buildSoulRoot(root: string, ownerHandle: string, slug: string) {
const ownerSegment = normalizeOwner(ownerHandle);
return `${root}/${ownerSegment}/${slug}`;
}
function encodePath(path: string) {
return path
.split("/")
.map((segment) => encodeURIComponent(segment))
.join("/");
}
function base64Url(value: string | Uint8Array) {
const buffer = typeof value === "string" ? Buffer.from(value) : Buffer.from(value);
return buffer.toString("base64").replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/g, "");
}
function toBase64(value: string) {
return Buffer.from(value).toString("base64");
}
function fromBase64(value: string) {
return Buffer.from(value, "base64").toString("utf8");
}
function isNotFoundError(error: unknown) {
return (
error instanceof Error && (error.message.includes("404") || error.message.includes("Not Found"))
);
}
+99
View File
@@ -1,11 +1,17 @@
import type { Doc } from "../_generated/dataModel";
import type { MutationCtx, QueryCtx } from "../_generated/server";
import { isPackageBlockedFromPublic } from "./packageSecurity";
export const GLOBAL_STATS_KEY = "default";
type SkillVisibilityFields = Pick<
Doc<"skills">,
"softDeletedAt" | "moderationStatus" | "moderationFlags"
> &
Partial<Pick<Doc<"skills">, "moderationVerdict">>;
type PackageVisibilityFields = Pick<
Doc<"packageSearchDigest">,
"softDeletedAt" | "family" | "channel" | "scanStatus"
>;
type GlobalStatsReadCtx = Pick<MutationCtx | QueryCtx, "db">;
@@ -16,6 +22,7 @@ export function isPublicSkillDoc<T extends SkillVisibilityFields>(
): skill is T {
if (!skill || skill.softDeletedAt) return false;
if (skill.moderationStatus && skill.moderationStatus !== "active") return false;
if (skill.moderationVerdict === "malicious") return false;
if (skill.moderationFlags?.includes("blocked.malware")) return false;
return true;
}
@@ -30,6 +37,26 @@ export function getPublicSkillVisibilityDelta(
return afterPublic ? 1 : -1;
}
export function isPublicPluginDoc<T extends PackageVisibilityFields>(
pkg: T | null | undefined,
): pkg is T {
if (!pkg || pkg.softDeletedAt) return false;
if (pkg.family !== "code-plugin" && pkg.family !== "bundle-plugin") return false;
if (pkg.channel === "private") return false;
if (isPackageBlockedFromPublic(pkg.scanStatus)) return false;
return true;
}
export function getPublicPluginVisibilityDelta(
before: PackageVisibilityFields | null | undefined,
after: PackageVisibilityFields | null | undefined,
) {
const beforePublic = isPublicPluginDoc(before);
const afterPublic = isPublicPluginDoc(after);
if (beforePublic === afterPublic) return 0;
return afterPublic ? 1 : -1;
}
function getErrorMessage(error: unknown) {
if (typeof error === "string") return error;
if (error && typeof error === "object" && "message" in error) {
@@ -81,6 +108,34 @@ export async function setGlobalPublicSkillsCount(
}
}
export async function setGlobalPublicPluginsCount(
ctx: GlobalStatsWriteCtx,
count: number,
now = Date.now(),
) {
const normalizedCount = Math.max(0, Math.trunc(Number.isFinite(count) ? count : 0));
try {
const existing = await ctx.db
.query("globalStats")
.withIndex("by_key", (q) => q.eq("key", GLOBAL_STATS_KEY))
.unique();
if (existing) {
await ctx.db.patch(existing._id, { activePluginsCount: normalizedCount, updatedAt: now });
} else {
await ctx.db.insert("globalStats", {
key: GLOBAL_STATS_KEY,
activeSkillsCount: 0,
activePluginsCount: normalizedCount,
updatedAt: now,
});
}
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return;
throw error;
}
}
export async function adjustGlobalPublicSkillsCount(
ctx: GlobalStatsWriteCtx,
delta: number,
@@ -116,6 +171,37 @@ export async function adjustGlobalPublicSkillsCount(
await ctx.db.patch(existing._id, { activeSkillsCount: nextCount, updatedAt: now });
}
export async function adjustGlobalPublicPluginsCount(
ctx: GlobalStatsWriteCtx,
delta: number,
now = Date.now(),
) {
const normalizedDelta = Math.trunc(Number.isFinite(delta) ? delta : 0);
if (normalizedDelta === 0) return;
let existing:
| {
_id: Doc<"globalStats">["_id"];
activePluginsCount?: number;
}
| null
| undefined;
try {
existing = await ctx.db
.query("globalStats")
.withIndex("by_key", (q) => q.eq("key", GLOBAL_STATS_KEY))
.unique();
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return;
throw error;
}
if (!existing || existing.activePluginsCount === undefined) return;
const nextCount = Math.max(0, existing.activePluginsCount + normalizedDelta);
await ctx.db.patch(existing._id, { activePluginsCount: nextCount, updatedAt: now });
}
export async function readGlobalPublicSkillsCount(ctx: GlobalStatsReadCtx) {
try {
const stats = await ctx.db
@@ -128,3 +214,16 @@ export async function readGlobalPublicSkillsCount(ctx: GlobalStatsReadCtx) {
throw error;
}
}
export async function readGlobalPublicPluginsCount(ctx: GlobalStatsReadCtx) {
try {
const stats = await ctx.db
.query("globalStats")
.withIndex("by_key", (q) => q.eq("key", GLOBAL_STATS_KEY))
.unique();
return stats?.activePluginsCount ?? null;
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return null;
throw error;
}
}
+214
View File
@@ -0,0 +1,214 @@
import { describe, expect, it } from "vitest";
import { buildSkillInstallResolution } from "./installResolver";
const baseSkill = {
slug: "aiq-deploy",
displayName: "AIQ Deploy",
latestVersionSummary: null,
installKind: "github" as const,
githubPath: "skills/aiq-deploy",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy",
githubCurrentStatus: "present" as const,
githubScanStatus: "clean" as const,
githubRemovedAt: undefined,
};
const source = {
repo: "NVIDIA/skills",
defaultBranch: "main",
};
describe("buildSkillInstallResolution", () => {
it("returns an archive descriptor for hosted direct uploads", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
slug: "direct-skill",
displayName: "Direct Skill",
latestVersionSummary: { version: "1.2.3" },
},
source: null,
});
expect(resolution).toEqual({
ok: true,
slug: "direct-skill",
installKind: "archive",
archive: {
version: "1.2.3",
downloadUrl: "https://clawhub.ai/api/v1/download?slug=direct-skill&version=1.2.3",
},
});
});
it("returns a pinned GitHub descriptor when current upstream state is scan-clean", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: baseSkill,
source,
});
expect(resolution).toEqual({
ok: true,
slug: "aiq-deploy",
installKind: "github",
github: {
repo: "NVIDIA/skills",
path: "skills/aiq-deploy",
commit: "1".repeat(40),
contentHash: "hash-aiq-deploy",
sourceUrl: `https://github.com/NVIDIA/skills/tree/${"1".repeat(40)}/skills/aiq-deploy`,
},
});
});
it("allows GitHub-backed installs when upstream content changed and the current hash is clean", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy-v2",
},
source,
});
expect(resolution).toMatchObject({
ok: true,
installKind: "github",
github: {
commit: "2".repeat(40),
contentHash: "hash-aiq-deploy-v2",
sourceUrl: `https://github.com/NVIDIA/skills/tree/${"2".repeat(40)}/skills/aiq-deploy`,
},
});
});
it("allows GitHub-backed installs when only unrelated repository content changed", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: baseSkill.githubCurrentContentHash,
},
source,
});
expect(resolution).toMatchObject({
ok: true,
installKind: "github",
github: {
commit: "2".repeat(40),
contentHash: "hash-aiq-deploy",
sourceUrl: `https://github.com/NVIDIA/skills/tree/${"2".repeat(40)}/skills/aiq-deploy`,
},
});
});
it.each([
{
name: "upstream path is missing",
patch: { githubCurrentStatus: "missing" as const },
reason: "github_upstream_missing",
status: 410,
},
{
name: "skill was pulled upstream",
patch: { githubRemovedAt: 456 },
reason: "github_upstream_removed",
status: 410,
},
{
name: "scan is pending",
patch: { githubScanStatus: "pending" as const },
reason: "github_verification_pending",
status: 423,
},
{
name: "scan failed",
patch: { githubScanStatus: "failed" as const },
reason: "github_scan_failed",
status: 403,
},
{
name: "scan is suspicious",
patch: { githubScanStatus: "suspicious" as const },
reason: "github_scan_failed",
status: 403,
},
])("blocks GitHub-backed installs when $name", ({ patch, reason, status }) => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: { ...baseSkill, ...patch },
source,
});
expect(resolution).toMatchObject({
ok: false,
slug: "aiq-deploy",
reason,
status,
});
});
it("explains pending GitHub-backed verification clearly", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubScanStatus: "pending",
},
source,
});
expect(resolution).toMatchObject({
ok: false,
slug: "aiq-deploy",
reason: "github_verification_pending",
status: 423,
message:
"GitHub-backed skill security scan is in progress. Try again shortly, or rerun with --force-install to install the unverified upstream commit.",
});
});
it("allows force-install for pending GitHub-backed verification", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubScanStatus: "pending",
},
source,
forceInstall: true,
});
expect(resolution).toMatchObject({
ok: true,
installKind: "github",
github: {
commit: "1".repeat(40),
contentHash: "hash-aiq-deploy",
},
});
});
it("does not force-install failed GitHub-backed scans", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubScanStatus: "failed",
},
source,
forceInstall: true,
});
expect(resolution).toMatchObject({
ok: false,
reason: "github_scan_failed",
status: 403,
});
});
});
+180
View File
@@ -0,0 +1,180 @@
export type GitHubSkillScanStatus = "clean" | "suspicious" | "malicious" | "pending" | "failed";
export type GitHubCurrentStatus = "present" | "missing" | "unknown";
export type InstallResolverSkill = {
slug: string;
displayName: string;
latestVersionSummary?: { version: string } | null;
installKind?: "github";
githubPath?: string;
githubCurrentCommit?: string;
githubCurrentContentHash?: string;
githubCurrentStatus?: GitHubCurrentStatus;
githubScanStatus?: GitHubSkillScanStatus;
githubRemovedAt?: number;
};
export type InstallResolverSource = {
repo: string;
defaultBranch?: string | null;
};
export type SkillInstallResolution =
| {
ok: true;
slug: string;
installKind: "archive";
archive: {
version: string;
downloadUrl: string;
};
}
| {
ok: true;
slug: string;
installKind: "github";
github: {
repo: string;
path: string;
commit: string;
contentHash: string;
sourceUrl: string;
};
}
| {
ok: false;
slug: string;
reason:
| "archive_version_missing"
| "github_source_missing"
| "github_upstream_removed"
| "github_upstream_missing"
| "github_upstream_unknown"
| "github_verification_pending"
| "github_scan_failed";
message: string;
status: 403 | 409 | 410 | 423;
};
export function buildSkillInstallResolution({
origin,
skill,
source,
forceInstall = false,
}: {
origin: string;
skill: InstallResolverSkill;
source: InstallResolverSource | null;
forceInstall?: boolean;
}): SkillInstallResolution {
if (skill.installKind !== "github") {
const version = skill.latestVersionSummary?.version;
if (!version) {
return block(skill.slug, "archive_version_missing", 409);
}
const url = new URL("/api/v1/download", origin);
url.searchParams.set("slug", skill.slug);
url.searchParams.set("version", version);
return {
ok: true,
slug: skill.slug,
installKind: "archive",
archive: {
version,
downloadUrl: url.toString(),
},
};
}
if (skill.githubRemovedAt) {
return block(skill.slug, "github_upstream_removed", 410);
}
if (skill.githubCurrentStatus === "missing") {
return block(skill.slug, "github_upstream_missing", 410);
}
if (
skill.githubScanStatus === "failed" ||
skill.githubScanStatus === "malicious" ||
skill.githubScanStatus === "suspicious"
) {
return block(skill.slug, "github_scan_failed", 403);
}
if (!source || !skill.githubPath) {
return block(skill.slug, "github_source_missing", 409);
}
if (
skill.githubCurrentStatus !== "present" ||
!skill.githubCurrentCommit ||
!skill.githubCurrentContentHash
) {
return block(skill.slug, "github_upstream_unknown", 423);
}
if (
skill.githubScanStatus !== "clean" &&
!(forceInstall && skill.githubScanStatus === "pending")
) {
return block(skill.slug, "github_verification_pending", 423);
}
return {
ok: true,
slug: skill.slug,
installKind: "github",
github: {
repo: source.repo,
path: skill.githubPath,
commit: skill.githubCurrentCommit,
contentHash: skill.githubCurrentContentHash,
sourceUrl: buildGitHubTreeUrl(source.repo, skill.githubCurrentCommit, skill.githubPath),
},
};
}
function block(
slug: string,
reason: Extract<SkillInstallResolution, { ok: false }>["reason"],
status: Extract<SkillInstallResolution, { ok: false }>["status"],
): SkillInstallResolution {
return {
ok: false,
slug,
reason,
status,
message: INSTALL_BLOCK_MESSAGES[reason],
};
}
const INSTALL_BLOCK_MESSAGES: Record<
Extract<SkillInstallResolution, { ok: false }>["reason"],
string
> = {
archive_version_missing: "Hosted skill has no downloadable version.",
github_source_missing: "GitHub-backed skill source metadata is incomplete.",
github_upstream_removed: "GitHub-backed skill has been removed upstream.",
github_upstream_missing: "GitHub-backed skill path is missing upstream.",
github_upstream_unknown: "GitHub-backed skill needs an upstream freshness check before install.",
github_verification_pending:
"GitHub-backed skill security scan is in progress. Try again shortly, or rerun with --force-install to install the unverified upstream commit.",
github_scan_failed: "GitHub-backed skill failed ClawHub security scanning.",
};
function buildGitHubTreeUrl(repo: string, commit: string, path: string) {
return `https://github.com/${encodeURIComponentRepo(repo)}/tree/${commit}/${encodeURIComponentPath(
path,
)}`;
}
function encodeURIComponentRepo(repo: string) {
return repo
.split("/")
.map((segment) => encodeURIComponent(segment))
.join("/");
}
function encodeURIComponentPath(path: string) {
return path
.split("/")
.map((segment) => encodeURIComponent(segment))
.join("/");
}
+25 -24
View File
@@ -223,7 +223,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("suspicious");
});
it("flags browser automation that puts passwords in argv", () => {
it("does not duplicate SkillSpector credential-browser automation analysis", () => {
const result = runStaticModerationScan({
slug: "email-daily-summary",
displayName: "Email Daily Summary",
@@ -242,11 +242,11 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("clean");
});
it("flags persisted browser-use eval against authenticated mail contexts", () => {
it("does not duplicate SkillSpector persisted browser eval analysis", () => {
const result = runStaticModerationScan({
slug: "email-daily-summary",
displayName: "Email Daily Summary",
@@ -266,8 +266,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("clean");
});
it("does not flag ordinary browser-use navigation docs", () => {
@@ -290,7 +290,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("clean");
});
it("blocks stealth browser automation that advertises bot-protection bypass and persistent sessions", () => {
it("does not duplicate SkillSpector browser automation analysis in the static scanner", () => {
const result = runStaticModerationScan({
slug: "stealth-browser",
displayName: "Stealth Browser",
@@ -311,8 +311,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("malicious.stealth_browser_abuse");
expect(result.status).toBe("malicious");
expect(result.reasonCodes).not.toContain("malicious.stealth_browser_abuse");
expect(result.status).toBe("clean");
});
it("flags wallet mnemonics passed as CLI argv", () => {
@@ -497,7 +497,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("suspicious");
});
it("flags Playwright file URL renders of interpolated SVG", () => {
it("does not duplicate SkillSpector browser file-render analysis", () => {
const result = runStaticModerationScan({
slug: "office-quotes",
displayName: "Office Quotes",
@@ -520,8 +520,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.browser_file_render");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.browser_file_render");
expect(result.status).toBe("clean");
});
it("does not flag Playwright file renders with JavaScript disabled", () => {
@@ -1754,7 +1754,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("clean");
});
it("flags shell positional input passed directly to browser typing", () => {
it("does not duplicate SkillSpector browser typing analysis", () => {
const result = runStaticModerationScan({
slug: "wechat-helper",
displayName: "WeChat Helper",
@@ -1775,11 +1775,11 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("clean");
});
it("checks every positional shell assignment before browser typing", () => {
it("does not inspect shell positional assignments for browser typing", () => {
const result = runStaticModerationScan({
slug: "wechat-helper",
displayName: "WeChat Helper",
@@ -1799,8 +1799,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("clean");
});
it("allows browser typing after basic shell input validation", () => {
@@ -1872,7 +1872,7 @@ describe("moderationEngine", () => {
expect(snapshot.reasonCodes).toEqual([]);
});
it("keeps static suspicious findings as evidence while VT and LLM decide the verdict", () => {
it("keeps static suspicious findings out of top-level moderation snapshots", () => {
const snapshot = buildModerationSnapshot({
staticScan: {
status: "suspicious",
@@ -1897,7 +1897,7 @@ describe("moderationEngine", () => {
expect(snapshot.verdict).toBe("clean");
expect(snapshot.reasonCodes).toEqual([]);
expect(snapshot.evidence.length).toBe(1);
expect(snapshot.evidence).toEqual([]);
});
it("does not let static suspicious findings alone drive the aggregate verdict", () => {
@@ -1925,7 +1925,7 @@ describe("moderationEngine", () => {
expect(snapshot.verdict).toBe("clean");
expect(snapshot.reasonCodes).toEqual([]);
expect(snapshot.evidence.length).toBe(1);
expect(snapshot.evidence).toEqual([]);
});
it("lets Codex clear static malicious findings", () => {
@@ -1948,7 +1948,7 @@ describe("moderationEngine", () => {
expect(snapshot.evidence).toEqual([]);
});
it("keeps static malicious findings when Codex has no completed verdict", () => {
it("keeps static malicious findings internal when Codex has no completed verdict", () => {
const snapshot = buildModerationSnapshot({
staticScan: {
status: "malicious",
@@ -1962,8 +1962,9 @@ describe("moderationEngine", () => {
llmStatus: "error",
});
expect(snapshot.verdict).toBe("malicious");
expect(snapshot.reasonCodes).toContain("malicious.crypto_mining");
expect(snapshot.verdict).toBe("clean");
expect(snapshot.reasonCodes).toEqual([]);
expect(snapshot.evidence).toEqual([]);
});
it("lets legacy completed benign Codex verdicts clear static malicious findings", () => {
+2 -195
View File
@@ -94,8 +94,6 @@ const GOOGLE_SHEETS_SPREADSHEET_URL_PATTERN =
/https?:\/\/[^\s"'`]*\/spreadsheets\/([A-Za-z0-9_-]{20,})\/[^\s"'`]*/i;
const DESTRUCTIVE_DELETE_PATTERN =
/\brm\s+-[A-Za-z]*r[A-Za-z]*f[A-Za-z]*\s+(["']?)(\/root\/\.openclaw\/|\/home\/[^/\s"'`]+\/\.openclaw\/|\/Users\/[^/\s"'`]+\/\.openclaw\/|~\/\.openclaw\/|\$HOME\/\.openclaw\/|\$\{HOME\}\/\.openclaw\/|\/etc\/|\/usr\/|\/opt\/|\/Library\/|\/Applications\/)[^\s"'`;|&)]*\1/i;
const SHELL_POSITIONAL_ASSIGNMENT_PATTERN =
/^\s*([A-Z_][A-Z0-9_]*)=(["']?)\$(?:[1-9][0-9]*|@|\*)\2\s*(?:#.*)?$/gm;
const SECRET_ASSIGNMENT_PATTERN =
/\b(?:[A-Za-z0-9]+[_\s-]+)*(?:(?:api|client|consumer)[_\s-]?(?:secret|key|token)|secret[_\s-]?key|access[_\s-]?(?:token|key|secret|grant)|auth[_\s-]?token|bearer(?:[_\s-]?token)?|private[_\s-]?key|service[_\s-]?role[_\s-]?key|github[_\s-]?(?:pat|token)|(?:openrouter|supabase|storj)[_\s-]?(?:key|token|secret|access[_\s-]?grant)|password)\b\s*[:=]\s*["'`]?([A-Za-z0-9][A-Za-z0-9._~+/=-]{15,})["'`]?/i;
const AUTH_HEADER_SECRET_PATTERN =
@@ -111,12 +109,6 @@ const HOST_PLATFORM_SOURCE_CONTEXT_PATTERN =
const HOST_PLATFORM_PATCH_COMMAND_PATTERN =
/\b(?:sed\s+-i|perl\s+-0?pi|cp\s+|cat\s+>|python3?\b.{0,120}(?:write|replace))/i;
const HOST_PLATFORM_REBUILD_PATTERN = /\b(?:pnpm\s+build|npm\s+run\s+build|bun\s+run\s+build)\b/i;
const BROWSER_USE_PASSWORD_ARGV_PATTERN =
/\bbrowser-use\s+input\b[^\n]*(?:password|passwd|\$[A-Z_]*(?:PASSWORD|PASS|PWD)[A-Z0-9_]*|<password>|\{password\})/i;
const BROWSER_USE_AUTH_EVAL_PATTERN = /\bbrowser-use\s+(?:eval|python)\b/i;
const AUTHENTICATED_MAIL_CONTEXT_PATTERN = /\b(?:mail\.google\.com|gmail|webmail|mailbox|inbox)\b/i;
const PERSISTENCE_SCHEDULER_PATTERN =
/\b(?:launchctl\s+load|crontab\b|LaunchAgents\/|systemctl\s+(?:--user\s+)?enable)\b/i;
const SECRET_ARGV_WARNING_PATTERN =
/\b(?:do\s+not|don't|avoid|never|reject)\b[^\n]{0,120}\b(?:argv|argument|from-mnemonic|private[-_\s]?key|seed[-\s]?phrase|mnemonic)\b/i;
const FROM_MNEMONIC_ARGV_PATTERN =
@@ -135,17 +127,6 @@ const PYTHON_BASE64_FILE_READ_PATTERN =
/base64\.b64encode\s*\(\s*(?:[A-Za-z_][A-Za-z0-9_]*\.read_bytes\s*\(\s*\)|Path\s*\([^)]*\)\.read_bytes\s*\(\s*\)|open\s*\([^)]*["']rb["'][\s\S]{0,120}\.read\s*\(\s*\))/i;
const PYTHON_NETWORK_UPLOAD_PATTERN =
/\b(?:requests|session|self\.session|client|httpx\.(?:post|request))\.post\s*\([\s\S]{0,1600}(?:json\s*=|data\s*=|files\s*=|headers\s*=|Authorization)/i;
const PLAYWRIGHT_CHROMIUM_PATTERN = /\b(?:playwright\.)?chromium\.launch\s*\(/i;
const FILE_URL_BROWSER_NAVIGATION_PATTERN = /\bpage\.goto\s*\([^)]*file:\/\//i;
const SVG_HTML_INTERPOLATION_PATTERN =
/(?:<body>[\s\S]{0,240}\$\{[^}]*svg[^}]*\}|writeFile(?:Sync)?\s*\([^)]*\.html[^)]*\$\{[^}]*svg[^}]*\}|\$\{[^}]*svg[^}]*\}[\s\S]{0,240}<\/body>)/i;
const BROWSER_JS_DISABLED_PATTERN =
/javaScriptEnabled\s*:\s*false|Content-Security-Policy|script-src\s+['"]?none/i;
const STEALTH_BROWSER_CONTEXT_PATTERN =
/\b(?:stealth|anti[-\s]?detect|undetected|fingerprint spoof|navigator\.webdriver)\b/i;
const BOT_PROTECTION_BYPASS_PATTERN = /\b(?:captcha|cloudflare|turnstile|bot detection|waf)\b/i;
const BROWSER_SESSION_PERSISTENCE_PATTERN =
/\b(?:persistent_context|userDataDir|storageState|session persistence|persist(?:ed)? cookies?)\b/i;
const AGENT_OUTPUT_DIR_ARGUMENT_PATTERN =
/add_argument\s*\(\s*["']--outdir["']|args\.outdir|output_path\s*=\s*Path\s*\(\s*args\.outdir\s*\)/i;
const FFMPEG_FORCE_OUTPUT_PATTERN =
@@ -268,31 +249,6 @@ function findCredentialExposureInstruction(content: string) {
return null;
}
function findBrowserCredentialAutomation(content: string) {
const lines = content.split("\n");
for (let i = 0; i < lines.length; i += 1) {
const line = lines[i] ?? "";
if (BROWSER_USE_PASSWORD_ARGV_PATTERN.test(line)) {
return { line: i + 1, text: line };
}
}
if (
BROWSER_USE_AUTH_EVAL_PATTERN.test(content) &&
AUTHENTICATED_MAIL_CONTEXT_PATTERN.test(content) &&
PERSISTENCE_SCHEDULER_PATTERN.test(content)
) {
for (let i = 0; i < lines.length; i += 1) {
const line = lines[i] ?? "";
if (BROWSER_USE_AUTH_EVAL_PATTERN.test(line) || PERSISTENCE_SCHEDULER_PATTERN.test(line)) {
return { line: i + 1, text: line };
}
}
}
return null;
}
function redactSecretArgvEvidence(line: string) {
return line.replace(SECRET_ARGV_REDACTION_PATTERN, "$1$2[REDACTED]$2");
}
@@ -401,49 +357,6 @@ function isScopedOpenClawDelete(line: string, slug?: string) {
return false;
}
function hasShellVariableValidation(content: string, variable: string, useIndex: number) {
const escaped = variable.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const beforeUse = content.slice(0, useIndex);
const variableReference = String.raw`(?:\$\{${escaped}\}|\$${escaped})`;
const lengthCheck = new RegExp(
String.raw`\$\{#${escaped}\}\s*(?:-[a-z]\s+)?(?:[<>!=]=?|-[gl][te])`,
"m",
);
const controlCharStrip = new RegExp(
String.raw`(?:tr\s+-d\s+["']?\\(?:000|x00).{0,80}\\(?:037|x1[fF]|177|x7[fF])|${escaped}\s*=.*tr\s+-d)`,
"s",
);
const explicitValidation = new RegExp(
String.raw`(?:validate|sanitize|strip|clean)[A-Za-z0-9_ -]{0,60}${variableReference}|${variableReference}.{0,60}(?:validate|sanitize|strip|clean)`,
"is",
);
return (
lengthCheck.test(beforeUse) ||
controlCharStrip.test(beforeUse) ||
explicitValidation.test(beforeUse)
);
}
function findUnsafeBrowserTextInput(content: string) {
for (const assignment of content.matchAll(SHELL_POSITIONAL_ASSIGNMENT_PATTERN)) {
const variable = assignment[1];
if (!variable) continue;
const escaped = variable.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const browserTextPattern = new RegExp(
String.raw`\bbrowser\s+action=act\b[^\n]*\bkind=["']?type["']?[^\n]*\btext=(?:"\$${escaped}"|'\$${escaped}'|\$${escaped})(?![A-Za-z0-9_])`,
"i",
);
const match = content.match(browserTextPattern);
if (!match || match.index === undefined) continue;
if (hasShellVariableValidation(content, variable, match.index)) continue;
return findLineAtIndex(content, match.index);
}
return null;
}
function addFinding(
findings: ModerationFinding[],
finding: Omit<ModerationFinding, "evidence"> & { evidence: string },
@@ -602,21 +515,6 @@ function findJsSensitiveFileNetworkSend(content: string) {
return null;
}
function findUnsafeBrowserFileRender(content: string) {
if (!PLAYWRIGHT_CHROMIUM_PATTERN.test(content)) return null;
if (!FILE_URL_BROWSER_NAVIGATION_PATTERN.test(content)) return null;
if (!SVG_HTML_INTERPOLATION_PATTERN.test(content)) return null;
if (BROWSER_JS_DISABLED_PATTERN.test(content)) return null;
return findFirstLine(content, FILE_URL_BROWSER_NAVIGATION_PATTERN);
}
function findStealthBrowserAbuse(content: string) {
if (!STEALTH_BROWSER_CONTEXT_PATTERN.test(content)) return null;
if (!BOT_PROTECTION_BYPASS_PATTERN.test(content)) return null;
if (!BROWSER_SESSION_PERSISTENCE_PATTERN.test(content)) return null;
return findFirstLine(content, STEALTH_BROWSER_CONTEXT_PATTERN);
}
function findUnsafeAgentControlledFileWrite(content: string) {
if (!AGENT_OUTPUT_DIR_ARGUMENT_PATTERN.test(content)) return null;
if (!FFMPEG_FORCE_OUTPUT_PATTERN.test(content)) return null;
@@ -836,18 +734,6 @@ function scanCodeFile(
});
}
const unsafeBrowserTextInput = findUnsafeBrowserTextInput(content);
if (unsafeBrowserTextInput) {
addFinding(findings, {
code: REASON_CODES.UNSAFE_BROWSER_TEXT_INPUT,
severity: "warn",
file: path,
line: unsafeBrowserTextInput.line,
message: "Shell positional input is typed into browser automation without validation.",
evidence: unsafeBrowserTextInput.text,
});
}
const hostPlatformSourcePatch = findHostPlatformSourcePatch(content);
if (hostPlatformSourcePatch) {
addFinding(findings, {
@@ -860,32 +746,6 @@ function scanCodeFile(
});
}
const unsafeBrowserFileRender = findUnsafeBrowserFileRender(content);
if (unsafeBrowserFileRender) {
addFinding(findings, {
code: REASON_CODES.BROWSER_FILE_RENDER,
severity: "critical",
file: path,
line: unsafeBrowserFileRender.line,
message:
"Browser automation renders interpolated SVG/HTML from a file URL with JavaScript enabled.",
evidence: unsafeBrowserFileRender.text,
});
}
const stealthBrowserAbuse = findStealthBrowserAbuse(content);
if (stealthBrowserAbuse) {
addFinding(findings, {
code: REASON_CODES.STEALTH_BROWSER_ABUSE,
severity: "critical",
file: path,
line: stealthBrowserAbuse.line,
message:
"Browser automation advertises stealth/anti-detection behavior with bot-protection bypass and persistent sessions.",
evidence: stealthBrowserAbuse.text,
});
}
const unsafeAgentControlledFileWrite = findUnsafeAgentControlledFileWrite(content);
if (unsafeAgentControlledFileWrite) {
addFinding(findings, {
@@ -1085,18 +945,6 @@ function scanMarkdownFile(
});
}
const browserCredentialAutomation = findBrowserCredentialAutomation(content);
if (browserCredentialAutomation) {
addFinding(findings, {
code: REASON_CODES.BROWSER_CREDENTIAL_AUTOMATION,
severity: "critical",
file: path,
line: browserCredentialAutomation.line,
message: "Browser automation instructions expose credentials or persist authenticated eval.",
evidence: browserCredentialAutomation.text,
});
}
const secretArgvExposure = findSecretArgvExposure(content);
if (secretArgvExposure) {
addFinding(findings, {
@@ -1122,19 +970,6 @@ function scanMarkdownFile(
});
}
const stealthBrowserAbuse = findStealthBrowserAbuse(content);
if (stealthBrowserAbuse) {
addFinding(findings, {
code: REASON_CODES.STEALTH_BROWSER_ABUSE,
severity: "critical",
file: path,
line: stealthBrowserAbuse.line,
message:
"Browser automation advertises stealth/anti-detection behavior with bot-protection bypass and persistent sessions.",
evidence: stealthBrowserAbuse.text,
});
}
if (hasMaliciousInstallPrompt(content)) {
const match = findFirstLine(
content,
@@ -1163,18 +998,6 @@ function scanMarkdownFile(
});
}
const unsafeBrowserTextInput = findUnsafeBrowserTextInput(content);
if (unsafeBrowserTextInput) {
addFinding(findings, {
code: REASON_CODES.UNSAFE_BROWSER_TEXT_INPUT,
severity: "warn",
file: path,
line: unsafeBrowserTextInput.line,
message: "Shell positional input is typed into browser automation without validation.",
evidence: unsafeBrowserTextInput.text,
});
}
if (
/ignore\s+(all\s+)?previous\s+instructions/i.test(content) ||
/system\s*prompt\s*[:=]/i.test(content)
@@ -1264,18 +1087,6 @@ function scanManifestFile(path: string, content: string, findings: ModerationFin
}
}
function dedupeEvidence(evidence: ModerationFinding[]) {
const seen = new Set<string>();
const out: ModerationFinding[] = [];
for (const item of evidence) {
const key = `${item.code}:${item.file}:${item.line}:${item.message}`;
if (seen.has(key)) continue;
seen.add(key);
out.push(item);
}
return out.slice(0, 40);
}
function normalizedSeverityRank(severity: string | undefined) {
switch (severity?.trim().toLowerCase()) {
case "critical":
@@ -1435,12 +1246,8 @@ export function buildModerationSnapshot(params: {
}): ModerationSnapshot {
const llmStatus = params.llmStatus ?? params.llmAnalysis?.status;
const codexStatus = completedCodexStatus(llmStatus, params.llmAnalysis);
const staticCodes = codexStatus
? []
: (params.staticScan?.reasonCodes ?? []).filter((code) => code.startsWith("malicious."));
const evidence = [...(params.staticScan?.findings ?? [])];
const reasonCodes = [...staticCodes];
const reasonCodes: string[] = [];
addLlmStatusReason(reasonCodes, codexStatus, params.llmAnalysis);
const normalizedCodes = normalizeReasonCodes(reasonCodes);
@@ -1448,7 +1255,7 @@ export function buildModerationSnapshot(params: {
return {
verdict,
reasonCodes: normalizedCodes,
evidence: dedupeEvidence(evidence),
evidence: [],
summary: summarizeReasonCodes(normalizedCodes),
engineVersion: MODERATION_ENGINE_VERSION,
evaluatedAt: Date.now(),

Some files were not shown because too many files have changed in this diff Show More