Compare commits

..
Author SHA1 Message Date
Patrick Erichsen 6fd3dcabf9 docs: add clawhub 0.20.2 release notes 2026-06-11 10:36:13 -07:00
Momoandclawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com> 7166883d7d fix: tolerate duplicate CLI device user codes (#2589)
* fix: tolerate duplicate CLI device user codes

* fix: harden duplicate device code handling

* fix: tolerate duplicate CLI device user codes

---------

Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
2026-06-11 18:18:38 +08:00
Jason (Json) 4496acf9f0 fix: remove homepage suggestion chips (#2590) 2026-06-11 15:46:16 +10:00
Jesse Merhi d6500794c6 feat: rank skills by installs and dedupe install telemetry (#2535) 2026-06-11 14:22:11 +10:00
joshavant a1c91b265e chore: remove temporary auth identity cleanup tool 2026-06-10 19:19:55 -05:00
joshavant cd47bbe060 chore: add temporary auth identity cleanup tool 2026-06-10 19:12:34 -05:00
joshavant 31c69ff518 chore: remove temporary auth identity cleanup tool 2026-06-10 18:57:04 -05:00
Patrick Erichsen c22a2d295b fix: normalize ClawScan review visibility 2026-06-10 16:51:30 -07:00
joshavant f0b0185649 chore: add temporary auth identity cleanup tool 2026-06-10 18:46:50 -05:00
joshavant 7742ac7f52 chore: remove temporary identity remediation tool 2026-06-10 15:34:22 -05:00
joshavant e4ef192ce6 chore: add temporary identity remediation tool 2026-06-10 15:32:09 -05:00
joshavant 207ab1758e chore: remove github auth remediation tool 2026-06-10 14:34:21 -05:00
Patrick Erichsen 8446d65224 chore: remove migration lifecycle skill (#2592) 2026-06-10 12:32:57 -07:00
joshavant 58051dffcb chore: add github auth remediation tool 2026-06-10 14:14:08 -05:00
joshavant 0b6466184b fix: reject malformed github oauth ids 2026-06-10 13:14:04 -05:00
joshavant b3f98b4686 fix: prevent GitHub email account linking 2026-06-10 12:03:16 -05:00
Patrick Erichsen 79d9a86e6f ci: dispatch plugin inspector scans on pin bumps (#2584)
* ci: dispatch plugin inspector bulk scans on pin bump

* ci: rename plugin inspector bulk scan workflow

* fix: include CLI plugin inspector pin in scan dispatch

* chore: remove stale plugin inspector cron comment

* fix: dispatch inspector scans from main pushes
2026-06-09 21:45:29 -07:00
Patrick Erichsen 27faf50992 feat: surface author remediation for plugin validation
Adds author-remediation metadata and ClawHub validation UI/CLI/email handling for Plugin Inspector findings.
2026-06-09 21:09:25 -07:00
Patrick Erichsen e9571f3aa4 docs: remove stale skill folder guidance
Remove obsolete AGENTS.md guidance that treated repo skill additions as a standing contributor-policy issue.
2026-06-09 21:08:16 -07:00
Patrick Erichsen 18ec428fc5 docs: correct org delete moderation note (#2582) 2026-06-09 21:06:50 -07:00
Patrick Erichsen 0724abccbc Add guarded admin email and reserve clawhub handle (#2580)
* feat: add guarded admin email command

* fix: reserve clawhub publisher handle

* fix: harden admin org deletion and email audit
2026-06-09 20:56:48 -07:00
Patrick Erichsen 7b71c15ee1 fix: delegate browser automation scanning to SkillSpector
Remove ClawHub static-scanner browser automation rules because SkillSpector owns that analysis lane. Keep static moderation focused on non-browser-specific concrete source/sink evidence and update policy/spec wording accordingly.

Linear: CLAW-252
2026-06-09 20:55:08 -07:00
Patrick Erichsen d2921c0600 feat: add owner skill deletion (#2548) 2026-06-09 20:50:17 -07:00
Patrick Erichsen f5bf1b61e6 chore: remove stale maintenance backfills
Remove completed/stale maintenance backfills and their obsolete package artifact HTTP/CLI/schema surface. Keep the targeted legacy publisher ownership repair path.
2026-06-09 18:47:30 -07:00
Patrick Erichsen 6447101397 chore: remove clawhub ui proof skill 2026-06-09 16:29:58 -07:00
Patrick Erichsen d12adf2f37 chore: remove dependency registry cleanup 2026-06-09 16:29:13 -07:00
Patrick Erichsen 741599d20e fix: remove redundant ban email support note 2026-06-09 15:20:26 -07:00
Patrick Erichsen 9244087d00 chore: add admin alias for ClawHub operator CLI 2026-06-09 15:20:26 -07:00
Patrick Erichsen 23edf7d19d fix: split dependency cleanup pagination (#2573) 2026-06-09 11:32:52 -07:00
Jesse Merhi 1876698093 fix: simplify package listing cursors
* fix: simplify package listing cursors

* test: avoid asserting package cursor internals
2026-06-10 00:54:52 +10:00
Jason (Json)andJesse Merhi ad640d2903 fix: sort plugins by downloads with family index
* fix: sort plugin downloads by family index

* fix: reset legacy package download cursors

---------

Co-authored-by: Jesse Merhi <79823012+jesse-merhi@users.noreply.github.com>
2026-06-09 23:18:36 +10:00
Jesse Merhi c0143b1cd1 fix: retire dependency registry scans 2026-06-09 22:28:13 +10:00
clawsweeper[bot]chlinyclawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>momothemage
8970a46acf fix: use consistent installedAt timestamp for origin.json and lockfile (#2569)
Summary:
- The PR captures one `installedAt` value per install/update path and reuses it for both `.clawhub/origin.json` and the lockfile, with regression tests for timestamp equality.
- Reproducibility: yes. Source inspection of current `main` shows separate timestamp writes in `cmdInstall` an ... d report provides the CLI and `jq` reproduction path, though I did not execute it in this read-only review.

Automerge notes:
- PR branch already contained follow-up commit before automerge: test: add regression tests for installedAt timestamp equality
- PR branch already contained follow-up commit before automerge: fix: use consistent installedAt timestamp for origin.json and lockfile

Validation:
- ClawSweeper review passed for head b941ee37d6.
- Required merge gates passed before the squash merge.

Prepared head SHA: b941ee37d6
Review: https://github.com/openclaw/clawhub/pull/2569#issuecomment-4657649958

Co-authored-by: chliny <chliny11@gmail.com>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: momothemage
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
2026-06-09 08:16:53 +00:00
Jesse Merhi a3de2360b4 docs: clarify ClawHub vulnerability disclosure policy (#2567) 2026-06-09 17:38:56 +10:00
clawsweeper[bot]momothemageclawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>momothemage
620445f080 fix: slim package manage context (#2564)
Summary:
- The branch narrows `packages.getManageContext` to package/release identifier fields, adds unit and local-auth payload-capture coverage, and adjusts a WebCrypto digest helper.
- Reproducibility: yes. Source inspection of current main shows `getManageContext` returning the full package document and public release object, and the PR adds a focused test for the slim response shape.

Automerge notes:
- PR branch already contained follow-up commit before automerge: fix: slim package manage context

Validation:
- ClawSweeper review passed for head 5b5834bfc6.
- Required merge gates passed before the squash merge.

Prepared head SHA: 5b5834bfc6
Review: https://github.com/openclaw/clawhub/pull/2564#issuecomment-4655984471

Co-authored-by: momothemage <niuzhengnan@163.com>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: momothemage
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
2026-06-09 04:35:14 +00:00
Patrick Erichsen 89a89af183 fix: suppress inspector gap findings in clawhub 2026-06-08 21:29:31 -07:00
Patrick Erichsen 2d367c60ed Revert "Revert "chore: disable plugin inspector nightly schedule (#2558)" (#2…" (#2563)
This reverts commit 8b39d00247.
2026-06-08 20:51:15 -07:00
Patrick Erichsen 8b39d00247 Revert "chore: disable plugin inspector nightly schedule (#2558)" (#2562)
This reverts commit e09bf7f0fd.
2026-06-08 20:35:53 -07:00
Patrick Erichsen e6a8bce394 Fix package publish validation flow (#2561)
* feat: support prebuilt package publish artifacts

* fix: finalize package publish validation flow

* fix: polish plugin inspector findings email
2026-06-08 20:35:25 -07:00
Patrick Erichsen e09bf7f0fd chore: disable plugin inspector nightly schedule (#2558) 2026-06-08 20:09:23 -07:00
Patrick Erichsen c9bb130235 feat: support prebuilt package publish artifacts 2026-06-08 19:52:32 -07:00
Patrick Erichsen be70ae5e9f Publish public Plugin Inspector findings (#2525)
* feat: gate plugin publishes with inspector warnings

* fix: harden plugin publish inspector gate

* feat: publish public plugin inspector findings

* fix: refine plugin inspector publish errors

* feat: add package validation command

* fix: refine plugin validation findings UI

* fix: scan only latest plugin releases nightly

* feat: add dry run for nightly plugin inspection

* test: fix plugin validation tab e2e matcher
2026-06-08 19:41:48 -07:00
Vincent Koc 9321c83b93 fix: block moderated skill version metadata
Block exact skill-version metadata and scan responses when the requested version is the moderated source version. Apply the same guard to package-compatible skill version metadata and cover public-null fallback cases.
2026-06-09 11:23:55 +09:00
Patrick Erichsen 2a3e08e1bd fix: include clawscan summaries in blocked version emails (#2554) 2026-06-08 18:14:51 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 0ead9a60cf build(deps-dev): bump the development-minor-and-patch group with 3 updates (#2553)
Bumps the development-minor-and-patch group with 3 updates: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node), [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) and [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint).


Updates `@types/node` from 25.9.1 to 25.9.2
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `oxfmt` from 0.53.0 to 0.54.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.54.0/npm/oxfmt)

Updates `oxlint` from 1.68.0 to 1.69.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.69.0/npm/oxlint)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-version: 25.9.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
- dependency-name: oxfmt
  dependency-version: 0.54.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
- dependency-name: oxlint
  dependency-version: 1.69.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-08 15:02:38 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> f0274e41c4 build(deps): bump the production-minor-and-patch group with 13 updates (#2552)
Bumps the production-minor-and-patch group with 13 updates:

| Package | From | To |
| --- | --- | --- |
| [@radix-ui/react-avatar](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/avatar) | `1.1.11` | `1.1.12` |
| [@radix-ui/react-dialog](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/dialog) | `1.1.15` | `1.1.16` |
| [@radix-ui/react-dropdown-menu](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/dropdown-menu) | `2.1.16` | `2.1.17` |
| [@radix-ui/react-label](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/label) | `2.1.8` | `2.1.9` |
| [@radix-ui/react-select](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/select) | `2.2.6` | `2.3.0` |
| [@radix-ui/react-separator](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/separator) | `1.1.8` | `1.1.9` |
| [@radix-ui/react-slot](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/slot) | `1.2.4` | `1.2.5` |
| [@radix-ui/react-toggle-group](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/toggle-group) | `1.1.11` | `1.1.12` |
| [@radix-ui/react-tooltip](https://github.com/radix-ui/primitives/tree/HEAD/packages/react/tooltip) | `1.2.8` | `1.2.9` |
| [@tanstack/react-router](https://github.com/TanStack/router/tree/HEAD/packages/react-router) | `1.170.12` | `1.170.15` |
| [@tanstack/react-start](https://github.com/TanStack/router/tree/HEAD/packages/react-start) | `1.168.21` | `1.168.25` |
| [semver](https://github.com/npm/node-semver) | `7.8.2` | `7.8.3` |
| [undici](https://github.com/nodejs/undici) | `7.27.1` | `7.27.2` |


Updates `@radix-ui/react-avatar` from 1.1.11 to 1.1.12
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/avatar/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/avatar)

Updates `@radix-ui/react-dialog` from 1.1.15 to 1.1.16
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/dialog/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/dialog)

Updates `@radix-ui/react-dropdown-menu` from 2.1.16 to 2.1.17
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/dropdown-menu/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/dropdown-menu)

Updates `@radix-ui/react-label` from 2.1.8 to 2.1.9
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/label/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/label)

Updates `@radix-ui/react-select` from 2.2.6 to 2.3.0
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/select/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/select)

Updates `@radix-ui/react-separator` from 1.1.8 to 1.1.9
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/separator/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/separator)

Updates `@radix-ui/react-slot` from 1.2.4 to 1.2.5
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/slot/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/slot)

Updates `@radix-ui/react-toggle-group` from 1.1.11 to 1.1.12
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/toggle-group/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/toggle-group)

Updates `@radix-ui/react-tooltip` from 1.2.8 to 1.2.9
- [Changelog](https://github.com/radix-ui/primitives/blob/main/packages/react/tooltip/CHANGELOG.md)
- [Commits](https://github.com/radix-ui/primitives/commits/HEAD/packages/react/tooltip)

Updates `@tanstack/react-router` from 1.170.12 to 1.170.15
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-router@1.170.15/packages/react-router)

Updates `@tanstack/react-start` from 1.168.21 to 1.168.25
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-start/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-start@1.168.25/packages/react-start)

Updates `semver` from 7.8.2 to 7.8.3
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v7.8.2...v7.8.3)

Updates `undici` from 7.27.1 to 7.27.2
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.27.1...v7.27.2)

---
updated-dependencies:
- dependency-name: "@radix-ui/react-avatar"
  dependency-version: 1.1.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-dialog"
  dependency-version: 1.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-dropdown-menu"
  dependency-version: 2.1.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-label"
  dependency-version: 2.1.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-select"
  dependency-version: 2.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-separator"
  dependency-version: 1.1.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-slot"
  dependency-version: 1.2.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-toggle-group"
  dependency-version: 1.1.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@radix-ui/react-tooltip"
  dependency-version: 1.2.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-router"
  dependency-version: 1.170.15
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-start"
  dependency-version: 1.168.25
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: semver
  dependency-version: 7.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: undici
  dependency-version: 7.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-08 15:02:35 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 315cf9603f build(deps): bump github/codeql-action in the github-actions group (#2551)
Bumps the github-actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 4.36.1 to 4.36.2
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/87557b9c84dde89fdd9b10e88954ac2f4248e463...8aad20d150bbac5944a9f9d289da16a4b0d87c1e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.36.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-08 14:54:30 -07:00
Peter Steinberger 2c623b3beb fix: keep dev persona handles stable 2026-06-08 20:27:25 +01:00
Peter Steinberger 01612b4f15 style: fix changelog spacing 2026-06-08 20:11:37 +01:00
Peter Steinberger 295db69eba chore: require Node.js 22 for CLI packages 2026-06-08 20:01:44 +01:00
Patrick Erichsen 1fa4f9049d feat: add plugin export endpoint (#2547)
Add an authenticated /api/v1/plugins/export route that mirrors the skills export shape, supports an optional plugin family filter, defaults to both code and bundle plugins, and emits ZIP archives with manifest, error, and per-plugin metadata entries.

Autoreview findings addressed:

- [P1] Do not mark partially consumed plugin pages done

  Keep merged plugin export family cursor state active while buffered rows remain, and cover the pagination regression.

- [P1] Block release-level security states in plugin export

  Apply the package release download security block before reading release storage blobs.

- [P2] Avoid colliding with exported plugin metadata

  Move generated plugin metadata under __clawhub_export/ so plugin file paths cannot overwrite it.
2026-06-07 22:52:04 -07:00
Patrick Erichsen 5c19da48d0 fix: reserve admin owner handle (#2546) 2026-06-07 20:27:58 -07:00
Patrick Erichsen fc0a85e150 fix: make account recovery purge idempotent (#2543) 2026-06-06 19:08:23 -07:00
Patrick Erichsen 3a2568f751 fix: schedule account deletion digest sync safely 2026-06-06 18:41:44 -07:00
Patrick Erichsen 8e1bad4ac1 feat: improve account appeal notifications (#2520)
* feat: improve account appeal notifications

Adds Resend-backed account-action emails for bans/restores, public-safe ban reason summaries, scanner-triggered appeal scan guidance, and inline banned-account sign-in appeal messaging.

Autoreview findings addressed:

- [P2] Wire restored-account email into unban flow

- [P2] Add the scan upload test to the CI selection

* feat: quarantine malicious artifact uploads

* fix: warn on repeated malicious rejections

* fix: route banned sign-ins to appeal page

* feat: add stored scan report downloads

* chore: bump clawhub cli to 0.20.0

* test: cover malicious skill ban journey

* test: align deletion cleanup expectations after merge

* fix: hide deleted account packages before cleanup

* fix: finalize appeals moderation flow
2026-06-06 16:55:23 -07:00
Patrick Erichsen 8f7c21ac02 test: cover account recreation after deletion (#2540) 2026-06-06 16:47:19 -07:00
Patrick Erichsen 999131e2e7 fix: purge auth-locked deleted accounts
* fix: purge auth-locked deleted accounts

* test: wait for account deletion resources

* fix: mark packages inactive during deletion purge

* test: expect package soft delete before purge

* test: ignore post-delete auth timeout noise
2026-06-06 16:07:19 -07:00
Patrick Erichsen 571cc70c7a test: limit local-auth playwright retries (#2538) 2026-06-06 14:58:18 -07:00
Patrick Erichsen 3ead985f82 fix: restore post-merge CI gates (#2537) 2026-06-06 14:50:46 -07:00
Patrick Erichsen 1117aa4340 feat: add cohort temporal abuse scoring (#2536) 2026-06-06 14:26:46 -07:00
Patrick Erichsen 9ce5e702b5 feat: hard delete account-owned resources (#2533)
* feat: hard delete account-owned resources

* fix: expose account recovery purge dry-run candidates
2026-06-06 14:17:50 -07:00
Patrick Erichsen c632b697bb fix: hide banned user publisher profiles (#2528)
* fix: hide banned user publisher profiles

* docs: require real ClawHub screenshot proof

* test: stabilize local auth profile visibility checks
2026-06-05 17:33:03 -07:00
Patrick Erichsen d2525b179a fix: stop copying publisher ids to skill embeddings (#2532) 2026-06-05 17:28:55 -07:00
Patrick Erichsen 8ebae39140 feat: add temporal publisher abuse scan (#2531) 2026-06-05 17:14:57 -07:00
Patrick Erichsen 821494b9dd fix(cli): accept legacy verify json flag
Accept legacy skill verify --json usage as a hidden compatibility no-op, add a regression e2e for the flattened verifier response, and prepare clawhub@0.19.2.
2026-06-05 16:30:19 -07:00
Patrick Erichsen 94413a60fb feat: show plugin downloads in lists
Show plugin download counts in lists, add downloads sorting, and remove plugin family badges from plugin list/grid rows.
2026-06-05 16:18:00 -07:00
Patrick Erichsen 9cbf98297c fix: exclude official orgs from publisher abuse (#2527) 2026-06-05 15:59:10 -07:00
Patrick Erichsen 94ded18dec feat: add org deletion cascade
Closes #2516.
2026-06-05 14:52:31 -07:00
Patrick Erichsen 9897850074 fix: add targeted legacy publisher ownership repair
Adds a targeted legacy owner-publisher projection repair mutation for one user and covers the repair path in maintenance tests.
2026-06-05 14:24:29 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 159118dc59 build(deps): bump the github-actions group with 3 updates (#2473)
Bumps the github-actions group with 3 updates: [useblacksmith/begin-testbox](https://github.com/useblacksmith/begin-testbox), [github/codeql-action](https://github.com/github/codeql-action) and [trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog).


Updates `useblacksmith/begin-testbox` from d0e04585c26905fdd92c94a09c159544c7ee1b67 to 233448af4bfdc6fca509a7f0974411ac6d8a8043
- [Commits](https://github.com/useblacksmith/begin-testbox/compare/d0e04585c26905fdd92c94a09c159544c7ee1b67...233448af4bfdc6fca509a7f0974411ac6d8a8043)

Updates `github/codeql-action` from 4.36.0 to 4.36.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/7211b7c8077ea37d8641b6271f6a365a22a5fbfa...87557b9c84dde89fdd9b10e88954ac2f4248e463)

Updates `trufflesecurity/trufflehog` from 3.95.3 to 3.95.5
- [Release notes](https://github.com/trufflesecurity/trufflehog/releases)
- [Commits](https://github.com/trufflesecurity/trufflehog/compare/v3.95.3...v3.95.5)

---
updated-dependencies:
- dependency-name: useblacksmith/begin-testbox
  dependency-version: 233448af4bfdc6fca509a7f0974411ac6d8a8043
  dependency-type: direct:production
  dependency-group: github-actions
- dependency-name: github/codeql-action
  dependency-version: 4.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
- dependency-name: trufflesecurity/trufflehog
  dependency-version: 3.95.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-05 14:11:39 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 843f19e640 build(deps): bump commander from 14.0.3 to 15.0.0 (#2476)
Bumps [commander](https://github.com/tj/commander.js) from 14.0.3 to 15.0.0.
- [Release notes](https://github.com/tj/commander.js/releases)
- [Changelog](https://github.com/tj/commander.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tj/commander.js/compare/v14.0.3...v15.0.0)

---
updated-dependencies:
- dependency-name: commander
  dependency-version: 15.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-05 14:07:07 -07:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Peter Steinberger
4b882b2f43 build(deps): bump the production-minor-and-patch group across 1 directory with 12 updates (#2510)
* build(deps): bump the production-minor-and-patch group across 1 directory with 12 updates

Bumps the production-minor-and-patch group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [@convex-dev/auth](https://github.com/get-convex/convex-auth) | `0.0.92` | `0.0.93` |
| [@shikijs/rehype](https://github.com/shikijs/shiki/tree/HEAD/packages/rehype) | `4.1.0` | `4.2.0` |
| [@tanstack/react-router](https://github.com/TanStack/router/tree/HEAD/packages/react-router) | `1.170.8` | `1.170.12` |
| [@tanstack/react-start](https://github.com/TanStack/router/tree/HEAD/packages/react-start) | `1.168.13` | `1.168.21` |
| [convex](https://github.com/get-convex/convex-backend/tree/HEAD/npm-packages/convex) | `1.39.1` | `1.40.0` |
| [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.16.0` | `1.17.0` |
| [react](https://github.com/facebook/react/tree/HEAD/packages/react) | `19.2.6` | `19.2.7` |
| [react-dom](https://github.com/facebook/react/tree/HEAD/packages/react-dom) | `19.2.6` | `19.2.7` |
| [semver](https://github.com/npm/node-semver) | `7.8.1` | `7.8.2` |
| [shiki](https://github.com/shikijs/shiki/tree/HEAD/packages/shiki) | `4.1.0` | `4.2.0` |
| [undici](https://github.com/nodejs/undici) | `7.26.0` | `7.27.1` |
| [@clack/prompts](https://github.com/bombshell-dev/clack/tree/HEAD/packages/prompts) | `1.4.0` | `1.5.1` |



Updates `@convex-dev/auth` from 0.0.92 to 0.0.93
- [Changelog](https://github.com/get-convex/convex-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/get-convex/convex-auth/compare/v0.0.92...v0.0.93)

Updates `@shikijs/rehype` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/shikijs/shiki/releases)
- [Commits](https://github.com/shikijs/shiki/commits/v4.2.0/packages/rehype)

Updates `@tanstack/react-router` from 1.170.8 to 1.170.12
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-router/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-router@1.170.12/packages/react-router)

Updates `@tanstack/react-start` from 1.168.13 to 1.168.21
- [Release notes](https://github.com/TanStack/router/releases)
- [Changelog](https://github.com/TanStack/router/blob/main/packages/react-start/CHANGELOG.md)
- [Commits](https://github.com/TanStack/router/commits/@tanstack/react-start@1.168.21/packages/react-start)

Updates `convex` from 1.39.1 to 1.40.0
- [Release notes](https://github.com/get-convex/convex-backend/releases)
- [Changelog](https://github.com/get-convex/convex-backend/blob/main/npm-packages/convex/CHANGELOG.md)
- [Commits](https://github.com/get-convex/convex-backend/commits/HEAD/npm-packages/convex)

Updates `lucide-react` from 1.16.0 to 1.17.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/1.17.0/packages/lucide-react)

Updates `react` from 19.2.6 to 19.2.7
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v19.2.7/packages/react)

Updates `react-dom` from 19.2.6 to 19.2.7
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v19.2.7/packages/react-dom)

Updates `semver` from 7.8.1 to 7.8.2
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v7.8.1...v7.8.2)

Updates `shiki` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/shikijs/shiki/releases)
- [Commits](https://github.com/shikijs/shiki/commits/v4.2.0/packages/shiki)

Updates `undici` from 7.26.0 to 7.27.1
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.26.0...v7.27.1)

Updates `@clack/prompts` from 1.4.0 to 1.5.1
- [Release notes](https://github.com/bombshell-dev/clack/releases)
- [Changelog](https://github.com/bombshell-dev/clack/blob/main/packages/prompts/CHANGELOG.md)
- [Commits](https://github.com/bombshell-dev/clack/commits/@clack/prompts@1.5.1/packages/prompts)

---
updated-dependencies:
- dependency-name: "@clack/prompts"
  dependency-version: 1.5.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: "@convex-dev/auth"
  dependency-version: 0.0.93
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@shikijs/rehype"
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-router"
  dependency-version: 1.170.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: "@tanstack/react-start"
  dependency-version: 1.168.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: convex
  dependency-version: 1.40.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: lucide-react
  dependency-version: 1.17.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: react
  dependency-version: 19.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: react-dom
  dependency-version: 19.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: semver
  dependency-version: 7.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-minor-and-patch
- dependency-name: shiki
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
- dependency-name: undici
  dependency-version: 7.27.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* ci: update convex local auth runner

* ci: keep local auth deployment marker prefixed

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-06-05 13:50:43 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 634667c2c8 build(deps-dev): bump the development-minor-and-patch group across 1 directory with 5 updates (#2475)
Bumps the development-minor-and-patch group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [@vitest/coverage-v8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8) | `4.1.7` | `4.1.8` |
| [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) | `0.51.0` | `0.53.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) | `1.66.0` | `1.68.0` |
| [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.0.14` | `8.0.16` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.7` | `4.1.8` |



Updates `@vitest/coverage-v8` from 4.1.7 to 4.1.8
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/coverage-v8)

Updates `oxfmt` from 0.51.0 to 0.53.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxfmt_v0.53.0/npm/oxfmt)

Updates `oxlint` from 1.66.0 to 1.68.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.68.0/npm/oxlint)

Updates `vite` from 8.0.14 to 8.0.16
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.0.16/packages/vite)

Updates `vitest` from 4.1.7 to 4.1.8
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/vitest)

---
updated-dependencies:
- dependency-name: "@vitest/coverage-v8"
  dependency-version: 4.1.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
- dependency-name: oxfmt
  dependency-version: 0.53.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
- dependency-name: oxlint
  dependency-version: 1.68.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-minor-and-patch
- dependency-name: vite
  dependency-version: 8.0.16
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
- dependency-name: vitest
  dependency-version: 4.1.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-05 13:26:14 -07:00
Patrick Erichsen bd30b182d7 feat: show skill scan queue progress 2026-06-05 11:35:08 -07:00
Patrick Erichsen 2d719feef5 fix: sync github status through skill digests 2026-06-04 21:02:54 -07:00
Patrick Erichsen d3e1059266 fix: remove org badge from publisher profiles 2026-06-04 21:02:54 -07:00
Patrick Erichsen 1a968d8243 fix: clean up publisher catalog rows (#2513)
* fix: clean up publisher catalog rows

* fix: type github sync soft deletes
2026-06-04 20:21:59 -07:00
Patrick Erichsen 59667cc1a5 fix: keep pending github skills visible (#2514) 2026-06-04 20:20:13 -07:00
Patrick Erichsen 5b0dd96530 fix: soft delete removed github skills (#2512) 2026-06-04 19:46:57 -07:00
Patrick Erichsen 8e3858a31d chore: release clawhub cli 0.19.1
Release clawhub CLI 0.19.1 so npm includes GitHub-backed skill install support.
2026-06-04 19:26:34 -07:00
Patrick Erichsen fe4acc8e10 feat: surface github sync issues (#2509) 2026-06-04 19:15:39 -07:00
Patrick Erichsen f2e6e87756 test: gate public routes in ci smoke (#2508) 2026-06-04 19:09:39 -07:00
Patrick Erichsen 3de69919de Add official org selector to GitHub sync settings (#2505)
* feat: select official org for github sync sources

* fix: rename github sync publisher field
2026-06-04 19:08:23 -07:00
Patrick Erichsen 7629433248 fix: skip publisher handle conflicts in ownership repair (#2507) 2026-06-04 19:06:38 -07:00
Patrick Erichsen b82ad43eae fix: align org publisher catalog tabs (#2506) 2026-06-04 19:05:33 -07:00
Patrick Erichsen b7ee5c8e62 fix: backfill legacy publisher ownership (#2503)
* fix: backfill legacy publisher ownership

* test: update publisher official mocks
2026-06-04 18:57:42 -07:00
Patrick Erichsen 1c8543ade6 feat: add official org mod commands (#2504) 2026-06-04 18:47:12 -07:00
Patrick Erichsen 7cd0ef362d fix: select publisher previews by installs (#2502) 2026-06-04 18:39:47 -07:00
Patrick Erichsen 9cf9e12bdd GitHub-backed skill source sync (#2498)
* feat: add github-backed skill seed schema

* feat: render github-backed skill manifests

* feat: add github-backed skill install resolver

* feat: add github-backed skill source sync

* fix: handle github resolver blocks

* fix: harden github source sync

* fix: expose github install blocks

* feat: support GitHub-backed skill sync

* fix: handle blocked github skill content

* fix: harden github sync rollout

* fix: bound github sync payloads

* feat: allow forcing pending github skill installs

* fix: remove official publisher seed rollout

* chore: format clawhub e2e test
2026-06-04 18:20:43 -07:00
Patrick Erichsen 112e25e28c fix: retry incomplete github sign-ins (#2501) 2026-06-04 17:18:32 -07:00
Jesse Merhi d786374b77 feat: add shared download metering (#2485) 2026-06-04 16:48:08 -07:00
Jesse Merhi 3fbe27560e feat: add publisher abuse review dashboard (#2471) 2026-06-05 09:05:50 +10:00
Patrick Erichsen 33539b6df3 fix: point account recovery to github issues (#2499) 2026-06-04 14:41:52 -07:00
clawsweeper[bot]clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>momothemage
354f12a033 fix(httpApiV1): surface 400-class delete validation errors instead of opaque 500 (#2496)
Summary:
- The branch narrows the soft-delete bad-request substring whitelist from any `reserved` message to the ClawHu ... rvation phrase and adds regression tests for the intended 400 path and an unrelated reserved-word 500 path.
- Reproducibility: yes. by source inspection: current main maps any cleaned soft-delete error containing `rese ...  while preserving the package route-reservation case. I did not execute tests during this read-only review.

Automerge notes:
- No ClawSweeper repair was needed after automerge opt-in.

Validation:
- ClawSweeper review passed for head a58687c54e.
- Required merge gates passed before the squash merge.

Prepared head SHA: a58687c54e
Review: https://github.com/openclaw/clawhub/pull/2496#issuecomment-4620039541

Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: momothemage
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
2026-06-04 07:43:22 +00:00
Momo b0ea80df6c fix(httpApiV1): surface 400-class delete validation errors instead of opaque 500 (#2488)
Merged via squash.

Prepared head SHA: cc6ac9d31a
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Reviewed-by: @momothemage
2026-06-04 15:32:30 +08:00
282 changed files with 50408 additions and 6150 deletions
+149 -25
View File
@@ -1,68 +1,95 @@
---
name: clawhub-moderation
description: "Use for ClawHub staff moderation actions with the repo-local clawhub-mod tool: unhide skills, ban or unban users, change roles, and verify moderation state."
description: "Use for ClawHub staff moderation actions with the repo-local clawhub-mod/admin tool: skills, users, org publishers, plugin packages, trusted publishers, official publishers, and guarded staff email."
---
# ClawHub Moderation
Use the repo-local `clawhub-mod` tool from a checked-out ClawHub repo. It wraps
Use the repo-local admin tool from a checked-out ClawHub repo. It wraps
the existing ClawHub CLI auth/config and HTTP API surfaces. Do not call Convex
internal mutations directly for staff actions.
## Safety Rules
- Require an explicit target from the user: skill slug, user handle, or user id.
- Require a reason for `skills unhide`, `users ban`, and `users unban`.
- Require a reason for destructive, restorative, ownership, or moderation writes.
- Before any write, show the exact command and ask for confirmation unless the
user already said to proceed or supplied `--yes`.
- For `email send`, the user must explicitly ask for the email and sign off on
the final recipient, subject, and body. Dry-run is fine for drafting. Never
send until both are true, and only use `--send --confirm-user-request
--confirm-user-signoff` after that approval.
- Prefer handles for humans. Use `--id` only when the user provides a user id.
- Never bypass API-token auth, server role checks, or audit logging.
- After the write, verify state with the CLI/API and report the result.
## Commands
## Command Map
Run from the ClawHub repo root:
```sh
bun run mod -- --help
bun run admin -- --help
```
Authenticate or validate the current token:
```sh
bun run mod -- login
bun run mod -- whoami
bun run admin -- login
bun run admin -- whoami
```
Unhide a skill after moderator review:
Current top-level command groups:
```sh
bun run mod -- skills unhide <slug> --reason "<reason>" --yes
```text
auth
users
plugins|plugin
packages|package
org
email
skills|skill
```
List and triage skill reports:
### Skills
```sh
bun run mod -- skills reports --status open
bun run mod -- skills triage-report <report-id> --status confirmed --action hide --note "<note>" --yes
`bun run admin -- skills --help` exposes:
```text
unhide <slug>
rescan <slug>
reports
triage-report <report-id>
```
Ban a user:
Examples:
```sh
bun run mod -- users ban <handleOrId> --reason "<reason>" --yes
bun run admin -- skills unhide <slug> --reason "<reason>" --yes
bun run admin -- skills rescan <slug> --reason "<reason>" --yes
bun run admin -- skills reports --status open
bun run admin -- skills triage-report <report-id> --status confirmed --action hide --note "<note>" --yes
```
Unban a user:
### Users
```sh
bun run mod -- users unban <handleOrId> --reason "<reason>" --yes
`bun run admin -- users --help` exposes:
```text
ban <handleOrId>
unban <handleOrId>
set-role <handleOrId> <role>
reclassify-ban <handleOrId>
remediate-autobans
```
Change a user role:
Examples:
```sh
bun run mod -- users set-role <handleOrId> <user|moderator|admin> --yes
bun run admin -- users ban <handleOrId> --reason "<reason>" --yes
bun run admin -- users unban <handleOrId> --reason "<reason>" --yes
bun run admin -- users set-role <handleOrId> <user|moderator|admin> --yes
bun run admin -- users reclassify-ban <handleOrId> --reason "<reason>" --apply --yes
bun run admin -- users remediate-autobans --apply --reason "<reason>"
```
Use `--id` when `<handleOrId>` is a user id. Use `--fuzzy` only when the user
@@ -71,15 +98,108 @@ has asked for fuzzy handle resolution or the exact handle is ambiguous.
The old top-level aliases still exist for user commands:
```sh
bun run mod -- ban-user <handleOrId> --reason "<reason>" --yes
bun run mod -- unban-user <handleOrId> --reason "<reason>" --yes
bun run admin -- ban-user <handleOrId> --reason "<reason>" --yes
bun run admin -- unban-user <handleOrId> --reason "<reason>" --yes
```
### Org Publishers
`bun run admin -- org --help` exposes:
```text
official
create <handle>
remove-member <handle> <member>
delete <handle>
repair-scoped-packages <csv>
```
Examples:
```sh
bun run admin -- org official list
bun run admin -- org official add <handle> --reason "<reason>" --yes
bun run admin -- org official remove <handle> --reason "<reason>" --yes
bun run admin -- org create <handle> --display-name "<name>" --member <user-handle> --role owner
bun run admin -- org remove-member <handle> <member-handle>
bun run admin -- org delete <handle> --reason "<reason>" # dry-run
bun run admin -- org delete <handle> --reason "<reason>" --apply
bun run admin -- org repair-scoped-packages <csv> # dry-run
bun run admin -- org repair-scoped-packages <csv> --apply
```
`org create` requires `--member`; it must not add the moderator running the
command as an implicit owner. `org delete` only works for empty org publishers
and defaults to dry-run.
### Plugin Packages
`bun run admin -- packages --help` exposes:
```text
moderate <name>
status|moderation-status <name>
queue|moderation-queue
reports
triage-report <report-id>
transfer <name>
repair-name <name>
migrations
set-migration <bundled-plugin-id>
trusted-publisher
```
Examples:
```sh
bun run admin -- packages status <name>
bun run admin -- packages transfer <name> --to <owner> --reason "<reason>" # dry-run
bun run admin -- packages transfer <name> --to <owner> --reason "<reason>" --apply
bun run admin -- packages repair-name <name> --next-name <name> --reason "<reason>"
bun run admin -- packages trusted-publisher get <name>
bun run admin -- packages trusted-publisher set <name> --repository <owner/repo> --workflow-filename <file>
```
### Staff Email
`bun run admin -- email send --help` exposes:
```text
--to <email>
--user <handle>
--subject <subject>
--body-file <path>
--body <text>
--send
--confirm-user-request
--confirm-user-signoff
--json
```
Draft only:
```sh
bun run admin -- email send --user <handle> --subject "<subject>" --body-file <path>
```
Send only after explicit request and sign-off:
```sh
bun run admin -- email send --user <handle> --subject "<subject>" --body-file <path> --send --confirm-user-request --confirm-user-signoff
```
The server sends through the production noreply provider and writes an audit log
only after admin auth succeeds.
## Verification
- For skills, inspect the page/API status after `skills unhide`.
- For users, prefer `bun run mod -- whoami` for the current token and user
search/admin surfaces for target accounts where available.
- For users, prefer user search/admin surfaces for target accounts where
available.
- For orgs and packages, use the public publisher/plugin pages and the relevant
CLI status command after a write.
- For email, verify the CLI response and audit expectation; do not send a second
email just to test delivery.
- If verification is blocked by auth or missing admin access, report the command
result and the verification blocker plainly.
@@ -94,3 +214,7 @@ bun run mod -- unban-user <handleOrId> --reason "<reason>" --yes
hides owned skills, soft-deletes comments, and writes audit logs.
- `unban-user` is admin-only. It clears ban state and restores skills that were
hidden by the matching ban flow; revoked API tokens stay revoked.
- `packages transfer` preserves the package row, stats, releases, and history;
it changes the owner publisher.
- `org delete` soft-deletes an empty org publisher and retains member rows for
history; it refuses orgs with active skills or packages.
-93
View File
@@ -1,93 +0,0 @@
---
name: clawhub-ui-proof
description: Use when ClawHub UI changes need visual proof, before/after comparison, new-feature screenshots, temporary Playwright scenarios, or Crabbox desktop recordings.
---
# ClawHub UI Proof
Use `proof:ui` for human-readable UI evidence. The agent should write a
temporary scenario for the feature instead of manually clicking through the UI.
## Pick A Mode
- Use `--mode before-after` for bug fixes, regressions, changed copy, changed
layout, or anything where main-vs-candidate comparison helps. This is the
default and runs baseline `origin/main` plus the candidate worktree.
- Use `--mode feature` for new pages, new workflows, or new UI states that do
not exist on main. This runs only the candidate lane.
- Every proof lane runs full-stack by default: the lane's Git checkout starts
its own local Convex backend, pushes that lane's functions/schema, and builds
the frontend against that lane-local Convex URL. Add
`--seed-command '<command>'` when the scenario needs fixtures.
- Dev auth is opt-in. Use `--dev-auth` or explicit `--env KEY=VALUE` entries
only for scenarios that need development auth controls.
- Do not use `proof:ui` to inspect contributor-provided screenshots, videos, or
logs. Review those artifacts directly and cite what they prove or fail to
prove.
## Scenario Shape
Create a temporary scenario under `.artifacts/proof-scenarios/`:
```js
export default async function scenario({ baseURL, expect, page, proof }) {
await proof.step("01 skills list", async () => {
await page.goto(`${baseURL}/skills`);
await expect(page.getByText("Skills")).toBeVisible();
});
}
```
Each `proof.step()` captures a screenshot after the step. The runner compares
`origin/main` to the current worktree by default in `before-after` mode.
## Commands
Dry-run the plan first. Before/after mode is the default:
```sh
bun run proof:ui -- --mode before-after --scenario .artifacts/proof-scenarios/my-fix.pw.ts --dry-run
```
For new feature proof, run candidate-only:
```sh
bun run proof:ui -- --mode feature --scenario .artifacts/proof-scenarios/my-feature.pw.ts --dry-run
```
Run real desktop proof on a Crabbox-owned provider:
```sh
bun run proof:ui -- --mode before-after --scenario .artifacts/proof-scenarios/my-fix.pw.ts --provider hetzner
```
Run proof with seeded lane-local Convex fixtures:
```sh
bun run proof:ui -- --mode before-after --seed-command 'bunx convex run --no-push devSeed:seedNixSkills' --scenario .artifacts/proof-scenarios/my-fix.pw.ts --provider hetzner
```
Artifacts are written under `.artifacts/clawhub-ui-proof/<timestamp>/` with
screenshots, videos when available, `summary.json`, and `report.md`. Feature
mode has only candidate artifacts. Promote only broadly useful scenarios into
committed `e2e/proofs/`.
## Publish To A PR
When UI proof should appear on a GitHub PR, publish the completed proof run
instead of posting local paths:
```sh
bun run proof:publish -- --proof-dir .artifacts/clawhub-ui-proof/<timestamp> --target-pr <number>
```
`proof:publish` copies the selected screenshots, video preview GIFs when
present, MP4s, `summary.json`, and `report.md` to the `qa-artifacts` branch,
then upserts a marker-backed PR comment with inline screenshots/previews and
linked MP4s. Use `--dry-run` first when drafting or checking the comment body.
## Share In GitHub Issues
When proof images or screenshots should appear in GitHub issues, share
`here.now` links instead of uploading image attachments directly to GitHub.
Include a short note about what the linked image proves.
+11
View File
@@ -16,5 +16,16 @@ AUTH_GITHUB_SECRET=
JWT_PRIVATE_KEY=
JWKS=
# Local dev personas
DEV_AUTH_ENABLED=
DEV_AUTH_CONVEX_DEPLOYMENT=
DEV_AUTH_SITE_URL=
DEV_AUTH_SECRET=
# Embeddings
OPENAI_API_KEY=
# Transactional email
RESEND_API_KEY=
CLAWHUB_SECURITY_EMAIL=security@notifications.openclaw.ai
CLAWHUB_SECURITY_EMAIL_FROM=ClawHub Security <noreply@notifications.openclaw.ai>
+15
View File
@@ -1,5 +1,19 @@
version: 2
updates:
- package-ecosystem: "bun"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
time: "09:00"
timezone: "America/Los_Angeles"
open-pull-requests-limit: 3
labels:
- dependencies
- needs-cli-release
allow:
- dependency-name: "@openclaw/plugin-inspector"
- package-ecosystem: "bun"
directory: "/"
schedule:
@@ -9,6 +23,7 @@ updates:
timezone: "America/Los_Angeles"
open-pull-requests-limit: 10
ignore:
- dependency-name: "@openclaw/plugin-inspector"
- dependency-name: "@auth/core"
update-types:
- "version-update:semver-minor"
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Begin Testbox
uses: useblacksmith/begin-testbox@d0e04585c26905fdd92c94a09c159544c7ee1b67
uses: useblacksmith/begin-testbox@233448af4bfdc6fca509a7f0974411ac6d8a8043
with:
testbox_id: ${{ inputs.testbox_id }}
@@ -0,0 +1,285 @@
name: ClawHub CLI GitHub Release
on:
workflow_dispatch:
inputs:
tag:
description: Release tag to create or repair, for example v0.17.0
required: true
type: string
main_run_id:
description: Optional successful main CI run id to include in release proof
required: false
type: string
preflight_run_id:
description: Optional successful CLI npm preflight run id to include in release proof
required: false
type: string
publish_run_id:
description: Optional successful CLI npm publish run id to include in release proof
required: false
type: string
update_existing:
description: Update an existing GitHub Release instead of failing
required: true
default: false
type: boolean
concurrency:
group: clawhub-cli-github-release-${{ inputs.tag }}
cancel-in-progress: false
permissions: {}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NODE_VERSION: "24.x"
jobs:
create_or_update_github_release:
runs-on: ubuntu-latest
permissions:
actions: read
contents: write
steps:
- name: Checkout release tooling
uses: actions/checkout@v6
with:
ref: ${{ github.ref }}
path: release-tools
- name: Checkout release tag
uses: actions/checkout@v6
with:
ref: refs/tags/${{ inputs.tag }}
fetch-depth: 0
path: release
- name: Setup Node
uses: actions/setup-node@v6
with:
node-version: ${{ env.NODE_VERSION }}
registry-url: https://registry.npmjs.org
- name: Validate release tag and package metadata
env:
RELEASE_TAG: ${{ inputs.tag }}
RELEASE_MAIN_REF: origin/main
run: |
set -euo pipefail
RELEASE_SHA="$(git rev-parse HEAD)"
echo "RELEASE_SHA=$RELEASE_SHA" >> "$GITHUB_ENV"
git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main
if ! git merge-base --is-ancestor "$RELEASE_SHA" "$RELEASE_MAIN_REF"; then
echo "Tagged commit ${RELEASE_SHA} is not contained in ${RELEASE_MAIN_REF}." >&2
exit 1
fi
node --input-type=module <<'EOF'
import { readFileSync } from "node:fs";
const releaseTag = process.env.RELEASE_TAG ?? "";
const pkg = JSON.parse(readFileSync("./packages/clawhub/package.json", "utf8"));
const version = String(pkg.version ?? "").trim();
const errors = [];
if (pkg.name !== "clawhub") {
errors.push(`packages/clawhub/package.json name must be "clawhub"; found "${pkg.name ?? ""}".`);
}
if (!/^\d+\.\d+\.\d+$/.test(version)) {
errors.push(`packages/clawhub/package.json version must be stable semver (X.Y.Z); found "${version || "<missing>"}".`);
}
if (!/^v\d+\.\d+\.\d+$/.test(releaseTag)) {
errors.push(`Release tag must match vX.Y.Z; found "${releaseTag || "<missing>"}".`);
}
if (releaseTag !== `v${version}`) {
errors.push(`Release tag ${releaseTag} does not match packages/clawhub/package.json version ${version}; expected v${version}.`);
}
if (errors.length > 0) {
for (const error of errors) console.error(error);
process.exit(1);
}
console.log(`Release metadata OK for clawhub@${version} (${releaseTag}).`);
EOF
working-directory: release
- name: Resolve release metadata
env:
RELEASE_TAG: ${{ inputs.tag }}
run: |
set -euo pipefail
PACKAGE_VERSION="$(node --input-type=module <<'EOF'
import { readFileSync } from "node:fs";
const pkg = JSON.parse(readFileSync("./packages/clawhub/package.json", "utf8"));
process.stdout.write(String(pkg.version ?? "").trim());
EOF
)"
NPM_DIST_JSON=""
for attempt in {1..12}; do
if NPM_DIST_JSON="$(npm view "clawhub@${PACKAGE_VERSION}" dist.tarball dist.integrity --json 2>/tmp/npm-view-error)" && [[ -n "$NPM_DIST_JSON" ]]; then
break
fi
if [[ "$attempt" == "12" ]]; then
cat /tmp/npm-view-error >&2 || true
exit 1
fi
sleep 5
done
NPM_TARBALL="$(NPM_DIST_JSON="$NPM_DIST_JSON" node --input-type=module <<'EOF'
const dist = JSON.parse(process.env.NPM_DIST_JSON ?? "{}");
process.stdout.write(String(dist["dist.tarball"] ?? ""));
EOF
)"
NPM_INTEGRITY="$(NPM_DIST_JSON="$NPM_DIST_JSON" node --input-type=module <<'EOF'
const dist = JSON.parse(process.env.NPM_DIST_JSON ?? "{}");
process.stdout.write(String(dist["dist.integrity"] ?? ""));
EOF
)"
if [[ -z "$NPM_TARBALL" || -z "$NPM_INTEGRITY" ]]; then
echo "npm dist metadata for clawhub@${PACKAGE_VERSION} is incomplete." >&2
exit 1
fi
{
echo "PACKAGE_VERSION=$PACKAGE_VERSION"
echo "NPM_PACKAGE_URL=https://www.npmjs.com/package/clawhub/v/${PACKAGE_VERSION}"
echo "NPM_TARBALL_URL=$NPM_TARBALL"
echo "NPM_INTEGRITY=$NPM_INTEGRITY"
echo "RELEASE_TITLE=clawhub ${PACKAGE_VERSION}"
} >> "$GITHUB_ENV"
working-directory: release
- name: Resolve proof workflow run URLs
env:
GH_TOKEN: ${{ github.token }}
MAIN_RUN_ID: ${{ inputs.main_run_id }}
PREFLIGHT_RUN_ID: ${{ inputs.preflight_run_id }}
PUBLISH_RUN_ID: ${{ inputs.publish_run_id }}
run: |
set -euo pipefail
resolve_run_url() {
local env_name="$1"
local run_id="$2"
local expected_workflow="$3"
local expected_event="$4"
local expected_branch="$5"
if [[ -z "$run_id" ]]; then
return 0
fi
local run_json
run_json="$(gh run view "$run_id" --repo "$GITHUB_REPOSITORY" --json conclusion,event,headBranch,headSha,url,workflowName)"
RUN_JSON="$run_json" RUN_ID="$run_id" EXPECTED_WORKFLOW="$expected_workflow" EXPECTED_EVENT="$expected_event" EXPECTED_BRANCH="$expected_branch" node --input-type=module <<'EOF'
const run = JSON.parse(process.env.RUN_JSON);
const expectedWorkflow = process.env.EXPECTED_WORKFLOW;
if (expectedWorkflow && run.workflowName !== expectedWorkflow) {
console.error(`Run ${process.env.RUN_ID} must be ${expectedWorkflow}; got ${run.workflowName ?? "<missing>"}.`);
process.exit(1);
}
if (run.conclusion !== "success") {
console.error(`Run ${process.env.RUN_ID} must have conclusion=success; got ${run.conclusion ?? "<missing>"}.`);
process.exit(1);
}
if (run.headSha !== process.env.RELEASE_SHA) {
console.error(`Run ${process.env.RUN_ID} must use release SHA ${process.env.RELEASE_SHA}; got ${run.headSha ?? "<missing>"}.`);
process.exit(1);
}
if (process.env.EXPECTED_EVENT && run.event !== process.env.EXPECTED_EVENT) {
console.error(`Run ${process.env.RUN_ID} must have event=${process.env.EXPECTED_EVENT}; got ${run.event ?? "<missing>"}.`);
process.exit(1);
}
if (process.env.EXPECTED_BRANCH && run.headBranch !== process.env.EXPECTED_BRANCH) {
console.error(`Run ${process.env.RUN_ID} must have headBranch=${process.env.EXPECTED_BRANCH}; got ${run.headBranch ?? "<missing>"}.`);
process.exit(1);
}
process.stdout.write(run.url);
EOF
echo "${env_name}=$(RUN_JSON="$run_json" RUN_ID="$run_id" EXPECTED_WORKFLOW="$expected_workflow" node --input-type=module <<'EOF'
const run = JSON.parse(process.env.RUN_JSON);
process.stdout.write(run.url);
EOF
)" >> "$GITHUB_ENV"
}
resolve_run_url MAIN_RUN_URL "$MAIN_RUN_ID" "CI" "" ""
resolve_run_url PREFLIGHT_RUN_URL "$PREFLIGHT_RUN_ID" "ClawHub CLI NPM Release" "workflow_dispatch" "main"
resolve_run_url PUBLISH_RUN_URL "$PUBLISH_RUN_ID" "ClawHub CLI NPM Release" "workflow_dispatch" "main"
- name: Verify preflight proof artifact
if: ${{ inputs.preflight_run_id != '' }}
env:
GH_TOKEN: ${{ github.token }}
PREFLIGHT_RUN_ID: ${{ inputs.preflight_run_id }}
RELEASE_TAG: ${{ inputs.tag }}
run: |
set -euo pipefail
PROOF_DIR="$RUNNER_TEMP/clawhub-cli-github-release-preflight-proof"
rm -rf "$PROOF_DIR"
mkdir -p "$PROOF_DIR"
gh run download "$PREFLIGHT_RUN_ID" \
--repo "$GITHUB_REPOSITORY" \
--name "clawhub-cli-npm-preflight-${RELEASE_TAG}" \
--dir "$PROOF_DIR"
if [[ "$(tr -d '\r\n' < "$PROOF_DIR/release-tag.txt")" != "$RELEASE_TAG" ]]; then
echo "Preflight artifact tag does not match ${RELEASE_TAG}." >&2
exit 1
fi
if [[ "$(tr -d '\r\n' < "$PROOF_DIR/release-sha.txt")" != "$RELEASE_SHA" ]]; then
echo "Preflight artifact SHA does not match ${RELEASE_SHA}." >&2
exit 1
fi
if [[ "$(tr -d '\r\n' < "$PROOF_DIR/package-version.txt")" != "$PACKAGE_VERSION" ]]; then
echo "Preflight artifact version does not match ${PACKAGE_VERSION}." >&2
exit 1
fi
- name: Build release notes
env:
RELEASE_TAG: ${{ inputs.tag }}
run: |
set -euo pipefail
node ../release-tools/scripts/extract-changelog-release.mjs --tag "$RELEASE_TAG" --changelog CHANGELOG.md > ../release-body.md
{
echo
echo "### Release Proof"
echo
echo "- npm: ${NPM_PACKAGE_URL}"
echo "- tarball: ${NPM_TARBALL_URL}"
echo "- integrity: ${NPM_INTEGRITY}"
if [[ -n "${MAIN_RUN_URL:-}" ]]; then
echo "- main CI: ${MAIN_RUN_URL}"
fi
if [[ -n "${PREFLIGHT_RUN_URL:-}" ]]; then
echo "- npm preflight: ${PREFLIGHT_RUN_URL}"
fi
if [[ -n "${PUBLISH_RUN_URL:-}" ]]; then
echo "- npm publish: ${PUBLISH_RUN_URL}"
fi
} >> ../release-body.md
working-directory: release
- name: Create or update GitHub Release
env:
GH_TOKEN: ${{ github.token }}
RELEASE_TAG: ${{ inputs.tag }}
UPDATE_EXISTING: ${{ inputs.update_existing }}
run: |
set -euo pipefail
if gh release view "$RELEASE_TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
if [[ "$UPDATE_EXISTING" != "true" ]]; then
echo "GitHub Release ${RELEASE_TAG} already exists. Rerun with update_existing=true to repair it." >&2
exit 1
fi
gh release edit "$RELEASE_TAG" \
--repo "$GITHUB_REPOSITORY" \
--title "$RELEASE_TITLE" \
--notes-file release-body.md
else
gh release create "$RELEASE_TAG" \
--repo "$GITHUB_REPOSITORY" \
--title "$RELEASE_TITLE" \
--notes-file release-body.md
fi
+2 -2
View File
@@ -88,13 +88,13 @@ jobs:
- name: Initialize CodeQL
if: ${{ github.event_name != 'workflow_dispatch' || inputs.profile == 'all' || inputs.profile == matrix.category }}
uses: github/codeql-action/init@7211b7c8077ea37d8641b6271f6a365a22a5fbfa # v4
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
with:
languages: ${{ matrix.language }}
config-file: ${{ matrix.config_file }}
- name: Analyze
if: ${{ github.event_name != 'workflow_dispatch' || inputs.profile == 'all' || inputs.profile == matrix.category }}
uses: github/codeql-action/analyze@7211b7c8077ea37d8641b6271f6a365a22a5fbfa # v4
uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
with:
category: "/codeql-light/${{ matrix.category }}"
+229 -5
View File
@@ -61,11 +61,14 @@ on:
description: Optional source path inside the repository for monorepo package publishes.
required: false
type: string
clawhub_version:
description: Legacy npm CLI version input. Kept for compatibility; the workflow now runs the checked-out source.
package_artifact_name:
description: Optional Actions artifact name containing a prebuilt ClawPack .tgz to publish.
required: false
type: string
package_artifact_path:
description: Optional path to the .tgz inside package_artifact_name. Defaults to the only .tgz in the artifact.
required: false
type: string
default: latest
secrets:
clawhub_token:
required: false
@@ -87,6 +90,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
actions: read
contents: read
id-token: write
outputs:
@@ -218,7 +222,56 @@ jobs:
PY
echo "CLAWHUB_CONFIG_PATH=$RUNNER_TEMP/clawhub-config.json" >> "$GITHUB_ENV"
- name: Download prebuilt package artifact
if: inputs.package_artifact_name != ''
uses: actions/download-artifact@v8
with:
name: ${{ inputs.package_artifact_name }}
path: ${{ runner.temp }}/prebuilt-package-artifact
- name: Resolve prebuilt package artifact
id: resolve_artifact
env:
INPUT_PACKAGE_ARTIFACT_NAME: ${{ inputs.package_artifact_name }}
INPUT_PACKAGE_ARTIFACT_PATH: ${{ inputs.package_artifact_path }}
run: |
python3 - <<'PY'
import os
from pathlib import Path
artifact_name = os.environ["INPUT_PACKAGE_ARTIFACT_NAME"].strip()
output_path = Path(os.environ["GITHUB_OUTPUT"])
if not artifact_name:
with output_path.open("a", encoding="utf-8") as fh:
fh.write("package_artifact_path=\n")
raise SystemExit(0)
artifact_root = Path(os.environ["RUNNER_TEMP"]) / "prebuilt-package-artifact"
requested_path = os.environ["INPUT_PACKAGE_ARTIFACT_PATH"].strip()
if requested_path:
candidate = (artifact_root / requested_path).resolve()
if artifact_root.resolve() not in candidate.parents and candidate != artifact_root.resolve():
raise SystemExit(f"Prebuilt artifact path escapes downloaded artifact: {requested_path}")
if not candidate.is_file():
raise SystemExit(f"Prebuilt package artifact path not found: {requested_path}")
else:
candidates = sorted(path for path in artifact_root.rglob("*.tgz") if path.is_file())
if not candidates:
raise SystemExit(f"Prebuilt package artifact {artifact_name!r} did not contain a .tgz file.")
if len(candidates) > 1:
joined = ", ".join(str(path.relative_to(artifact_root)) for path in candidates)
raise SystemExit(
"Prebuilt package artifact contains multiple .tgz files; set package_artifact_path. "
f"Found: {joined}"
)
candidate = candidates[0]
with output_path.open("a", encoding="utf-8") as fh:
fh.write(f"package_artifact_path={candidate}\n")
PY
- name: Resolve publish command
id: resolve_publish
env:
INPUT_SOURCE: ${{ inputs.source }}
INPUT_REF: ${{ inputs.ref }}
@@ -230,9 +283,11 @@ jobs:
INPUT_SOURCE_COMMIT: ${{ inputs.source_commit }}
INPUT_SOURCE_REF: ${{ inputs.source_ref }}
INPUT_SOURCE_PATH: ${{ inputs.source_path }}
PREBUILT_PACKAGE_ARTIFACT_PATH: ${{ steps.resolve_artifact.outputs.package_artifact_path }}
INPUT_SITE: ${{ inputs.site }}
INPUT_REGISTRY: ${{ inputs.registry }}
CLAWHUB_TOKEN: ${{ secrets.clawhub_token }}
GITHUB_TOKEN: ${{ github.token }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_REF: ${{ github.ref }}
@@ -243,6 +298,78 @@ jobs:
import os
import shlex
from pathlib import Path
from urllib.error import HTTPError
from urllib.parse import quote, urlparse
from urllib.request import Request, urlopen
def split_ref_path(value):
if not value:
return "", ""
if ":" not in value:
return value, ""
ref, path = value.split(":", 1)
return ref, path.strip("/")
def github_commit_exists(repo, ref):
token = os.environ.get("GITHUB_TOKEN", "").strip()
headers = {
"Accept": "application/vnd.github+json",
"User-Agent": "clawhub-package-publish",
}
if token:
headers["Authorization"] = f"Bearer {token}"
request = Request(
f"https://api.github.com/repos/{repo}/commits/{quote(ref, safe='')}",
headers=headers,
)
try:
with urlopen(request, timeout=10) as response:
return 200 <= response.status < 300
except HTTPError as error:
if error.code in (404, 422):
return False
raise
def resolve_github_url_ref_and_path(repo, kind, segments):
min_path_segments = 1 if kind == "blob" else 0
max_ref_segments = len(segments) - min_path_segments
for ref_segment_count in range(max_ref_segments, 0, -1):
ref = "/".join(segments[:ref_segment_count])
path = "/".join(segments[ref_segment_count:]).strip("/")
if kind == "blob" and not path:
continue
if not github_commit_exists(repo, ref):
continue
if kind == "blob":
path = "/".join(path.split("/")[:-1]).strip("/")
return ref, path
raise SystemExit(f"GitHub ref not found in source URL for {repo}")
def parse_github_source(value):
raw = value.strip()
if raw.startswith("github:"):
raw = raw[len("github:"):]
if raw.startswith("https://") or raw.startswith("http://"):
parsed = urlparse(raw)
if parsed.netloc.lower() != "github.com":
return None
parts = [part for part in parsed.path.strip("/").split("/") if part]
if len(parts) < 2:
return None
repo_name = parts[1][:-4] if parts[1].endswith(".git") else parts[1]
repo = f"{parts[0]}/{repo_name}"
if len(parts) >= 4 and parts[2] in {"tree", "blob"}:
ref, path = resolve_github_url_ref_and_path(repo, parts[2], parts[3:])
return {"repo": repo, "ref": ref, "path": path}
return {"repo": repo, "ref": "", "path": ""}
source_part, at, ref_part = raw.partition("@")
repo_parts = source_part.split("/")
if len(repo_parts) != 2 or not repo_parts[0] or not repo_parts[1]:
return None
repo_name = repo_parts[1][:-4] if repo_parts[1].endswith(".git") else repo_parts[1]
ref, path = split_ref_path(ref_part if at else "")
return {"repo": f"{repo_parts[0]}/{repo_name}", "ref": ref, "path": path}
source = os.environ["INPUT_SOURCE"].strip()
if not source:
@@ -254,6 +381,36 @@ jobs:
is_local_source = source.startswith(".") or source.startswith("/") or Path(source).exists()
if ref and "@" not in source and not source.startswith("http") and not is_local_source:
source = f"{source}@{ref}"
source_path = os.environ["INPUT_SOURCE_PATH"].strip()
prebuilt_artifact_path = os.environ["PREBUILT_PACKAGE_ARTIFACT_PATH"].strip()
if source_path and prebuilt_artifact_path:
raise SystemExit("Prebuilt artifact mode does not accept source_path; publish the already-packed artifact instead.")
inspect_checkout_repository = ""
inspect_checkout_ref = ""
inspect_local_root = str(Path(os.environ["GITHUB_WORKSPACE"]).resolve())
inspect_subdir = source_path
if prebuilt_artifact_path:
inspect_local_root = str((Path(os.environ["RUNNER_TEMP"]) / "prebuilt-package-inspect").resolve())
inspect_subdir = ""
elif is_local_source:
inspect_local_root = str(Path(source).resolve())
else:
github_source = parse_github_source(source)
source_ref_differs_from_checkout = (
bool(github_source and github_source["ref"])
and github_source["ref"] != os.environ["GITHUB_SHA"]
)
if github_source and (
github_source["repo"] != os.environ["GITHUB_REPOSITORY"]
or source_ref_differs_from_checkout
):
inspect_checkout_repository = github_source["repo"]
inspect_checkout_ref = github_source["ref"]
inspect_local_root = str((Path(os.environ["GITHUB_WORKSPACE"]) / "clawhub-publish-source").resolve())
inspect_subdir = source_path or github_source["path"]
elif github_source:
inspect_subdir = source_path or github_source["path"]
cli_entry = (
Path(os.environ["GITHUB_WORKSPACE"])
@@ -266,12 +423,13 @@ jobs:
if not cli_entry.exists():
raise SystemExit(f"Missing ClawHub CLI entrypoint at {cli_entry}")
cmd_source = prebuilt_artifact_path or source
cmd = [
"bun",
str(cli_entry),
"package",
"publish",
source,
cmd_source,
"--site",
os.environ["INPUT_SITE"],
"--registry",
@@ -294,7 +452,16 @@ jobs:
source_repo = os.environ["INPUT_SOURCE_REPO"].strip()
source_commit = os.environ["INPUT_SOURCE_COMMIT"].strip()
source_ref = os.environ["INPUT_SOURCE_REF"].strip()
source_path = os.environ["INPUT_SOURCE_PATH"].strip()
if prebuilt_artifact_path:
if not source_repo and not source_commit:
source_repo = os.environ["GITHUB_REPOSITORY"].strip()
source_commit = os.environ["GITHUB_SHA"].strip()
elif not source_repo or not source_commit:
raise SystemExit(
"Prebuilt artifact mode requires source_repo and source_commit together when overriding source attribution."
)
if not source_ref:
source_ref = os.environ["GITHUB_REF"].strip()
if source_repo:
cmd += ["--source-repo", source_repo]
if source_commit:
@@ -318,8 +485,65 @@ jobs:
path.write_text("#!/usr/bin/env bash\nset -euo pipefail\n" + shell_line + "\n", encoding="utf-8")
path.chmod(0o755)
print(shell_line)
output_path = Path(os.environ["GITHUB_OUTPUT"])
with output_path.open("a", encoding="utf-8") as fh:
fh.write(f"inspect_checkout_repository={inspect_checkout_repository}\n")
fh.write(f"inspect_checkout_ref={inspect_checkout_ref}\n")
fh.write(f"inspect_local_root={inspect_local_root}\n")
fh.write(f"inspect_subdir={inspect_subdir}\n")
PY
- name: Extract prebuilt package artifact for plugin validation
if: steps.resolve_artifact.outputs.package_artifact_path != ''
env:
PREBUILT_PACKAGE_ARTIFACT_PATH: ${{ steps.resolve_artifact.outputs.package_artifact_path }}
INSPECT_LOCAL_ROOT: ${{ steps.resolve_publish.outputs.inspect_local_root }}
run: |
set -euo pipefail
mkdir -p "$INSPECT_LOCAL_ROOT"
tar -xzf "$PREBUILT_PACKAGE_ARTIFACT_PATH" -C "$INSPECT_LOCAL_ROOT" --strip-components=1
- name: Checkout publish source for plugin inspector
if: steps.resolve_publish.outputs.inspect_checkout_repository != ''
uses: actions/checkout@v6
with:
repository: ${{ steps.resolve_publish.outputs.inspect_checkout_repository }}
ref: ${{ steps.resolve_publish.outputs.inspect_checkout_ref }}
path: clawhub-publish-source
- name: Run plugin validation
env:
INSPECT_LOCAL_ROOT: ${{ steps.resolve_publish.outputs.inspect_local_root }}
INSPECT_SUBDIR: ${{ steps.resolve_publish.outputs.inspect_subdir }}
run: |
set -euo pipefail
inspect_root="$(python3 - <<'PY'
import os
from pathlib import Path
root = Path(os.environ["INSPECT_LOCAL_ROOT"]).resolve()
subdir = os.environ["INSPECT_SUBDIR"].strip()
inspect_root = (root / subdir).resolve() if subdir else root
if inspect_root != root and root not in inspect_root.parents:
raise SystemExit(f"Inspector source path escapes publish source: {subdir}")
print(inspect_root)
PY
)"
if [ ! -f "$inspect_root/package.json" ] && [ ! -f "$inspect_root/openclaw.plugin.json" ]; then
echo "::warning::Plugin Inspector skipped because $inspect_root is not a plugin root."
exit 0
fi
bun "$GITHUB_WORKSPACE/clawhub-source/packages/clawhub/src/cli.ts" package validate "$inspect_root" --out "$RUNNER_TEMP/plugin-inspector"
- name: Upload plugin inspector reports
if: always()
uses: actions/upload-artifact@v7
with:
name: plugin-inspector-report
path: ${{ runner.temp }}/plugin-inspector
if-no-files-found: ignore
- name: Run package publish
run: |
set -euo pipefail
@@ -0,0 +1,63 @@
name: Plugin Inspector Bulk Scan
on:
workflow_dispatch:
inputs:
batch_size:
description: "Maximum plugin releases to scan"
required: false
default: "25"
dry_run:
description: "Preview impact without persisting findings or sending emails"
required: false
default: "false"
type: boolean
dry_run_max_batches:
description: "Maximum preview batches to scan when dry_run is enabled"
required: false
default: "20"
source_pr:
description: "Merged PR number that triggered this scan, when dispatched automatically"
required: false
default: ""
source_sha:
description: "Merged commit SHA that triggered this scan, when dispatched automatically"
required: false
default: ""
permissions:
contents: read
jobs:
scan:
name: Scan published plugins
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Run plugin inspector bulk scan
env:
CLAWHUB_SITE_URL: ${{ vars.CLAWHUB_SITE_URL || 'https://clawhub.ai' }}
CLAWHUB_PLUGIN_INSPECTOR_WORKER_TOKEN: ${{ secrets.CLAWHUB_PLUGIN_INSPECTOR_WORKER_TOKEN }}
PLUGIN_INSPECTOR_BATCH_SIZE: ${{ inputs.batch_size || '25' }}
PLUGIN_INSPECTOR_DRY_RUN: ${{ inputs.dry_run && '1' || '0' }}
PLUGIN_INSPECTOR_DRY_RUN_MAX_BATCHES: ${{ inputs.dry_run_max_batches || '20' }}
PLUGIN_INSPECTOR_SOURCE_PR: ${{ inputs.source_pr || '' }}
PLUGIN_INSPECTOR_SOURCE_SHA: ${{ inputs.source_sha || '' }}
PLUGIN_INSPECTOR_ARTIFACT_DIR: plugin-inspector-bulk-scan-reports
run: bun scripts/package-inspector-nightly-scan.ts
- name: Upload inspector reports
if: always()
uses: actions/upload-artifact@v4
with:
name: plugin-inspector-bulk-scan-reports
path: plugin-inspector-bulk-scan-reports
if-no-files-found: warn
@@ -0,0 +1,55 @@
name: Plugin Inspector Pin Bump Dispatch
on:
push:
branches:
- main
paths:
- package.json
- packages/clawhub/package.json
- bun.lock
permissions:
actions: write
contents: read
jobs:
dispatch-plugin-inspector-bulk-scan:
name: Dispatch Plugin Inspector bulk scan
runs-on: ubuntu-latest
steps:
- name: Checkout main commit
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}
fetch-depth: 0
- name: Fetch previous main commit
env:
BASE_SHA: ${{ github.event.before }}
run: git fetch --no-tags --depth=1 origin "$BASE_SHA"
- name: Detect pinned Plugin Inspector change
id: detect
env:
BASE_SHA: ${{ github.event.before }}
HEAD_SHA: ${{ github.sha }}
run: node scripts/github/plugin-inspector-pin-change.mjs --base "$BASE_SHA" --head "$HEAD_SHA"
- name: Dispatch Plugin Inspector bulk scan
if: ${{ steps.detect.outputs.changed == 'true' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
gh workflow run plugin-inspector-bulk-scan.yml \
--ref main \
-f batch_size=25 \
-f dry_run=false \
-f dry_run_max_batches=20 \
-f source_sha=${{ github.sha }}
- name: Explain skipped dispatch
if: ${{ steps.detect.outputs.changed != 'true' }}
env:
DISPATCH_SKIP_REASON: ${{ steps.detect.outputs.reason }}
run: printf '%s\n' "$DISPATCH_SKIP_REASON"
@@ -0,0 +1,84 @@
name: ClawHub Scheduled Live Checks
on:
schedule:
- cron: "17 5 * * *"
workflow_dispatch:
inputs:
github-repo:
description: GitHub skills repo to use for the source-backed canary
required: false
default: openclaw/agent-skills
github-skill:
description: Skill slug to verify from the GitHub skills repo
required: false
default: handoff
concurrency:
group: clawhub-scheduled-live-checks-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
env:
VITE_CONVEX_URL: https://example.invalid
jobs:
github-backed-skills:
name: GitHub-backed skills canary
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/setup-bun
- name: Run GitHub-backed skills live canary
env:
CLAWHUB_LIVE_GITHUB_CANARY: "1"
CLAWHUB_LIVE_GITHUB_REPO: ${{ inputs.github-repo || 'openclaw/agent-skills' }}
CLAWHUB_LIVE_GITHUB_SKILL: ${{ inputs.github-skill || 'handoff' }}
GITHUB_TOKEN: ${{ github.token }}
run: bunx vitest run convex/githubSkillSync.live.test.ts
open-failure-issue:
name: Open failure issue
needs: github-backed-skills
if: ${{ always() && needs.github-backed-skills.result == 'failure' }}
runs-on: ubuntu-latest
permissions:
issues: write
steps:
- name: Open or update failure issue
env:
GH_TOKEN: ${{ github.token }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
WORKFLOW_NAME: ${{ github.workflow }}
run: |
set -euo pipefail
marker_token="clawhub-scheduled-live-checks-failure"
marker="<!-- $marker_token -->"
title="ClawHub scheduled live checks failing"
issue_number="$(gh issue list \
--repo "$GITHUB_REPOSITORY" \
--state open \
--search "$marker_token in:body" \
--json number \
--jq '.[0].number // empty')"
body_file="$(mktemp)"
cat > "$body_file" <<EOF
$marker
The scheduled ClawHub live checks failed.
Workflow: $WORKFLOW_NAME
Run: $RUN_URL
EOF
if [[ -n "$issue_number" ]]; then
gh issue comment "$issue_number" --repo "$GITHUB_REPOSITORY" --body-file "$body_file"
else
gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$body_file"
fi
+1 -1
View File
@@ -51,7 +51,7 @@ jobs:
id: trufflehog
# Use a concrete released ref that resolves in upstream action registry.
# v3 (major tag) is not published by trufflesecurity/trufflehog.
uses: trufflesecurity/trufflehog@v3.95.3
uses: trufflesecurity/trufflehog@v3.95.5
with:
path: ./
base: ${{ steps.scan_range.outputs.base }}
+1 -12
View File
@@ -3,6 +3,7 @@ node_modules
.bun-build
*.bun-build
.artifacts/
artifacts/
.cache/
.data/
bin/docs-list
@@ -37,18 +38,6 @@ convex/_generated/ai/*
!convex/_generated/ai/guidelines.md
!convex/_generated/ai/ai-files.state.json
skills-lock.json
*/skills/*
!.agents/skills/
!.agents/skills/convex*/
!.agents/skills/convex*/**
!.agents/skills/clawhub-ui-proof/
!.agents/skills/clawhub-ui-proof/**
!.agents/skills/clawhub-pr-maintainer/
!.agents/skills/clawhub-pr-maintainer/**
!.agents/skills/clawhub-moderation/
!.agents/skills/clawhub-moderation/**
!.agents/skills/autoreview/
!.agents/skills/autoreview/**
skills/*
.codex/*
!.codex/environments/
+8 -1
View File
@@ -51,6 +51,8 @@ Specialized corpus, scanner, security-worker, UI proof, proof publishing, Crabbo
- Tests live in `src/**` and `convex/lib/**`.
- Coverage threshold: 80% global (lines/functions/branches/statements).
- Example: `convex/lib/skills.test.ts`.
- When adding or changing Convex functions, do not rely only on mocked `ctx` tests for behavior that depends on Convex runtime semantics such as pagination, indexes, validators, auth identity, internal/public function boundaries, scheduler/cron behavior, actions calling queries/mutations, HTTP actions, storage, or OCC/transaction behavior. Add or run a real Convex validation path, such as `convex dev --once`, `convex run`, an HTTP action smoke, or a local-auth Playwright flow, covering the changed behavior. Mocked `ctx.db` / `ctx.runQuery` tests are still fine for pure business logic, but they do not count as Convex runtime validation.
- For local UI state testing, prefer creating realistic backend state through seed logic plus a DevPersonaFab entry for the associated test user. Avoid one-off manual DB edits when the state is likely to be reused, such as org membership, official publisher access, moderation holds, or publishing permissions.
## Commit & Pull Request Guidelines
@@ -60,9 +62,9 @@ Specialized corpus, scanner, security-worker, UI proof, proof publishing, Crabbo
- Before commit/PR handoff for non-trivial code changes, use `$autoreview` until no accepted/actionable findings remain, unless equivalent manual review already happened, the change is trivial/docs-only, or the user opts out.
- Before opening a PR for source or test changes, run the targeted tests for the touched behavior and `bun run ci:unit` (`VITE_CONVEX_URL=https://example.invalid bun run coverage`) unless the change is docs/config-only or the user explicitly asks to rely on CI. For runtime, build, or package changes, also run the matching broader gate when it covers the touched surface: `bun run ci:types-build`, `bun run ci:packages`, `bun run ci:e2e-http`, or `bun run ci:playwright-smoke`.
- PRs: include summary + test commands run. Add screenshots for UI changes.
- Screenshot proof MUST come from a real running ClawHub instance in a real browser. Do not use generated HTML mockups, synthetic terminal cards, or manually composed images as proof. For route/status/backend visibility bugs, run ClawHub locally with the relevant Convex code and fixture state, capture the actual browser page, and state the local URL and fixture used.
- Before merging any PR, verify TypeScript cleanly with `bunx tsc -p packages/schema/tsconfig.json --noEmit` and `bunx tsc -p packages/clawhub/tsconfig.json --noEmit`; if Convex code changed, also run the repo typecheck path used by deploy so `bunx convex deploy` will not fail on `tsc`.
- GitHub comments: for multiline `gh` comments/close messages, use `--body-file`, `--input`, or stdin/heredoc with real newlines; never pass literal `\\n` in shell strings.
- Reject PRs that add skills into source code/repo content directly (for example under `skills/` or seed-only additions intended as published skills). Skills must be uploaded/published via CLI.
- Repo-local developer skills under `.agents/skills/` are allowed only when they are ClawHub-specific, such as Convex, moderation, PR maintainer, or UI proof workflows. Keep generic shared skills such as `crabbox` and `autoreview` in the global `agent-skills` install, not this repo. Keep top-level `skills/` reserved for installed/published skill content and ignored by git.
## Production Release
@@ -101,6 +103,11 @@ Specialized corpus, scanner, security-worker, UI proof, proof publishing, Crabbo
- For non-interactive prod deploys, use `bunx convex deploy -y` to skip confirmation.
- If `bunx convex run --env-file .env.local ...` returns `401 MissingAccessToken` despite `bunx convex login`, workaround: omit `--env-file` and use `--deployment-name <name>` / `--prod`.
## Convex Migrations & Backfills
- Any Convex production data migration, backfill, destructive cleanup, schema narrowing, or table reshaping must start with the `convex-migration-helper` skill. Default to `@convex-dev/migrations` for production data changes because it provides batching, dry runs, resume/progress tracking, and safer operator UX. Exceptions require an explicit note explaining why the component is unnecessary, plus equivalent dry-run support, cursor batching, resume/progress behavior, confirmation for destructive writes, and real Convex runtime validation.
- After a migration or cleanup is verified complete, remove temporary migration functions/code in a follow-up PR unless they are intentionally retained as ongoing maintenance tooling.
## Convex Query & Bandwidth Rules
- **Always use `.withIndex()` instead of `.filter()` for fields that can be indexed.** `.filter()` causes full table scans — every doc is read and billed. Even a single `.filter()` on a 16K-row table reads ~16 MB per call.
+24 -1
View File
@@ -1,6 +1,29 @@
# Changelog
## Unreleased
## 0.20.2 - 2026-06-11
### Changes
- CLI packages now require Node.js 22 or newer, dropping the EOL Node 20 runtime floor.
- CLI: add `clawhub package validate <source>` for local plugin validation with author-facing Plugin Inspector findings, remediation text, and report artifacts.
## 0.20.0 - 2026-06-06
### Changes
- CLI/API: replace local `clawhub scan` uploads with stored submitted-version scan report downloads, including owner-authorized `clawhub scan download <name> --version <version>` support for blocked skill and plugin submissions.
## 0.19.2 - 2026-06-05
### Fixes
- CLI: accept the legacy `clawhub skill verify --json` flag as a hidden compatibility no-op while continuing to print JSON by default.
## 0.19.1 - 2026-06-05
### Fixes
- CLI: install source-backed GitHub skills from the deployed `/api/v1/skills/:slug/install` resolver so `clawhub install` works for skills without hosted ClawHub versions.
## 0.19.0 - 2026-06-03
+1 -1
View File
@@ -74,7 +74,7 @@ Docs: [`docs/quickstart.md`](docs/quickstart.md), [`docs/cli.md`](docs/cli.md).
## Telemetry
ClawHub tracks minimal **install telemetry** (to compute install counts) when you run `clawhub sync` while logged in.
ClawHub tracks minimal **install telemetry** (to compute install counts) when you run `clawhub install` while logged in.
Disable via:
```bash
+13
View File
@@ -9,6 +9,19 @@ Good ClawHub advisory reports include bugs in:
- authentication, authorization, or API tokens
- scanning, moderation, or report handling
Because ClawHub is a hosted cloud application, ClawHub service vulnerabilities
are not publicly disclosed by default. They are publicly disclosed when there is
evidence of real user impact or when users need to take action.
Examples of real user impact include confirmed exploitation, exposure of user
data or secrets, malicious content reaching users because of a platform failure,
or any issue that requires users to rotate credentials, update local software, or
take other protective action.
Vulnerabilities in user-installed software are publicly disclosed, such as
ClawHub CLI packages, binaries, libraries, or other release artifacts that users
need to update locally.
Do not use ClawHub advisories for vulnerabilities in a third-party skill or
plugin's own source code. Report those directly to the publisher or source
repository linked from the ClawHub listing.
+256 -355
View File
File diff suppressed because it is too large Load Diff
+28 -2
View File
@@ -17,7 +17,9 @@ import type * as crons from "../crons.js";
import type * as depRegistryScan from "../depRegistryScan.js";
import type * as devSeed from "../devSeed.js";
import type * as devSeedExtra from "../devSeedExtra.js";
import type * as downloadMetrics from "../downloadMetrics.js";
import type * as downloads from "../downloads.js";
import type * as emailsNode from "../emailsNode.js";
import type * as functions from "../functions.js";
import type * as githubAccountAgeBackfill from "../githubAccountAgeBackfill.js";
import type * as githubBackups from "../githubBackups.js";
@@ -26,6 +28,8 @@ import type * as githubIdentity from "../githubIdentity.js";
import type * as githubImport from "../githubImport.js";
import type * as githubRestore from "../githubRestore.js";
import type * as githubRestoreMutations from "../githubRestoreMutations.js";
import type * as githubSkillSources from "../githubSkillSources.js";
import type * as githubSkillSync from "../githubSkillSync.js";
import type * as githubSoulBackups from "../githubSoulBackups.js";
import type * as githubSoulBackupsNode from "../githubSoulBackupsNode.js";
import type * as http from "../http.js";
@@ -53,8 +57,9 @@ import type * as lib_changelog from "../lib/changelog.js";
import type * as lib_clawpack from "../lib/clawpack.js";
import type * as lib_commentScamPrompt from "../lib/commentScamPrompt.js";
import type * as lib_contentTypes from "../lib/contentTypes.js";
import type * as lib_depRegistryScan from "../lib/depRegistryScan.js";
import type * as lib_devAuth from "../lib/devAuth.js";
import type * as lib_devSeed from "../lib/devSeed.js";
import type * as lib_emails from "../lib/emails.js";
import type * as lib_embeddingVisibility from "../lib/embeddingVisibility.js";
import type * as lib_embeddings from "../lib/embeddings.js";
import type * as lib_githubAccount from "../lib/githubAccount.js";
@@ -65,16 +70,19 @@ import type * as lib_githubIdentity from "../lib/githubIdentity.js";
import type * as lib_githubImport from "../lib/githubImport.js";
import type * as lib_githubProfileSync from "../lib/githubProfileSync.js";
import type * as lib_githubRestoreHelpers from "../lib/githubRestoreHelpers.js";
import type * as lib_githubSkillSync from "../lib/githubSkillSync.js";
import type * as lib_githubSoulBackup from "../lib/githubSoulBackup.js";
import type * as lib_globalStats from "../lib/globalStats.js";
import type * as lib_httpHeaders from "../lib/httpHeaders.js";
import type * as lib_httpRateLimit from "../lib/httpRateLimit.js";
import type * as lib_httpUtils from "../lib/httpUtils.js";
import type * as lib_installResolver from "../lib/installResolver.js";
import type * as lib_leaderboards from "../lib/leaderboards.js";
import type * as lib_manualOverrides from "../lib/manualOverrides.js";
import type * as lib_moderation from "../lib/moderation.js";
import type * as lib_moderationEngine from "../lib/moderationEngine.js";
import type * as lib_moderationReasonCodes from "../lib/moderationReasonCodes.js";
import type * as lib_observabilityEvents from "../lib/observabilityEvents.js";
import type * as lib_officialPublishers from "../lib/officialPublishers.js";
import type * as lib_openaiResponse from "../lib/openaiResponse.js";
import type * as lib_packageRegistry from "../lib/packageRegistry.js";
@@ -85,6 +93,7 @@ import type * as lib_public from "../lib/public.js";
import type * as lib_publicRouteReservations from "../lib/publicRouteReservations.js";
import type * as lib_publishLimits from "../lib/publishLimits.js";
import type * as lib_publisherAbuseScoring from "../lib/publisherAbuseScoring.js";
import type * as lib_publisherCatalogDisplay from "../lib/publisherCatalogDisplay.js";
import type * as lib_publisherStats from "../lib/publisherStats.js";
import type * as lib_publishers from "../lib/publishers.js";
import type * as lib_reporting from "../lib/reporting.js";
@@ -115,9 +124,13 @@ import type * as lib_userSkillStats from "../lib/userSkillStats.js";
import type * as lib_webhooks from "../lib/webhooks.js";
import type * as llmEval from "../llmEval.js";
import type * as maintenance from "../maintenance.js";
import type * as managementDevSeed from "../managementDevSeed.js";
import type * as packageInspectorHttp from "../packageInspectorHttp.js";
import type * as packageInspectorNode from "../packageInspectorNode.js";
import type * as packagePublishTokens from "../packagePublishTokens.js";
import type * as packages from "../packages.js";
import type * as publisherAbuse from "../publisherAbuse.js";
import type * as publisherAbuseDevSeed from "../publisherAbuseDevSeed.js";
import type * as publishers from "../publishers.js";
import type * as rateLimits from "../rateLimits.js";
import type * as search from "../search.js";
@@ -159,7 +172,9 @@ declare const fullApi: ApiFromModules<{
depRegistryScan: typeof depRegistryScan;
devSeed: typeof devSeed;
devSeedExtra: typeof devSeedExtra;
downloadMetrics: typeof downloadMetrics;
downloads: typeof downloads;
emailsNode: typeof emailsNode;
functions: typeof functions;
githubAccountAgeBackfill: typeof githubAccountAgeBackfill;
githubBackups: typeof githubBackups;
@@ -168,6 +183,8 @@ declare const fullApi: ApiFromModules<{
githubImport: typeof githubImport;
githubRestore: typeof githubRestore;
githubRestoreMutations: typeof githubRestoreMutations;
githubSkillSources: typeof githubSkillSources;
githubSkillSync: typeof githubSkillSync;
githubSoulBackups: typeof githubSoulBackups;
githubSoulBackupsNode: typeof githubSoulBackupsNode;
http: typeof http;
@@ -195,8 +212,9 @@ declare const fullApi: ApiFromModules<{
"lib/clawpack": typeof lib_clawpack;
"lib/commentScamPrompt": typeof lib_commentScamPrompt;
"lib/contentTypes": typeof lib_contentTypes;
"lib/depRegistryScan": typeof lib_depRegistryScan;
"lib/devAuth": typeof lib_devAuth;
"lib/devSeed": typeof lib_devSeed;
"lib/emails": typeof lib_emails;
"lib/embeddingVisibility": typeof lib_embeddingVisibility;
"lib/embeddings": typeof lib_embeddings;
"lib/githubAccount": typeof lib_githubAccount;
@@ -207,16 +225,19 @@ declare const fullApi: ApiFromModules<{
"lib/githubImport": typeof lib_githubImport;
"lib/githubProfileSync": typeof lib_githubProfileSync;
"lib/githubRestoreHelpers": typeof lib_githubRestoreHelpers;
"lib/githubSkillSync": typeof lib_githubSkillSync;
"lib/githubSoulBackup": typeof lib_githubSoulBackup;
"lib/globalStats": typeof lib_globalStats;
"lib/httpHeaders": typeof lib_httpHeaders;
"lib/httpRateLimit": typeof lib_httpRateLimit;
"lib/httpUtils": typeof lib_httpUtils;
"lib/installResolver": typeof lib_installResolver;
"lib/leaderboards": typeof lib_leaderboards;
"lib/manualOverrides": typeof lib_manualOverrides;
"lib/moderation": typeof lib_moderation;
"lib/moderationEngine": typeof lib_moderationEngine;
"lib/moderationReasonCodes": typeof lib_moderationReasonCodes;
"lib/observabilityEvents": typeof lib_observabilityEvents;
"lib/officialPublishers": typeof lib_officialPublishers;
"lib/openaiResponse": typeof lib_openaiResponse;
"lib/packageRegistry": typeof lib_packageRegistry;
@@ -227,6 +248,7 @@ declare const fullApi: ApiFromModules<{
"lib/publicRouteReservations": typeof lib_publicRouteReservations;
"lib/publishLimits": typeof lib_publishLimits;
"lib/publisherAbuseScoring": typeof lib_publisherAbuseScoring;
"lib/publisherCatalogDisplay": typeof lib_publisherCatalogDisplay;
"lib/publisherStats": typeof lib_publisherStats;
"lib/publishers": typeof lib_publishers;
"lib/reporting": typeof lib_reporting;
@@ -257,9 +279,13 @@ declare const fullApi: ApiFromModules<{
"lib/webhooks": typeof lib_webhooks;
llmEval: typeof llmEval;
maintenance: typeof maintenance;
managementDevSeed: typeof managementDevSeed;
packageInspectorHttp: typeof packageInspectorHttp;
packageInspectorNode: typeof packageInspectorNode;
packagePublishTokens: typeof packagePublishTokens;
packages: typeof packages;
publisherAbuse: typeof publisherAbuse;
publisherAbuseDevSeed: typeof publisherAbuseDevSeed;
publishers: typeof publishers;
rateLimits: typeof rateLimits;
search: typeof search;
+47 -2
View File
@@ -3,7 +3,9 @@ import type { Id } from "./_generated/dataModel";
import {
BANNED_REAUTH_MESSAGE,
DELETED_ACCOUNT_REAUTH_MESSAGE,
createGitHubAuthProvider,
handleDeletedUserSignIn,
normalizeGitHubProfileId,
} from "./auth";
function makeCtx({
@@ -128,7 +130,7 @@ describe("handleDeletedUserSignIn", () => {
expect(ctx.db.patch).not.toHaveBeenCalled();
});
it("includes the moderator ban reason in the sign-in error", async () => {
it("does not leak the moderator ban reason in the sign-in error", async () => {
const { ctx } = makeCtx({
user: { deletedAt: 123, banReason: "Chargeback fraud" },
banRecords: [{ action: "user.ban" }],
@@ -136,6 +138,49 @@ describe("handleDeletedUserSignIn", () => {
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(`${BANNED_REAUTH_MESSAGE} Reason: Chargeback fraud`);
).rejects.toThrow(BANNED_REAUTH_MESSAGE);
});
});
describe("GitHub auth provider", () => {
it("does not link ClawHub accounts by GitHub profile email", () => {
const provider = createGitHubAuthProvider() as {
options?: { allowDangerousEmailAccountLinking?: boolean };
};
expect(provider.options?.allowDangerousEmailAccountLinking).toBe(false);
});
it("normalizes numeric GitHub profile ids", () => {
expect(normalizeGitHubProfileId(123456)).toBe("123456");
expect(normalizeGitHubProfileId("789012")).toBe("789012");
});
it("rejects missing or nonnumeric GitHub profile ids", () => {
expect(() => normalizeGitHubProfileId(undefined)).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
expect(() => normalizeGitHubProfileId("undefined")).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
expect(() => normalizeGitHubProfileId("github-user")).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
});
it("fails closed when the GitHub provider receives a malformed profile", () => {
const provider = createGitHubAuthProvider() as {
options?: { profile?: (profile: Record<string, unknown>) => Record<string, unknown> };
};
expect(() => provider.options?.profile?.({ message: "Bad credentials" })).toThrow(
"GitHub OAuth profile is missing a valid numeric id",
);
expect(provider.options?.profile?.({ id: 123456, login: "fixture-user" })).toEqual({
id: "123456",
name: "fixture-user",
email: undefined,
image: undefined,
});
});
});
+44 -21
View File
@@ -9,19 +9,48 @@ import { isLocalDevAuthEnabled } from "./lib/devAuth";
import { shouldScheduleGitHubProfileSync } from "./lib/githubProfileSync";
export const BANNED_REAUTH_MESSAGE =
"This account has been banned and cannot sign in. If you believe this is a mistake, please contact security@openclaw.ai and we will review it.";
"This account has been banned and cannot sign in. If you believe this is a mistake, appeal this decision: https://appeals.openclaw.ai/.";
export const DELETED_ACCOUNT_REAUTH_MESSAGE =
"This account has been permanently deleted and cannot be restored.";
const REAUTH_BLOCKING_BAN_ACTIONS = new Set(["user.ban", "user.autoban.malware"]);
const DEV_PERSONAS = new Set(["owner", "user", "admin"]);
const DEV_PERSONAS = new Set(["owner", "user", "admin", "officialOrgMember", "abusePublisher"]);
function getBannedReauthMessage(reason: string | undefined) {
const normalizedReason = reason?.trim();
if (!normalizedReason || normalizedReason.toLowerCase() === "malware auto-ban") {
return BANNED_REAUTH_MESSAGE;
export function normalizeGitHubProfileId(profileId: unknown) {
const id =
typeof profileId === "number" && Number.isSafeInteger(profileId)
? String(profileId)
: typeof profileId === "string"
? profileId.trim()
: null;
if (!id || !/^\d+$/.test(id)) {
throw new Error("GitHub OAuth profile is missing a valid numeric id");
}
return `${BANNED_REAUTH_MESSAGE} Reason: ${normalizedReason}`;
return id;
}
export function createGitHubAuthProvider() {
return GitHub({
clientId: process.env.AUTH_GITHUB_ID ?? "",
clientSecret: process.env.AUTH_GITHUB_SECRET ?? "",
// GitHub's OAuth email must not be treated as a ClawHub account key. The
// immutable GitHub provider account id is the only account-linking key.
allowDangerousEmailAccountLinking: false,
profile(profile) {
return {
id: normalizeGitHubProfileId(profile.id),
name: profile.login,
email: profile.email ?? undefined,
image: profile.avatar_url,
};
},
});
}
function getBannedReauthMessage(_reason: string | undefined) {
return BANNED_REAUTH_MESSAGE;
}
export async function handleDeletedUserSignIn(
@@ -75,26 +104,20 @@ export async function handleDeletedUserSignIn(
export const { auth, signIn, signOut, store, isAuthenticated } = convexAuth({
providers: [
GitHub({
clientId: process.env.AUTH_GITHUB_ID ?? "",
clientSecret: process.env.AUTH_GITHUB_SECRET ?? "",
profile(profile) {
return {
id: String(profile.id),
name: profile.login,
email: profile.email ?? undefined,
image: profile.avatar_url,
};
},
}),
createGitHubAuthProvider(),
ConvexCredentials({
id: "dev-persona",
authorize: async (credentials, ctx) => {
if (!isLocalDevAuthEnabled()) throw new Error("Dev auth is disabled");
const devAuthSecret =
typeof credentials.devAuthSecret === "string" ? credentials.devAuthSecret : undefined;
if (!isLocalDevAuthEnabled(process.env, devAuthSecret)) {
throw new Error("Dev auth is disabled");
}
const persona = typeof credentials.persona === "string" ? credentials.persona : "";
if (!DEV_PERSONAS.has(persona)) throw new Error("Unknown dev persona");
const userId: Id<"users"> = await ctx.runMutation(internal.users.upsertDevPersonaInternal, {
persona: persona as "owner" | "user" | "admin",
persona: persona as "owner" | "user" | "admin" | "officialOrgMember" | "abusePublisher",
devAuthSecret,
});
return { userId };
},
+263
View File
@@ -0,0 +1,263 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("./lib/access", () => ({
requireUser: vi.fn(),
}));
const { requireUser } = await import("./lib/access");
const { approve, deny } = await import("./cliDeviceAuth");
const approveHandler = (approve as unknown as { _handler: Function })._handler;
const denyHandler = (deny as unknown as { _handler: Function })._handler;
function makeCtx(rows: Array<Record<string, unknown>>) {
const take = vi.fn().mockResolvedValue(rows);
const order = vi.fn().mockReturnValue({ take });
const withIndex = vi.fn().mockReturnValue({ order });
const query = vi.fn().mockReturnValue({ withIndex });
const get = vi.fn().mockResolvedValue(null);
const insert = vi.fn().mockResolvedValue("inserted:id");
const patch = vi.fn().mockResolvedValue(undefined);
const replace = vi.fn().mockResolvedValue(undefined);
const delete_ = vi.fn().mockResolvedValue(undefined);
const normalizeId = vi.fn().mockReturnValue(null);
return {
ctx: { db: { get, insert, query, patch, replace, delete: delete_, normalizeId } },
order,
take,
withIndex,
query,
patch,
};
}
describe("cliDeviceAuth approval", () => {
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(new Date("2026-06-10T12:00:00Z"));
vi.mocked(requireUser).mockResolvedValue({
userId: "users:approver",
user: { _id: "users:approver" },
} as never);
});
afterEach(() => {
vi.useRealTimers();
vi.mocked(requireUser).mockReset();
});
it("approves the newest active pending row when duplicate user codes exist", async () => {
const now = Date.now();
const { ctx, order, patch, take } = makeCtx([
{
_id: "cliDeviceCodes:new",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
{
_id: "cliDeviceCodes:old",
_creationTime: now - 10_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 10_000,
expiresAt: now + 60_000,
},
]);
const result = await approveHandler(ctx, { userCode: "q639-nbsx" });
expect(result).toEqual({
ok: true,
userCode: "Q639-NBSX",
expiresAt: now + 60_000,
});
expect(order).toHaveBeenCalledWith("desc");
expect(take).toHaveBeenCalledWith(50);
expect(patch).toHaveBeenCalledTimes(1);
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:new", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
});
it("uses descending index order when duplicate creation timestamps disagree", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:new",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 20_000,
expiresAt: now + 60_000,
},
{
_id: "cliDeviceCodes:old",
_creationTime: now - 10_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 500,
expiresAt: now + 60_000,
},
]);
await approveHandler(ctx, { userCode: "Q639-NBSX" });
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:new", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
});
it("expires stale pending rows before approving an active row", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:expired",
_creationTime: now - 100_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 100_000,
expiresAt: now - 1,
},
{
_id: "cliDeviceCodes:active",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
]);
await approveHandler(ctx, { userCode: "Q639-NBSX" });
expect(patch).toHaveBeenNthCalledWith(1, "cliDeviceCodes:expired", { status: "expired" });
expect(patch).toHaveBeenNthCalledWith(2, "cliDeviceCodes:active", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
});
it("preserves stale terminal rows while expiring stale pending duplicates", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:approved",
_creationTime: now - 100_000,
status: "approved",
userCode: "Q639-NBSX",
createdAt: now - 100_000,
expiresAt: now - 1,
approvedAt: now - 90_000,
approvedByUserId: "users:previous",
},
{
_id: "cliDeviceCodes:denied",
_creationTime: now - 90_000,
status: "denied",
userCode: "Q639-NBSX",
createdAt: now - 90_000,
expiresAt: now - 1,
deniedAt: now - 80_000,
},
{
_id: "cliDeviceCodes:consumed",
_creationTime: now - 80_000,
status: "consumed",
userCode: "Q639-NBSX",
createdAt: now - 80_000,
expiresAt: now - 1,
consumedAt: now - 70_000,
},
{
_id: "cliDeviceCodes:expired",
_creationTime: now - 70_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 70_000,
expiresAt: now - 1,
},
{
_id: "cliDeviceCodes:active",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
]);
await approveHandler(ctx, { userCode: "Q639-NBSX" });
expect(patch).toHaveBeenNthCalledWith(1, "cliDeviceCodes:expired", { status: "expired" });
expect(patch).toHaveBeenNthCalledWith(2, "cliDeviceCodes:active", {
status: "approved",
approvedByUserId: "users:approver",
approvedAt: now,
});
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:approved", expect.anything());
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:denied", expect.anything());
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:consumed", expect.anything());
});
it("denies the newest active pending row when duplicate user codes exist", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:new",
_creationTime: now - 1_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 1_000,
expiresAt: now + 60_000,
},
{
_id: "cliDeviceCodes:old",
_creationTime: now - 10_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 10_000,
expiresAt: now + 60_000,
},
]);
await expect(denyHandler(ctx, { userCode: "Q639-NBSX" })).resolves.toEqual({ ok: true });
expect(patch).toHaveBeenCalledTimes(1);
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:new", {
status: "denied",
deniedAt: now,
});
});
it("does not deny stale pending rows after expiring them", async () => {
const now = Date.now();
const { ctx, patch } = makeCtx([
{
_id: "cliDeviceCodes:expired",
_creationTime: now - 100_000,
status: "pending",
userCode: "Q639-NBSX",
createdAt: now - 100_000,
expiresAt: now - 1,
},
]);
await expect(denyHandler(ctx, { userCode: "Q639-NBSX" })).resolves.toEqual({ ok: true });
expect(patch).toHaveBeenCalledTimes(1);
expect(patch).toHaveBeenCalledWith("cliDeviceCodes:expired", { status: "expired" });
expect(patch).not.toHaveBeenCalledWith("cliDeviceCodes:expired", {
status: "denied",
deniedAt: now,
});
});
});
+45 -15
View File
@@ -1,10 +1,13 @@
import { v } from "convex/values";
import type { Doc } from "./_generated/dataModel";
import type { MutationCtx } from "./_generated/server";
import { internalMutation, mutation } from "./functions";
import { requireUser } from "./lib/access";
import { generateToken, hashToken } from "./lib/tokens";
const DEVICE_CODE_TTL_MS = 15 * 60_000;
const DEVICE_POLL_INTERVAL_SECONDS = 5;
const MAX_DUPLICATE_USER_CODE_ROWS = 50;
const USER_CODE_ALPHABET = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789";
export const createInternal = internalMutation({
@@ -88,17 +91,13 @@ export const approve = mutation({
if (!normalized) throw new Error("Code required");
const userCodeHash = await hashToken(normalized);
const row = await ctx.db
.query("cliDeviceCodes")
.withIndex("by_user_code_hash", (q) => q.eq("userCodeHash", userCodeHash))
.unique();
if (!row) throw new Error("Device code not found");
const now = Date.now();
if (row.expiresAt <= now) {
if (row.status !== "expired") await ctx.db.patch(row._id, { status: "expired" });
throw new Error("Device code expired");
}
const rows = await expireStaleRows(ctx, await getRowsByUserCodeHash(ctx, userCodeHash), now);
const row =
pickLatestRow(rows, now, "pending") ?? pickLatestRow(rows, now) ?? pickLatestRow(rows);
if (!row) throw new Error("Device code not found");
if (row.expiresAt <= now) throw new Error("Device code expired");
if (row.status === "expired") throw new Error("Device code expired");
if (row.status === "consumed") throw new Error("Device code already used");
if (row.status === "approved") throw new Error("Device code already authorized");
if (row.status === "denied") throw new Error("Device code was denied");
@@ -119,12 +118,11 @@ export const deny = mutation({
const normalized = normalizeUserCode(args.userCode);
if (!normalized) throw new Error("Code required");
const userCodeHash = await hashToken(normalized);
const row = await ctx.db
.query("cliDeviceCodes")
.withIndex("by_user_code_hash", (q) => q.eq("userCodeHash", userCodeHash))
.unique();
if (!row) throw new Error("Device code not found");
const now = Date.now();
const rows = await expireStaleRows(ctx, await getRowsByUserCodeHash(ctx, userCodeHash), now);
const row =
pickLatestRow(rows, now, "pending") ?? pickLatestRow(rows, now) ?? pickLatestRow(rows);
if (!row) throw new Error("Device code not found");
if (row.status === "approved") throw new Error("Device code already authorized");
if (row.status === "pending") {
await ctx.db.patch(row._id, { status: "denied", deniedAt: now });
@@ -140,6 +138,38 @@ function normalizeUserCode(value: string) {
.replace(/[^A-Z0-9]/g, "");
}
async function getRowsByUserCodeHash(ctx: MutationCtx, userCodeHash: string) {
return await ctx.db
.query("cliDeviceCodes")
.withIndex("by_user_code_hash", (q) => q.eq("userCodeHash", userCodeHash))
.order("desc")
.take(MAX_DUPLICATE_USER_CODE_ROWS);
}
async function expireStaleRows(ctx: MutationCtx, rows: Array<Doc<"cliDeviceCodes">>, now: number) {
const nextRows: Array<Doc<"cliDeviceCodes">> = [];
for (const row of rows) {
if (row.status === "pending" && row.expiresAt <= now) {
await ctx.db.patch(row._id, { status: "expired" });
nextRows.push({ ...row, status: "expired" });
} else {
nextRows.push(row);
}
}
return nextRows;
}
function pickLatestRow(
rows: Array<Doc<"cliDeviceCodes">>,
now?: number,
status?: Doc<"cliDeviceCodes">["status"],
) {
return rows.find(
(row) =>
(now === undefined || row.expiresAt > now) && (status === undefined || row.status === status),
);
}
function generateUserCode() {
const bytes = new Uint8Array(8);
crypto.getRandomValues(bytes);
+58
View File
@@ -0,0 +1,58 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from "vitest";
const mocks = vi.hoisted(() => {
const interval = vi.fn();
const githubSkillSyncRef = Symbol("github-skill-source-sync");
return { interval, githubSkillSyncRef };
});
vi.mock("convex/server", () => ({
cronJobs: () => ({
interval: mocks.interval,
}),
}));
vi.mock("./_generated/api", () => ({
internal: {
githubBackupsNode: { syncGitHubBackupsInternal: Symbol("github-backup-sync") },
githubSkillSync: { syncGitHubSkillSourcesInternal: mocks.githubSkillSyncRef },
leaderboards: { rebuildTrendingLeaderboardAction: Symbol("trending-leaderboard") },
statsMaintenance: {
runSkillStatBackfillInternal: Symbol("skill-stats-backfill"),
updateGlobalStatsAction: Symbol("global-stats-update"),
},
skillStatEvents: { processSkillStatEventsAction: Symbol("skill-stat-events") },
packages: {
processPackageStatEventsInternal: Symbol("package-stat-events"),
backfillPackageReleaseScansInternal: Symbol("package-scan-backfill"),
},
publisherAbuse: {
runPublisherAbuseScoreRunInternal: Symbol("publisher-abuse-score-refresh"),
},
vt: {
pollPendingScans: Symbol("vt-pending-scans"),
backfillActiveSkillsVTCache: Symbol("vt-cache-backfill"),
},
securityScan: {
pruneExpiredSkillScanRequestsInternal: Symbol("skill-scan-request-prune"),
},
downloads: { pruneDownloadDedupesInternal: Symbol("download-dedupe-prune") },
downloadMetrics: {
pruneDownloadMetricDedupesInternal: Symbol("download-metric-dedupe-prune"),
},
},
}));
describe("crons", () => {
it("runs GitHub skill source sync every 15 minutes", async () => {
await import("./crons");
expect(mocks.interval).toHaveBeenCalledWith(
"github-skill-source-sync",
{ minutes: 15 },
mocks.githubSkillSyncRef,
{},
);
});
});
+28
View File
@@ -10,6 +10,13 @@ crons.interval(
{ batchSize: 50, maxBatches: 5 },
);
crons.interval(
"github-skill-source-sync",
{ minutes: 15 },
internal.githubSkillSync.syncGitHubSkillSourcesInternal,
{},
);
crons.interval(
"trending-leaderboard",
{ minutes: 60 },
@@ -64,6 +71,20 @@ crons.interval(
{ batchSize: 250, maxPages: 5, trigger: "cron" },
);
crons.interval(
"publisher-temporal-abuse-scan",
{ hours: 24 },
internal.publisherAbuse.runTemporalPublisherAbuseScanInternal,
{
mode: "current",
dryRun: false,
candidateLimit: 1000,
batchSize: 50,
maxPages: 20,
trigger: "cron",
},
);
crons.interval("vt-pending-scans", { minutes: 5 }, internal.vt.pollPendingScans, {
batchSize: 100,
});
@@ -93,4 +114,11 @@ crons.interval(
{},
);
crons.interval(
"download-metric-dedupe-prune",
{ hours: 24 },
internal.downloadMetrics.pruneDownloadMetricDedupesInternal,
{},
);
export default crons;
+10
View File
@@ -0,0 +1,10 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
const { checkDependencyRegistriesHandler } = await import("./depRegistryScan");
describe("dependency registry scan drain", () => {
it("keeps legacy scheduled jobs harmless after the scanner is retired", async () => {
await expect(checkDependencyRegistriesHandler()).resolves.toBeNull();
});
});
+4 -263
View File
@@ -1,270 +1,11 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc, Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { internalAction, internalMutation, internalQuery } from "./functions";
import {
dedupeDeps,
depRegistryUrl,
parseDependencyFile,
SUPPORTED_DEP_REGISTRIES,
summarizeDepRegistryChecks,
type DepEntry,
type DepRegistryResult,
type DepRegistryUnresolved,
type SupportedDepRegistry,
} from "./lib/depRegistryScan";
import { readStorageText } from "./lib/packageRegistry";
import { internalAction } from "./_generated/server";
const REQUEST_TIMEOUT_MS = 8_000;
const MAX_RETRIES = 2;
const BACKOFF_BASE_MS = 750;
const INTER_REQUEST_DELAY_MS = 100;
const MAX_DEPENDENCIES_PER_SCAN = 120;
const CACHE_TTL_EXISTS_MS = 30 * 24 * 60 * 60 * 1_000;
const CACHE_TTL_NOT_EXISTS_MS = 7 * 24 * 60 * 60 * 1_000;
const registryValidator = v.union(v.literal("pypi"), v.literal("npm"), v.literal("cargo"));
type RegistryCheck =
| { kind: "found"; httpStatus: number }
| { kind: "missing"; httpStatus: number }
| { kind: "unresolved"; reason: string };
function isSupportedRegistry(value: string): value is SupportedDepRegistry {
return (SUPPORTED_DEP_REGISTRIES as readonly string[]).includes(value);
}
async function wait(ms: number) {
await new Promise((resolve) => setTimeout(resolve, ms));
}
async function checkRegistry(dep: DepEntry): Promise<RegistryCheck> {
const headers: Record<string, string> = { Accept: "application/json" };
if (dep.registry === "cargo") {
headers["User-Agent"] = "ClawHub-DepRegistryScan/1.0 (https://clawhub.ai)";
}
let lastStatus: number | undefined;
for (let attempt = 0; attempt <= MAX_RETRIES; attempt += 1) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
const response = await fetch(depRegistryUrl(dep.registry, dep.name), {
method: "GET",
headers,
signal: controller.signal,
});
clearTimeout(timeout);
lastStatus = response.status;
if (response.status === 200) return { kind: "found", httpStatus: response.status };
if (response.status === 404) return { kind: "missing", httpStatus: response.status };
if (response.status !== 429 && response.status < 500) {
return {
kind: "unresolved",
reason: `unexpected HTTP ${response.status}`,
};
}
} catch (error) {
clearTimeout(timeout);
if (attempt === MAX_RETRIES) {
return {
kind: "unresolved",
reason: error instanceof Error ? error.message : "network error",
};
}
}
if (attempt < MAX_RETRIES) {
await wait(2 ** attempt * BACKOFF_BASE_MS);
}
}
return {
kind: "unresolved",
reason: lastStatus ? `HTTP ${lastStatus}` : "network error",
};
}
async function extractDependencies(ctx: Pick<ActionCtx, "storage">, version: Doc<"skillVersions">) {
const entries: DepEntry[] = [];
for (const file of version.files) {
const basename = file.path.split("/").pop()?.toLowerCase() ?? "";
if (
basename !== "requirements.txt" &&
basename !== "requirements-dev.txt" &&
basename !== "requirements_dev.txt" &&
basename !== "requirements-test.txt" &&
basename !== "requirements_test.txt" &&
basename !== "package.json" &&
basename !== "cargo.toml" &&
basename !== "pyproject.toml"
) {
continue;
}
const content = await readStorageText(ctx, file.storageId);
entries.push(...parseDependencyFile(file.path, content));
}
return dedupeDeps(entries);
}
export const lookupCacheInternal = internalQuery({
args: {
registry: registryValidator,
name: v.string(),
},
handler: async (ctx, args): Promise<Doc<"depRegistryCache"> | null> => {
return ctx.db
.query("depRegistryCache")
.withIndex("by_registry_name", (q) => q.eq("registry", args.registry).eq("name", args.name))
.unique();
},
});
export const upsertCacheInternal = internalMutation({
args: {
registry: registryValidator,
name: v.string(),
exists: v.boolean(),
httpStatus: v.number(),
checkedAt: v.number(),
},
handler: async (ctx, args) => {
const existing = await ctx.db
.query("depRegistryCache")
.withIndex("by_registry_name", (q) => q.eq("registry", args.registry).eq("name", args.name))
.unique();
const patch = {
registry: args.registry,
name: args.name,
exists: args.exists,
httpStatus: args.httpStatus,
checkedAt: args.checkedAt,
};
if (existing) {
await ctx.db.patch(existing._id, patch);
} else {
await ctx.db.insert("depRegistryCache", patch);
}
},
});
export const getRetryableVersionIdsInternal = internalQuery({
args: {
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const limit = Math.min(Math.max(args.limit ?? 25, 1), 100);
const versions = await ctx.db
.query("skillVersions")
.withIndex("by_dep_registry_scan_status_and_created", (q) =>
q.eq("depRegistryScanStatus", "error"),
)
.order("desc")
.take(limit);
return versions.map((version) => version._id);
},
});
async function checkWithCache(ctx: ActionCtx, dep: DepEntry) {
const now = Date.now();
const cached = (await ctx.runQuery(internal.depRegistryScan.lookupCacheInternal, {
registry: dep.registry,
name: dep.name,
})) as Doc<"depRegistryCache"> | null;
if (cached) {
const ttl = cached.exists ? CACHE_TTL_EXISTS_MS : CACHE_TTL_NOT_EXISTS_MS;
if (now - cached.checkedAt < ttl) {
return cached.exists
? ({ kind: "found", httpStatus: cached.httpStatus } as const)
: ({ kind: "missing", httpStatus: cached.httpStatus } as const);
}
}
const check = await checkRegistry(dep);
if (check.kind !== "unresolved") {
await ctx.runMutation(internal.depRegistryScan.upsertCacheInternal, {
registry: dep.registry,
name: dep.name,
exists: check.kind === "found",
httpStatus: check.httpStatus,
checkedAt: now,
});
}
return check;
export async function checkDependencyRegistriesHandler(): Promise<null> {
return null;
}
export const checkDependencyRegistries = internalAction({
args: { versionId: v.id("skillVersions") },
handler: async (ctx, args) => {
const version = (await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: args.versionId,
})) as Doc<"skillVersions"> | null;
if (!version) return null;
if (version.depRegistryAnalysis && version.depRegistryAnalysis.status !== "error") {
return version.depRegistryAnalysis;
}
const deps = await extractDependencies(ctx, version);
const checkableDeps = deps.slice(0, MAX_DEPENDENCIES_PER_SCAN);
const deferredDeps = deps.slice(MAX_DEPENDENCIES_PER_SCAN);
const results: DepRegistryResult[] = [];
const unresolved: DepRegistryUnresolved[] = deferredDeps.map((dep) => ({
...dep,
reason: "dependency scan limit reached",
}));
for (const dep of checkableDeps) {
if (!isSupportedRegistry(dep.registry)) continue;
const check = await checkWithCache(ctx, dep);
if (check.kind === "unresolved") {
unresolved.push({ ...dep, reason: check.reason });
} else {
results.push({
...dep,
exists: check.kind === "found",
httpStatus: check.httpStatus,
});
}
await wait(INTER_REQUEST_DELAY_MS);
}
const analysis = summarizeDepRegistryChecks({
results,
unresolved,
checkedAt: Date.now(),
});
await ctx.runMutation(internal.skills.updateVersionDepRegistryAnalysisInternal, {
versionId: args.versionId,
depRegistryAnalysis: analysis,
});
return analysis;
},
});
export const rescanErrorDepRegistryVersions = internalAction({
args: {
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const versionIds = (await ctx.runQuery(
internal.depRegistryScan.getRetryableVersionIdsInternal,
{ limit: args.batchSize ?? 25 },
)) as Id<"skillVersions">[];
let scheduled = 0;
for (const versionId of versionIds) {
await ctx.scheduler.runAfter(
scheduled * 2_000,
internal.depRegistryScan.checkDependencyRegistries,
{
versionId,
},
);
scheduled += 1;
}
return { scheduled };
},
handler: checkDependencyRegistriesHandler,
});
+258
View File
@@ -4,7 +4,10 @@ import {
currentUserSeedPackageName,
currentUserSeedSkillSlug,
seedFeaturedPluginPackagesMutation,
seedGitHubBackedSkillSourceMutation,
seedLocalFixtures,
seedLocalModerationFixturesHandler,
seedPublicCorpusBatchMutation,
seedSkillMutation,
} from "./devSeed";
@@ -18,6 +21,15 @@ const seedSkillMutationHandler = (
const seedFeaturedPluginPackagesHandler = (
seedFeaturedPluginPackagesMutation as unknown as WrappedHandler<Record<string, unknown>>
)._handler;
const seedGitHubBackedSkillSourceHandler = (
seedGitHubBackedSkillSourceMutation as unknown as WrappedHandler<Record<string, unknown>>
)._handler;
const seedLocalFixturesHandler = (
seedLocalFixtures as unknown as WrappedHandler<{ reset?: boolean }>
)._handler;
const seedPublicCorpusBatchHandler = (
seedPublicCorpusBatchMutation as unknown as WrappedHandler<Record<string, unknown>>
)._handler;
function chainEq(constraints: Record<string, unknown>) {
return {
@@ -146,6 +158,33 @@ function seedSkillArgs(storageId: string) {
}
describe("devSeed local fixtures", () => {
it("does not preconfigure GitHub-backed source fixtures in the local seed action", async () => {
const mutationCalls: Array<{ args: Record<string, unknown> }> = [];
let storageCounter = 0;
const ctx = {
storage: {
store: async () => `storage:${++storageCounter}`,
},
runMutation: async (_ref: unknown, args: Record<string, unknown>) => {
mutationCalls.push({ args });
return { ok: true, seeded: ["local-moderation-fixtures"], skipped: [] };
},
};
const result = await seedLocalFixturesHandler(ctx as never, { reset: true });
expect(mutationCalls).toHaveLength(1);
expect(mutationCalls[0]?.args).toMatchObject({
reset: true,
});
expect(result).toEqual(
expect.objectContaining({
ok: true,
results: [expect.objectContaining({ slug: "local-moderation-fixtures" })],
}),
);
});
it("seeds core skill fixtures for an explicit local user without creating @local", async () => {
const { db, tables } = createDb();
const userId = (await db.insert("users", {
@@ -190,6 +229,210 @@ describe("devSeed local fixtures", () => {
);
});
it("does not copy publisher ownership onto public corpus skill embeddings", async () => {
const { db, tables } = createDb();
await seedPublicCorpusBatchHandler(
createMutationCtx(db) as never,
{
rows: [
{
kind: "skill",
slug: "corpus-demo",
displayName: "Corpus Demo",
version: "0.1.0",
skillMd: "---\ndescription: Corpus demo\n---\n# Corpus demo",
storageId: "storage:corpus-demo",
embedding: [0, 1, 2],
dummyOwner: {
handle: "corpus-owner",
displayName: "Corpus Owner",
image: "https://example.invalid/avatar.png",
},
},
],
} as never,
);
expect(tables.skills?.[0]).toEqual(
expect.objectContaining({
slug: "corpus-demo",
ownerPublisherId: tables.publishers?.[0]?._id,
}),
);
expect(tables.skillEmbeddings?.[0]).not.toHaveProperty("ownerPublisherId");
});
it("seeds a GitHub-backed source and skills without creating mirrored versions", async () => {
const { db, tables } = createDb();
const userId = (await db.insert("users", {
handle: "nvidia-dev",
displayName: "NVIDIA Dev",
role: "user",
createdAt: 1,
updatedAt: 1,
})) as Id<"users">;
const result = await seedGitHubBackedSkillSourceHandler(
createMutationCtx(db) as never,
{
ownerUserId: userId,
repo: "NVIDIA/skills",
defaultBranch: "main",
displayManifestKind: "skills.sh",
displayManifestHash: "manifest-sha256",
displayManifestCommit: "0".repeat(40),
displayManifestFetchedAt: 123,
displayManifestStatus: "ok",
displayManifest: {
notGrouped: "bottom",
groupings: [
{
title: "Agentic AI",
description: "Agentic AI skills.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
],
},
skills: [
{
slug: "aiq-deploy",
displayName: "AIQ Deploy",
summary: "Deploy AgentIQ workflows.",
githubPath: "skills/aiq-deploy",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy",
githubScanStatus: "clean",
githubCurrentCheckedAt: 456,
},
{
slug: "nemoclaw-user-configure-security",
displayName: "NeMoClaw User Configure Security",
summary: "Configure NeMoClaw user security.",
githubPath: "skills/nemoclaw-user-configure-security",
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: "hash-nemoclaw",
githubScanStatus: "clean",
githubCurrentCheckedAt: 789,
githubRemovedAt: 900,
},
],
} as never,
);
expect(result).toMatchObject({
ok: true,
seeded: ["aiq-deploy", "nemoclaw-user-configure-security"],
skipped: [],
});
expect(tables.githubSkillSources).toHaveLength(1);
expect(tables.githubSkillSources?.[0]).toEqual(
expect.objectContaining({
repo: "NVIDIA/skills",
ownerPublisherId: tables.publishers?.[0]?._id,
defaultBranch: "main",
displayManifestKind: "skills.sh",
displayManifestHash: "manifest-sha256",
displayManifestCommit: "0".repeat(40),
displayManifestFetchedAt: 123,
displayManifestStatus: "ok",
displayManifest: {
notGrouped: "bottom",
groupings: [
{
title: "Agentic AI",
description: "Agentic AI skills.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
],
},
}),
);
expect(tables.skills).toHaveLength(2);
expect(tables.skills).toEqual(
expect.arrayContaining([
expect.objectContaining({
slug: "aiq-deploy",
installKind: "github",
githubSourceId: tables.githubSkillSources?.[0]?._id,
githubPath: "skills/aiq-deploy",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy",
githubScanStatus: "clean",
githubCurrentCheckedAt: 456,
latestVersionId: undefined,
latestVersionSummary: undefined,
tags: {},
stats: expect.objectContaining({ versions: 0 }),
}),
expect.objectContaining({
slug: "nemoclaw-user-configure-security",
installKind: "github",
githubSourceId: tables.githubSkillSources?.[0]?._id,
githubPath: "skills/nemoclaw-user-configure-security",
githubCurrentContentHash: "hash-nemoclaw",
githubRemovedAt: 900,
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
moderationVerdict: undefined,
isSuspicious: false,
latestVersionId: undefined,
tags: {},
}),
]),
);
expect(tables.skillVersions ?? []).toHaveLength(0);
});
it("keeps unscanned GitHub-backed skills hidden from public listings", async () => {
const { db, tables } = createDb();
await seedGitHubBackedSkillSourceHandler(
createMutationCtx(db) as never,
{
repo: "NVIDIA/skills",
displayManifestStatus: "ok",
skills: [
{
slug: "pending-github-skill",
displayName: "Pending GitHub Skill",
githubPath: "skills/pending-github-skill",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-pending",
githubScanStatus: "pending",
},
{
slug: "failed-scan-github-skill",
displayName: "Failed Scan GitHub Skill",
githubPath: "skills/failed-scan-github-skill",
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: "hash-failed-scan",
githubScanStatus: "failed",
},
],
} as never,
);
expect(tables.skills).toEqual(
expect.arrayContaining([
expect.objectContaining({
slug: "pending-github-skill",
moderationStatus: "hidden",
moderationReason: "pending.scan",
moderationVerdict: undefined,
isSuspicious: false,
}),
expect.objectContaining({
slug: "failed-scan-github-skill",
moderationStatus: "hidden",
moderationReason: "scanner.failed",
moderationVerdict: undefined,
isSuspicious: false,
}),
]),
);
});
it("seeds moderation and plugin fixtures for an explicit local user with scoped identifiers", async () => {
const { db, tables } = createDb();
const userId = (await db.insert("users", {
@@ -258,6 +501,21 @@ describe("devSeed local fixtures", () => {
scannedPluginName,
]);
expect(tables.packages?.every((pkg) => pkg.ownerUserId === userId)).toBe(true);
expect(tables.packageInspectorWarnings).toEqual(
expect.arrayContaining([
expect.objectContaining({
packageName: scannedPluginName,
findingKind: "warning",
code: "legacy-before-agent-start",
}),
expect.objectContaining({
packageName: scannedPluginName,
findingKind: "error",
code: "missing-expected-seam",
scanSource: "nightly",
}),
]),
);
});
it("retires legacy @local-owner seed publishers so dev-auth users can claim the handle", async () => {
+1034 -2
View File
File diff suppressed because it is too large Load Diff
+249
View File
@@ -0,0 +1,249 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from "vitest";
import {
__test,
pruneDownloadMetricDedupesInternal,
recordDownloadMetricInternal,
} from "./downloadMetrics";
type WrappedHandler<TArgs, TResult> = {
_handler: (ctx: unknown, args: TArgs) => Promise<TResult>;
};
const recordDownloadMetricHandler = (
recordDownloadMetricInternal as unknown as WrappedHandler<
{
target: { kind: "skill"; id: string } | { kind: "package"; id: string };
identityKind: "user" | "ip";
identityHash: string;
dayStart: number;
occurredAt?: number;
},
void
>
)._handler;
const pruneDownloadMetricDedupesHandler = (
pruneDownloadMetricDedupesInternal as unknown as WrappedHandler<
Record<string, never>,
{ deleted: number; hasMore: boolean }
>
)._handler;
function makeQueryBuilder() {
const builder = {
eq: vi.fn(() => builder),
lt: vi.fn(() => builder),
};
return builder;
}
type QueryBuilder = ReturnType<typeof makeQueryBuilder>;
function makeDb(
existingByTable: Record<string, unknown> = {},
rowsByTable: Record<string, Array<{ _id: string }>> = {},
) {
const indexCalls: Array<{ table: string; indexName: string; builder: QueryBuilder }> = [];
const insert = vi.fn();
const unique = vi.fn(async function uniqueForTable(this: { table: string }) {
return existingByTable[this.table] ?? null;
});
const take = vi.fn(async function takeForTable(this: { table: string }, limit: number) {
return (rowsByTable[this.table] ?? []).slice(0, limit);
});
const query = vi.fn((table: string) => ({
withIndex: vi.fn((indexName: string, buildQuery: (q: unknown) => unknown) => {
const builder = makeQueryBuilder();
buildQuery(builder);
indexCalls.push({ table, indexName, builder });
return {
unique: unique.bind({ table }),
take: take.bind({ table }),
};
}),
}));
const delete_ = vi.fn();
return {
db: {
query,
get: vi.fn(),
insert,
patch: vi.fn(),
replace: vi.fn(),
delete: delete_,
normalizeId: vi.fn(),
system: {
get: vi.fn(),
query: vi.fn(),
},
},
insert,
delete_,
take,
indexCalls,
};
}
describe("download metric helpers", () => {
afterEach(() => {
vi.useRealTimers();
});
it("uses a day bucket for download dedupe", () => {
expect(__test.getDayStart(86_400_000 - 1)).toBe(0);
expect(__test.getDayStart(86_400_000)).toBe(86_400_000);
});
it("prefers user identity and falls back to IP identity", () => {
const request = new Request("https://example.com", {
headers: { "cf-connecting-ip": "203.0.113.10" },
});
expect(__test.getDownloadIdentity(request, "users:one")).toEqual({
identityKind: "user",
identityValue: "users:one",
});
expect(__test.getDownloadIdentity(request, null)).toEqual({
identityKind: "ip",
identityValue: "203.0.113.10",
});
});
it("does not create a metering identity when user and IP are missing", () => {
expect(__test.getDownloadIdentity(new Request("https://example.com"), null)).toBeNull();
});
it("records one authenticated skill download and emits the existing skill stat event", async () => {
const { db, insert, indexCalls } = makeDb();
await recordDownloadMetricHandler(
{ db },
{
target: { kind: "skill", id: "skills:one" },
identityKind: "user",
identityHash: "hash-user",
dayStart: 86_400_000,
occurredAt: 86_500_000,
},
);
expect(indexCalls[0]?.table).toBe("downloadMetricDedupes");
expect(indexCalls[0]?.indexName).toBe("by_target_identity_day");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("targetKind", "skill");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("targetId", "skills:one");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("identityKind", "user");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("identityHash", "hash-user");
expect(indexCalls[0]?.builder.eq).toHaveBeenCalledWith("dayStart", 86_400_000);
expect(insert).toHaveBeenCalledWith(
"downloadMetricDedupes",
expect.objectContaining({
targetKind: "skill",
targetId: "skills:one",
identityKind: "user",
identityHash: "hash-user",
dayStart: 86_400_000,
}),
);
expect(insert).toHaveBeenCalledWith(
"skillStatEvents",
expect.objectContaining({
skillId: "skills:one",
kind: "download",
occurredAt: 86_500_000,
}),
);
expect(insert).not.toHaveBeenCalledWith("packageStatEvents", expect.anything());
});
it("records one anonymous package download and emits the existing package stat event", async () => {
const { db, insert } = makeDb();
await recordDownloadMetricHandler(
{ db },
{
target: { kind: "package", id: "packages:one" },
identityKind: "ip",
identityHash: "hash-ip",
dayStart: 86_400_000,
occurredAt: 86_500_000,
},
);
expect(insert).toHaveBeenCalledWith(
"downloadMetricDedupes",
expect.objectContaining({
targetKind: "package",
targetId: "packages:one",
identityKind: "ip",
identityHash: "hash-ip",
dayStart: 86_400_000,
}),
);
expect(insert).toHaveBeenCalledWith(
"packageStatEvents",
expect.objectContaining({
packageId: "packages:one",
kind: "download",
occurredAt: 86_500_000,
}),
);
expect(insert).not.toHaveBeenCalledWith("skillStatEvents", expect.anything());
});
it("ignores duplicate identities in the same target/day bucket", async () => {
const { db, insert } = makeDb({
downloadMetricDedupes: { _id: "downloadMetricDedupes:existing" },
});
await recordDownloadMetricHandler(
{ db },
{
target: { kind: "skill", id: "skills:one" },
identityKind: "ip",
identityHash: "hash-ip",
dayStart: 86_400_000,
},
);
expect(insert).not.toHaveBeenCalled();
});
it("prunes stale dedupe rows by day bucket", async () => {
vi.setSystemTime(30 * 86_400_000);
const { db, delete_, take, indexCalls } = makeDb(
{},
{
downloadMetricDedupes: [
{ _id: "downloadMetricDedupes:one" },
{ _id: "downloadMetricDedupes:two" },
],
},
);
const result = await pruneDownloadMetricDedupesHandler({ db }, {});
expect(result).toEqual({ deleted: 2, hasMore: false });
expect(indexCalls[0]?.table).toBe("downloadMetricDedupes");
expect(indexCalls[0]?.indexName).toBe("by_day");
expect(take).toHaveBeenCalledWith(200);
expect(delete_).toHaveBeenCalledWith("downloadMetricDedupes:one");
expect(delete_).toHaveBeenCalledWith("downloadMetricDedupes:two");
});
it("reschedules stale dedupe pruning when one bounded batch fills", async () => {
vi.setSystemTime(30 * 86_400_000);
const rows = Array.from({ length: 200 }, (_, index) => ({
_id: `downloadMetricDedupes:${index}`,
}));
const { db, delete_ } = makeDb({}, { downloadMetricDedupes: rows });
const runAfter = vi.fn();
const result = await pruneDownloadMetricDedupesHandler({ db, scheduler: { runAfter } }, {});
expect(result).toEqual({ deleted: 200, hasMore: true });
expect(delete_).toHaveBeenCalledTimes(200);
expect(runAfter).toHaveBeenCalledWith(0, expect.anything(), {});
});
});
+137
View File
@@ -0,0 +1,137 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import { internalMutation } from "./functions";
import { getClientIp } from "./lib/httpRateLimit";
import { hashToken } from "./lib/tokens";
import { insertStatEvent } from "./skillStatEvents";
const DAY_MS = 86_400_000;
const DEDUPE_RETENTION_MS = 14 * DAY_MS;
const PRUNE_BATCH_SIZE = 200;
const identityKindValidator = v.union(v.literal("user"), v.literal("ip"));
const targetValidator = v.union(
v.object({ kind: v.literal("skill"), id: v.id("skills") }),
v.object({ kind: v.literal("package"), id: v.id("packages") }),
);
type DownloadIdentityKind = "user" | "ip";
type DownloadIdentity = {
identityKind: DownloadIdentityKind;
identityValue: string;
};
export function getDownloadIdentity(
request: Request,
userId: string | null,
): DownloadIdentity | null {
if (userId) return { identityKind: "user", identityValue: userId };
const ip = getClientIp(request);
if (!ip) return null;
return { identityKind: "ip", identityValue: ip };
}
export async function buildDownloadMetricArgs(params: {
target: { kind: "skill"; id: Id<"skills"> } | { kind: "package"; id: Id<"packages"> };
identity: DownloadIdentity;
now: number;
}) {
return {
target: params.target,
identityKind: params.identity.identityKind,
identityHash: await hashToken(
`${params.identity.identityKind}:${params.identity.identityValue}`,
),
dayStart: getDayStart(params.now),
occurredAt: params.now,
};
}
export const recordDownloadMetricInternal = internalMutation({
args: {
target: targetValidator,
identityKind: identityKindValidator,
identityHash: v.string(),
dayStart: v.number(),
occurredAt: v.optional(v.number()),
},
handler: async (ctx, args) => {
const targetId = args.target.id;
const existing = await ctx.db
.query("downloadMetricDedupes")
.withIndex("by_target_identity_day", (q) =>
q
.eq("targetKind", args.target.kind)
.eq("targetId", targetId)
.eq("identityKind", args.identityKind)
.eq("identityHash", args.identityHash)
.eq("dayStart", args.dayStart),
)
.unique();
if (existing) return;
const now = Date.now();
await ctx.db.insert("downloadMetricDedupes", {
targetKind: args.target.kind,
targetId,
identityKind: args.identityKind,
identityHash: args.identityHash,
dayStart: args.dayStart,
createdAt: now,
});
if (args.target.kind === "skill") {
await insertStatEvent(ctx, {
skillId: args.target.id,
kind: "download",
occurredAt: args.occurredAt,
});
return;
}
await ctx.db.insert("packageStatEvents", {
packageId: args.target.id,
kind: "download",
occurredAt: args.occurredAt ?? now,
processedAt: undefined,
});
},
});
export const pruneDownloadMetricDedupesInternal = internalMutation({
args: {},
handler: async (ctx) => {
const cutoffDayStart = getDayStart(Date.now() - DEDUPE_RETENTION_MS);
const stale = await ctx.db
.query("downloadMetricDedupes")
.withIndex("by_day", (q) => q.lt("dayStart", cutoffDayStart))
.take(PRUNE_BATCH_SIZE);
for (const entry of stale) {
await ctx.db.delete(entry._id);
}
const hasMore = stale.length === PRUNE_BATCH_SIZE;
if (hasMore) {
await ctx.scheduler.runAfter(
0,
internal.downloadMetrics.pruneDownloadMetricDedupesInternal,
{},
);
}
return { deleted: stale.length, hasMore };
},
});
function getDayStart(timestamp: number) {
return Math.floor(timestamp / DAY_MS) * DAY_MS;
}
export const __test = {
getDayStart,
getDownloadIdentity,
};
+150 -14
View File
@@ -21,6 +21,19 @@ const okRate = () => ({
resetAt: Date.now() + 60_000,
});
function stubZipResponse() {
class MockResponse {
status: number;
headers: Headers;
constructor(_body?: BodyInit | null, init?: ResponseInit) {
this.status = init?.status ?? 200;
this.headers = new Headers(init?.headers);
}
}
vi.stubGlobal("Response", MockResponse as unknown as typeof Response);
}
describe("downloads helpers", () => {
afterEach(() => {
vi.unstubAllEnvs();
@@ -63,16 +76,7 @@ describe("downloads helpers", () => {
});
it("schedules zip download stats outside the response path", async () => {
class MockResponse {
status: number;
headers: Headers;
constructor(_body?: BodyInit | null, init?: ResponseInit) {
this.status = init?.status ?? 200;
this.headers = new Headers(init?.headers);
}
}
vi.stubGlobal("Response", MockResponse as unknown as typeof Response);
stubZipResponse();
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
@@ -127,9 +131,10 @@ describe("downloads helpers", () => {
if (!args || typeof args !== "object") return false;
const value = args as Record<string, unknown>;
return (
value.skillId === "skills:1" &&
typeof value.target === "object" &&
typeof value.identityHash === "string" &&
typeof value.hourStart === "number"
value.identityKind === "ip" &&
typeof value.dayStart === "number"
);
});
expect(recordCalls).toHaveLength(1);
@@ -137,9 +142,11 @@ describe("downloads helpers", () => {
expect(recordCalls[0]?.[0]).toBeGreaterThanOrEqual(0);
expect(recordCalls[0]?.[0]).toBeLessThan(60_000);
expect(recordCalls[0]?.[2]).toEqual({
skillId: "skills:1",
target: { kind: "skill", id: "skills:1" },
identityKind: "ip",
identityHash: expect.any(String),
hourStart: expect.any(Number),
dayStart: expect.any(Number),
occurredAt: expect.any(Number),
});
});
@@ -202,4 +209,133 @@ describe("downloads helpers", () => {
expect(await response.text()).toBe("Version not found");
expect(storageGet).not.toHaveBeenCalled();
});
it("uses API token user identity for zip download stats when present", async () => {
stubZipResponse();
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
if ("tokenHash" in args) {
return { _id: "apiTokens:1", revokedAt: undefined };
}
if ("tokenId" in args) {
return { _id: "users:token", deletedAt: undefined, deactivatedAt: undefined };
}
if ("slug" in args) {
return {
skill: {
_id: "skills:1",
ownerUserId: "users:1",
slug: "demo",
tags: {},
latestVersionId: "skillVersions:1",
},
moderationInfo: null,
};
}
if ("versionId" in args) {
return {
_id: "skillVersions:1",
skillId: "skills:1",
version: "1.0.0",
createdAt: 3,
files: [{ path: "SKILL.md", storageId: "_storage:1" }],
softDeletedAt: undefined,
};
}
return null;
});
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
return { tokenTouched: "tokenId" in args };
});
const runAfter = vi.fn();
const storageGet = vi.fn().mockResolvedValue(new Blob(["hello"], { type: "text/markdown" }));
const response = await downloadZipHandler(
{
runQuery,
runMutation,
scheduler: { runAfter },
storage: { get: storageGet },
} as unknown as ActionCtx,
new Request("https://example.com/api/v1/download?slug=demo", {
headers: {
authorization: "Bearer clh_test",
"cf-connecting-ip": "1.2.3.4",
},
}),
);
expect(response.status).toBe(200);
expect(runAfter).toHaveBeenCalledWith(
expect.any(Number),
expect.anything(),
expect.objectContaining({
target: { kind: "skill", id: "skills:1" },
identityKind: "user",
identityHash: expect.stringMatching(/^[a-f0-9]{64}$/),
}),
);
});
it("returns zip downloads when download metering is scheduled", async () => {
stubZipResponse();
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
if ("slug" in args) {
return {
skill: {
_id: "skills:1",
ownerUserId: "users:1",
slug: "demo",
tags: {},
latestVersionId: "skillVersions:1",
},
moderationInfo: null,
};
}
if ("versionId" in args) {
return {
_id: "skillVersions:1",
skillId: "skills:1",
version: "1.0.0",
createdAt: 3,
files: [{ path: "SKILL.md", storageId: "_storage:1" }],
softDeletedAt: undefined,
};
}
return null;
});
const runMutation = vi.fn(async (_mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
return { mutationRecorded: true };
});
const runAfter = vi.fn();
const storageGet = vi.fn().mockResolvedValue(new Blob(["hello"], { type: "text/markdown" }));
const response = await downloadZipHandler(
{
runQuery,
runMutation,
scheduler: { runAfter },
storage: { get: storageGet },
} as unknown as ActionCtx,
new Request("https://example.com/api/v1/download?slug=demo", {
headers: { "cf-connecting-ip": "1.2.3.4" },
}),
);
expect(response.status).toBe(200);
expect(runAfter).toHaveBeenCalledWith(
expect.any(Number),
expect.anything(),
expect.objectContaining({
target: { kind: "skill", id: "skills:1" },
identityKind: "ip",
identityHash: expect.stringMatching(/^[a-f0-9]{64}$/),
}),
);
});
});
+20 -9
View File
@@ -1,12 +1,14 @@
import { v } from "convex/values";
import { api, internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import { buildDownloadMetricArgs, getDownloadIdentity } from "./downloadMetrics";
import { httpAction, internalMutation } from "./functions";
import { getOptionalActiveAuthUserIdFromAction } from "./lib/access";
import { getOptionalApiTokenUserId } from "./lib/apiTokenAuth";
import { corsHeaders, mergeHeaders } from "./lib/httpHeaders";
import { applyRateLimit, getClientIp } from "./lib/httpRateLimit";
import { getPublicSkillFileAccessBlock, isSkillVersionForSkill } from "./lib/skillFileAccess";
import { buildDeterministicZip } from "./lib/skillZip";
import { hashToken } from "./lib/tokens";
import { insertStatEvent } from "./skillStatEvents";
const HOUR_MS = 3_600_000;
@@ -98,17 +100,17 @@ export async function downloadZipHandler(
const zipBlob = new Blob([zipArray], { type: "application/zip" });
try {
const userId = await getOptionalApiTokenUserId(ctx, request);
const identity = getDownloadIdentityValue(request, userId ? String(userId) : null);
const userId = await getOptionalDownloadUserId(ctx, request);
const identity = getDownloadIdentity(request, userId ? String(userId) : null);
if (identity) {
await ctx.scheduler.runAfter(
Math.floor(Math.random() * DOWNLOAD_STAT_JITTER_MS),
internal.downloads.recordDownloadInternal,
{
skillId: skill._id,
identityHash: await hashToken(identity),
hourStart: getHourStart(Date.now()),
},
internal.downloadMetrics.recordDownloadMetricInternal,
await buildDownloadMetricArgs({
target: { kind: "skill", id: skill._id },
identity,
now: Date.now(),
}),
);
}
} catch {
@@ -196,6 +198,15 @@ export function getDownloadIdentityValue(request: Request, userId: string | null
return `ip:${ip}`;
}
async function getOptionalDownloadUserId(
ctx: Parameters<Parameters<typeof httpAction>[0]>[0],
request: Request,
): Promise<Id<"users"> | null> {
const apiTokenUserId = await getOptionalApiTokenUserId(ctx, request);
if (apiTokenUserId) return apiTokenUserId;
return (await getOptionalActiveAuthUserIdFromAction(ctx)) ?? null;
}
export const __test = {
getHourStart,
getDownloadIdentityValue,
+160
View File
@@ -0,0 +1,160 @@
"use node";
import { mkdir, appendFile } from "node:fs/promises";
import { dirname } from "node:path";
import { v } from "convex/values";
import { Resend } from "resend";
import { internalAction } from "./functions";
import {
buildBanNotificationEmail,
buildMaliciousArtifactEmail,
buildRestoredAccountEmail,
type NotificationArtifact,
} from "./lib/emails";
const DEFAULT_FROM = "ClawHub Security <noreply@notifications.openclaw.ai>";
const DEFAULT_REPLY_TO = "security@notifications.openclaw.ai";
const notificationArtifactValidator = v.object({
kind: v.union(v.literal("skill"), v.literal("plugin")),
name: v.string(),
});
type SendEmailArgs = {
idempotencyKey: string;
to: string;
subject: string;
text: string;
html: string;
};
function getEmailConfig() {
return {
apiKey: process.env.RESEND_API_KEY,
from: process.env.CLAWHUB_SECURITY_EMAIL_FROM || DEFAULT_FROM,
replyTo: process.env.CLAWHUB_SECURITY_EMAIL || DEFAULT_REPLY_TO,
};
}
async function sendTransactionalEmail(args: SendEmailArgs) {
const captureFile = process.env.CLAWHUB_EMAIL_CAPTURE_FILE?.trim();
if (captureFile) {
await mkdir(dirname(captureFile), { recursive: true });
await appendFile(
captureFile,
`${JSON.stringify({ ...args, capturedAt: Date.now() })}\n`,
"utf8",
);
return { ok: true as const, id: "local-capture" };
}
const config = getEmailConfig();
if (!config.apiKey) {
console.warn(`[emails] RESEND_API_KEY is not configured; skipped ${args.idempotencyKey}`);
return { ok: false as const, reason: "missing_api_key" as const };
}
try {
const resend = new Resend(config.apiKey);
const result = await resend.emails.send(
{
from: config.from,
to: args.to,
replyTo: config.replyTo,
subject: args.subject,
text: args.text,
html: args.html,
},
{ idempotencyKey: args.idempotencyKey },
);
if (result.error) {
console.error("[emails] Resend error", result.error);
return { ok: false as const, reason: "resend_error" as const };
}
return { ok: true as const, id: result.data?.id ?? null };
} catch (error) {
console.error("[emails] Send failed", error);
return { ok: false as const, reason: "send_error" as const };
}
}
export const sendBanNotificationInternal = internalAction({
args: {
userId: v.id("users"),
bannedAt: v.number(),
to: v.string(),
handle: v.optional(v.string()),
source: v.union(v.literal("manual"), v.literal("autoban")),
reason: v.optional(v.string()),
trigger: v.optional(v.string()),
artifact: v.optional(notificationArtifactValidator),
},
handler: async (_ctx, args) => {
const email = buildBanNotificationEmail({
handle: args.handle,
source: args.source,
reason: args.reason,
trigger: args.trigger,
artifact: args.artifact as NotificationArtifact | undefined,
});
return await sendTransactionalEmail({
idempotencyKey: `ban:${args.userId}:${args.bannedAt}`,
to: args.to,
subject: email.subject,
text: email.text,
html: email.html,
});
},
});
export const sendRestoredAccountNotificationInternal = internalAction({
args: {
userId: v.id("users"),
restoredAt: v.number(),
to: v.string(),
handle: v.optional(v.string()),
restoredListings: v.optional(v.array(notificationArtifactValidator)),
},
handler: async (_ctx, args) => {
const email = buildRestoredAccountEmail({
handle: args.handle,
restoredListings: args.restoredListings as NotificationArtifact[] | undefined,
});
return await sendTransactionalEmail({
idempotencyKey: `account-restored:${args.userId}:${args.restoredAt}`,
to: args.to,
subject: email.subject,
text: email.text,
html: email.html,
});
},
});
export const sendMaliciousArtifactNotificationInternal = internalAction({
args: {
userId: v.id("users"),
findingAt: v.number(),
to: v.string(),
handle: v.optional(v.string()),
artifact: notificationArtifactValidator,
version: v.optional(v.string()),
trigger: v.optional(v.string()),
findingSummary: v.optional(v.string()),
},
handler: async (_ctx, args) => {
const email = buildMaliciousArtifactEmail({
handle: args.handle,
artifact: args.artifact as NotificationArtifact,
version: args.version,
trigger: args.trigger,
findingSummary: args.findingSummary,
});
return await sendTransactionalEmail({
idempotencyKey: `malicious-artifact:${args.userId}:${args.findingAt}:${args.artifact.kind}:${args.artifact.name}:${args.version ?? ""}`,
to: args.to,
subject: email.subject,
text: email.text,
html: email.html,
});
},
});
+101
View File
@@ -9,6 +9,8 @@ import {
repointPackageLatestRelease,
scheduleGitHubBackupDeletionForSkill,
scheduleOwnerPublisherDigestSync,
shouldScheduleOwnerPublisherDigestSyncForPublisherChange,
shouldScheduleOwnerUserPackageDigestSyncForUserChange,
syncPackageSearchDigestForPackageId,
syncPackageSearchDigestsForOwnerPublisherId,
syncPackageSearchDigestsForOwnerUserId,
@@ -53,6 +55,20 @@ describe("package digest sync", () => {
moderationStatus: "active",
}),
).toBe(true);
expect(
isGitHubMirrorEligibleSkillDoc({
softDeletedAt: undefined,
moderationStatus: "active",
moderationVerdict: "malicious",
}),
).toBe(false);
expect(
isGitHubMirrorEligibleSkillDoc({
softDeletedAt: undefined,
moderationStatus: "active",
moderationFlags: ["blocked.malware"],
}),
).toBe(false);
expect(
isGitHubMirrorEligibleSkillDoc({
softDeletedAt: undefined,
@@ -614,7 +630,92 @@ describe("package digest sync", () => {
});
});
describe("user package digest scheduling", () => {
const user = {
_id: "users:owner",
handle: "owner",
deletedAt: undefined,
deactivatedAt: undefined,
};
it("schedules package digest sync when an active user's handle changes", () => {
expect(
shouldScheduleOwnerUserPackageDigestSyncForUserChange({
id: "users:owner",
operation: "update",
oldDoc: user,
newDoc: { ...user, handle: "renamed" },
} as never),
).toBe(true);
});
it("skips redundant package digest sync when a user becomes deactivated", () => {
expect(
shouldScheduleOwnerUserPackageDigestSyncForUserChange({
id: "users:owner",
operation: "update",
oldDoc: user,
newDoc: { ...user, handle: null, deactivatedAt: 1_700_000_000_000 },
} as never),
).toBe(false);
});
it("skips unchanged user updates", () => {
expect(
shouldScheduleOwnerUserPackageDigestSyncForUserChange({
id: "users:owner",
operation: "update",
oldDoc: user,
newDoc: { ...user },
} as never),
).toBe(false);
});
});
describe("publisher digest scheduling", () => {
const publisherChangeDoc = {
_id: "publishers:demo",
kind: "org",
handle: "demo",
displayName: "Demo",
image: null,
deletedAt: undefined,
deactivatedAt: undefined,
};
it("schedules digest sync when an active publisher's profile changes", () => {
expect(
shouldScheduleOwnerPublisherDigestSyncForPublisherChange({
id: "publishers:demo",
operation: "update",
oldDoc: publisherChangeDoc,
newDoc: { ...publisherChangeDoc, displayName: "Renamed Demo" },
} as never),
).toBe(true);
});
it("skips redundant digest sync when a publisher becomes deactivated", () => {
expect(
shouldScheduleOwnerPublisherDigestSyncForPublisherChange({
id: "publishers:demo",
operation: "update",
oldDoc: publisherChangeDoc,
newDoc: { ...publisherChangeDoc, handle: null, deactivatedAt: 1_700_000_000_000 },
} as never),
).toBe(false);
});
it("skips unchanged publisher updates", () => {
expect(
shouldScheduleOwnerPublisherDigestSyncForPublisherChange({
id: "publishers:demo",
operation: "update",
oldDoc: publisherChangeDoc,
newDoc: { ...publisherChangeDoc },
} as never),
).toBe(false);
});
it("schedules package and skill digest sync in separate background mutations", async () => {
const ctx = {
scheduler: {
+102 -41
View File
@@ -1,5 +1,5 @@
import { customCtx, customMutation } from "convex-helpers/server/customFunctions";
import { Triggers } from "convex-helpers/server/triggers";
import { Triggers, type Change } from "convex-helpers/server/triggers";
import { v } from "convex/values";
import semver from "semver";
import { internal } from "./_generated/api";
@@ -14,6 +14,7 @@ import {
httpAction,
} from "./_generated/server";
import type { MutationCtx } from "./_generated/server";
import { isPublicSkillDoc } from "./lib/globalStats";
import {
deletePackageSearchDigests,
extractPackageDigestFields,
@@ -152,22 +153,28 @@ export async function syncPackageSearchDigestForPackageId(
}
export async function syncPackageSearchDigestsForOwnerUserId(
ctx: PackageDigestSyncCtx,
ctx: PackageDigestSyncCtx & OwnerPublisherDigestScheduleCtx,
ownerUserId: Id<"users"> | null | undefined,
cursor: string | null = null,
) {
if (!ownerUserId) return;
let cursor: string | null = null;
try {
while (true) {
const page = await ctx.db
.query("packages")
.withIndex("by_owner", (q) => q.eq("ownerUserId", ownerUserId))
.paginate({ cursor, numItems: 100 });
for (const pkg of page.page) {
await syncPackageSearchDigest(ctx, pkg);
}
if (page.isDone) break;
cursor = page.continueCursor;
const page = await ctx.db
.query("packages")
.withIndex("by_owner", (q) => q.eq("ownerUserId", ownerUserId))
.paginate({ cursor, numItems: OWNER_PUBLISHER_DIGEST_PAGE_SIZE });
for (const pkg of page.page) {
await syncPackageSearchDigest(ctx, pkg);
}
if (!page.isDone && ctx.scheduler && page.continueCursor) {
await ctx.scheduler.runAfter(
0,
internal.functions.syncPackageSearchDigestsForOwnerUserIdInternal,
{
ownerUserId,
cursor: page.continueCursor,
},
);
}
} catch (error) {
if (isMissingTableError(error, "packages")) return;
@@ -223,21 +230,28 @@ async function syncSkillSearchDigestForSkill(
}
export function isGitHubMirrorEligibleSkillDoc(
skill: Pick<Doc<"skills">, "softDeletedAt" | "moderationStatus"> | null | undefined,
skill:
| Pick<
Doc<"skills">,
"softDeletedAt" | "moderationStatus" | "moderationFlags" | "moderationVerdict"
>
| null
| undefined,
) {
if (!skill || skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
return isPublicSkillDoc(skill);
}
export async function scheduleGitHubBackupDeletionForSkill(
ctx: GitHubBackupDeletionCtx,
skill: Pick<
Doc<"skills">,
"slug" | "ownerPublisherId" | "ownerUserId" | "softDeletedAt" | "moderationStatus"
| "slug"
| "ownerPublisherId"
| "ownerUserId"
| "softDeletedAt"
| "moderationStatus"
| "moderationFlags"
| "moderationVerdict"
>,
) {
const owner = await getOwnerPublisher(ctx, {
@@ -295,6 +309,69 @@ export async function scheduleOwnerPublisherDigestSync(
);
}
export async function scheduleOwnerUserPackageDigestSync(
ctx: OwnerPublisherDigestScheduleCtx,
ownerUserId: Id<"users"> | null | undefined,
) {
if (!ownerUserId || !ctx.scheduler) return;
await ctx.scheduler.runAfter(
0,
internal.functions.syncPackageSearchDigestsForOwnerUserIdInternal,
{
ownerUserId,
},
);
}
export function shouldScheduleOwnerUserPackageDigestSyncForUserChange(
change: Change<DataModel, "users">,
) {
if (change.operation === "delete") return true;
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return false;
}
if (change.operation === "update" && (change.newDoc.deletedAt || change.newDoc.deactivatedAt)) {
return false;
}
return true;
}
export function shouldScheduleOwnerPublisherDigestSyncForPublisherChange(
change: Change<DataModel, "publishers">,
) {
if (change.operation === "delete") return true;
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.kind === change.newDoc.kind &&
change.oldDoc.displayName === change.newDoc.displayName &&
change.oldDoc.image === change.newDoc.image &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return false;
}
if (change.operation === "update" && (change.newDoc.deletedAt || change.newDoc.deactivatedAt)) {
return false;
}
return true;
}
export const syncPackageSearchDigestsForOwnerUserIdInternal = rawInternalMutation({
args: {
ownerUserId: v.id("users"),
cursor: v.optional(v.union(v.string(), v.null())),
},
handler: async (ctx, args) => {
await syncPackageSearchDigestsForOwnerUserId(ctx, args.ownerUserId, args.cursor ?? null);
},
});
export const syncPackageSearchDigestsForOwnerPublisherIdInternal = rawInternalMutation({
args: {
ownerPublisherId: v.id("publishers"),
@@ -331,6 +408,7 @@ export async function repointPackageLatestRelease(
if (!packageId || !affectedReleaseId) return;
const pkg = await ctx.db.get(packageId);
if (!pkg) return;
if (pkg.softDeletedAt) return;
const nextTags = Object.fromEntries(
Object.entries(pkg.tags).filter(([, releaseId]) => releaseId !== affectedReleaseId),
@@ -431,30 +509,13 @@ triggers.register("packageReleases", async (ctx, change) => {
});
triggers.register("users", async (ctx, change) => {
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return;
}
if (!shouldScheduleOwnerUserPackageDigestSyncForUserChange(change)) return;
const ownerUserId = change.operation === "delete" ? change.id : change.newDoc._id;
await syncPackageSearchDigestsForOwnerUserId(ctx, ownerUserId);
await scheduleOwnerUserPackageDigestSync(ctx, ownerUserId);
});
triggers.register("publishers", async (ctx, change) => {
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.kind === change.newDoc.kind &&
change.oldDoc.displayName === change.newDoc.displayName &&
change.oldDoc.image === change.newDoc.image &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return;
}
if (!shouldScheduleOwnerPublisherDigestSyncForPublisherChange(change)) return;
const ownerPublisherId = change.operation === "delete" ? change.id : change.newDoc._id;
await scheduleOwnerPublisherDigestSync(ctx, ownerPublisherId);
});
+9 -11
View File
@@ -1,8 +1,9 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { Doc, Id } from "./_generated/dataModel";
import { action, internalMutation, internalQuery } from "./functions";
import { assertRole, requireUserFromAction } from "./lib/access";
import { isPublicSkillDoc } from "./lib/globalStats";
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
@@ -104,16 +105,13 @@ export const getGitHubBackupPageInternal = internalQuery({
},
});
function isPubliclyAvailableSkill(skill: {
softDeletedAt?: number;
moderationStatus?: string | null;
}) {
if (skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
function isPubliclyAvailableSkill(
skill: Pick<
Doc<"skillSearchDigest">,
"softDeletedAt" | "moderationStatus" | "moderationFlags" | "moderationVerdict"
>,
) {
return isPublicSkillDoc(skill);
}
function isStaleCursorError(error: unknown) {
+2 -6
View File
@@ -14,6 +14,7 @@ import {
listGitHubSkillBackupEntries,
normalizeOwner,
} from "./lib/githubBackup";
import { isPublicSkillDoc } from "./lib/globalStats";
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
@@ -273,12 +274,7 @@ async function pruneDeletedSkillBackups(
}
function isMirrorEligibleSkill(skill: Doc<"skills"> | null): skill is Doc<"skills"> {
if (!skill || skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
return isPublicSkillDoc(skill);
}
async function deleteBackupIfNeeded(
+241
View File
@@ -0,0 +1,241 @@
import { ConvexError } from "convex/values";
import { beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("./lib/access", () => ({
requireUser: vi.fn(),
}));
vi.mock("./lib/publishers", async () => {
const actual = await vi.importActual<typeof import("./lib/publishers")>("./lib/publishers");
return {
...actual,
requirePublisherRole: vi.fn(),
};
});
const { requireUser } = await import("./lib/access");
const { requirePublisherRole } = await import("./lib/publishers");
const { deleteForPublisherHandler } = await import("./githubSkillSources");
const { buildSkillInstallResolution } = await import("./lib/installResolver");
type Row = Record<string, unknown> & { _id: string };
function chainEq(constraints: Record<string, unknown>) {
return {
eq(field: string, value: unknown) {
constraints[field] = value;
return chainEq(constraints);
},
};
}
function matches(doc: Row, constraints: Record<string, unknown>) {
return Object.entries(constraints).every(([key, value]) => doc[key] === value);
}
function createDb(initial: Record<string, Row[]> = {}) {
const tables: Record<string, Row[]> = Object.fromEntries(
Object.entries(initial).map(([table, rows]) => [table, [...rows]]),
);
const list = (table: string) => {
tables[table] ??= [];
return tables[table];
};
const db = {
get: async (id: string) => {
const table = id.split(":")[0] ?? "";
return list(table).find((row) => row._id === id) ?? null;
},
patch: async (id: string, patch: Record<string, unknown>) => {
const table = id.split(":")[0] ?? "";
const row = list(table).find((candidate) => candidate._id === id);
if (!row) return;
for (const [key, value] of Object.entries(patch)) {
if (value === undefined) delete row[key];
else row[key] = value;
}
},
insert: async (table: string, doc: Record<string, unknown>) => {
const id = `${table}:${list(table).length + 1}`;
list(table).push({ _id: id, ...doc });
return id;
},
delete: async (id: string) => {
const table = id.split(":")[0] ?? "";
const rows = list(table);
const index = rows.findIndex((row) => row._id === id);
if (index >= 0) rows.splice(index, 1);
},
query: (table: string) => ({
withIndex: (_indexName: string, build: (q: ReturnType<typeof chainEq>) => unknown) => {
const constraints: Record<string, unknown> = {};
build(chainEq(constraints));
const matched = () => list(table).filter((row) => matches(row, constraints));
return {
collect: async () => matched(),
unique: async () => matched()[0] ?? null,
};
},
}),
};
return { db, tables };
}
describe("githubSkillSources.deleteForPublisherHandler", () => {
beforeEach(() => {
vi.mocked(requireUser).mockResolvedValue({ userId: "users:owner" } as never);
vi.mocked(requirePublisherRole).mockResolvedValue(undefined as never);
});
it("deletes a source and removes only GitHub-backed skills from that source", async () => {
const { db, tables } = createDb({
githubSkillSources: [
{
_id: "githubSkillSources:matt",
repo: "mattpocock/skills",
ownerPublisherId: "publishers:openclaw",
createdAt: 1,
updatedAt: 2,
},
],
githubSkillContents: [
{
_id: "githubSkillContents:one",
skillId: "skills:github",
githubSourceId: "githubSkillSources:matt",
},
],
skills: [
{
_id: "skills:github",
slug: "source-backed",
displayName: "Source Backed",
installKind: "github",
githubSourceId: "githubSkillSources:matt",
githubPath: "skills/source-backed",
githubCurrentCommit: "a".repeat(40),
githubCurrentContentHash: "hash-source-backed",
githubCurrentStatus: "present",
githubScanStatus: "clean",
ownerUserId: "users:owner",
ownerPublisherId: "publishers:openclaw",
forkOf: undefined,
tags: {},
capabilityTags: undefined,
badges: {},
stats: {
comments: 0,
downloads: 0,
installsAllTime: 0,
installsCurrent: 0,
stars: 0,
versions: 0,
},
moderationStatus: "active",
moderationFlags: [],
isSuspicious: false,
createdAt: 1,
updatedAt: 2,
softDeletedAt: undefined,
},
{
_id: "skills:direct",
slug: "direct-upload",
displayName: "Direct Upload",
ownerPublisherId: "publishers:openclaw",
softDeletedAt: undefined,
},
{
_id: "skills:other-source",
slug: "other-source",
displayName: "Other Source",
installKind: "github",
githubSourceId: "githubSkillSources:other",
githubPath: "skills/other-source",
githubCurrentCommit: "b".repeat(40),
githubCurrentContentHash: "hash-other-source",
githubCurrentStatus: "present",
githubScanStatus: "clean",
ownerPublisherId: "publishers:openclaw",
softDeletedAt: undefined,
},
],
});
await expect(
deleteForPublisherHandler({ db } as never, {
ownerPublisherId: "publishers:openclaw" as never,
sourceId: "githubSkillSources:matt" as never,
now: 123,
}),
).resolves.toEqual({ ok: true, deletedSkills: 1 });
expect(requirePublisherRole).toHaveBeenCalledWith(
expect.anything(),
expect.objectContaining({
publisherId: "publishers:openclaw",
userId: "users:owner",
allowed: ["admin"],
}),
);
expect(tables.githubSkillSources).toHaveLength(0);
expect(tables.githubSkillContents).toHaveLength(0);
const deletedSkill = tables.skills.find((skill) => skill._id === "skills:github");
expect(deletedSkill).toMatchObject({
softDeletedAt: 123,
githubRemovedAt: 123,
githubCurrentStatus: "missing",
updatedAt: 123,
});
expect(tables.skillSearchDigest).toEqual([
expect.objectContaining({
skillId: "skills:github",
githubCurrentStatus: "missing",
githubScanStatus: "clean",
softDeletedAt: 123,
}),
]);
expect(
buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: deletedSkill as never,
source: null,
}),
).toMatchObject({
ok: false,
reason: "github_upstream_removed",
status: 410,
});
expect(tables.skills.find((skill) => skill._id === "skills:direct")).toMatchObject({
softDeletedAt: undefined,
});
expect(tables.skills.find((skill) => skill._id === "skills:other-source")).toMatchObject({
githubCurrentStatus: "present",
softDeletedAt: undefined,
});
});
it("rejects deleting a source from another publisher", async () => {
const { db } = createDb({
githubSkillSources: [
{
_id: "githubSkillSources:matt",
repo: "mattpocock/skills",
ownerPublisherId: "publishers:other",
createdAt: 1,
updatedAt: 2,
},
],
});
await expect(
deleteForPublisherHandler({ db } as never, {
ownerPublisherId: "publishers:openclaw" as never,
sourceId: "githubSkillSources:matt" as never,
now: 123,
}),
).rejects.toBeInstanceOf(ConvexError);
});
});
+206
View File
@@ -0,0 +1,206 @@
import { ConvexError, v } from "convex/values";
import type { Doc, Id } from "./_generated/dataModel";
import type { MutationCtx, QueryCtx } from "./_generated/server";
import { internalQuery, mutation, query } from "./functions";
import { requireUser } from "./lib/access";
import { adjustGlobalPublicSkillsCount, getPublicSkillVisibilityDelta } from "./lib/globalStats";
import { isOfficialPublisher } from "./lib/officialPublishers";
import { isPublisherActive, isPublisherRoleAllowed, requirePublisherRole } from "./lib/publishers";
import { syncSkillSearchDigestForSkill } from "./lib/skillSearchDigest";
type PublicGitHubSkillSource = Pick<
Doc<"githubSkillSources">,
| "_id"
| "repo"
| "defaultBranch"
| "lastSyncStatus"
| "lastSyncError"
| "lastSyncErrorAt"
| "displayManifestStatus"
| "displayManifestFetchedAt"
| "displayManifestCommit"
| "lastSyncIssues"
| "lastSyncInvalidSkills"
| "createdAt"
| "updatedAt"
> & {
ownerPublisher: Pick<Doc<"publishers">, "_id" | "handle" | "displayName"> | null;
skills: Array<
Pick<Doc<"skills">, "_id" | "slug" | "displayName" | "githubPath" | "githubCurrentStatus">
>;
};
export const getByIdInternal = internalQuery({
args: { sourceId: v.id("githubSkillSources") },
handler: async (ctx, args) => ctx.db.get(args.sourceId),
});
async function toPublicGitHubSkillSource(
ctx: Pick<QueryCtx, "db">,
source: Doc<"githubSkillSources">,
): Promise<PublicGitHubSkillSource> {
const skills = await ctx.db
.query("skills")
.withIndex("by_github_source", (q) => q.eq("githubSourceId", source._id))
.collect();
const visibleGitHubSkills = skills
.filter((skill) => skill.installKind === "github" && !skill.softDeletedAt)
.sort((a, b) => a.displayName.localeCompare(b.displayName))
.map((skill) => ({
_id: skill._id,
slug: skill.slug,
displayName: skill.displayName,
githubPath: skill.githubPath,
githubCurrentStatus: skill.githubCurrentStatus,
}));
const ownerPublisher = source.ownerPublisherId ? await ctx.db.get(source.ownerPublisherId) : null;
return {
_id: source._id as Id<"githubSkillSources">,
repo: source.repo,
ownerPublisher: ownerPublisher
? {
_id: ownerPublisher._id,
handle: ownerPublisher.handle,
displayName: ownerPublisher.displayName,
}
: null,
defaultBranch: source.defaultBranch,
lastSyncStatus: source.lastSyncStatus,
lastSyncError: source.lastSyncError,
lastSyncErrorAt: source.lastSyncErrorAt,
displayManifestStatus: source.displayManifestStatus,
displayManifestFetchedAt: source.displayManifestFetchedAt,
displayManifestCommit: source.displayManifestCommit,
lastSyncIssues: source.lastSyncIssues,
lastSyncInvalidSkills: source.lastSyncInvalidSkills,
createdAt: source.createdAt,
updatedAt: source.updatedAt,
skills: visibleGitHubSkills,
};
}
export const listForPublisher = query({
args: { ownerPublisherId: v.id("publishers") },
handler: async (ctx, args): Promise<PublicGitHubSkillSource[]> => {
const { userId } = await requireUser(ctx);
await requirePublisherRole(ctx, {
publisherId: args.ownerPublisherId,
userId,
allowed: ["admin"],
});
const sources = await ctx.db
.query("githubSkillSources")
.withIndex("by_owner_publisher", (q) => q.eq("ownerPublisherId", args.ownerPublisherId))
.collect();
const sortedSources = sources.sort((a, b) => b.updatedAt - a.updatedAt);
return await Promise.all(sortedSources.map((source) => toPublicGitHubSkillSource(ctx, source)));
},
});
export const listForManageableOfficialPublishers = query({
args: {},
handler: async (ctx): Promise<PublicGitHubSkillSource[]> => {
const { userId } = await requireUser(ctx);
const memberships = await ctx.db
.query("publisherMembers")
.withIndex("by_user", (q) => q.eq("userId", userId))
.collect();
const ownerPublisherIds: Id<"publishers">[] = [];
for (const membership of memberships) {
if (!isPublisherRoleAllowed(membership.role, ["admin"])) continue;
const publisher = await ctx.db.get(membership.publisherId);
if (
!publisher ||
publisher.kind !== "org" ||
!isPublisherActive(publisher) ||
!(await isOfficialPublisher(ctx, publisher))
) {
continue;
}
ownerPublisherIds.push(publisher._id);
}
const sourceGroups = await Promise.all(
ownerPublisherIds.map((ownerPublisherId) =>
ctx.db
.query("githubSkillSources")
.withIndex("by_owner_publisher", (q) => q.eq("ownerPublisherId", ownerPublisherId))
.collect(),
),
);
const sortedSources = sourceGroups.flat().sort((a, b) => b.updatedAt - a.updatedAt);
return await Promise.all(sortedSources.map((source) => toPublicGitHubSkillSource(ctx, source)));
},
});
export async function deleteForPublisherHandler(
ctx: MutationCtx,
args: {
ownerPublisherId: Id<"publishers">;
sourceId: Id<"githubSkillSources">;
now?: number;
},
) {
const { userId } = await requireUser(ctx);
await requirePublisherRole(ctx, {
publisherId: args.ownerPublisherId,
userId,
allowed: ["admin"],
});
const source = await ctx.db.get(args.sourceId);
if (!source || source.ownerPublisherId !== args.ownerPublisherId) {
throw new ConvexError("GitHub source not found.");
}
const now = args.now ?? Date.now();
const contents = await ctx.db
.query("githubSkillContents")
.withIndex("by_github_source", (q) => q.eq("githubSourceId", args.sourceId))
.collect();
for (const content of contents) {
await ctx.db.delete(content._id);
}
const skills = await ctx.db
.query("skills")
.withIndex("by_github_source", (q) => q.eq("githubSourceId", args.sourceId))
.collect();
let deletedSkills = 0;
let publicSkillDelta = 0;
for (const skill of skills) {
if (skill.installKind !== "github") continue;
const nextSkill: Doc<"skills"> = {
...skill,
softDeletedAt: skill.softDeletedAt ?? now,
githubCurrentStatus: "missing",
githubRemovedAt: skill.githubRemovedAt ?? now,
updatedAt: now,
};
publicSkillDelta += getPublicSkillVisibilityDelta(skill, nextSkill);
await ctx.db.patch(skill._id, {
softDeletedAt: nextSkill.softDeletedAt,
githubCurrentStatus: nextSkill.githubCurrentStatus,
githubRemovedAt: nextSkill.githubRemovedAt,
updatedAt: now,
});
await syncSkillSearchDigestForSkill(ctx, nextSkill);
deletedSkills += 1;
}
if (publicSkillDelta !== 0) {
await adjustGlobalPublicSkillsCount(ctx, publicSkillDelta, now);
}
await ctx.db.delete(args.sourceId);
return { ok: true as const, deletedSkills };
}
export const deleteForPublisher: ReturnType<typeof mutation> = mutation({
args: {
ownerPublisherId: v.id("publishers"),
sourceId: v.id("githubSkillSources"),
},
handler: async (ctx, args) => deleteForPublisherHandler(ctx, args),
});
+249
View File
@@ -0,0 +1,249 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
import {
applyGitHubSkillSourceSyncHandler,
applyGitHubSkillVerificationResultHandler,
configurePublicGitHubSkillSourceHandler,
upsertGitHubSkillContentHandler,
verifyGitHubSkillHandler,
} from "./githubSkillSync";
import { buildSkillInstallResolution } from "./lib/installResolver";
type Row = Record<string, unknown> & { _id: string };
function chainEq(constraints: Record<string, unknown>) {
return {
eq(field: string, value: unknown) {
constraints[field] = value;
return chainEq(constraints);
},
};
}
function matches(doc: Row, constraints: Record<string, unknown>) {
return Object.entries(constraints).every(([key, value]) => doc[key] === value);
}
function createDb(initial: Record<string, Row[]> = {}) {
const tables: Record<string, Row[]> = Object.fromEntries(
Object.entries(initial).map(([table, rows]) => [table, [...rows]]),
);
const counters: Record<string, number> = {};
const list = (table: string) => {
tables[table] ??= [];
return tables[table];
};
const db = {
get: async (id: string) => {
const table = id.split(":")[0] ?? "";
return list(table).find((row) => row._id === id) ?? null;
},
insert: async (table: string, doc: Record<string, unknown>) => {
counters[table] = (counters[table] ?? 0) + 1;
const inserted = {
_id: `${table}:new-${counters[table]}`,
_creationTime: counters[table],
...doc,
};
list(table).push(inserted);
return inserted._id;
},
patch: async (id: string, patch: Record<string, unknown>) => {
const table = id.split(":")[0] ?? "";
const row = list(table).find((candidate) => candidate._id === id);
if (!row) return;
for (const [key, value] of Object.entries(patch)) {
if (value === undefined) delete row[key];
else row[key] = value;
}
},
query: (table: string) => ({
withIndex: (_indexName: string, build?: (q: ReturnType<typeof chainEq>) => unknown) => {
const constraints: Record<string, unknown> = {};
build?.(chainEq(constraints));
const matched = () => list(table).filter((row) => matches(row, constraints));
return {
collect: async () => matched(),
unique: async () => matched()[0] ?? null,
};
},
}),
};
return { db, tables };
}
function getSkill(tables: Record<string, Row[]>, slug: string) {
const skill = tables.skills?.find((row) => row.slug === slug);
if (!skill) throw new Error(`Live GitHub canary did not discover skill: ${slug}`);
return skill;
}
function resolveInstallFromTables(tables: Record<string, Row[]>, slug: string) {
const skill = getSkill(tables, slug);
const source =
typeof skill.githubSourceId === "string"
? (tables.githubSkillSources?.find((row) => row._id === skill.githubSourceId) ?? null)
: null;
return buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: skill as never,
source: source as never,
});
}
const liveCanaryEnabled = process.env.CLAWHUB_LIVE_GITHUB_CANARY === "1";
const itIfLive = liveCanaryEnabled ? it : it.skip;
describe("GitHub-backed skills live canary", () => {
itIfLive(
"discovers and verifies an installable skill from a real GitHub repo",
{ timeout: 45_000 },
async () => {
const repo = process.env.CLAWHUB_LIVE_GITHUB_REPO?.trim() || "openclaw/agent-skills";
const skillSlug = process.env.CLAWHUB_LIVE_GITHUB_SKILL?.trim() || "handoff";
const { db, tables } = createDb({
globalStats: [
{
_id: "globalStats:default",
key: "default",
activeSkillsCount: 0,
updatedAt: 1,
},
],
});
const scheduler = { runAfter: async () => undefined };
let now = Date.now();
const actionCtx = {
runQuery: async (_query: unknown, args: Record<string, unknown>) => {
if ("ownerPublisherId" in args && "actorUserId" in args) {
return {
ownerUserId: "users:live-owner",
existingSource:
tables.githubSkillSources?.find((source) => source.repo === repo) ?? null,
official: true,
};
}
if ("skillId" in args) {
const skill = tables.skills?.find((row) => row._id === args.skillId);
const source =
skill && typeof skill.githubSourceId === "string"
? tables.githubSkillSources?.find((row) => row._id === skill.githubSourceId)
: null;
return skill && source ? { skill, source } : null;
}
if ("sourceId" in args) {
return (tables.skills ?? []).flatMap((skill) => {
if (
skill.githubSourceId !== args.sourceId ||
skill.installKind !== "github" ||
skill.githubCurrentStatus !== "present" ||
typeof skill.githubPath !== "string" ||
typeof skill.githubCurrentContentHash !== "string"
) {
return [];
}
return [
{
skillId: skill._id,
githubPath: skill.githubPath,
githubCurrentContentHash: skill.githubCurrentContentHash,
},
];
});
}
throw new Error(`unexpected live canary query args: ${JSON.stringify(args)}`);
},
runMutation: async (_mutation: unknown, args: Record<string, unknown>) => {
if ("snapshot" in args) {
return await applyGitHubSkillSourceSyncHandler(
{ db, scheduler } as never,
{
...args,
now,
} as never,
);
}
if ("scanStatus" in args && "contentHash" in args) {
return await applyGitHubSkillVerificationResultHandler(
{ db } as never,
{
...args,
now,
} as never,
);
}
if ("discovered" in args && "commit" in args) {
return await upsertGitHubSkillContentHandler(
{ db } as never,
{
...args,
now,
} as never,
);
}
throw new Error(`unexpected live canary mutation args: ${JSON.stringify(args)}`);
},
auth: { getUserIdentity: async () => null },
};
const configured = await configurePublicGitHubSkillSourceHandler(
actionCtx as never,
{
ownerPublisherId: "publishers:live" as never,
repo,
},
fetch,
{ userId: "users:live-owner" as never },
);
expect(configured.stats.discovered).toBeGreaterThan(0);
expect(configured.manifestStatus === "missing" || configured.manifestStatus === "ok").toBe(
true,
);
expect(configured.commit).toMatch(/^[a-f0-9]{40}$/);
let skill = getSkill(tables, skillSlug);
expect(skill).toMatchObject({
installKind: "github",
githubPath: `skills/${skillSlug}`,
githubCurrentCommit: configured.commit,
githubCurrentStatus: "present",
githubScanStatus: "pending",
});
expect(resolveInstallFromTables(tables, skillSlug)).toMatchObject({
ok: false,
reason: "github_verification_pending",
});
now = Date.now();
const verified = await verifyGitHubSkillHandler(
actionCtx as never,
{
skillId: skill._id as never,
contentHash: skill.githubCurrentContentHash as string,
},
fetch,
);
expect(verified).toMatchObject({ ok: true, scanStatus: "clean" });
skill = getSkill(tables, skillSlug);
expect(skill).toMatchObject({
githubCurrentCommit: configured.commit,
githubScanStatus: "clean",
moderationStatus: "active",
});
expect(resolveInstallFromTables(tables, skillSlug)).toMatchObject({
ok: true,
installKind: "github",
github: {
repo,
path: `skills/${skillSlug}`,
commit: configured.commit,
contentHash: skill.githubCurrentContentHash,
},
});
},
);
});
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+40 -3
View File
@@ -8,7 +8,7 @@ import {
cliDeviceTokenHttp,
cliSkillDeleteHttp,
cliSkillUndeleteHttp,
cliTelemetrySyncHttp,
cliTelemetryInstallHttp,
cliUploadUrlHttp,
cliWhoamiHttp,
getSkillHttp,
@@ -17,6 +17,7 @@ import {
} from "./httpApi";
import {
exportSkillsV1Http,
exportPluginsV1Http,
listBundlePluginsV1Http,
listCodePluginsV1Http,
listPackagesV1Http,
@@ -50,12 +51,18 @@ import {
starsPostRouterV1Http,
transfersGetRouterV1Http,
banAppealContextV1Http,
usersGetRouterV1Http,
usersListV1Http,
usersPostRouterV1Http,
verifyDocsSessionV1Http,
whoamiV1Http,
} from "./httpApiV1";
import { preflightHandler } from "./httpPreflight";
import {
packageInspectorArtifactHttp,
packageInspectorClaimHttp,
packageInspectorResultsHttp,
} from "./packageInspectorHttp";
const http = httpRouter();
@@ -109,6 +116,12 @@ http.route({
handler: listPluginsV1Http,
});
http.route({
path: ApiRoutes.pluginsExport,
method: "GET",
handler: exportPluginsV1Http,
});
http.route({
path: ApiRoutes.codePlugins,
method: "GET",
@@ -181,6 +194,24 @@ http.route({
handler: mintPublishTokenV1Http,
});
http.route({
path: "/api/v1/package-inspector/claim",
method: "POST",
handler: packageInspectorClaimHttp,
});
http.route({
path: "/api/v1/package-inspector/artifact",
method: "GET",
handler: packageInspectorArtifactHttp,
});
http.route({
path: "/api/v1/package-inspector/results",
method: "POST",
handler: packageInspectorResultsHttp,
});
http.route({
pathPrefix: `${ApiRoutes.packages}/`,
method: "POST",
@@ -271,6 +302,12 @@ http.route({
handler: banAppealContextV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.users}/`,
method: "GET",
handler: usersGetRouterV1Http,
});
http.route({
path: ApiRoutes.users,
method: "GET",
@@ -356,9 +393,9 @@ http.route({
});
http.route({
path: LegacyApiRoutes.cliTelemetrySync,
path: LegacyApiRoutes.cliTelemetryInstall,
method: "POST",
handler: cliTelemetrySyncHttp,
handler: cliTelemetryInstallHttp,
});
http.route({
+20 -51
View File
@@ -243,7 +243,7 @@ describe("httpApi handlers", () => {
it("cliWhoamiHttp returns 401 on auth failure", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(
new Error(
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, contact security@openclaw.ai.",
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, open a GitHub issue: https://github.com/openclaw/clawhub/issues/new.",
),
);
const response = await __handlers.cliWhoamiHandler(
@@ -267,49 +267,40 @@ describe("httpApi handlers", () => {
expect(json.user.handle).toBe("p");
});
it("cliTelemetrySyncHttp forwards roots and returns ok", async () => {
it("cliTelemetryInstallHttp forwards one install event and returns ok", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const runMutation = vi.fn().mockResolvedValue(null);
const response = await __handlers.cliTelemetrySyncHandler(
const response = await __handlers.cliTelemetryInstallHandler(
makeCtx({ runMutation }),
new Request("https://x/api/cli/telemetry/sync", {
new Request("https://x/api/cli/telemetry/install", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
roots: [
{
rootId: "abc",
label: "~/skills",
skills: [{ slug: "weather", version: null }],
},
],
event: "install",
slug: "weather",
version: "1.0.0",
rootId: "abc",
rootLabel: "~/skills",
}),
}),
);
expect(response.status).toBe(200);
expect(await response.json()).toEqual({ ok: true });
expect(runMutation).toHaveBeenCalledTimes(1);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: "users:1",
slug: "weather",
version: "1.0.0",
rootId: "abc",
rootLabel: "~/skills",
});
});
it("cliTelemetrySyncHttp returns 400 on invalid payload", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation: vi.fn() }),
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ roots: "nope" }),
}),
);
expect(response.status).toBe(400);
});
it("cliTelemetrySyncHttp forwards skill versions when provided", async () => {
it("cliTelemetryInstallHttp rejects sync-shaped roots snapshots", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const runMutation = vi.fn().mockResolvedValue(null);
await __handlers.cliTelemetrySyncHandler(
const response = await __handlers.cliTelemetryInstallHandler(
makeCtx({ runMutation }),
new Request("https://x/api/cli/telemetry/sync", {
new Request("https://x/api/cli/telemetry/install", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
@@ -323,31 +314,9 @@ describe("httpApi handlers", () => {
}),
}),
);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: "users:1",
roots: [
{ rootId: "abc", label: "~/skills", skills: [{ slug: "weather", version: "1.0.0" }] },
],
});
});
it("cliTelemetrySyncHttp returns 400 on invalid json", async () => {
const request = new Request("https://x/api/cli/telemetry/sync", { method: "POST", body: "{" });
const response = await __handlers.cliTelemetrySyncHandler(makeCtx({}), request);
expect(response.status).toBe(400);
});
it("cliTelemetrySyncHttp returns 401 when unauthorized", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({}),
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ roots: [] }),
}),
);
expect(response.status).toBe(401);
expect(runMutation).not.toHaveBeenCalled();
});
it("cliDeviceCodeHttp rate limits and creates a device code", async () => {
+12 -16
View File
@@ -1,9 +1,9 @@
import {
ApiCliSkillDeleteResponseSchema,
ApiCliTelemetrySyncResponseSchema,
ApiCliTelemetryInstallResponseSchema,
CliTelemetryInstallRequestSchema,
CliPublishRequestSchema,
CliSkillDeleteRequestSchema,
CliTelemetrySyncRequestSchema,
parseArk,
} from "clawhub-schema";
import { api, internal } from "./_generated/api";
@@ -227,7 +227,7 @@ export const cliSkillUndeleteHttp = httpAction((ctx, request) =>
cliSkillDeleteHandler(ctx, request, false),
);
async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
async function cliTelemetryInstallHandler(ctx: ActionCtx, request: Request) {
let body: unknown;
try {
body = await request.json();
@@ -237,19 +237,15 @@ async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
try {
const { userId } = await requireApiTokenUser(ctx, request);
const args = parseArk(CliTelemetrySyncRequestSchema, body, "Telemetry payload");
await ctx.runMutation(internal.telemetry.reportCliSyncInternal, {
const args = parseArk(CliTelemetryInstallRequestSchema, body, "Install telemetry payload");
await ctx.runMutation(internal.telemetry.reportCliInstallInternal, {
userId,
roots: args.roots.map((root) => ({
rootId: root.rootId,
label: root.label,
skills: root.skills.map((skill) => ({
slug: skill.slug,
version: skill.version ?? undefined,
})),
})),
slug: args.slug,
version: args.version,
rootId: args.rootId,
rootLabel: args.rootLabel,
});
const ok = parseArk(ApiCliTelemetrySyncResponseSchema, { ok: true }, "Telemetry response");
const ok = parseArk(ApiCliTelemetryInstallResponseSchema, { ok: true }, "Telemetry response");
return json(ok);
} catch (error) {
const message = error instanceof Error ? error.message : "Telemetry failed";
@@ -258,7 +254,7 @@ async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
}
}
export const cliTelemetrySyncHttp = httpAction(cliTelemetrySyncHandler);
export const cliTelemetryInstallHttp = httpAction(cliTelemetryInstallHandler);
async function cliDeviceCodeHandler(ctx: ActionCtx, request: Request) {
if (request.method !== "POST") return text("Method not allowed", 405);
@@ -392,7 +388,7 @@ export const __handlers = {
cliUploadUrlHandler,
cliPublishHandler,
cliSkillDeleteHandler,
cliTelemetrySyncHandler,
cliTelemetryInstallHandler,
cliDeviceCodeHandler,
cliDeviceTokenHandler,
};
File diff suppressed because it is too large Load Diff
+49
View File
@@ -6,6 +6,7 @@ import {
formatUserFacingErrorMessage,
parseMultipartSkillScan,
resolveVersionTagsBatch,
softDeleteErrorToResponse,
} from "./httpApiV1/shared";
function makeCtx() {
@@ -32,6 +33,54 @@ describe("http API v1 shared helpers", () => {
).toBe("Publisher not found");
});
it("maps soft-delete validation failures to 400 with cleaned messages", async () => {
const response = softDeleteErrorToResponse(
"package",
new Error(
"[CONVEX M] [Request ID: abc] Server Error Called by client Uncaught ConvexError: Package name must be lowercase and npm-safe (example: @scope/name or plugin-name)",
),
{},
);
expect(response.status).toBe(400);
await expect(response.text()).resolves.toBe(
"Package name must be lowercase and npm-safe (example: @scope/name or plugin-name)",
);
});
it("maps reserved package route validation failures to 400 with cleaned messages", async () => {
const response = softDeleteErrorToResponse(
"package",
new Error(
'[CONVEX M] [Request ID: abc] Server Error Called by client Uncaught ConvexError: Package name "publish" is reserved for ClawHub routes. Use a scoped name or choose a different package name.',
),
{},
);
expect(response.status).toBe(400);
await expect(response.text()).resolves.toBe(
'Package name "publish" is reserved for ClawHub routes. Use a scoped name or choose a different package name.',
);
});
it("keeps unknown soft-delete failures generic 500s", async () => {
const response = softDeleteErrorToResponse("soul", new Error("boom"), {});
expect(response.status).toBe(500);
await expect(response.text()).resolves.toBe("Internal Server Error");
});
it("keeps unrelated reserved-word failures generic 500s", async () => {
const response = softDeleteErrorToResponse(
"package",
new Error("database reserved capacity exceeded"),
{},
);
expect(response.status).toBe(500);
await expect(response.text()).resolves.toBe("Internal Server Error");
});
it("resolves latest tags without reading version documents", async () => {
const ctx = makeCtx();
const versionId = "skillVersions:latest" as Id<"skillVersions">;
+10
View File
@@ -1,6 +1,7 @@
import { httpAction } from "./functions";
import { verifyDocsSessionV1Handler } from "./httpApiV1/docsSessionV1";
import {
exportPluginsV1Handler,
listBundlePluginsV1Handler,
listCodePluginsV1Handler,
listPackagesV1Handler,
@@ -40,6 +41,7 @@ import { starsDeleteRouterV1Handler, starsPostRouterV1Handler } from "./httpApiV
import { transfersGetRouterV1Handler } from "./httpApiV1/transfersV1";
import {
banAppealContextV1Handler,
usersGetRouterV1Handler,
usersListV1Handler,
usersPostRouterV1Handler,
} from "./httpApiV1/usersV1";
@@ -47,6 +49,7 @@ import { whoamiV1Handler } from "./httpApiV1/whoamiV1";
export const listPackagesV1Http = httpAction(listPackagesV1Handler);
export const listPluginsV1Http = httpAction(listPluginsV1Handler);
export const exportPluginsV1Http = httpAction(exportPluginsV1Handler);
export const packagesGetRouterV1Http = httpAction(packagesGetRouterV1Handler);
export const packagesPostRouterV1Http = httpAction(packagesPostRouterV1Handler);
export const packagesDeleteRouterV1Http = httpAction(packagesDeleteRouterV1Handler);
@@ -84,6 +87,7 @@ export const starsDeleteRouterV1Http = httpAction(starsDeleteRouterV1Handler);
export const transfersGetRouterV1Http = httpAction(transfersGetRouterV1Handler);
export const whoamiV1Http = httpAction(whoamiV1Handler);
export const usersGetRouterV1Http = httpAction(usersGetRouterV1Handler);
export const usersPostRouterV1Http = httpAction(usersPostRouterV1Handler);
export const usersListV1Http = httpAction(usersListV1Handler);
export const banAppealContextV1Http = httpAction(banAppealContextV1Handler);
@@ -91,6 +95,7 @@ export const banAppealContextV1Http = httpAction(banAppealContextV1Handler);
export const __handlers = {
listPackagesV1Handler,
listPluginsV1Handler,
exportPluginsV1Handler,
packagesGetRouterV1Handler,
packagesPostRouterV1Handler,
packagesDeleteRouterV1Handler,
@@ -108,6 +113,10 @@ export const __handlers = {
skillsGetRouterV1Handler,
publishSkillV1Handler,
skillSecurityVerdictsV1Handler,
skillScanSubmitV1Handler,
skillScanGetRouterV1Handler,
skillScanBatchSubmitV1Handler,
skillScanBatchStatusV1Handler,
skillsPostRouterV1Handler,
skillsDeleteRouterV1Handler,
exportSkillsV1Handler,
@@ -120,6 +129,7 @@ export const __handlers = {
starsDeleteRouterV1Handler,
transfersGetRouterV1Handler,
whoamiV1Handler,
usersGetRouterV1Handler,
usersPostRouterV1Handler,
usersListV1Handler,
banAppealContextV1Handler,
+576 -56
View File
@@ -1,5 +1,4 @@
import {
PackageArtifactBackfillRequestSchema,
ApiV1PackageOfficialMigrationListResponseSchema,
ApiV1PackageOfficialMigrationResponseSchema,
ApiV1PackageModerationStatusResponseSchema,
@@ -27,8 +26,9 @@ import {
import { api, internal } from "../_generated/api";
import type { Doc, Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { buildDownloadMetricArgs, getDownloadIdentity } from "../downloadMetrics";
import { getOptionalActiveAuthUserIdFromAction } from "../lib/access";
import { getOptionalApiTokenUserId } from "../lib/apiTokenAuth";
import { getOptionalApiTokenUserId, requireApiTokenUser } from "../lib/apiTokenAuth";
import { parseClawPack, sha256Base64, sha256Hex } from "../lib/clawpack";
import {
fetchGitHubRepositoryIdentity,
@@ -53,9 +53,19 @@ import {
MAX_PUBLISH_FILE_BYTES,
MAX_PUBLISH_TOTAL_BYTES,
} from "../lib/publishLimits";
import { getPublicSkillFileAccessBlock, isSkillVersionForSkill } from "../lib/skillFileAccess";
import {
getPublicSkillFileAccessBlock,
getPublicSkillVersionAccessBlock,
getSkillFileModerationInfoFromSkill,
isSkillVersionForSkill,
} from "../lib/skillFileAccess";
import { isMacJunkPath, isTextFile } from "../lib/skills";
import { buildDeterministicPackageZip } from "../lib/skillZip";
import {
buildDeterministicPackageZip,
buildMergedExportZip,
validateFilePath,
type MergedExportManifestEntry,
} from "../lib/skillZip";
import { generateToken, hashToken } from "../lib/tokens";
import {
MAX_RAW_FILE_BYTES,
@@ -88,6 +98,7 @@ const apiRefs = api as unknown as {
const internalRefs = internal as unknown as {
packages: {
getByNameForViewerInternal: unknown;
listPluginExportPageInternal: unknown;
listPageForViewerInternal: unknown;
searchForViewerInternal: unknown;
listVersionsForViewerInternal: unknown;
@@ -120,9 +131,11 @@ const internalRefs = internal as unknown as {
resolvePackageAppealForUserInternal: unknown;
listOfficialPluginMigrationsInternal: unknown;
upsertOfficialPluginMigrationForUserInternal: unknown;
backfillPackageArtifactKindsInternal: unknown;
listPackageModerationQueueInternal: unknown;
};
downloadMetrics: {
recordDownloadMetricInternal: unknown;
};
packagePublishTokens: {
createInternal: unknown;
};
@@ -204,6 +217,20 @@ async function runMutationRef<T>(ctx: ActionCtx, ref: unknown, args: unknown): P
return (await ctx.runMutation(ref as never, args as never)) as T;
}
async function chunkedParallel<T, R>(
items: T[],
chunkSize: number,
fn: (item: T) => Promise<R>,
): Promise<R[]> {
const results: R[] = [];
for (let i = 0; i < items.length; i += chunkSize) {
const chunk = items.slice(i, i + chunkSize);
const chunkResults = await Promise.all(chunk.map(fn));
results.push(...chunkResults);
}
return results;
}
async function getOptionalViewerUserIdForRequest(ctx: ActionCtx, request: Request) {
const apiTokenUserId = await getOptionalApiTokenUserId(ctx, request);
if (apiTokenUserId) return apiTokenUserId;
@@ -226,7 +253,13 @@ function normalizeCapabilityTagSegment(value: string) {
}
const PACKAGE_FAMILY_VALUES = ["skill", "code-plugin", "bundle-plugin"] as const;
const PLUGIN_EXPORT_FAMILY_VALUES = ["code-plugin", "bundle-plugin"] as const;
const PACKAGE_CHANNEL_VALUES = ["official", "community", "private"] as const;
const PACKAGE_LIST_SORT_VALUES = ["updated", "downloads"] as const;
const MAX_PLUGIN_EXPORT_FILE_COUNT = 10_000;
const MAX_PLUGIN_EXPORT_PAGE_LIMIT = 250;
const DEFAULT_PLUGIN_EXPORT_PAGE_LIMIT = 250;
const MAX_PLUGIN_EXPORT_TOTAL_BYTES = 256 * 1024 * 1024;
function invalidQueryParamMessage(name: string) {
return `Invalid ${name} query parameter`;
@@ -388,6 +421,7 @@ type PackageListQueryArgs = {
executesCode?: boolean;
capabilityTag?: string;
category?: string;
sort?: (typeof PACKAGE_LIST_SORT_VALUES)[number];
viewerUserId?: Id<"users">;
paginationOpts: { cursor: string | null; numItems: number };
};
@@ -423,6 +457,7 @@ type SkillVersionLike = {
type ReleaseLike = {
_id: Id<"packageReleases">;
packageId: Id<"packages">;
version: string;
createdAt: number;
changelog: string;
@@ -458,6 +493,23 @@ type ReleaseLike = {
softDeletedAt?: number;
};
type PluginExportFamily = (typeof PLUGIN_EXPORT_FAMILY_VALUES)[number];
type PluginExportDigest = {
packageId: Id<"packages">;
name: string;
displayName: string;
family: PluginExportFamily;
latestReleaseId?: Id<"packageReleases">;
latestVersion?: string | null;
createdAt: number;
updatedAt: number;
stats?: Record<string, unknown> | null;
ownerUserId: Id<"users">;
ownerHandle?: string | null;
ownerDisplayName?: string | null;
};
type PackageTrustedPublisherLike = {
_id: Id<"packageTrustedPublishers">;
packageId: Id<"packages">;
@@ -643,9 +695,11 @@ function releaseArtifactUrls(request: Request, packageName: string, release: Rel
async function streamClawPackRelease(
ctx: ActionCtx,
request: Request,
rateHeaders: HeadersInit,
pkg: PublicPackageDocLike,
release: ReleaseLike,
viewerUserId: Id<"users"> | null,
statKind: "download" | "install" = "download",
) {
const securityBlock = getReleaseSecurityBlock(release);
@@ -656,13 +710,24 @@ async function streamClawPackRelease(
const blob = await ctx.storage.get(release.clawpackStorageId);
if (!blob) return text("ClawPack artifact not found", 404, rateHeaders);
try {
const statMutation =
statKind === "install"
? internalRefs.packages.recordPackageInstallInternal
: internalRefs.packages.recordPackageDownloadInternal;
await runMutationRef(ctx, statMutation, {
packageId: pkg._id,
});
if (statKind === "install") {
await runMutationRef(ctx, internalRefs.packages.recordPackageInstallInternal, {
packageId: pkg._id,
});
}
const identity = getDownloadIdentity(request, viewerUserId ? String(viewerUserId) : null);
if (identity) {
await runMutationRef(
ctx,
internalRefs.downloadMetrics.recordDownloadMetricInternal,
await buildDownloadMetricArgs({
target: { kind: "package", id: pkg._id },
identity,
now: Date.now(),
}),
);
}
} catch {
// Best-effort metric path; never fail package downloads.
}
@@ -719,6 +784,7 @@ type CatalogListItem = {
capabilityTags?: string[];
executesCode?: boolean;
verificationTier?: string | null;
stats?: { downloads: number; installs: number; stars: number; versions: number };
};
type CatalogSearchEntry = {
@@ -924,6 +990,18 @@ function compareCatalogItems(a: CatalogListItem, b: CatalogListItem) {
return a.name.localeCompare(b.name);
}
function compareCatalogItemsForSort(
a: CatalogListItem,
b: CatalogListItem,
sort: (typeof PACKAGE_LIST_SORT_VALUES)[number] | undefined,
) {
if (sort === "downloads") {
const downloads = (b.stats?.downloads ?? 0) - (a.stats?.downloads ?? 0);
if (downloads !== 0) return downloads;
}
return compareCatalogItems(a, b);
}
function compareCatalogSearchEntries(a: CatalogSearchEntry, b: CatalogSearchEntry) {
return (
(a.rankTier ?? Number.POSITIVE_INFINITY) - (b.rankTier ?? Number.POSITIVE_INFINITY) ||
@@ -1335,6 +1413,8 @@ async function listPackages(
if (!highlightedOnlyParam.ok) return text(highlightedOnlyParam.message, 400, rate.headers);
const executesCode = parseBooleanQueryParam(url.searchParams, "executesCode");
if (!executesCode.ok) return text(executesCode.message, 400, rate.headers);
const sortParam = parseEnumQueryParam(url.searchParams, "sort", PACKAGE_LIST_SORT_VALUES);
if (!sortParam.ok) return text(sortParam.message, 400, rate.headers);
const category = url.searchParams.get("category")?.trim() || undefined;
if (category && !isPluginCategorySlug(category)) {
return text("Invalid plugin category", 400, rate.headers);
@@ -1361,6 +1441,7 @@ async function listPackages(
highlightedOnly: highlightedOnly || undefined,
executesCode: executesCode.value,
capabilityTag,
sort: sortParam.value,
paginationOpts: { cursor, numItems: limit },
});
return json(
@@ -1394,6 +1475,7 @@ async function listPackages(
executesCode: executesCode.value,
capabilityTag,
category,
sort: sortParam.value,
viewerUserId: viewerUserId ?? undefined,
paginationOpts: { cursor: pageCursor, numItems },
});
@@ -1414,6 +1496,7 @@ async function listPackages(
highlightedOnly: highlightedOnly || undefined,
executesCode: executesCode.value,
capabilityTag,
sort: sortParam.value,
paginationOpts: { cursor: pageCursor, numItems },
});
return {
@@ -1427,7 +1510,8 @@ async function listPackages(
if (!packageCandidate && !skillCandidate) break;
if (
!skillCandidate ||
(packageCandidate && compareCatalogItems(packageCandidate, skillCandidate) <= 0)
(packageCandidate &&
compareCatalogItemsForSort(packageCandidate, skillCandidate, sortParam.value) <= 0)
) {
items.push(packageCandidate!);
packageSource.index += 1;
@@ -1479,6 +1563,7 @@ async function listPackages(
executesCode: executesCode.value,
capabilityTag,
category,
sort: sortParam.value,
viewerUserId: viewerUserId ?? undefined,
paginationOpts: { cursor: pageCursor, numItems },
});
@@ -1507,7 +1592,8 @@ async function listPackages(
if (
!bundlePluginCandidate ||
(codePluginCandidate &&
compareCatalogItems(codePluginCandidate, bundlePluginCandidate) <= 0)
compareCatalogItemsForSort(codePluginCandidate, bundlePluginCandidate, sortParam.value) <=
0)
) {
items.push(codePluginCandidate!);
codePluginSource.index += 1;
@@ -1548,6 +1634,7 @@ async function listPackages(
executesCode: executesCode.value,
capabilityTag,
category,
sort: sortParam.value,
viewerUserId: viewerUserId ?? undefined,
paginationOpts: { cursor, numItems: limit },
} satisfies PackageListQueryArgs);
@@ -1562,6 +1649,388 @@ export async function listPackagesV1Handler(ctx: ActionCtx, request: Request) {
return await listPackages(ctx, request, undefined, { includeSkills: true });
}
type PluginsExportPhase =
| "list_plugins"
| "build_empty_zip"
| "load_releases"
| "plan_blobs"
| "load_blobs"
| "assemble_entries"
| "build_zip";
type PluginsExportLogContext = {
phase: PluginsExportPhase;
startDate: number;
endDate: number;
family: PluginExportFamily | null;
limit: number;
cursorPresent: boolean;
pageLength: number;
hasMore: boolean | null;
nextCursorPresent: boolean | null;
releaseCount: number;
blobTaskCount: number;
blobCount: number;
zipEntryCount: number;
manifestCount: number;
exportErrorCount: number;
totalExportBytes: number;
};
function logPluginsExportFailure(context: PluginsExportLogContext, error: unknown) {
console.error("plugins_export_failed", {
...context,
errorName: error instanceof Error ? error.name : typeof error,
errorMessage:
error instanceof Error ? error.message.slice(0, 500) : String(error).slice(0, 500),
});
}
function parsePluginExportFamily(value: string | null) {
const family = value?.trim();
if (!family) return undefined;
return PLUGIN_EXPORT_FAMILY_VALUES.includes(family as PluginExportFamily)
? (family as PluginExportFamily)
: null;
}
function isReleaseForPackage(release: ReleaseLike, digest: PluginExportDigest) {
return release.packageId === digest.packageId;
}
function pluginExportRoot(digest: PluginExportDigest) {
return `${digest.family}/${digest.name}`;
}
function pluginExportMetaPath(digest: PluginExportDigest) {
return `__clawhub_export/${pluginExportRoot(digest)}/plugin_meta.json`;
}
export async function exportPluginsV1Handler(ctx: ActionCtx, request: Request) {
try {
await requireApiTokenUser(ctx, request);
} catch (err) {
return text(err instanceof Error ? err.message : "Unauthorized", 401);
}
const rate = await applyRateLimit(ctx, request, "export");
if (!rate.ok) return rate.response;
const url = new URL(request.url);
const startDate = toOptionalNumber(url.searchParams.get("startDate"));
const endDate = toOptionalNumber(url.searchParams.get("endDate"));
const requestedLimit = toOptionalNumber(url.searchParams.get("limit"));
const cursor = url.searchParams.get("cursor")?.trim() || undefined;
const family = parsePluginExportFamily(url.searchParams.get("family"));
if (family === null) {
return text("family must be code-plugin or bundle-plugin", 400, rate.headers);
}
if (startDate == null || endDate == null) {
return text(
"startDate and endDate query parameters are required (Unix milliseconds)",
400,
rate.headers,
);
}
if (startDate > endDate) {
return text("startDate must be <= endDate", 400, rate.headers);
}
if (requestedLimit != null && requestedLimit > MAX_PLUGIN_EXPORT_PAGE_LIMIT) {
return text(`limit must be <= ${MAX_PLUGIN_EXPORT_PAGE_LIMIT}`, 400, rate.headers);
}
const limit = Math.max(1, requestedLimit ?? DEFAULT_PLUGIN_EXPORT_PAGE_LIMIT);
const logContext: PluginsExportLogContext = {
phase: "list_plugins",
startDate,
endDate,
family: family ?? null,
limit,
cursorPresent: Boolean(cursor),
pageLength: 0,
hasMore: null,
nextCursorPresent: null,
releaseCount: 0,
blobTaskCount: 0,
blobCount: 0,
zipEntryCount: 0,
manifestCount: 0,
exportErrorCount: 0,
totalExportBytes: 0,
};
let result: {
page: PluginExportDigest[];
nextCursor: string | null;
hasMore: boolean;
};
try {
result = await runQueryRef(ctx, internalRefs.packages.listPluginExportPageInternal, {
startDate,
endDate,
cursor,
numItems: limit,
family,
});
} catch (err) {
if (err instanceof Error && err.message.includes("Invalid cursor format")) {
return text("Invalid cursor format", 400, rate.headers);
}
logPluginsExportFailure(logContext, err);
throw err;
}
logContext.pageLength = result.page.length;
logContext.hasMore = result.hasMore;
logContext.nextCursorPresent = Boolean(result.nextCursor);
const familyLabel = family ?? "all";
if (result.page.length === 0) {
try {
logContext.phase = "build_empty_zip";
const emptyZip = buildMergedExportZip([], []);
return new Response(emptyZip as unknown as BodyInit, {
status: 200,
headers: mergeHeaders(rate.headers, {
"Content-Type": "application/zip",
"Content-Disposition": `attachment; filename="plugins-export-${familyLabel}-${startDate}-${endDate}-empty.zip"`,
"X-Next-Cursor": result.nextCursor ?? "",
"X-Has-More": String(result.hasMore),
"X-Total-Returned": "0",
"X-Date-Range": `${startDate}-${endDate}`,
"X-Export-Errors": "0",
}),
});
} catch (err) {
logPluginsExportFailure(logContext, err);
throw err;
}
}
const exportErrors: Array<{ package: string; error: string }> = [];
try {
logContext.phase = "load_releases";
const releases = await chunkedParallel(result.page, 100, (digest) =>
digest.latestReleaseId
? runQueryRef<ReleaseLike | null>(ctx, internalRefs.packages.getReleaseByIdInternal, {
releaseId: digest.latestReleaseId,
})
: Promise.resolve(null),
);
logContext.releaseCount = releases.filter(Boolean).length;
const exportableReleases: Array<ReleaseLike | null> = Array.from(
{ length: result.page.length },
() => null,
);
type BlobTask = { digestIndex: number; fileIndex: number; storageId: Id<"_storage"> };
const blobTasks: BlobTask[] = [];
logContext.phase = "plan_blobs";
for (let i = 0; i < result.page.length; i++) {
const digest = result.page[i];
const release = releases[i] ?? null;
if (!digest.latestReleaseId || !release) {
exportErrors.push({
package: digest.name,
error: `release not found (latestReleaseId: ${digest.latestReleaseId ?? "null"})`,
});
continue;
}
if (!isReleaseForPackage(release, digest)) {
exportErrors.push({
package: digest.name,
error: `release not found (latestReleaseId: ${digest.latestReleaseId})`,
});
continue;
}
if (release.softDeletedAt) {
exportErrors.push({
package: digest.name,
error: `release not available (latestReleaseId: ${digest.latestReleaseId})`,
});
continue;
}
const securityBlock = getReleaseSecurityBlock(release);
if (securityBlock) {
exportErrors.push({
package: digest.name,
error: `release blocked: ${securityBlock.message}`,
});
continue;
}
if (!release.files || release.files.length === 0) {
exportErrors.push({
package: digest.name,
error: `release has no files (latestReleaseId: ${digest.latestReleaseId})`,
});
continue;
}
if (!validateFilePath(pluginExportRoot(digest))) {
exportErrors.push({
package: digest.name,
error: "invalid package export path (fails Zip Slip validation)",
});
continue;
}
exportableReleases[i] = release;
for (let j = 0; j < release.files.length; j++) {
if (blobTasks.length >= MAX_PLUGIN_EXPORT_FILE_COUNT) {
exportErrors.push({
package: digest.name,
error: `file count cap exceeded (${MAX_PLUGIN_EXPORT_FILE_COUNT})`,
});
break;
}
blobTasks.push({
digestIndex: i,
fileIndex: j,
storageId: release.files[j].storageId,
});
}
}
logContext.blobTaskCount = blobTasks.length;
logContext.exportErrorCount = exportErrors.length;
logContext.phase = "load_blobs";
const blobs = await chunkedParallel(blobTasks, 50, (task) => ctx.storage.get(task.storageId));
logContext.blobCount = blobs.length;
const zipEntries: Array<{ path: string; bytes: Uint8Array }> = [];
const manifest: Array<
MergedExportManifestEntry & {
family: PluginExportFamily;
packageName: string;
latestReleaseId: string | null;
artifactKind: ReleaseLike["artifactKind"] | null;
}
> = [];
let totalExportBytes = 0;
const blobsByDigest = new Map<number, Map<number, Blob | null>>();
for (let k = 0; k < blobTasks.length; k++) {
const task = blobTasks[k];
if (!blobsByDigest.has(task.digestIndex)) {
blobsByDigest.set(task.digestIndex, new Map());
}
blobsByDigest.get(task.digestIndex)!.set(task.fileIndex, blobs[k]);
}
logContext.phase = "assemble_entries";
for (let i = 0; i < result.page.length; i++) {
const digest = result.page[i];
const release = exportableReleases[i];
if (!release?.files) continue;
const exportRoot = pluginExportRoot(digest);
if (!validateFilePath(exportRoot)) continue;
const digestBlobs = blobsByDigest.get(i);
if (!digestBlobs) continue;
let fileCount = 0;
for (let j = 0; j < release.files.length; j++) {
const filePath = release.files[j].path;
if (!validateFilePath(filePath)) {
exportErrors.push({
package: digest.name,
error: `invalid file path: "${filePath}" (fails Zip Slip validation)`,
});
continue;
}
const blob = digestBlobs.get(j);
if (!blob) {
exportErrors.push({
package: digest.name,
error: `blob not found for file "${filePath}" (storageId: ${release.files[j].storageId})`,
});
continue;
}
const buffer = new Uint8Array(await blob.arrayBuffer());
if (totalExportBytes + buffer.byteLength > MAX_PLUGIN_EXPORT_TOTAL_BYTES) {
exportErrors.push({
package: digest.name,
error: `byte cap exceeded (${MAX_PLUGIN_EXPORT_TOTAL_BYTES}) at file "${filePath}"`,
});
continue;
}
totalExportBytes += buffer.byteLength;
zipEntries.push({ path: `${exportRoot}/${filePath}`, bytes: buffer });
fileCount++;
}
const pluginMeta = {
name: digest.name,
displayName: digest.displayName,
family: digest.family,
version: release.version ?? digest.latestVersion ?? null,
latestReleaseId: digest.latestReleaseId ?? null,
artifactKind: release.artifactKind ?? null,
createdAt: digest.createdAt,
updatedAt: digest.updatedAt,
stats: digest.stats ?? null,
owner: {
handle: digest.ownerHandle ?? null,
displayName: digest.ownerDisplayName ?? null,
},
};
zipEntries.push({
path: pluginExportMetaPath(digest),
bytes: new TextEncoder().encode(JSON.stringify(pluginMeta, null, 2)),
});
manifest.push({
publisher: digest.ownerHandle ?? String(digest.ownerUserId),
slug: digest.name,
packageName: digest.name,
family: digest.family,
version: release.version ?? digest.latestVersion ?? null,
displayName: digest.displayName,
createdAt: digest.createdAt,
updatedAt: digest.updatedAt,
stats: digest.stats ?? null,
fileCount,
latestReleaseId: digest.latestReleaseId ?? null,
artifactKind: release.artifactKind ?? null,
});
}
if (exportErrors.length > 0) {
zipEntries.push({
path: "_errors.json",
bytes: new TextEncoder().encode(JSON.stringify(exportErrors, null, 2)),
});
}
logContext.zipEntryCount = zipEntries.length;
logContext.manifestCount = manifest.length;
logContext.exportErrorCount = exportErrors.length;
logContext.totalExportBytes = totalExportBytes;
logContext.phase = "build_zip";
const zipBytes = buildMergedExportZip(zipEntries, manifest);
return new Response(zipBytes as unknown as BodyInit, {
status: 200,
headers: mergeHeaders(rate.headers, {
"Content-Type": "application/zip",
"Content-Disposition": `attachment; filename="plugins-export-${familyLabel}-${startDate}-${endDate}.zip"`,
"X-Next-Cursor": result.nextCursor ?? "",
"X-Has-More": String(result.hasMore),
"X-Total-Returned": String(manifest.length),
"X-Date-Range": `${startDate}-${endDate}`,
"X-Export-Errors": String(exportErrors.length),
}),
});
} catch (err) {
logPluginsExportFailure(logContext, err);
throw err;
}
}
export async function listPluginsV1Handler(ctx: ActionCtx, request: Request) {
return await listPackages(ctx, request, undefined, {
includeSkills: false,
@@ -1734,42 +2203,6 @@ export async function mintPublishTokenV1Handler(ctx: ActionCtx, request: Request
export async function packagesPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const segments = getPathSegments(request, "/api/v1/packages/");
if (segments[0] === "backfill" && segments[1] === "artifacts" && segments.length === 2) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const auth = await requireApiTokenUserOrResponse(ctx, request, rate.headers);
if (!auth.ok) return auth.response;
try {
const body = parseArk(
PackageArtifactBackfillRequestSchema,
await request.json().catch(() => ({})),
"Package artifact backfill payload",
) as {
cursor?: string | null;
batchSize?: number;
dryRun?: boolean;
};
const result = await runMutationRef(
ctx,
internalRefs.packages.backfillPackageArtifactKindsInternal,
{
actorUserId: auth.userId,
...(body.cursor !== undefined ? { cursor: body.cursor } : {}),
...(typeof body.batchSize === "number" ? { batchSize: body.batchSize } : {}),
...(typeof body.dryRun === "boolean" ? { dryRun: body.dryRun } : {}),
},
);
return json(result, 200, rate.headers);
} catch (error) {
return text(
error instanceof Error ? error.message : "Package artifact backfill failed",
400,
rate.headers,
);
}
}
if (segments[0] === "migrations" && segments.length === 1) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
@@ -2409,10 +2842,50 @@ async function getSkillDetailForRequest(ctx: ActionCtx, slug: string) {
isMalwareBlocked?: boolean | null;
isHiddenByMod?: boolean | null;
isRemoved?: boolean | null;
sourceVersionId?: Id<"skillVersions"> | null;
} | null;
} | null;
}
type PackageExactVersionModeratedSkill = Pick<
Doc<"skills">,
| "_id"
| "softDeletedAt"
| "latestVersionId"
| "tags"
| "moderationStatus"
| "moderationReason"
| "moderationFlags"
| "moderationVerdict"
| "moderationSourceVersionId"
>;
async function getUnavailableSkillPackageVersionBlock(
ctx: ActionCtx,
slug: string,
versionName: string,
) {
const skill = await runQueryRef<PackageExactVersionModeratedSkill | null>(
ctx,
internalRefs.skills.getSkillBySlugInternal,
{ slug },
);
if (!skill || skill.softDeletedAt) return null;
const version = (await runQueryRef(ctx, internalRefs.skills.getVersionBySkillAndVersionInternal, {
skillId: skill._id,
version: versionName,
})) as SkillVersionLike | null;
if (!version || !isSkillVersionForSkill(version, skill._id)) return null;
if (version.softDeletedAt) return { status: 410, message: "Version not available" };
return getPublicSkillVersionAccessBlock(
getSkillFileModerationInfoFromSkill(skill),
version._id,
skill.latestVersionId ?? skill.tags?.latest,
);
}
async function getSkillVersionForRequest(
ctx: ActionCtx,
skill: Pick<SkillPackageDocLike, "_id" | "latestVersionId" | "tags">,
@@ -2772,7 +3245,21 @@ export async function packagesGetRouterV1Handler(ctx: ActionCtx, request: Reques
const skillDetail = detail?.package
? null
: await getSkillDetailForRequest(ctx, normalizedPackageName);
if (!detail?.package && !skillDetail?.skill) return text("Package not found", 404, rate.headers);
const isExactVersionRequest =
packageSegments[0] === "versions" && packageSegments[1] && packageSegments.length === 2;
if (!detail?.package && !skillDetail?.skill) {
if (isExactVersionRequest) {
const moderationBlock = await getUnavailableSkillPackageVersionBlock(
ctx,
normalizedPackageName,
packageSegments[1],
);
if (moderationBlock) {
return text(moderationBlock.message, moderationBlock.status, rate.headers);
}
}
return text("Package not found", 404, rate.headers);
}
const packageDetail = detail?.package ? detail : null;
const publicPackage = packageDetail?.package ?? null;
const packageOwner = packageDetail?.owner ?? null;
@@ -2906,7 +3393,14 @@ export async function packagesGetRouterV1Handler(ctx: ActionCtx, request: Reques
if (!release) return text("Version not found", 404, rate.headers);
if (packageSegments[3] === "download") {
if (release.artifactKind === "npm-pack") {
return await streamClawPackRelease(ctx, rate.headers, publicPackage!, release);
return await streamClawPackRelease(
ctx,
request,
rate.headers,
publicPackage!,
release,
viewerUserId ?? null,
);
}
const url = new URL(
`/api/v1/packages/${encodePackagePath(publicPackage!.name)}/download`,
@@ -2947,6 +3441,15 @@ export async function packagesGetRouterV1Handler(ctx: ActionCtx, request: Reques
},
)) as SkillVersionLike | null;
if (!version || version.softDeletedAt) return text("Version not found", 404, rate.headers);
const effectiveLatestVersionId =
skillDetail.skill.latestVersionId ?? skillDetail.skill.tags?.latest;
const moderationBlock = getPublicSkillVersionAccessBlock(
skillDetail.moderationInfo,
version._id,
effectiveLatestVersionId,
);
if (moderationBlock)
return text(moderationBlock.message, moderationBlock.status, rate.headers);
const tags = await resolveSkillTags(ctx, skillDetail.skill._id, skillDetail.skill.tags);
return json(
{
@@ -3105,9 +3608,18 @@ export async function packagesGetRouterV1Handler(ctx: ActionCtx, request: Reques
const zip = buildDeterministicPackageZip(entries);
const [zipSha256, zipSha256Base64] = await Promise.all([sha256Hex(zip), sha256Base64(zip)]);
try {
await runMutationRef(ctx, internalRefs.packages.recordPackageDownloadInternal, {
packageId: publicPackage!._id,
});
const identity = getDownloadIdentity(request, viewerUserId ? String(viewerUserId) : null);
if (identity) {
await runMutationRef(
ctx,
internalRefs.downloadMetrics.recordDownloadMetricInternal,
await buildDownloadMetricArgs({
target: { kind: "package", id: publicPackage!._id },
identity,
now: Date.now(),
}),
);
}
} catch {
// Best-effort metric path; never fail package downloads.
}
@@ -3253,7 +3765,15 @@ export async function npmMirrorGetHandler(ctx: ActionCtx, request: Request) {
const tarballName = path.rest[1]!;
const release = releases.find((candidate) => candidate.npmTarballName === tarballName);
if (!release) return text("ClawPack artifact not found", 404, rate.headers);
return await streamClawPackRelease(ctx, rate.headers, detail.package, release, "install");
return await streamClawPackRelease(
ctx,
request,
rate.headers,
detail.package,
release,
viewerUserId ?? null,
"install",
);
}
if (path.rest.length > 0) return text("Not found", 404, rate.headers);
+23 -5
View File
@@ -514,22 +514,40 @@ export function parsePublishBody(body: unknown) {
};
}
// Substrings that indicate user-input validation failures from the underlying
// mutations (e.g. `normalizePackageName` ConvexErrors). These are surfaced as
// 400s with the cleaned message so CLI/API clients can see the actual reason
// instead of an opaque 500.
const SOFT_DELETE_BAD_REQUEST_HINTS = [
"slug required",
"package name required",
"package name must be",
"must be lowercase",
"npm-safe",
"reserved for clawhub routes",
"version required",
] as const;
export function softDeleteErrorToResponse(
entity: "skill" | "soul" | "package",
error: unknown,
headers: HeadersInit,
) {
const message = error instanceof Error ? error.message : `${entity} delete failed`;
const lower = message.toLowerCase();
const rawMessage = error instanceof Error ? error.message : `${entity} delete failed`;
const cleaned = cleanUserFacingErrorMessage(rawMessage) || rawMessage;
const lower = cleaned.toLowerCase();
if (lower.includes("unauthorized"))
return text(formatAuthzMessage(error, "Unauthorized"), 401, headers);
if (lower.includes("forbidden"))
return text(formatAuthzMessage(error, "Forbidden"), 403, headers);
if (lower.includes("not found")) return text(message, 404, headers);
if (lower.includes("slug required")) return text("Slug required", 400, headers);
if (lower.includes("not found")) return text(cleaned, 404, headers);
if (SOFT_DELETE_BAD_REQUEST_HINTS.some((hint) => lower.includes(hint))) {
return text(cleaned, 400, headers);
}
// Unknown: server-side failure. Keep body generic.
// Unknown: server-side failure. Keep the body generic; only known
// user-input validation failures above surface the cleaned mutation message.
return text("Internal Server Error", 500, headers);
}
+284 -86
View File
@@ -21,13 +21,24 @@ import { getOptionalApiTokenUserId, requireApiTokenUser } from "../lib/apiTokenA
import { mergeHeaders } from "../lib/httpHeaders";
import { applyRateLimit } from "../lib/httpRateLimit";
import { parseBooleanQueryParam, resolveBooleanQueryParam } from "../lib/httpUtils";
import {
buildSkillInstallResolution,
type InstallResolverSkill,
type InstallResolverSource,
type SkillInstallResolution,
} from "../lib/installResolver";
import type {
LlmAgenticRiskFinding,
LlmEvalDimension,
LlmRiskSummary,
} from "../lib/securityPrompt";
import { selectGeneratedSkillCardFile, sourceSkillVersionFiles } from "../lib/skillCards";
import { getPublicSkillFileAccessBlock, isSkillVersionForSkill } from "../lib/skillFileAccess";
import {
getPublicSkillFileAccessBlock,
getPublicSkillVersionAccessBlock,
getSkillFileModerationInfoFromSkill,
isSkillVersionForSkill,
} from "../lib/skillFileAccess";
import {
buildDeterministicZip,
buildMergedExportZip,
@@ -42,7 +53,6 @@ import {
getPathSegments,
json,
parseJsonPayload,
parseMultipartSkillScan,
parseMultipartPublish,
parsePublishBody,
publicApiOrigin,
@@ -186,6 +196,7 @@ type GetBySlugResult = {
summary?: string;
engineVersion?: string;
updatedAt?: number;
sourceVersionId?: Id<"skillVersions"> | null;
reason?: string;
} | null;
} | null;
@@ -284,10 +295,13 @@ type SkillSecuritySnapshot = {
};
const internalRefs = internal as unknown as {
githubSkillSources: {
getByIdInternal: unknown;
};
securityScan: {
createUploadedSkillScanRequestInternal: unknown;
createPublishedSkillScanRequestInternal: unknown;
enqueueBulkSkillRescanBatchForAdminInternal: unknown;
getStoredScanReportForUserInternal: unknown;
getSkillScanRequestForUserInternal: unknown;
getBulkSkillRescanBatchStatusForAdminInternal: unknown;
requestSkillRescanForUserInternal: unknown;
@@ -297,6 +311,10 @@ const internalRefs = internal as unknown as {
};
skills: {
getSecurityVerdictTargetInternal: unknown;
getVerifyTargetBySlugInternal: unknown;
getSkillBySlugInternal: unknown;
getVersionByIdInternal: unknown;
getVersionBySkillAndVersionInternal: unknown;
reportSkillForUserInternal: unknown;
listSkillReportsInternal: unknown;
triageSkillReportForUserInternal: unknown;
@@ -324,10 +342,6 @@ function isMultipartRequest(request: Request) {
);
}
async function deleteStoredScanFiles(ctx: ActionCtx, files: Array<{ storageId: Id<"_storage"> }>) {
await Promise.allSettled(files.map((file) => ctx.storage.delete(file.storageId)));
}
function encodeJsonEntry(value: unknown) {
return new TextEncoder().encode(`${JSON.stringify(value, null, 2)}\n`);
}
@@ -362,14 +376,26 @@ function buildSkillScanReportZip(status: Record<string, unknown>) {
`Scan ID: ${scanIdText}`,
`Status: ${statusText}`,
"",
"This archive uses the ClawHub security-audit export shape:",
"This archive contains the stored security scan results for the submitted ClawHub version.",
"",
"- manifest.json",
"- clawscan.json",
"- skillspector.json",
"- static-analysis.json",
"- virustotal.json",
"- README.md",
"## How to read this report",
"",
"Start with `clawscan.json`. ClawScan is the primary security verdict for the submitted artifact. Its `summary` field is the short explanation of what triggered the result, and `guidance` explains what to change before uploading a fixed version.",
"",
"- `malicious` means ClawHub blocked the submitted version from public install surfaces.",
"- `suspicious` means ClawHub found behavior that needs review before users should rely on it.",
"- `clean` means ClawHub did not find blocking security issues in this scan.",
"",
"VirusTotal results are supporting reputation telemetry. They can help explain a risk signal, but they are not the sole source of ClawHub's final verdict.",
"",
"## Files",
"",
"- `manifest.json`: artifact identity, scan status, timestamps, and writeback state.",
"- `clawscan.json`: final ClawScan verdict, summary, guidance, and findings.",
"- `skillspector.json`: SkillSpector structure and agentic-risk signals when available.",
"- `static-analysis.json`: deterministic scanner findings, reason codes, and static summary.",
"- `virustotal.json`: external reputation counts and status when available.",
"- `README.md`: this interpretation guide.",
"",
].join("\n");
@@ -386,6 +412,16 @@ function buildSkillScanReportZip(status: Record<string, unknown>) {
]);
}
function safeScanReportFilenamePart(value: string) {
return (
value
.replace(/^@/, "")
.replaceAll("/", "-")
.replaceAll(/[^a-zA-Z0-9._-]+/g, "-")
.replace(/^-+|-+$/g, "") || "artifact"
);
}
async function handleSkillScanBatchSubmit(ctx: ActionCtx, request: Request, headers: HeadersInit) {
const auth = await requireApiTokenUserOrResponse(ctx, request, headers);
if (!auth.ok) return auth.response;
@@ -629,16 +665,6 @@ type VerifySecurityVersion = {
| "checkedAt"
> &
Partial<Pick<NonNullable<Doc<"skillVersions">["skillSpectorAnalysis"]>, "summary" | "error">>;
depRegistryAnalysis?: Pick<
NonNullable<Doc<"skillVersions">["depRegistryAnalysis"]>,
"status" | "summary" | "checkedAt"
> &
Partial<
Pick<
NonNullable<Doc<"skillVersions">["depRegistryAnalysis"]>,
"notFoundPackages" | "unresolvedPackages"
>
>;
};
type SecurityVerdictTargetResult = {
@@ -688,9 +714,6 @@ function buildVerifySecurity(version: VerifySecurityVersion) {
const skillSpectorStatus = version.skillSpectorAnalysis
? normalizeVerificationStatus(version.skillSpectorAnalysis.status)
: null;
const depStatus = version.depRegistryAnalysis
? normalizeVerificationStatus(version.depRegistryAnalysis.status)
: null;
const status = clawStatus;
return {
@@ -746,16 +769,7 @@ function buildVerifySecurity(version: VerifySecurityVersion) {
checkedAt: version.skillSpectorAnalysis.checkedAt ?? null,
}
: null,
dependencyRegistry: version.depRegistryAnalysis
? {
status: depStatus ?? "pending",
rawStatus: version.depRegistryAnalysis.status,
summary: version.depRegistryAnalysis.summary ?? null,
notFoundPackages: version.depRegistryAnalysis.notFoundPackages ?? [],
unresolvedPackages: version.depRegistryAnalysis.unresolvedPackages ?? [],
checkedAt: version.depRegistryAnalysis.checkedAt ?? null,
}
: null,
dependencyRegistry: null,
},
};
}
@@ -788,7 +802,7 @@ function buildVerifyReasons(args: {
securityStatus: NormalizedSecurityStatus;
}) {
const reasons: string[] = [];
if (!args.cardAvailable) reasons.push("card.missing");
if (!args.cardAvailable && !args.isMalwareBlocked) reasons.push("card.missing");
reasons.push(
...buildSecurityVerdictReasons({
isMalwareBlocked: args.isMalwareBlocked,
@@ -822,7 +836,6 @@ function getVerifySecurityCheckedAt(security: ReturnType<typeof buildVerifySecur
security.signals.staticScan?.checkedAt,
security.signals.virusTotal?.checkedAt,
security.signals.skillSpector?.checkedAt,
security.signals.dependencyRegistry?.checkedAt,
].filter((value): value is number => typeof value === "number");
return candidates.length > 0 ? Math.max(...candidates) : null;
}
@@ -869,14 +882,7 @@ function buildSecurityVerdictSummary(security: ReturnType<typeof buildVerifySecu
checkedAt: security.signals.skillSpector.checkedAt,
}
: null,
dependencyRegistry: security.signals.dependencyRegistry
? {
status: security.signals.dependencyRegistry.status,
rawStatus: security.signals.dependencyRegistry.rawStatus,
summary: security.signals.dependencyRegistry.summary,
checkedAt: security.signals.dependencyRegistry.checkedAt,
}
: null,
dependencyRegistry: null,
},
};
}
@@ -1113,35 +1119,11 @@ export async function skillScanSubmitV1Handler(ctx: ActionCtx, request: Request)
try {
if (isMultipartRequest(request)) {
const multipart = await parseMultipartSkillScan(ctx, request, (payload) => {
const parsed = parseArk(
ApiV1SkillScanSubmitRequestSchema,
payload,
"Skill scan payload",
) as {
source: { kind: "upload" } | { kind: "published"; slug: string; version?: string };
update?: boolean;
};
if (parsed.source.kind !== "upload") {
throw new Error("multipart scan payload must use source.kind=upload");
}
if (parsed.update === true) {
throw new Error("update is not valid for uploaded scans");
}
return parsed;
});
const result = await runMutationRef(
ctx,
internalRefs.securityScan.createUploadedSkillScanRequestInternal,
{
actorUserId: auth.userId,
files: multipart.files,
},
).catch(async (error) => {
await deleteStoredScanFiles(ctx, multipart.files);
throw error;
});
return json(result, 202, rate.headers);
return text(
"Local upload scans are no longer supported. Upload a version, then use `clawhub scan download <slug> --version <version>` to retrieve stored scan results.",
410,
rate.headers,
);
}
const body = parseArk(
@@ -1153,7 +1135,11 @@ export async function skillScanSubmitV1Handler(ctx: ActionCtx, request: Request)
update?: boolean;
};
if (body.source.kind === "upload") {
return text("uploaded scans must use multipart/form-data", 400, rate.headers);
return text(
"Local upload scans are no longer supported. Upload a version, then use `clawhub scan download <slug> --version <version>` to retrieve stored scan results.",
410,
rate.headers,
);
}
const result = await runMutationRef(
ctx,
@@ -1187,6 +1173,32 @@ export async function skillScanGetRouterV1Handler(ctx: ActionCtx, request: Reque
if (!scanId) return text("scanId required", 400, rate.headers);
try {
if (segments.length === 2 && scanId === "download") {
const name = (segments[1] ?? "").trim();
const url = new URL(request.url);
const version = url.searchParams.get("version")?.trim() ?? "";
const kind = url.searchParams.get("kind")?.trim() === "plugin" ? "plugin" : "skill";
if (!name) return text("name required", 400, rate.headers);
if (!version) return text("version required", 400, rate.headers);
const status = (await runQueryRef(
ctx,
internalRefs.securityScan.getStoredScanReportForUserInternal,
{
actorUserId: auth.userId,
kind,
name,
version,
},
)) as Record<string, unknown>;
const zip = buildSkillScanReportZip(status);
const headers = mergeHeaders(rate.headers, {
"Content-Type": "application/zip",
"Content-Disposition": `attachment; filename="clawhub-scan-${safeScanReportFilenamePart(name)}-${safeScanReportFilenamePart(version)}.zip"`,
});
return new Response(zip, { status: 200, headers });
}
const status = (await runQueryRef(
ctx,
internalRefs.securityScan.getSkillScanRequestForUserInternal,
@@ -1466,6 +1478,80 @@ async function describeOwnerVisibleSkillState(
return null;
}
function shouldExposeHiddenGitHubInstallBlock(
skill: InstallResolverSkill & {
installKind?: "github";
moderationStatus?: "active" | "hidden" | "removed";
moderationReason?: string;
},
resolution: SkillInstallResolution,
) {
if (skill.installKind !== "github" || resolution.ok) return false;
if (skill.moderationStatus !== "hidden") return false;
const reason = skill.moderationReason ?? "";
return (
reason === "pending.scan" ||
reason === "scanner.failed" ||
reason === "scanner.llm.malicious" ||
reason.startsWith("github.")
);
}
type ExactVersionModeratedSkill = Pick<
Doc<"skills">,
| "_id"
| "softDeletedAt"
| "latestVersionId"
| "tags"
| "moderationStatus"
| "moderationReason"
| "moderationFlags"
| "moderationVerdict"
| "moderationSourceVersionId"
>;
async function getUnavailableSkillVersionBlock(
ctx: ActionCtx,
slug: string,
selector?: { versionName?: string; tagName?: string },
) {
const skill = await runQueryRef<ExactVersionModeratedSkill | null>(
ctx,
internalRefs.skills.getSkillBySlugInternal,
{ slug },
);
if (!skill || skill.softDeletedAt) return null;
const latestVersionId = skill.latestVersionId ?? skill.tags?.latest;
const selectedVersionId = selector?.tagName ? skill.tags?.[selector.tagName] : latestVersionId;
if (!selector?.versionName && !selectedVersionId) return null;
const version = selector?.versionName
? await runQueryRef<PublicSkillVersionResponse | null>(
ctx,
internalRefs.skills.getVersionBySkillAndVersionInternal,
{
skillId: skill._id,
version: selector.versionName,
},
)
: await runQueryRef<PublicSkillVersionResponse | null>(
ctx,
internalRefs.skills.getVersionByIdInternal,
{
versionId: selectedVersionId,
},
);
if (!version || !isSkillVersionForSkill(version, skill._id)) return null;
if (version.softDeletedAt) return { status: 410, message: "Version not available" };
return getPublicSkillVersionAccessBlock(
getSkillFileModerationInfoFromSkill(skill),
version._id,
skill.latestVersionId ?? skill.tags?.latest,
);
}
export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
@@ -1530,6 +1616,60 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
return json(result, 200, rate.headers);
}
if (second === "install" && segments.length === 2) {
const url = new URL(request.url);
const forceInstall = parseBooleanQueryParam(url.searchParams.get("forceInstall"));
const skill = (await runQueryRef<
| (InstallResolverSkill & {
_id: Id<"skills">;
githubSourceId?: Id<"githubSkillSources">;
softDeletedAt?: number;
moderationStatus?: "active" | "hidden" | "removed";
moderationReason?: string;
moderationFlags?: string[];
})
| null
>(ctx, internalRefs.skills.getSkillBySlugInternal, { slug })) as
| (InstallResolverSkill & {
_id: Id<"skills">;
githubSourceId?: Id<"githubSkillSources">;
softDeletedAt?: number;
moderationStatus?: "active" | "hidden" | "removed";
moderationReason?: string;
moderationFlags?: string[];
})
| null;
if (!skill || skill.softDeletedAt || skill.moderationStatus === "removed") {
return text("Skill not found", 404, rate.headers);
}
const source =
skill.installKind === "github" && skill.githubSourceId
? ((await runQueryRef(ctx, internalRefs.githubSkillSources.getByIdInternal, {
sourceId: skill.githubSourceId,
})) as InstallResolverSource | null)
: null;
const resolution = buildSkillInstallResolution({
origin: publicApiOrigin(request),
skill,
source,
forceInstall,
});
const publicSkillResult = (await ctx.runQuery(api.skills.getBySlug, {
slug,
})) as GetBySlugResult;
const publiclyVisible = publicSkillResult?.skill?._id === skill._id;
if (!publiclyVisible) {
if (!resolution.ok && shouldExposeHiddenGitHubInstallBlock(skill, resolution)) {
return json(resolution, resolution.status, rate.headers);
}
return text("Skill not found", 404, rate.headers);
}
return json(resolution, resolution.ok ? 200 : resolution.status, rate.headers);
}
if (segments.length === 1) {
const result = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult;
if (!result?.skill) {
@@ -1704,7 +1844,15 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
if (second === "versions" && third && segments.length === 3) {
const skillResult = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult;
if (!skillResult?.skill) return text("Skill not found", 404, rate.headers);
if (!skillResult?.skill) {
const moderationBlock = await getUnavailableSkillVersionBlock(ctx, slug, {
versionName: third,
});
if (moderationBlock) {
return text(moderationBlock.message, moderationBlock.status, rate.headers);
}
return text("Skill not found", 404, rate.headers);
}
const version = (await ctx.runQuery(api.skills.getVersionBySkillAndVersion, {
skillId: skillResult.skill._id,
@@ -1712,6 +1860,16 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
})) as PublicSkillVersionResponse | null;
if (!version) return text("Version not found", 404, rate.headers);
if (version.softDeletedAt) return text("Version not available", 410, rate.headers);
const effectiveLatestVersionId =
skillResult.skill.latestVersionId ?? skillResult.skill.tags?.latest;
const moderationBlock = getPublicSkillVersionAccessBlock(
skillResult.moderationInfo,
version._id,
effectiveLatestVersionId,
);
if (moderationBlock) {
return text(moderationBlock.message, moderationBlock.status, rate.headers);
}
const security = buildSkillSecuritySnapshot(version);
return json(
@@ -1744,6 +1902,13 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
const result = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult;
if (!result?.skill) {
const moderationBlock = await getUnavailableSkillVersionBlock(ctx, slug, {
versionName: versionParam,
tagName: tagParam,
});
if (moderationBlock) {
return text(moderationBlock.message, moderationBlock.status, rate.headers);
}
const hidden = await describeOwnerVisibleSkillState(ctx, request, slug);
if (hidden) return text(hidden.message, hidden.status, rate.headers);
return text("Skill not found", 404, rate.headers);
@@ -1769,9 +1934,33 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
}
if (version.softDeletedAt) return text("Version not available", 410, rate.headers);
const effectiveLatestVersionId = result.skill.latestVersionId ?? result.skill.tags?.latest;
const moderationBlock = getPublicSkillVersionAccessBlock(
result.moderationInfo,
version._id,
effectiveLatestVersionId,
);
if (moderationBlock) {
return text(moderationBlock.message, moderationBlock.status, rate.headers);
}
let moderationSourceVersion: PublicSkillVersionResponse | null = result.latestVersion;
const moderationSourceVersionId = result.moderationInfo?.sourceVersionId;
if (moderationSourceVersionId) {
if (version._id === moderationSourceVersionId) {
moderationSourceVersion = version;
} else if (result.latestVersion?._id !== moderationSourceVersionId) {
const sourceVersion = (await ctx.runQuery(api.skills.getVersionById, {
versionId: moderationSourceVersionId,
})) as PublicSkillVersionResponse | null;
moderationSourceVersion = isSkillVersionForSkill(sourceVersion, result.skill._id)
? sourceVersion
: null;
}
}
const security = buildSkillSecuritySnapshot(version);
const moderationMatchesRequestedVersion = Boolean(
result.latestVersion && result.latestVersion._id === version._id,
moderationSourceVersion && moderationSourceVersion._id === version._id,
);
return json(
@@ -1788,10 +1977,10 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
moderation: result.moderationInfo
? {
scope: "skill",
sourceVersion: result.latestVersion
sourceVersion: moderationSourceVersion
? {
version: result.latestVersion.version,
createdAt: result.latestVersion.createdAt,
version: moderationSourceVersion.version,
createdAt: moderationSourceVersion.createdAt,
}
: null,
matchesRequestedVersion: moderationMatchesRequestedVersion,
@@ -1815,7 +2004,11 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
const tagParam = url.searchParams.get("tag")?.trim();
if (versionParam && tagParam) return text("Use either version or tag", 400, rate.headers);
const skillResult = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult;
const skillResult = (await runQueryRef<GetBySlugResult>(
ctx,
internalRefs.skills.getVerifyTargetBySlugInternal,
{ slug },
)) as GetBySlugResult;
if (!skillResult?.skill) {
const hidden = await describeOwnerVisibleSkillState(ctx, request, slug);
if (hidden) return text(hidden.message, hidden.status, rate.headers);
@@ -1854,11 +2047,14 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
const bundleFingerprints = fingerprintEntries
.filter((entry) => entry.kind === "generated-bundle")
.map((entry) => entry.fingerprint);
const generatedCardFile = await selectGeneratedSkillCardFile(version.files, bundleFingerprints);
const isMalwareBlocked = skillResult.moderationInfo?.isMalwareBlocked ?? false;
const generatedCardFile = isMalwareBlocked
? null
: await selectGeneratedSkillCardFile(version.files, bundleFingerprints);
const security = buildVerifySecurity(version);
const reasons = buildVerifyReasons({
cardAvailable: Boolean(generatedCardFile),
isMalwareBlocked: skillResult.moderationInfo?.isMalwareBlocked ?? false,
isMalwareBlocked,
securityPassed: security.passed,
securityStatus: security.status,
});
@@ -1895,7 +2091,9 @@ export async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request)
: {
available: false,
path: "skill-card.md",
url: buildCardUrl(request, skillResult.skill.slug, version.version),
url: isMalwareBlocked
? null
: buildCardUrl(request, skillResult.skill.slug, version.version),
sha256: null,
size: null,
contentType: null,
+247
View File
@@ -14,11 +14,17 @@ import {
const usersV1InternalRefs = internal as unknown as {
publishers: {
addOfficialPublisherInternal: unknown;
deleteEmptyOrgPublisherInternal: unknown;
listOfficialPublishersInternal: unknown;
removeOrgPublisherMemberInternal: unknown;
removeOfficialPublisherInternal: unknown;
};
users: {
getBanAppealContextByGitHubProviderAccountIdInternal: unknown;
getByHandleInternal: unknown;
recordStaffEmailAttemptAuditInternal: unknown;
recordStaffEmailSentAuditInternal: unknown;
remediateAutobansInternal: unknown;
reclassifyBanInternal: unknown;
unbanUserForBanAppealServiceInternal: unknown;
@@ -83,7 +89,10 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
action !== "ban-appeal-unban" &&
action !== "reclaim" &&
action !== "reserve" &&
action !== "email" &&
action !== "publisher" &&
action !== "publisher-delete" &&
action !== "publisher-official" &&
action !== "publisher-member"
) {
return text("Not found", 404, rate.headers);
@@ -133,12 +142,30 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
return handleAdminReserve(ctx, payload, actorUserId, rate.headers);
}
if (action === "email") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminStaffEmail(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminEnsurePublisher(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher-delete") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminDeletePublisher(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher-official") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminOfficialPublisherPost(ctx, payload, actorUserId, rate.headers);
}
if (action === "publisher-member") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
@@ -241,6 +268,159 @@ export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request)
}
}
async function handleAdminDeletePublisher(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
const reason = typeof payload.reason === "string" ? payload.reason.trim() : "";
const dryRun = payload.dryRun !== false;
if (!handle) return text("Missing handle", 400, headers);
if (!reason) return text("Missing reason", 400, headers);
if (reason.length > 500) return text("Reason too long (max 500 chars)", 400, headers);
try {
const result = await runUsersV1MutationRef(
ctx,
usersV1InternalRefs.publishers.deleteEmptyOrgPublisherInternal,
{
actorUserId,
handle,
reason,
dryRun,
},
);
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Publisher delete failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, headers);
}
if (message.toLowerCase().includes("not found")) {
return text(message, 404, headers);
}
return text(message, 400, headers);
}
}
async function handleAdminStaffEmail(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const toEmail = typeof payload.toEmail === "string" ? payload.toEmail.trim().toLowerCase() : "";
const userHandle =
typeof payload.userHandle === "string"
? payload.userHandle.trim().replace(/^@+/, "").toLowerCase()
: "";
const subject = typeof payload.subject === "string" ? payload.subject.trim() : "";
const body = typeof payload.body === "string" ? payload.body.trim() : "";
const confirmUserRequest = payload.confirmUserRequest === true;
const confirmUserSignoff = payload.confirmUserSignoff === true;
if (toEmail && userHandle) return text("Pass toEmail or userHandle, not both", 400, headers);
if (!toEmail && !userHandle) return text("Missing toEmail or userHandle", 400, headers);
if (toEmail && !/^[^@\s]+@[^@\s]+\.[^@\s]+$/.test(toEmail)) {
return text("Invalid toEmail", 400, headers);
}
if (!subject) return text("Missing subject", 400, headers);
if (subject.length > 200) return text("Subject too long (max 200 chars)", 400, headers);
if (!body) return text("Missing body", 400, headers);
if (body.length > 20_000) return text("Body too long (max 20000 chars)", 400, headers);
if (!confirmUserRequest || !confirmUserSignoff) {
return text(
"Staff email requires explicit user request and user sign-off on final recipient, subject, and body.",
400,
headers,
);
}
let recipientEmail = toEmail;
let recipientUserId: Id<"users"> | undefined;
let recipientHandle: string | null | undefined;
if (userHandle) {
const user = await runUsersV1QueryRef<{
_id?: Id<"users">;
handle?: string | null;
email?: string | null;
} | null>(ctx, usersV1InternalRefs.users.getByHandleInternal, { handle: userHandle });
if (!user?._id) return text("User not found", 404, headers);
if (!user.email?.trim()) return text("User has no email address", 400, headers);
recipientEmail = user.email.trim().toLowerCase();
recipientUserId = user._id;
recipientHandle = user.handle ?? userHandle;
}
const apiKey = process.env.RESEND_API_KEY?.trim();
if (!apiKey) return text("RESEND_API_KEY is not configured", 500, headers);
const from =
process.env.CLAWHUB_NOREPLY_FROM?.trim() ||
process.env.NOREPLY_EMAIL_FROM?.trim() ||
"ClawHub <noreply@clawhub.ai>";
const emailAudit = await runUsersV1MutationRef<{ auditLogId: Id<"auditLogs"> }>(
ctx,
usersV1InternalRefs.users.recordStaffEmailAttemptAuditInternal,
{
actorUserId,
toEmail: recipientEmail,
...(recipientUserId ? { recipientUserId } : {}),
...(recipientHandle ? { recipientHandle } : {}),
subject,
},
);
const response = await fetch("https://api.resend.com/emails", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
from,
to: [recipientEmail],
subject,
text: body,
}),
});
if (!response.ok) {
const errorText = await response.text().catch(() => "");
return text(
`Email provider failed (${response.status})${errorText ? `: ${errorText}` : ""}`,
502,
headers,
);
}
const providerJson = (await response.json().catch(() => null)) as { id?: unknown } | null;
const providerId = typeof providerJson?.id === "string" ? providerJson.id : null;
await runUsersV1MutationRef(ctx, usersV1InternalRefs.users.recordStaffEmailSentAuditInternal, {
actorUserId,
auditLogId: emailAudit.auditLogId,
providerId,
});
return json(
{
ok: true,
sent: true,
recipient: {
email: recipientEmail,
...(recipientUserId ? { userId: recipientUserId } : {}),
...(recipientHandle ? { handle: recipientHandle } : {}),
},
subject,
providerId,
},
200,
headers,
);
}
async function handleAdminReclassifyBan(
ctx: ActionCtx,
payload: unknown,
@@ -352,6 +532,37 @@ async function handleAdminRemediateAutobans(
}
}
export async function usersGetRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/users/");
if (segments.length !== 1 || segments[0] !== "publisher-official") {
return text("Not found", 404, rate.headers);
}
const authResult = await requireApiTokenUserOrResponse(ctx, request, rate.headers);
if (!authResult.ok) return authResult.response;
const admin = requireAdminOrResponse(authResult.user, rate.headers);
if (!admin.ok) return admin.response;
try {
const result = await runUsersV1QueryRef(
ctx,
usersV1InternalRefs.publishers.listOfficialPublishersInternal,
{ actorUserId: authResult.userId },
);
return json(result, 200, rate.headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Official publisher list failed";
if (message.toLowerCase().includes("forbidden")) return text("Forbidden", 403, rate.headers);
if (message.toLowerCase().includes("unauthorized")) {
return text("Unauthorized", 401, rate.headers);
}
return text(message, 400, rate.headers);
}
}
/**
* POST /api/v1/users/restore
* Admin-only: restore skills from GitHub backup for a user.
@@ -532,6 +743,42 @@ async function handleAdminReserve(
return json({ ok: true, results, succeeded, failed }, 200, headers);
}
async function handleAdminOfficialPublisherPost(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const action = typeof payload.action === "string" ? payload.action.trim().toLowerCase() : "";
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
const reason = typeof payload.reason === "string" ? payload.reason.trim() : "";
if (action !== "add" && action !== "remove") return text("Invalid action", 400, headers);
if (!handle) return text("Missing handle", 400, headers);
if (!reason) return text("Missing reason", 400, headers);
if (reason.length > 500) return text("Reason too long (max 500 chars)", 400, headers);
try {
const result = await runUsersV1MutationRef(
ctx,
action === "add"
? usersV1InternalRefs.publishers.addOfficialPublisherInternal
: usersV1InternalRefs.publishers.removeOfficialPublisherInternal,
{
actorUserId,
handle,
reason,
},
);
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Official publisher update failed";
if (message.toLowerCase().includes("forbidden")) return text("Forbidden", 403, headers);
if (message.toLowerCase().includes("unauthorized")) return text("Unauthorized", 401, headers);
if (message.toLowerCase().includes("not found")) return text(message, 404, headers);
return text(message, 400, headers);
}
}
async function handleAdminEnsurePublisher(
ctx: ActionCtx,
payload: Record<string, unknown>,
+97 -1
View File
@@ -1,4 +1,4 @@
import { describe, expect, it, vi } from "vitest";
import { beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("@convex-dev/auth/server", () => ({
getAuthUserId: vi.fn(),
@@ -9,6 +9,10 @@ const { getAuthUserId } = await import("@convex-dev/auth/server");
const { assertAdmin, assertModerator, assertRole, requireUser, requireUserFromAction } =
await import("./access");
beforeEach(() => {
vi.mocked(getAuthUserId).mockReset();
});
describe("access.requireUser", () => {
it("throws when auth is missing", async () => {
vi.mocked(getAuthUserId).mockResolvedValue(null);
@@ -59,6 +63,67 @@ describe("access.requireUser", () => {
expect(dbGet).toHaveBeenCalledWith("users:2");
expect(result).toEqual({ userId: "users:2", user });
});
it("uses the local dev impersonation user before browser auth", async () => {
const previousHandle = process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
const previousEnabled = process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
try {
process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = "local";
process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = "1";
vi.mocked(getAuthUserId).mockResolvedValue("users:browser" as never);
const user = { _id: "users:local", handle: "local", role: "admin" };
const unique = vi.fn().mockResolvedValue(user as never);
const withIndex = vi.fn().mockReturnValue({ unique });
const query = vi.fn().mockReturnValue({ withIndex });
const dbGet = vi.fn().mockResolvedValue(user as never);
const result = await requireUser({
db: { get: dbGet, query },
} as never);
expect(query).toHaveBeenCalledWith("users");
expect(withIndex).toHaveBeenCalledWith("handle", expect.any(Function));
expect(dbGet).toHaveBeenCalledWith("users:local");
expect(getAuthUserId).not.toHaveBeenCalled();
expect(result).toEqual({ userId: "users:local", user });
} finally {
if (previousHandle === undefined) delete process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
else process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = previousHandle;
if (previousEnabled === undefined) delete process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
else process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = previousEnabled;
}
});
it("does not use local dev impersonation in production deployments", async () => {
const previousHandle = process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
const previousEnabled = process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
const previousDeployment = process.env.CONVEX_DEPLOYMENT;
try {
process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = "local";
process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = "1";
process.env.CONVEX_DEPLOYMENT = "prod:wry-manatee-359";
vi.mocked(getAuthUserId).mockResolvedValue("users:browser" as never);
const user = { _id: "users:browser", handle: "browser", role: "user" };
const query = vi.fn();
const dbGet = vi.fn().mockResolvedValue(user as never);
const result = await requireUser({
db: { get: dbGet, query },
} as never);
expect(query).not.toHaveBeenCalled();
expect(getAuthUserId).toHaveBeenCalled();
expect(dbGet).toHaveBeenCalledWith("users:browser");
expect(result).toEqual({ userId: "users:browser", user });
} finally {
if (previousHandle === undefined) delete process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
else process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = previousHandle;
if (previousEnabled === undefined) delete process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
else process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = previousEnabled;
if (previousDeployment === undefined) delete process.env.CONVEX_DEPLOYMENT;
else process.env.CONVEX_DEPLOYMENT = previousDeployment;
}
});
});
describe("access.requireUserFromAction", () => {
@@ -111,6 +176,37 @@ describe("access.requireUserFromAction", () => {
expect(runQuery).toHaveBeenCalledTimes(1);
expect(result).toEqual({ userId: "users:9", user });
});
it("uses the local dev impersonation user before action auth", async () => {
const previousHandle = process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
const previousEnabled = process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
try {
process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = "local";
process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = "1";
vi.mocked(getAuthUserId).mockResolvedValue("users:browser" as never);
const user = { _id: "users:local", handle: "local", role: "admin" };
const runQuery = vi.fn(async (_query, args: { handle?: string; userId?: string }) => {
if (args.handle === "local") return user;
if (args.userId === "users:local") return user;
return null;
});
const result = await requireUserFromAction({ runQuery } as never);
expect(runQuery).toHaveBeenCalledTimes(2);
expect(runQuery).toHaveBeenNthCalledWith(1, expect.anything(), { handle: "local" });
expect(runQuery).toHaveBeenNthCalledWith(2, expect.anything(), {
userId: "users:local",
});
expect(getAuthUserId).not.toHaveBeenCalled();
expect(result).toEqual({ userId: "users:local", user });
} finally {
if (previousHandle === undefined) delete process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
else process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE = previousHandle;
if (previousEnabled === undefined) delete process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
else process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION = previousEnabled;
}
});
});
describe("access role assertions", () => {
+36 -7
View File
@@ -8,10 +8,23 @@ export type Role = "admin" | "moderator" | "user" | "mirror";
const DEV_IMPERSONATE_LOCAL_HANDLE = "local";
function readEnv(name: string) {
const value = process.env[name]?.trim();
const value = readKnownEnv(name)?.trim();
return value ? value : undefined;
}
function readKnownEnv(name: string) {
if (name === "CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE") {
return process.env.CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE;
}
if (name === "CLAW_HUB_ENABLE_DEV_IMPERSONATION") {
return process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
}
if (name === "CONVEX_DEPLOYMENT") {
return process.env.CONVEX_DEPLOYMENT;
}
return undefined;
}
function isDevImpersonationAllowed() {
const requestedHandle = readEnv("CLAW_HUB_DEV_IMPERSONATE_USER_HANDLE");
if (requestedHandle !== DEV_IMPERSONATE_LOCAL_HANDLE) return false;
@@ -52,39 +65,49 @@ async function getDevImpersonatedUserIdFromAction(
export async function getOptionalActiveAuthUserId(
ctx: MutationCtx | QueryCtx,
): Promise<Id<"users"> | undefined> {
const devUserId = await getDevImpersonatedUserId(ctx);
if (devUserId) return devUserId;
try {
const userId = await getAuthUserId(ctx);
if (!userId) return await getDevImpersonatedUserId(ctx);
if (!userId) return undefined;
const user = await ctx.db.get(userId);
if (!user || user.deletedAt || user.deactivatedAt) return undefined;
return userId;
} catch {
return await getDevImpersonatedUserId(ctx);
return undefined;
}
}
export async function getOptionalActiveAuthUserIdFromAction(
ctx: ActionCtx,
): Promise<Id<"users"> | undefined> {
const devUserId = await getDevImpersonatedUserIdFromAction(ctx);
if (devUserId) return devUserId;
try {
const userId = await getAuthUserId(ctx);
if (!userId) return await getDevImpersonatedUserIdFromAction(ctx);
if (!userId) return undefined;
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId });
if (!user || user.deletedAt || user.deactivatedAt) return undefined;
return userId;
} catch {
return await getDevImpersonatedUserIdFromAction(ctx);
return undefined;
}
}
export async function requireUser(ctx: MutationCtx | QueryCtx) {
const devUserId = await getDevImpersonatedUserId(ctx);
if (devUserId) {
const devUser = await ctx.db.get(devUserId);
if (!devUser || devUser.deletedAt || devUser.deactivatedAt) throw new Error("User not found");
return { userId: devUserId, user: devUser };
}
let userId: Id<"users"> | null | undefined = null;
try {
userId = await getAuthUserId(ctx);
} catch {
userId = null;
}
userId ??= await getDevImpersonatedUserId(ctx);
if (!userId) throw new Error("Unauthorized");
let user: Doc<"users"> | null;
try {
@@ -99,13 +122,19 @@ export async function requireUser(ctx: MutationCtx | QueryCtx) {
export async function requireUserFromAction(
ctx: ActionCtx,
): Promise<{ userId: Id<"users">; user: Doc<"users"> }> {
const devUserId = await getDevImpersonatedUserIdFromAction(ctx);
if (devUserId) {
const devUser = await ctx.runQuery(internal.users.getByIdInternal, { userId: devUserId });
if (!devUser || devUser.deletedAt || devUser.deactivatedAt) throw new Error("User not found");
return { userId: devUserId, user: devUser as Doc<"users"> };
}
let userId: Id<"users"> | null | undefined = null;
try {
userId = await getAuthUserId(ctx);
} catch {
userId = null;
}
userId ??= await getDevImpersonatedUserIdFromAction(ctx);
if (!userId) throw new Error("Unauthorized");
let user: Doc<"users"> | null;
try {
+1 -1
View File
@@ -34,7 +34,7 @@ export const MISSING_API_TOKEN_MESSAGE =
export const INVALID_API_TOKEN_MESSAGE =
"Unauthorized: API token is invalid or revoked. Run `clawhub login` again.";
export const BLOCKED_API_TOKEN_ACCOUNT_MESSAGE =
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, contact security@openclaw.ai.";
"Unauthorized: This ClawHub account is not in good standing and cannot use API tokens. If you believe this is a mistake, open a GitHub issue: https://github.com/openclaw/clawhub/issues/new.";
export async function requireApiTokenUser(
ctx: ActionCtx,
-100
View File
@@ -1,100 +0,0 @@
import { describe, expect, it } from "vitest";
import {
depRegistryUrl,
mergeDepRegistryFinding,
parseDependencyFile,
summarizeDepRegistryChecks,
} from "./depRegistryScan";
import { summarizeReasonCodes, verdictFromCodes } from "./moderationReasonCodes";
describe("depRegistryScan", () => {
it("parses registry dependency manifests and skips vendored or non-registry specs", () => {
expect(
parseDependencyFile(
"package.json",
JSON.stringify({
dependencies: {
"@types/node": "^24.0.0",
local: "file:../local",
remote: "github:owner/repo",
},
optionalDependencies: {
undici: "^7.0.0",
},
}),
),
).toEqual([
{ name: "@types/node", registry: "npm", source: "package.json" },
{ name: "undici", registry: "npm", source: "package.json" },
]);
expect(
parseDependencyFile("vendor/package.json", '{"dependencies":{"phantom":"1.0.0"}}'),
).toEqual([]);
expect(
parseDependencyFile(
"requirements.txt",
["requests>=2", "demo @ git+https://example.test/demo.git", "-r dev.txt"].join("\n"),
),
).toEqual([{ name: "requests", registry: "pypi", source: "requirements.txt" }]);
});
it("keeps npm scope names compatible with registry URL lookup", () => {
expect(depRegistryUrl("npm", "@types/node")).toBe("https://registry.npmjs.org/@types%2Fnode");
});
it("does not produce clean status when registry lookups are unresolved", () => {
const analysis = summarizeDepRegistryChecks({
checkedAt: 123,
results: [{ name: "requests", registry: "pypi", source: "requirements.txt", exists: true }],
unresolved: [
{
name: "maybe-real",
registry: "npm",
source: "package.json",
reason: "network error",
},
],
});
expect(analysis.status).toBe("error");
expect(analysis.notFoundPackages).toEqual([]);
expect(analysis.unresolvedPackages).toEqual(["maybe-real (npm)"]);
});
it("injects a static finding only for confirmed missing packages", () => {
const suspicious = summarizeDepRegistryChecks({
checkedAt: 456,
results: [
{
name: "phantom-package-xyz",
registry: "npm",
source: "package.json",
exists: false,
httpStatus: 404,
},
],
unresolved: [],
});
const merged = mergeDepRegistryFinding({
staticScan: undefined,
analysis: suspicious,
statusFromCodes: verdictFromCodes,
summarizeCodes: summarizeReasonCodes,
});
expect(merged.status).toBe("suspicious");
expect(merged.reasonCodes).toEqual(["suspicious.dep_not_found_on_registry"]);
expect(merged.findings[0]?.file).toBe("Dependency manifests");
const cleanAgain = mergeDepRegistryFinding({
staticScan: merged,
analysis: summarizeDepRegistryChecks({ checkedAt: 789, results: [], unresolved: [] }),
statusFromCodes: verdictFromCodes,
summarizeCodes: summarizeReasonCodes,
});
expect(cleanAgain.status).toBe("clean");
expect(cleanAgain.findings).toEqual([]);
});
});
-321
View File
@@ -1,321 +0,0 @@
import {
MODERATION_ENGINE_VERSION,
REASON_CODES,
type ModerationFinding,
type ModerationVerdict,
} from "./moderationReasonCodes";
export const SUPPORTED_DEP_REGISTRIES = ["pypi", "npm", "cargo"] as const;
export type SupportedDepRegistry = (typeof SUPPORTED_DEP_REGISTRIES)[number];
export type DepRegistryStatus = "clean" | "suspicious" | "error";
export type DepEntry = {
name: string;
registry: SupportedDepRegistry;
source: string;
};
export type DepRegistryResult = DepEntry & {
exists: boolean;
httpStatus?: number;
};
export type DepRegistryUnresolved = DepEntry & {
reason: string;
};
export type DepRegistryAnalysis = {
status: DepRegistryStatus;
results: DepRegistryResult[];
notFoundPackages: string[];
unresolvedPackages: string[];
summary: string;
checkedAt: number;
};
const DEP_FILE_PARSERS: Record<string, (content: string, path: string) => DepEntry[]> = {
"requirements.txt": parseRequirementsTxt,
"requirements-dev.txt": parseRequirementsTxt,
"requirements_dev.txt": parseRequirementsTxt,
"requirements-test.txt": parseRequirementsTxt,
"requirements_test.txt": parseRequirementsTxt,
"package.json": parsePackageJson,
"cargo.toml": parseCargoToml,
"pyproject.toml": parsePyprojectToml,
};
const NON_REGISTRY_NPM_SPEC_PREFIXES = [
"file:",
"link:",
"git+",
"git://",
"github:",
"bitbucket:",
"gist:",
"http:",
"https:",
"workspace:",
"npm:",
];
const VENDORED_PATH_PATTERNS = [
/(^|\/)node_modules\//,
/(^|\/)vendor\//,
/(^|\/)__pycache__\//,
/(^|\/)\.venv\//,
/(^|\/)venv\//,
/(^|\/)target\//,
/(^|\/)\.cargo\//,
/(^|\/)dist\//,
/(^|\/)build\//,
];
function normalizeName(name: string, registry: SupportedDepRegistry) {
const normalized = name.trim().toLowerCase();
return registry === "cargo" ? normalized.replaceAll("_", "-") : normalized;
}
export function isVendoredDependencyPath(path: string) {
return VENDORED_PATH_PATTERNS.some((pattern) => pattern.test(path));
}
export function parseDependencyFile(path: string, content: string): DepEntry[] {
if (isVendoredDependencyPath(path)) return [];
const basename = path.split("/").pop()?.toLowerCase() ?? "";
const parser = DEP_FILE_PARSERS[basename];
return parser ? dedupeDeps(parser(content, path)) : [];
}
export function dedupeDeps(entries: DepEntry[]) {
const seen = new Set<string>();
return entries.filter((entry) => {
const key = `${entry.registry}:${entry.name}`;
if (seen.has(key)) return false;
seen.add(key);
return true;
});
}
function stripInlineComment(line: string) {
return line.replace(/\s+#.*$/, "").trim();
}
function parseRequirementsTxt(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
for (const rawLine of content.split("\n")) {
const line = stripInlineComment(rawLine);
if (!line || line.startsWith("-")) continue;
if (/^(?:git\+|https?:|file:|\.{0,2}\/)/i.test(line)) continue;
if (/\s@\s/.test(line)) continue;
const match = line.match(/^([a-zA-Z0-9_][a-zA-Z0-9._-]*)/);
if (!match) continue;
entries.push({ name: normalizeName(match[1], "pypi"), registry: "pypi", source: path });
}
return entries;
}
function parsePackageJson(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
let pkg: Record<string, unknown>;
try {
pkg = JSON.parse(content) as Record<string, unknown>;
} catch {
return entries;
}
for (const field of ["dependencies", "devDependencies", "optionalDependencies"]) {
const deps = pkg[field];
if (!deps || typeof deps !== "object" || Array.isArray(deps)) continue;
for (const [rawName, rawSpec] of Object.entries(deps as Record<string, unknown>)) {
const spec = typeof rawSpec === "string" ? rawSpec.trim().toLowerCase() : "";
if (NON_REGISTRY_NPM_SPEC_PREFIXES.some((prefix) => spec.startsWith(prefix))) continue;
entries.push({ name: normalizeName(rawName, "npm"), registry: "npm", source: path });
}
}
return entries;
}
function parseCargoToml(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
let inDepSection = false;
for (const rawLine of content.split("\n")) {
const line = stripInlineComment(rawLine);
if (/^\[.*\]$/.test(line)) {
const section = line.replace(/[[\]\s]/g, "").toLowerCase();
inDepSection =
section === "dependencies" ||
section === "dev-dependencies" ||
section === "build-dependencies";
continue;
}
if (!inDepSection || !line) continue;
const match = line.match(/^([a-zA-Z0-9_][a-zA-Z0-9_-]*)\s*=/);
if (!match) continue;
entries.push({ name: normalizeName(match[1], "cargo"), registry: "cargo", source: path });
}
return entries;
}
function parsePyprojectToml(content: string, path: string): DepEntry[] {
const entries: DepEntry[] = [];
let inDepArray = false;
let inPoetryDepTable = false;
for (const rawLine of content.split("\n")) {
const line = stripInlineComment(rawLine);
if (/^\[.*\]$/.test(line)) {
inDepArray = false;
const section = line.replace(/[[\]\s]/g, "").toLowerCase();
inPoetryDepTable =
section === "tool.poetry.dependencies" ||
section === "tool.poetry.dev-dependencies" ||
section === "tool.poetry.group.dev.dependencies";
continue;
}
if (/^dependencies\s*=\s*\[/.test(line)) {
inDepArray = true;
const inline = line.match(/\[\s*(.*)\s*\]/);
if (inline) {
for (const item of extractQuotedStrings(inline[1])) addPyPiDependency(entries, item, path);
inDepArray = false;
}
continue;
}
if (inDepArray) {
if (line === "]") {
inDepArray = false;
continue;
}
const quoted = line.match(/^["']([^"']+)["']/);
if (quoted) addPyPiDependency(entries, quoted[1], path);
continue;
}
if (!inPoetryDepTable || !line) continue;
const match = line.match(/^([a-zA-Z0-9_][a-zA-Z0-9._-]*)\s*=/);
if (!match || match[1].toLowerCase() === "python") continue;
entries.push({ name: normalizeName(match[1], "pypi"), registry: "pypi", source: path });
}
return entries;
}
function addPyPiDependency(entries: DepEntry[], spec: string, path: string) {
if (/\s@\s/.test(spec)) return;
const match = spec.match(/^([a-zA-Z0-9_][a-zA-Z0-9._-]*)/);
if (!match) return;
entries.push({ name: normalizeName(match[1], "pypi"), registry: "pypi", source: path });
}
function extractQuotedStrings(s: string) {
return [...s.matchAll(/["']([^"']+)["']/g)].map((match) => match[1]);
}
export function depRegistryUrl(registry: SupportedDepRegistry, packageName: string) {
const encoded =
registry === "npm" && packageName.startsWith("@")
? `@${encodeURIComponent(packageName.slice(1))}`
: encodeURIComponent(packageName);
if (registry === "pypi") return `https://pypi.org/pypi/${encoded}/json`;
if (registry === "npm") return `https://registry.npmjs.org/${encoded}`;
return `https://crates.io/api/v1/crates/${encoded}`;
}
export function summarizeDepRegistryChecks(params: {
results: DepRegistryResult[];
unresolved: DepRegistryUnresolved[];
checkedAt?: number;
}): DepRegistryAnalysis {
const notFound = params.results.filter((result) => !result.exists);
const notFoundPackages = notFound.map((result) => `${result.name} (${result.registry})`);
const unresolvedPackages = params.unresolved.map(
(result) => `${result.name} (${result.registry})`,
);
const checkedAt = params.checkedAt ?? Date.now();
if (notFoundPackages.length > 0) {
const partial =
unresolvedPackages.length > 0
? ` ${unresolvedPackages.length} package(s) could not be checked and will be retried.`
: "";
return {
status: "suspicious",
results: params.results,
notFoundPackages,
unresolvedPackages,
summary: `${notFoundPackages.length} declared dependency package(s) were not found on their public registry: ${notFoundPackages.join(", ")}.${partial}`,
checkedAt,
};
}
if (unresolvedPackages.length > 0) {
return {
status: "error",
results: params.results,
notFoundPackages: [],
unresolvedPackages,
summary: `${unresolvedPackages.length} dependency package(s) could not be verified due to registry lookup errors. The scan will be retried.`,
checkedAt,
};
}
return {
status: "clean",
results: params.results,
notFoundPackages: [],
unresolvedPackages: [],
summary: `All ${params.results.length} declared dependency package(s) verified as present on their public registries.`,
checkedAt,
};
}
export function buildDepRegistryFinding(analysis: DepRegistryAnalysis): ModerationFinding | null {
if (analysis.status !== "suspicious" || analysis.notFoundPackages.length === 0) return null;
return {
code: REASON_CODES.DEP_NOT_FOUND,
severity: "critical",
file: "Dependency manifests",
line: 1,
message: `${analysis.notFoundPackages.length} package(s) referenced in dependency files do not exist on their public registries: ${analysis.notFoundPackages.join(", ")}`,
evidence:
"An attacker could register these phantom package names and inject malicious install-time code through dependency confusion.",
};
}
export function mergeDepRegistryFinding(params: {
staticScan:
| {
status: ModerationVerdict;
reasonCodes: string[];
findings: ModerationFinding[];
summary: string;
engineVersion: string;
checkedAt: number;
}
| undefined;
analysis: DepRegistryAnalysis;
statusFromCodes: (codes: string[]) => ModerationVerdict;
summarizeCodes: (codes: string[]) => string;
}) {
const base = params.staticScan ?? {
status: "clean" as ModerationVerdict,
reasonCodes: [],
findings: [],
summary: "No suspicious patterns detected.",
engineVersion: MODERATION_ENGINE_VERSION,
checkedAt: params.analysis.checkedAt,
};
const findings = base.findings.filter((finding) => finding.code !== REASON_CODES.DEP_NOT_FOUND);
const depFinding = buildDepRegistryFinding(params.analysis);
if (depFinding) findings.push(depFinding);
const reasonCodes = Array.from(new Set(findings.map((finding) => finding.code))).sort((a, b) =>
a.localeCompare(b),
);
return {
...base,
status: params.statusFromCodes(reasonCodes),
reasonCodes,
findings,
summary: params.summarizeCodes(reasonCodes),
checkedAt: params.analysis.checkedAt,
};
}
+67 -3
View File
@@ -1,6 +1,8 @@
import { describe, expect, it } from "vitest";
import { isLocalDevAuthEnabled } from "./devAuth";
const CLOUD_DEV_AUTH_SECRET = "dev-auth-secret-with-enough-entropy-123";
describe("isLocalDevAuthEnabled", () => {
it("requires the explicit dev auth flag", () => {
expect(
@@ -41,16 +43,78 @@ describe("isLocalDevAuthEnabled", () => {
).toBe(true);
});
it("rejects cloud dev deployments even when the dev auth flag is set", () => {
it("allows cloud dev deployments with an explicit localhost site and matching secret", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
},
CLOUD_DEV_AUTH_SECRET,
),
).toBe(true);
});
it("allows cloud dev deployments from the fallback marker when Convex deployment is blank", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_DEPLOYMENT: "",
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
},
CLOUD_DEV_AUTH_SECRET,
),
).toBe(true);
});
it("rejects cloud dev deployments when the secret is missing", () => {
expect(
isLocalDevAuthEnabled({
CONVEX_SITE_URL: "http://127.0.0.1:3211",
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
}),
).toBe(false);
});
it("rejects cloud dev deployments when the configured secret is too short", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_SITE_URL: "https://clever-rabbit-123.convex.cloud",
DEV_AUTH_CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: "short",
DEV_AUTH_SITE_URL: "http://127.0.0.1:3211",
},
"short",
),
).toBe(false);
});
it("rejects cloud dev deployments without an explicit localhost dev auth site", () => {
expect(
isLocalDevAuthEnabled(
{
CONVEX_SITE_URL: "http://127.0.0.1:3211",
DEV_AUTH_ENABLED: "1",
DEV_AUTH_SECRET: CLOUD_DEV_AUTH_SECRET,
CONVEX_DEPLOYMENT: "dev:clever-rabbit-123",
},
CLOUD_DEV_AUTH_SECRET,
),
).toBe(false);
});
it("rejects localhost site URLs without a local deployment marker", () => {
expect(
isLocalDevAuthEnabled({
+33 -3
View File
@@ -3,18 +3,48 @@ type DevAuthEnv = {
CONVEX_SITE_URL?: string;
DEV_AUTH_CONVEX_DEPLOYMENT?: string;
DEV_AUTH_ENABLED?: string;
DEV_AUTH_SECRET?: string;
DEV_AUTH_SITE_URL?: string;
};
export function isLocalDevAuthEnabled(env: DevAuthEnv = process.env) {
const MIN_CLOUD_DEV_AUTH_SECRET_LENGTH = 32;
export function isLocalDevAuthEnabled(env: DevAuthEnv = process.env, suppliedSecret?: string) {
if (env.DEV_AUTH_ENABLED !== "1") return false;
const deployment = env.CONVEX_DEPLOYMENT?.trim() || env.DEV_AUTH_CONVEX_DEPLOYMENT?.trim() || "";
return isLocalConvexDeployment(deployment) && isLocalhostUrl(env.CONVEX_SITE_URL);
const convexDeployment = env.CONVEX_DEPLOYMENT?.trim();
const devAuthDeployment = env.DEV_AUTH_CONVEX_DEPLOYMENT?.trim();
const deployment = convexDeployment || devAuthDeployment || "";
if (isLocalConvexDeployment(deployment)) {
return isLocalhostUrl(env.CONVEX_SITE_URL);
}
if (isDevConvexDeployment(deployment)) {
return isLocalhostUrl(env.DEV_AUTH_SITE_URL) && hasValidCloudDevAuthSecret(env, suppliedSecret);
}
return false;
}
function isLocalConvexDeployment(deployment: string) {
return deployment.startsWith("local:") || deployment.startsWith("anonymous:");
}
function isDevConvexDeployment(deployment: string) {
return deployment.startsWith("dev:");
}
function hasValidCloudDevAuthSecret(env: DevAuthEnv, suppliedSecret: string | undefined) {
const expected = env.DEV_AUTH_SECRET?.trim();
const actual = suppliedSecret?.trim();
return Boolean(
expected &&
actual &&
expected.length >= MIN_CLOUD_DEV_AUTH_SECRET_LENGTH &&
actual === expected,
);
}
function isLocalhostUrl(value: string | undefined) {
if (!value) return false;
try {
+18
View File
@@ -0,0 +1,18 @@
export function assertLocalDevSeedAllowed(seedName: string): void {
const deployment =
process.env.CONVEX_DEPLOYMENT?.trim() || process.env.DEV_AUTH_CONVEX_DEPLOYMENT?.trim() || "";
if (
deployment.startsWith("dev:") ||
deployment.startsWith("local:") ||
deployment.startsWith("anonymous:")
) {
return;
}
if (
!deployment &&
(process.env.DEV_AUTH_ENABLED === "1" || process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION === "1")
) {
return;
}
throw new Error(`${seedName} dev seed is disabled outside local/dev deployments`);
}
+253
View File
@@ -0,0 +1,253 @@
import { describe, expect, it } from "vitest";
import {
APPEALS_URL,
buildMaliciousArtifactEmail,
buildBanNotificationEmail,
buildPackageInspectorFindingsEmail,
buildRestoredAccountEmail,
} from "./emails";
describe("moderation notification email copy", () => {
it("builds public-safe malicious skill context with appeal but no local scan guidance", () => {
const email = buildBanNotificationEmail({
handle: "gingiris",
source: "autoban",
reason: "malicious.llm_malicious",
artifact: { kind: "skill", name: "gingiris-launch" },
trigger: "scanner.llm.malicious",
});
expect(email.subject).toBe("Your ClawHub account was disabled");
expect(email.context).toMatchObject({
appealUrl: APPEALS_URL,
artifact: { kind: "skill", name: "gingiris-launch" },
scannerLabel: "ClawScan",
findingSummary: "ClawScan classified the uploaded skill as malicious.",
});
expect(email.text).toContain("Skill: gingiris-launch");
expect(email.text).not.toContain("Scanner:");
expect(email.html).not.toContain("<strong>Scanner:</strong>");
expect(email.text).not.toContain("republishing");
expect(email.html).not.toContain("republishing");
expect(email.text).not.toContain("To support your appeal, include scan results");
expect(email.html).not.toContain("Include scan results with your appeal");
expect(email.text).toContain("Appeal: https://appeals.openclaw.ai/");
expect(email.html).not.toContain("If you already appealed");
expect(email.html).not.toContain("separate support email");
expect(email.text).not.toContain("clawhub scan ./my-skill --output clawhub-scan.zip");
expect(email.text).not.toContain("https://docs.openclaw.ai/clawhub/cli#scan-path");
});
it("does not leak raw manual moderator notes into outbound email", () => {
const email = buildBanNotificationEmail({
handle: "target",
source: "manual",
reason: "internal reviewer note: reporter=user_123 secret finding id=abc",
});
expect(email.context.findingSummary).toBe(
"ClawHub staff disabled the account after a security review.",
);
expect(email.text).not.toContain("internal reviewer note");
expect(email.text).not.toContain("reporter=user_123");
expect(email.html).not.toContain("secret finding id");
});
it("uses rate-limit copy without scan remediation guidance", () => {
const email = buildBanNotificationEmail({
handle: "publish-loop",
source: "manual",
reason: "rate limit triggered by automated CLI publishing",
});
expect(email.context).toMatchObject({
scannerLabel: null,
findingSummary: "Publishing automation triggered ClawHub rate-limit abuse controls.",
});
expect(email.text).toContain("Publishing automation");
expect(email.text).not.toContain("clawhub scan");
expect(email.text).not.toContain("Include scan results");
expect(email.html).not.toContain("Include scan results");
expect(email.html).not.toContain("fixed local copy");
});
it("builds restored-account copy that explains tokens stay revoked", () => {
const email = buildRestoredAccountEmail({
handle: "restored",
restoredListings: [
{ kind: "skill", name: "safe-one" },
{ kind: "plugin", name: "@scope/demo" },
],
});
expect(email.subject).toBe("Your ClawHub account was restored");
expect(email.text).toContain("Your ClawHub account can sign in again.");
expect(email.text).toContain("Skill: safe-one");
expect(email.text).toContain("Plugin: @scope/demo");
expect(email.text).toContain("Previously revoked API tokens stay revoked.");
});
it("builds malicious artifact copy without account appeal language", () => {
const email = buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "skill", name: "demo-skill" },
version: "1.2.3",
trigger: "malicious.llm_malicious",
findingSummary: "Attempts to exfiltrate credentials.",
});
expect(email.subject).toBe("ClawHub blocked a skill version");
expect(email.text).toContain("Reason: Attempts to exfiltrate credentials.");
expect(email.html).toContain("Attempts to exfiltrate credentials.");
expect(email.text).toContain("Skill: demo-skill");
expect(email.text).toContain("Version: 1.2.3");
expect(email.text).toContain("clawhub scan download demo-skill --version 1.2.3");
expect(email.text).toContain("Increment the version number before uploading the fixed skill.");
expect(email.text).toContain("https://docs.openclaw.ai/clawhub/moderation");
expect(email.text).not.toContain("clawhub scan ./my-skill --output clawhub-scan.zip");
expect(email.text).not.toContain("fixed local copy");
expect(email.text).toContain("Repeated malicious rejections may lead to account disablement");
expect(email.html).toContain("Repeated malicious rejections may lead to account disablement");
expect(email.text).not.toContain(APPEALS_URL);
expect(email.html).not.toContain(APPEALS_URL);
expect(email.html).not.toContain("appeal this decision");
});
it("falls back to generic malicious artifact copy when no ClawScan summary is available", () => {
const email = buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "skill", name: "demo-skill" },
version: "1.2.3",
trigger: "malicious.llm_malicious",
});
expect(email.text).toContain("Reason: ClawScan classified the uploaded artifact as malicious.");
});
it("keeps supplied ClawScan summaries to one email-safe line", () => {
const email = buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "skill", name: "demo-skill" },
version: "1.2.3",
findingSummary: ` ${"credential exfiltration ".repeat(30)}\nwith hidden tooling `,
});
const reasonLine = email.text.split("\n").find((line) => line.startsWith("Reason: "));
expect(reasonLine).toBeDefined();
expect(reasonLine).not.toContain("\n");
expect(reasonLine?.length).toBeLessThanOrEqual("Reason: ".length + 280);
expect(reasonLine).toContain("...");
});
it("builds plugin scan download copy with an explicit artifact kind", () => {
const email = buildMaliciousArtifactEmail({
handle: "publisher",
artifact: { kind: "plugin", name: "@scope/demo" },
version: "2.0.0",
trigger: "malicious.static",
});
expect(email.text).toContain("Plugin: @scope/demo");
expect(email.text).toContain("clawhub scan download @scope/demo --version 2.0.0 --kind plugin");
expect(email.text).toContain("Increment the version number before uploading the fixed plugin.");
});
it("builds plugin inspector warning copy with local validation guidance", () => {
const email = buildPackageInspectorFindingsEmail({
handle: "octocat",
packageName: "demo-plugin",
version: "1.0.0",
findings: [
{
findingKind: "warning",
code: "legacy-before-agent-start",
issueClass: "deprecation-warning",
severity: "P2",
message: "legacy before_agent_start hook is deprecated",
inspectorVersion: "0.4.0",
targetOpenClawVersion: "0.9.0",
scanSource: "publish",
authorRemediation: {
summary: "Replace the legacy before_agent_start hook with current prompt hooks.",
docsUrl:
"https://docs.openclaw.ai/clawhub/plugin-validation-fixes#legacy-before-agent-start",
},
},
],
});
expect(email.subject).toBe("Plugin Inspector findings for demo-plugin@1.0.0");
expect(email.text).toContain("Hi octocat,");
expect(email.text).toContain("We found 1 issue with version 1.0.0 of demo-plugin.");
expect(email.text).toContain("OpenClaw Version: 0.9.0");
expect(email.text).toContain("Address the findings below in your plugin package.");
expect(email.text).toContain("Run the validation command locally against your changes.");
expect(email.text).toContain("clawhub package validate <path-to-plugin>");
expect(email.text).toContain(
"- **WARNING** `legacy-before-agent-start` (deprecation-warning, P2)",
);
expect(email.text).toContain(" legacy before_agent_start hook is deprecated");
expect(email.text).toContain(" Fix:");
expect(email.text).toContain(
" Replace the legacy before_agent_start hook with current prompt hooks.",
);
expect(email.text).toContain(" Docs:");
expect(email.text).toContain(
" https://docs.openclaw.ai/clawhub/plugin-validation-fixes#legacy-before-agent-start",
);
expect(email.text).not.toContain("ClawHub Security");
expect(email.html).toContain("Validate a local fix");
expect(email.html).toContain("Hi octocat,");
expect(email.html).toContain("<strong>OpenClaw Version:</strong> 0.9.0");
expect(email.html).toContain("clawhub package validate &lt;path-to-plugin&gt;");
expect(email.html).toContain("legacy-before-agent-start");
expect(email.html).toContain("deprecation-warning · P2");
expect(email.html).toContain("Fix");
expect(email.html).toContain("Replace the legacy before_agent_start hook");
expect(email.html).toContain("<strong>Docs</strong>");
expect(email.html).toContain("plugin-validation-fixes#legacy-before-agent-start");
expect(email.html).not.toContain("plugin validation fix docs");
expect(email.html).not.toContain("ClawHub Security");
expect(email.text).not.toContain("Plugin Inspector: 0.4.0");
expect(email.text).not.toContain("Target OpenClaw:");
expect(email.html).not.toContain("<strong>Plugin Inspector:</strong>");
expect(email.html).not.toContain("<strong>Target OpenClaw:</strong>");
expect(email.html).not.toContain("Review:");
expect(email.html).not.toContain("plugin validation findings");
expect(email.html).not.toContain("https://clawhub.ai/plugins/demo-plugin#validation");
expect(email.html).not.toContain("Your plugin was published");
expect(email.html).not.toContain("published successfully");
});
it("builds plugin inspector error copy without publish-time wording", () => {
const email = buildPackageInspectorFindingsEmail({
packageName: "demo-plugin",
version: "1.0.1",
findings: [
{
findingKind: "error",
code: "missing-expected-seam",
issueClass: "compatibility-error",
severity: "P0",
level: "breakage",
message: "registerTool is no longer available",
inspectorVersion: "0.5.0",
targetOpenClawVersion: "0.10.0",
scanSource: "nightly",
},
],
});
expect(email.text).toContain("We found 1 issue with version 1.0.1 of demo-plugin.");
expect(email.text).toContain("Address the findings below in your plugin package.");
expect(email.text).toContain("Run the validation command locally against your changes.");
expect(email.text).toContain("clawhub package validate <path-to-plugin>");
expect(email.text).toContain("- **ERROR** `missing-expected-seam` (compatibility-error, P0)");
expect(email.text).not.toContain("Your plugin was published");
expect(email.text).not.toContain("was published, but");
expect(email.text).not.toContain("Some findings are errors");
expect(email.text).not.toContain("nightly");
expect(email.html).toContain("missing-expected-seam");
expect(email.html).toContain("compatibility-error · P0");
});
});
+504
View File
@@ -0,0 +1,504 @@
export const APPEALS_URL = "https://appeals.openclaw.ai/";
export const MODERATION_GUIDELINES_URL = "https://docs.openclaw.ai/clawhub/moderation";
export const MALICIOUS_REJECTION_ACCOUNT_WARNING =
"Repeated malicious rejections may lead to account disablement.";
const MAX_EMAIL_FINDING_SUMMARY_LENGTH = 280;
export type NotificationArtifact = {
kind: "skill" | "plugin";
name: string;
};
export type BanNotificationSource = "manual" | "autoban";
export type BanNotificationEmailArgs = {
handle?: string;
source: BanNotificationSource;
reason?: string;
trigger?: string;
artifact?: NotificationArtifact;
};
export type BanNotificationEmailContext = {
appealUrl: typeof APPEALS_URL;
artifact: NotificationArtifact | null;
scannerLabel: string | null;
findingSummary: string;
};
export type TransactionalEmail = {
subject: string;
context: BanNotificationEmailContext;
text: string;
html: string;
};
export type RestoredAccountEmailArgs = {
handle?: string;
restoredListings?: NotificationArtifact[];
};
export type MaliciousArtifactEmailArgs = {
handle?: string;
artifact: NotificationArtifact;
version?: string;
trigger?: string;
findingSummary?: string;
};
export type PackageInspectorEmailFinding = {
findingKind: "warning" | "error";
code: string;
issueClass?: string;
level?: string;
severity?: string;
message: string;
authorRemediation?: {
summary: string;
docsUrl?: string;
};
inspectorVersion?: string;
targetOpenClawVersion?: string;
scanSource?: "publish" | "nightly";
};
export type PackageInspectorFindingsEmailArgs = {
handle?: string;
packageName: string;
version: string;
findings: PackageInspectorEmailFinding[];
};
type BanReasonSummary = {
scannerLabel: string | null;
findingSummary: string;
};
function escapeHtml(value: string) {
return value
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
function normalizeReasonInput(args: Pick<BanNotificationEmailArgs, "reason" | "trigger">) {
return `${args.reason ?? ""} ${args.trigger ?? ""}`.trim().toLowerCase();
}
function summarizeBanReason(args: BanNotificationEmailArgs): BanReasonSummary {
const normalized = normalizeReasonInput(args);
if (args.source === "autoban") {
if (normalized.includes("virustotal") || normalized.includes("virus_total")) {
return {
scannerLabel: "VirusTotal",
findingSummary: "VirusTotal telemetry contributed to a malicious upload finding.",
};
}
if (normalized.includes("static")) {
return {
scannerLabel: "Static analysis",
findingSummary: "Static analysis flagged malicious upload patterns.",
};
}
if (
normalized.includes("clawscan") ||
normalized.includes("llm") ||
normalized.includes("malicious")
) {
return {
scannerLabel: "ClawScan",
findingSummary: "ClawScan classified the uploaded skill as malicious.",
};
}
return {
scannerLabel: "ClawHub security checks",
findingSummary: "ClawHub security checks classified the uploaded skill as malicious.",
};
}
if (/rate[-\s]?limit|publishing automation|automated(?: cli)? publishing/.test(normalized)) {
return {
scannerLabel: null,
findingSummary: "Publishing automation triggered ClawHub rate-limit abuse controls.",
};
}
return {
scannerLabel: null,
findingSummary: "ClawHub staff disabled the account after a security review.",
};
}
function artifactLabel(artifact: NotificationArtifact) {
return `${artifact.kind === "skill" ? "Skill" : "Plugin"}: ${artifact.name}`;
}
function greeting(handle: string | undefined) {
return handle?.trim() ? `Hi ${handle.trim()},` : "Hi,";
}
function emailShell(args: { preheader: string; title: string; body: string }) {
return `<!doctype html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>${escapeHtml(args.title)}</title>
</head>
<body style="margin:0;background:#ffffff;color:#1f2328;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif;">
<span style="display:none!important;visibility:hidden;opacity:0;color:transparent;height:0;width:0;overflow:hidden;">${escapeHtml(
args.preheader,
)}</span>
<table role="presentation" width="100%" cellspacing="0" cellpadding="0" style="background:#ffffff;margin:0;padding:24px 16px;">
<tr>
<td align="center">
<table role="presentation" width="100%" cellspacing="0" cellpadding="0" style="max-width:640px;background:#ffffff;">
<tr>
<td style="padding:0;font-size:15px;line-height:22px;color:#1f2328;">
${args.body}
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>`;
}
function textLink(href: string, label: string) {
return `<a href="${escapeHtml(href)}" style="color:#0969da;text-decoration:underline;">${escapeHtml(label)}</a>`;
}
function detailLine(label: string, value: string) {
return `<p style="margin:0 0 6px;font-size:15px;line-height:22px;color:#1f2328;"><strong>${escapeHtml(label)}:</strong> ${escapeHtml(value)}</p>`;
}
function sectionHeading(label: string) {
return `<p style="margin:18px 0 8px;font-size:15px;line-height:22px;color:#1f2328;"><strong>${escapeHtml(label)}</strong></p>`;
}
function paragraph(value: string) {
return `<p style="margin:0 0 14px;font-size:15px;line-height:22px;color:#1f2328;">${escapeHtml(value)}</p>`;
}
function bulletList(items: string[]) {
return `<ul style="margin:0 0 14px;padding-left:22px;font-size:15px;line-height:22px;color:#1f2328;">${items
.map((item) => `<li style="margin:0 0 6px;">${escapeHtml(item)}</li>`)
.join("")}</ul>`;
}
function commandBlock(command: string) {
return `<pre style="margin:8px 0 14px;padding:10px 12px;background:#f6f8fa;border:1px solid #d8dee4;border-radius:6px;white-space:pre-wrap;color:#1f2328;font-family:ui-monospace,SFMono-Regular,Consolas,'Liberation Mono',monospace;font-size:13px;line-height:20px;"><code>${escapeHtml(command)}</code></pre>`;
}
function buildScanDownloadCommand(args: MaliciousArtifactEmailArgs) {
const version = args.version?.trim() || "<version>";
const kindFlag = args.artifact.kind === "plugin" ? " --kind plugin" : "";
return `clawhub scan download ${args.artifact.name} --version ${version}${kindFlag}`;
}
function buildPluginValidateCommand() {
return "clawhub package validate <path-to-plugin>";
}
function normalizeEmailFindingSummary(value: string | undefined) {
const normalized = value?.replace(/\s+/g, " ").trim();
if (!normalized) return undefined;
if (normalized.length <= MAX_EMAIL_FINDING_SUMMARY_LENGTH) return normalized;
return `${normalized.slice(0, MAX_EMAIL_FINDING_SUMMARY_LENGTH - 3).trimEnd()}...`;
}
export function buildBanNotificationEmail(args: BanNotificationEmailArgs): TransactionalEmail {
const summary = summarizeBanReason(args);
const artifact = args.artifact ?? null;
const context: BanNotificationEmailContext = {
appealUrl: APPEALS_URL,
artifact,
scannerLabel: summary.scannerLabel,
findingSummary: summary.findingSummary,
};
const lines = [
greeting(args.handle),
"",
"Your ClawHub account was disabled.",
`Reason: ${context.findingSummary}`,
];
if (artifact) lines.push(artifactLabel(artifact));
lines.push(
"",
"What changed:",
"- Your ClawHub account cannot sign in.",
"- Existing API tokens for the account have been revoked.",
"- Published listings owned by the account may be hidden from public view.",
"",
`Appeal: ${APPEALS_URL}`,
);
lines.push("", "ClawHub Security");
const impactItems = [
"Your ClawHub account cannot sign in.",
"Existing API tokens for the account have been revoked.",
"Published listings owned by the account may be hidden from public view.",
];
const detailLines = [
detailLine("Reason", context.findingSummary),
...(artifact
? [detailLine(artifact.kind === "skill" ? "Skill" : "Plugin", artifact.name)]
: []),
].join("");
const html = emailShell({
title: "Your ClawHub account was disabled",
preheader: context.findingSummary,
body: [
paragraph(greeting(args.handle)),
paragraph("We disabled your ClawHub account after an account-safety review."),
detailLines,
sectionHeading("What changed"),
bulletList(impactItems),
`<p style="margin:0 0 14px;font-size:15px;line-height:22px;color:#1f2328;">You can ${textLink(APPEALS_URL, "appeal this decision")} if you believe this was a mistake.</p>`,
paragraph("ClawHub Security"),
].join(""),
});
return {
subject: "Your ClawHub account was disabled",
context,
text: lines.join("\n"),
html,
};
}
export function buildRestoredAccountEmail(args: RestoredAccountEmailArgs) {
const restoredListings = args.restoredListings ?? [];
const listingLines = restoredListings.map(artifactLabel);
const lines = [
greeting(args.handle),
"",
"Your ClawHub account can sign in again.",
"Previously revoked API tokens stay revoked. Create a new token before using the CLI or API again.",
];
if (listingLines.length > 0) {
lines.push("", "Restored listings:", ...listingLines);
}
lines.push("", "ClawHub Security");
const html = emailShell({
title: "Your ClawHub account was restored",
preheader: "Your ClawHub account can sign in again.",
body: [
paragraph(greeting(args.handle)),
paragraph("Your ClawHub account can sign in again."),
paragraph(
"Previously revoked API tokens stay revoked. Create a new token before using the CLI or API again.",
),
listingLines.length > 0
? `${sectionHeading("Restored listings")}${bulletList(listingLines)}`
: "",
`<p style="margin:0 0 14px;font-size:15px;line-height:22px;color:#1f2328;">Settings: ${textLink("https://clawhub.ai/settings", "open ClawHub settings")}</p>`,
paragraph("ClawHub Security"),
].join(""),
});
return {
subject: "Your ClawHub account was restored",
text: lines.join("\n"),
html,
};
}
export function buildMaliciousArtifactEmail(args: MaliciousArtifactEmailArgs) {
const artifactKind = args.artifact.kind === "skill" ? "skill" : "plugin";
const artifactLabelText = artifactLabel(args.artifact);
const scanDownloadCommand = buildScanDownloadCommand(args);
const findingSummary =
normalizeEmailFindingSummary(args.findingSummary) ??
(args.trigger?.includes("static") === true
? "Static analysis flagged malicious upload patterns."
: args.trigger?.includes("virustotal") === true || args.trigger?.includes("vt_") === true
? "VirusTotal telemetry contributed to a malicious upload finding."
: "ClawScan classified the uploaded artifact as malicious.");
const subject = `ClawHub blocked a ${artifactKind} version`;
const lines = [
greeting(args.handle),
"",
`ClawHub blocked a ${artifactKind} version after a security scan.`,
`Reason: ${findingSummary}`,
artifactLabelText,
];
if (args.version?.trim()) lines.push(`Version: ${args.version.trim()}`);
lines.push(
"",
"What changed:",
"- This version was not made public.",
"- Your account can still sign in.",
`- You can upload a fixed version of this ${artifactKind}.`,
`- ${MALICIOUS_REJECTION_ACCOUNT_WARNING}`,
"",
"Download the scan results for the blocked submitted version:",
scanDownloadCommand,
`Docs: ${MODERATION_GUIDELINES_URL}`,
`Increment the version number before uploading the fixed ${artifactKind}.`,
"",
"ClawHub Security",
);
const detailLines = [
detailLine("Reason", findingSummary),
detailLine(args.artifact.kind === "skill" ? "Skill" : "Plugin", args.artifact.name),
...(args.version?.trim() ? [detailLine("Version", args.version.trim())] : []),
].join("");
const html = emailShell({
title: subject,
preheader: `${artifactLabelText} was blocked by ClawHub security scans.`,
body: [
paragraph(greeting(args.handle)),
paragraph(`ClawHub blocked a ${artifactKind} version after a security scan.`),
detailLines,
sectionHeading("What changed"),
bulletList([
"This version was not made public.",
"Your account can still sign in.",
`You can upload a fixed version of this ${artifactKind}.`,
MALICIOUS_REJECTION_ACCOUNT_WARNING,
]),
sectionHeading("Review the blocked-version scan results"),
paragraph("Download the scan results for the blocked submitted version."),
commandBlock(scanDownloadCommand),
paragraph(`Increment the version number before uploading the fixed ${artifactKind}.`),
`<p style="margin:0 0 14px;font-size:15px;line-height:22px;color:#1f2328;">Docs: ${textLink(MODERATION_GUIDELINES_URL, "moderation and account safety")}</p>`,
paragraph("ClawHub Security"),
].join(""),
});
return {
subject,
text: lines.join("\n"),
html,
};
}
export function buildPackageInspectorFindingsEmail(args: PackageInspectorFindingsEmailArgs) {
const targetOpenClawVersion = args.findings.find(
(finding) => finding.targetOpenClawVersion,
)?.targetOpenClawVersion;
const validateCommand = buildPluginValidateCommand();
const subject = `Plugin Inspector findings for ${args.packageName}@${args.version}`;
const findingCount = args.findings.length;
const intro = `We found ${findingCount} ${findingCount === 1 ? "issue" : "issues"} with version ${args.version} of ${args.packageName}.`;
const nextSteps = [
"Address the findings below in your plugin package.",
"Run the validation command locally against your changes.",
"When validation passes, upload a new version.",
];
const findingLines = formatPackageInspectorFindingsText(args.findings);
const metadataLines = [
`Plugin: ${args.packageName}@${args.version}`,
targetOpenClawVersion ? `OpenClaw Version: ${targetOpenClawVersion}` : null,
].filter((line): line is string => line !== null);
const lines = [
greeting(args.handle),
"",
intro,
"",
...metadataLines,
"",
"Next steps:",
...nextSteps.map((item) => `- ${item}`),
"",
"Findings:",
...findingLines,
"",
"Validate a local fix:",
validateCommand,
];
const detailLines = [
detailLine("Plugin", `${args.packageName}@${args.version}`),
...(targetOpenClawVersion ? [detailLine("OpenClaw Version", targetOpenClawVersion)] : []),
].join("");
const html = emailShell({
title: "Plugin Inspector findings",
preheader: intro,
body: [
paragraph(greeting(args.handle)),
paragraph(intro),
detailLines,
sectionHeading("Next steps"),
bulletList(nextSteps),
sectionHeading("Findings"),
formatPackageInspectorFindingsHtml(args.findings),
sectionHeading("Validate a local fix"),
commandBlock(validateCommand),
].join(""),
});
return {
subject,
text: lines.join("\n"),
html,
};
}
function formatPackageInspectorFindingsText(findings: PackageInspectorEmailFinding[]) {
if (findings.length === 0) return ["- No findings were included."];
return findings.flatMap((finding) => {
const lines = [
`- **${finding.findingKind.toUpperCase()}** \`${finding.code}\`${formatFindingMetaText(finding)}`,
` ${finding.message}`,
];
if (finding.authorRemediation?.summary) {
lines.push(" Fix:");
lines.push(` ${finding.authorRemediation.summary}`);
if (finding.authorRemediation.docsUrl) {
lines.push(" Docs:");
lines.push(` ${finding.authorRemediation.docsUrl}`);
}
}
return lines;
});
}
function formatFindingMetaText(finding: PackageInspectorEmailFinding) {
const meta = [finding.issueClass, finding.severity].filter(Boolean).join(", ");
return meta ? ` (${meta})` : "";
}
function formatPackageInspectorFindingsHtml(findings: PackageInspectorEmailFinding[]) {
if (findings.length === 0) return paragraph("No findings were included.");
return findings
.map((finding) => {
const meta = [finding.issueClass, finding.severity].filter(Boolean).join(" · ");
const remediation = finding.authorRemediation?.summary
? `<div style="margin:10px 0 0;padding-top:10px;border-top:1px solid #d8dee4;">
<p style="margin:0 0 6px;font-size:15px;line-height:22px;color:#1f2328;"><strong>Fix</strong></p>
<p style="margin:0 0 6px;font-size:15px;line-height:22px;color:#1f2328;">${escapeHtml(finding.authorRemediation.summary)}</p>
${
finding.authorRemediation.docsUrl
? `<p style="margin:10px 0 6px;font-size:15px;line-height:22px;color:#1f2328;"><strong>Docs</strong></p>
<p style="margin:0;font-size:15px;line-height:22px;color:#1f2328;word-break:break-word;">${textLink(finding.authorRemediation.docsUrl, finding.authorRemediation.docsUrl)}</p>`
: ""
}
</div>`
: "";
return `<div style="margin:0 0 10px;padding:10px 12px;border:1px solid #d8dee4;border-radius:6px;background:#ffffff;">
<p style="margin:0 0 6px;font-size:15px;line-height:22px;color:#1f2328;">
<strong>${escapeHtml(finding.findingKind.toUpperCase())}</strong>
<code style="font-family:ui-monospace,SFMono-Regular,Consolas,'Liberation Mono',monospace;font-size:13px;background:#f6f8fa;border:1px solid #d8dee4;border-radius:4px;padding:1px 4px;">${escapeHtml(finding.code)}</code>
${meta ? `<span style="color:#57606a;">${escapeHtml(meta)}</span>` : ""}
</p>
<p style="margin:0;font-size:15px;line-height:22px;color:#1f2328;">${escapeHtml(finding.message)}</p>
${remediation}
</div>`;
})
.join("");
}
+1 -6
View File
@@ -91,12 +91,7 @@ export function parseGitHubImportUrl(input: string): GitHubImportUrl {
}
function canonicalGitHubImportUrl(url: URL) {
const canonical = new URL(url.toString());
canonical.username = "";
canonical.password = "";
canonical.search = "";
canonical.hash = "";
return `${canonical.origin}${canonical.pathname}`;
return `https://${url.hostname}${url.pathname}`;
}
export async function resolveGitHubCommit(
+566
View File
@@ -0,0 +1,566 @@
import { describe, expect, it } from "vitest";
import {
buildGitHubSkillSourceSnapshot,
buildGitHubSkillSyncPlan,
parseSkillsShDisplayManifest,
} from "./githubSkillSync";
const encoder = new TextEncoder();
function bytes(text: string) {
return encoder.encode(text);
}
function repoEntries(entries: Record<string, string>) {
return Object.fromEntries(Object.entries(entries).map(([path, text]) => [path, bytes(text)]));
}
describe("parseSkillsShDisplayManifest", () => {
it("keeps the supported skills.sh rendering fields and drops invalid groups", () => {
const result = parseSkillsShDisplayManifest(
JSON.stringify({
notGrouped: "top",
groupings: [
{
title: "Agentic AI",
description: "Agentic workflows.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
{ title: "Broken", skills: [123] },
{ description: "Missing title", skills: ["ignored"] },
],
}),
);
expect(result).toEqual({
status: "ok",
manifest: {
notGrouped: "top",
groupings: [
{
title: "Agentic AI",
description: "Agentic workflows.",
skills: ["aiq-deploy", "nemoclaw-user-configure-security"],
},
],
},
});
});
it("marks missing and invalid manifests so the UI can fall back", () => {
expect(parseSkillsShDisplayManifest(undefined)).toEqual({
status: "missing",
manifest: undefined,
});
expect(parseSkillsShDisplayManifest("{nope")).toEqual({
status: "invalid",
manifest: undefined,
});
expect(parseSkillsShDisplayManifest(JSON.stringify({ groupings: [] }))).toEqual({
status: "invalid",
manifest: undefined,
});
});
});
describe("buildGitHubSkillSourceSnapshot", () => {
it("discovers skill folders, parses SKILL.md metadata, and hashes exact folder bytes", async () => {
const baseEntries = repoEntries({
"skills/aiq-deploy/SKILL.md":
"---\nname: AIQ Deploy\nversion: 0.2.0\ndescription: Deploy AgentIQ workflows.\n---\n# AIQ Deploy\n",
"skills/aiq-deploy/skill-card.md": "# Card\n",
"skills/vision-helper/SKILL.md": "# Vision Helper\n",
"skills.sh.json": JSON.stringify({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
}),
});
const changedEntries = {
...baseEntries,
"skills/aiq-deploy/skill-card.md": bytes("# Card changed\n"),
};
const base = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: baseEntries,
});
const changed = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: changedEntries,
});
expect(base.manifestStatus).toBe("ok");
expect(base.manifest).toEqual({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
});
expect(base.skills).toEqual(
expect.arrayContaining([
expect.objectContaining({
slug: "aiq-deploy",
displayName: "AIQ Deploy",
summary: "Deploy AgentIQ workflows.",
upstreamVersion: "0.2.0",
path: "skills/aiq-deploy",
skillMarkdownPath: "skills/aiq-deploy/SKILL.md",
skillMarkdown:
"---\nname: AIQ Deploy\nversion: 0.2.0\ndescription: Deploy AgentIQ workflows.\n---\n# AIQ Deploy\n",
skillCardMarkdownPath: "skills/aiq-deploy/skill-card.md",
skillCardMarkdown: "# Card\n",
}),
expect.objectContaining({
slug: "vision-helper",
displayName: "Vision Helper",
path: "skills/vision-helper",
skillMarkdownPath: "skills/vision-helper/SKILL.md",
skillMarkdown: "# Vision Helper\n",
}),
]),
);
expect(changed.skills.find((skill) => skill.slug === "aiq-deploy")?.contentHash).not.toBe(
base.skills.find((skill) => skill.slug === "aiq-deploy")?.contentHash,
);
expect(changed.skills.find((skill) => skill.slug === "vision-helper")?.contentHash).toBe(
base.skills.find((skill) => skill.slug === "vision-helper")?.contentHash,
);
});
it("includes valid filenames containing dot-dot text in folder hashes", async () => {
const base = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
"skills/aiq-deploy/payload..sh": "echo safe\n",
}),
});
const changed = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
"skills/aiq-deploy/payload..sh": "echo changed\n",
}),
});
expect(changed.skills[0]?.contentHash).not.toBe(base.skills[0]?.contentHash);
});
it("rejects duplicate normalized skill slugs before syncing content", async () => {
await expect(
buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq_deploy/SKILL.md": "# AIQ Deploy A\n",
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy B\n",
}),
}),
).rejects.toThrow(/duplicate normalized slug/i);
});
it("prefers the top-level skills catalog folder over duplicate plugin copies", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
"plugins/nvidia-skills/skills/aiq-deploy/SKILL.md": "# Plugin Copy\n",
"skills.sh.json": JSON.stringify({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
}),
}),
});
expect(snapshot.skills.map((skill) => skill.path)).toEqual(["skills/aiq-deploy"]);
expect(snapshot.skills[0]?.displayName).toBe("AIQ Deploy");
});
it("rejects oversized cached markdown before writing Convex content docs", async () => {
await expect(
buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "1".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": `# AIQ Deploy\n${"x".repeat(513 * 1024)}`,
}),
}),
).rejects.toThrow(/too large to cache/i);
});
});
describe("buildGitHubSkillSyncPlan", () => {
it("marks changed upstream content pending", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy v2\n",
"skills.sh.json": JSON.stringify({
groupings: [{ title: "Agentic AI", skills: ["aiq-deploy"] }],
}),
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: "old-hash",
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches).toEqual([
expect.objectContaining({
skillId: "skills:aiq-deploy",
slug: "aiq-deploy",
patch: expect.objectContaining({
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: snapshot.skills[0]?.contentHash,
githubScanStatus: "pending",
moderationStatus: "active",
moderationReason: "pending.scan",
}),
}),
]);
expect(plan.skillInserts).toEqual([]);
expect(plan.stats.changed).toBe(1);
});
it("keeps clean scan status when only the repo commit changes", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "---\nversion: 0.2.0\n---\n# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
latestVersionSummary: {
version: "0.2.0",
createdAt: 7,
},
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: contentHash,
githubScanStatus: "clean",
moderationStatus: "active",
moderationVerdict: "clean",
});
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("updatedAt");
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("latestVersionSummary");
expect(plan.stats.unchanged).toBe(1);
});
it("updates existing skill ownership when a source is reassigned", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:new-owner",
ownerPublisherId: "publishers:new-owner",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: snapshot.skills[0]?.contentHash ?? "",
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
ownerUserId: "users:new-owner",
ownerPublisherId: "publishers:new-owner",
});
});
it("preserves pending scan status for unchanged pending content", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "pending",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentCommit: "3".repeat(40),
githubCurrentContentHash: contentHash,
githubScanStatus: "pending",
moderationStatus: "active",
moderationReason: "pending.scan",
});
expect(plan.stats.unchanged).toBe(1);
});
it("preserves terminal scan status for unchanged current content", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
moderationStatus: "hidden",
moderationReason: "scanner.llm.malicious",
});
});
it("preserves terminal scan status for unchanged current bytes", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/aiq-deploy/SKILL.md": "# AIQ Deploy\n",
}),
});
const contentHash = snapshot.skills[0]?.contentHash ?? "";
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "present",
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentContentHash: contentHash,
githubScanStatus: "malicious",
moderationStatus: "hidden",
moderationReason: "scanner.llm.malicious",
});
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("updatedAt");
expect(plan.stats.unchanged).toBe(1);
});
it("revives soft-deleted skills when a configured repo is synced again", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "mattpocock/skills",
defaultBranch: "main",
commit: "4".repeat(40),
entries: repoEntries({
"skills/engineering/tdd/SKILL.md": "# TDD\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:matt",
ownerUserId: "users:matt",
ownerPublisherId: "publishers:matt",
existingSkills: [
{
_id: "skills:tdd",
slug: "tdd",
displayName: "TDD",
githubPath: "skills/engineering/tdd",
githubCurrentStatus: "missing",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentStatus: "present",
githubRemovedAt: undefined,
softDeletedAt: undefined,
});
});
it("tombstones upstream removals instead of leaving stale installs active", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "2".repeat(40),
entries: repoEntries({
"skills/vision-helper/SKILL.md": "# Vision Helper\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches).toEqual([
expect.objectContaining({
skillId: "skills:aiq-deploy",
slug: "aiq-deploy",
patch: expect.objectContaining({
githubCurrentCommit: "2".repeat(40),
githubCurrentStatus: "missing",
githubRemovedAt: 123,
softDeletedAt: 123,
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
}),
}),
]);
expect(plan.skillInserts).toHaveLength(1);
expect(plan.stats.removed).toBe(1);
});
it("preserves first upstream removal time on later syncs", async () => {
const snapshot = await buildGitHubSkillSourceSnapshot({
repo: "NVIDIA/skills",
defaultBranch: "main",
commit: "3".repeat(40),
entries: repoEntries({
"skills/vision-helper/SKILL.md": "# Vision Helper\n",
}),
});
const plan = buildGitHubSkillSyncPlan({
sourceId: "githubSkillSources:nvidia",
ownerUserId: "users:nvidia",
ownerPublisherId: "publishers:nvidia",
existingSkills: [
{
_id: "skills:aiq-deploy",
slug: "aiq-deploy",
displayName: "AIQ Deploy",
githubPath: "skills/aiq-deploy",
githubCurrentStatus: "missing",
githubRemovedAt: 77,
githubScanStatus: "clean",
},
],
snapshot,
now: 123,
});
expect(plan.skillPatches[0]?.patch).toMatchObject({
githubCurrentCommit: "3".repeat(40),
githubCurrentStatus: "missing",
githubCurrentCheckedAt: 123,
githubRemovedAt: 77,
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
});
expect(plan.skillPatches[0]?.patch).not.toHaveProperty("updatedAt");
});
});
+670
View File
@@ -0,0 +1,670 @@
import { getFrontmatterValue, parseFrontmatter } from "./skills";
export type GitHubSkillScanStatus = "clean" | "suspicious" | "malicious" | "pending" | "failed";
export type GitHubCurrentStatus = "present" | "missing" | "unknown";
export type DisplayManifestStatus = "ok" | "missing" | "invalid" | "failed";
export type DisplayManifest = {
notGrouped?: "top" | "bottom";
groupings: Array<{
title: string;
description?: string;
skills: string[];
}>;
};
export type GitHubSkillSourceSnapshot = {
repo: string;
defaultBranch: string;
commit: string;
manifestStatus: DisplayManifestStatus;
manifestHash?: string;
manifest?: DisplayManifest;
skills: DiscoveredGitHubSkill[];
};
export type GitHubSkillSourceMetadataSnapshot = Omit<GitHubSkillSourceSnapshot, "skills"> & {
skills: DiscoveredGitHubSkillMetadata[];
};
export type DiscoveredGitHubSkill = {
slug: string;
displayName: string;
summary?: string;
upstreamVersion?: string;
path: string;
skillMarkdownPath: string;
skillMarkdown: string;
skillCardMarkdownPath?: string;
skillCardMarkdown?: string;
contentHash: string;
};
export type DiscoveredGitHubSkillMetadata = Omit<
DiscoveredGitHubSkill,
"skillMarkdown" | "skillCardMarkdown"
>;
export type ExistingGitHubSkillForSync = {
_id: string;
slug: string;
displayName: string;
summary?: string;
latestVersionSummary?: {
version: string;
createdAt: number;
};
githubPath?: string;
githubCurrentCommit?: string;
githubCurrentContentHash?: string;
githubCurrentStatus?: GitHubCurrentStatus;
githubScanStatus?: GitHubSkillScanStatus;
githubRemovedAt?: number;
softDeletedAt?: number;
};
export type GitHubBackedSkillModeration = {
moderationStatus: "active" | "hidden";
moderationReason?: string;
moderationVerdict?: "clean" | "suspicious" | "malicious";
moderationFlags: string[];
isSuspicious: boolean;
};
export type GitHubSkillPatchForSync = {
skillId: string;
slug: string;
patch: Record<string, unknown>;
};
export type GitHubSkillInsertForSync = {
slug: string;
doc: Record<string, unknown>;
};
export type GitHubSkillSyncPlan = {
sourcePatch: Record<string, unknown>;
skillPatches: GitHubSkillPatchForSync[];
skillInserts: GitHubSkillInsertForSync[];
stats: {
discovered: number;
inserted: number;
changed: number;
unchanged: number;
removed: number;
};
};
const SKILL_MARKDOWN_BASENAME = "skill.md";
const SKILL_CARD_MARKDOWN_BASENAME = "skill-card.md";
const MAX_STORED_MARKDOWN_BYTES = 512 * 1024;
const MAX_STORED_SKILL_CONTENT_BYTES = 768 * 1024;
export function parseSkillsShDisplayManifest(raw: string | undefined | null): {
status: DisplayManifestStatus;
manifest?: DisplayManifest;
} {
if (raw === undefined || raw === null) return { status: "missing", manifest: undefined };
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
return { status: "invalid", manifest: undefined };
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
return { status: "invalid", manifest: undefined };
}
const record = parsed as Record<string, unknown>;
const rawGroups = record.groupings;
if (!Array.isArray(rawGroups)) return { status: "invalid", manifest: undefined };
const groupings = rawGroups.flatMap((group): DisplayManifest["groupings"] => {
if (!group || typeof group !== "object" || Array.isArray(group)) return [];
const groupRecord = group as Record<string, unknown>;
const title = typeof groupRecord.title === "string" ? groupRecord.title.trim() : "";
const description =
typeof groupRecord.description === "string" ? groupRecord.description.trim() : "";
const skills = Array.isArray(groupRecord.skills)
? groupRecord.skills
.filter((value): value is string => typeof value === "string")
.map((value) => value.trim())
.filter(Boolean)
: [];
if (!title || skills.length === 0) return [];
return [
{
title,
...(description ? { description } : {}),
skills,
},
];
});
if (groupings.length === 0) return { status: "invalid", manifest: undefined };
const notGrouped =
record.notGrouped === "top" || record.notGrouped === "bottom" ? record.notGrouped : undefined;
return {
status: "ok",
manifest: {
...(notGrouped ? { notGrouped } : {}),
groupings,
},
};
}
export async function buildGitHubSkillSourceSnapshot({
repo,
defaultBranch,
commit,
entries,
}: {
repo: string;
defaultBranch: string;
commit: string;
entries: Record<string, Uint8Array>;
}): Promise<GitHubSkillSourceSnapshot> {
const normalizedEntries = normalizeEntryMap(entries);
const manifestBytes = normalizedEntries["skills.sh.json"];
const manifestText = manifestBytes ? decodeUtf8(manifestBytes) : undefined;
const parsedManifest = parseSkillsShDisplayManifest(manifestText);
const manifestHash = manifestBytes ? await sha256Hex(manifestBytes) : undefined;
const skillPaths = discoverSkillPaths(normalizedEntries);
const skills: DiscoveredGitHubSkill[] = [];
for (const skillMdPath of skillPaths) {
const path = parentPath(skillMdPath);
const markdownBytes = normalizedEntries[skillMdPath] ?? new Uint8Array();
assertStoredMarkdownSize(skillMdPath, markdownBytes);
const markdown = decodeUtf8(markdownBytes);
const frontmatter = parseFrontmatter(markdown);
const folderName = path.split("/").filter(Boolean).at(-1) ?? "";
const slug = slugFromPathSegment(folderName);
if (!slug) continue;
const frontmatterName = getFrontmatterValue(frontmatter, "name")?.trim();
const frontmatterDescription = getFrontmatterValue(frontmatter, "description")?.trim();
const frontmatterVersion = getFrontmatterValue(frontmatter, "version")?.trim();
const heading = firstMarkdownHeading(markdown);
const skillCardMarkdownPath = findFolderFilePath(
normalizedEntries,
path,
SKILL_CARD_MARKDOWN_BASENAME,
);
const skillCardBytes = skillCardMarkdownPath
? normalizedEntries[skillCardMarkdownPath]
: undefined;
if (skillCardMarkdownPath && skillCardBytes) {
assertStoredMarkdownSize(skillCardMarkdownPath, skillCardBytes);
assertStoredSkillContentSize(markdownBytes.byteLength + skillCardBytes.byteLength);
} else {
assertStoredSkillContentSize(markdownBytes.byteLength);
}
const skillCardMarkdown = skillCardBytes ? decodeUtf8(skillCardBytes) : undefined;
skills.push({
slug,
displayName: frontmatterName || heading || titleizeSlug(slug),
...(frontmatterDescription ? { summary: frontmatterDescription } : {}),
...(frontmatterVersion ? { upstreamVersion: frontmatterVersion } : {}),
path,
skillMarkdownPath: skillMdPath,
skillMarkdown: markdown,
...(skillCardMarkdownPath ? { skillCardMarkdownPath } : {}),
...(skillCardMarkdown !== undefined ? { skillCardMarkdown } : {}),
contentHash: await computeGitHubSkillFolderContentHash(normalizedEntries, path),
});
}
const sortedSkills = skills.sort((a, b) => a.path.localeCompare(b.path));
assertUniqueDiscoveredSlugs(sortedSkills);
return {
repo,
defaultBranch,
commit,
manifestStatus: parsedManifest.status,
...(manifestHash ? { manifestHash } : {}),
...(parsedManifest.manifest ? { manifest: parsedManifest.manifest } : {}),
skills: sortedSkills,
};
}
export async function computeGitHubSkillFolderContentHash(
entries: Record<string, Uint8Array>,
folderPath: string,
) {
const normalizedEntries = normalizeEntryMap(entries);
const root = folderPath ? `${folderPath}/` : "";
const lines: string[] = [];
for (const [path, content] of Object.entries(normalizedEntries).sort(([a], [b]) =>
a.localeCompare(b),
)) {
if (root && path !== folderPath && !path.startsWith(root)) continue;
if (!root && path.includes("/")) continue;
const relativePath = root ? path.slice(root.length) : path;
if (!relativePath) continue;
const fileHash = await sha256Hex(content);
lines.push(`${relativePath}\0${content.byteLength}\0${fileHash}`);
}
return sha256Hex(new TextEncoder().encode(lines.join("\n")));
}
export function buildGitHubSkillSyncPlan({
sourceId,
ownerUserId,
ownerPublisherId,
existingSkills,
snapshot,
now,
}: {
sourceId: string;
ownerUserId: string;
ownerPublisherId?: string;
existingSkills: ExistingGitHubSkillForSync[];
snapshot: GitHubSkillSourceSnapshot | GitHubSkillSourceMetadataSnapshot;
now: number;
}): GitHubSkillSyncPlan {
const sourcePatch = {
repo: snapshot.repo,
defaultBranch: snapshot.defaultBranch,
lastSyncStatus: "ok",
lastSyncError: undefined,
lastSyncErrorAt: undefined,
displayManifestKind: "skills.sh",
displayManifestHash: snapshot.manifestHash,
displayManifestCommit: snapshot.commit,
displayManifestFetchedAt: now,
displayManifestStatus: snapshot.manifestStatus,
displayManifest: snapshot.manifest,
...(ownerPublisherId ? { ownerPublisherId } : {}),
updatedAt: now,
};
const existingByPath = new Map(
existingSkills
.filter((skill) => skill.githubPath)
.map((skill) => [skill.githubPath as string, skill]),
);
const existingBySlug = new Map(existingSkills.map((skill) => [skill.slug, skill]));
const matchedSkillIds = new Set<string>();
const skillPatches: GitHubSkillPatchForSync[] = [];
const skillInserts: GitHubSkillInsertForSync[] = [];
const stats = {
discovered: snapshot.skills.length,
inserted: 0,
changed: 0,
unchanged: 0,
removed: 0,
};
for (const discovered of snapshot.skills) {
const existing = existingByPath.get(discovered.path) ?? existingBySlug.get(discovered.slug);
if (!existing) {
const scanStatus: GitHubSkillScanStatus = "pending";
const moderation = githubBackedSkillModeration(scanStatus);
skillInserts.push({
slug: discovered.slug,
doc: {
slug: discovered.slug,
displayName: discovered.displayName,
summary: discovered.summary,
ownerUserId,
ownerPublisherId,
installKind: "github",
githubSourceId: sourceId,
githubPath: discovered.path,
githubHasSkillCard: Boolean(discovered.skillCardMarkdownPath),
githubCurrentCommit: snapshot.commit,
githubCurrentContentHash: discovered.contentHash,
githubCurrentStatus: "present",
githubCurrentCheckedAt: now,
githubScanStatus: scanStatus,
githubRemovedAt: undefined,
latestVersionId: undefined,
latestVersionSummary: latestVersionSummary(discovered.upstreamVersion, now),
tags: {},
capabilityTags: [],
softDeletedAt: undefined,
badges: undefined,
statsDownloads: 0,
statsStars: 0,
statsInstallsCurrent: 0,
statsInstallsAllTime: 0,
stats: {
downloads: 0,
stars: 0,
installsCurrent: 0,
installsAllTime: 0,
versions: 0,
comments: 0,
},
...moderation,
createdAt: now,
updatedAt: now,
},
});
stats.inserted += 1;
continue;
}
matchedSkillIds.add(existing._id);
const currentContentUnchanged =
existing.githubCurrentStatus === "present" &&
existing.githubCurrentContentHash === discovered.contentHash;
const scanStatus: GitHubSkillScanStatus = currentContentUnchanged
? githubScanStatusForUnchangedContent(existing.githubScanStatus)
: "pending";
const moderation = githubBackedSkillModeration(scanStatus);
const nextLatestVersionSummary = latestVersionSummary(
discovered.upstreamVersion,
existing.latestVersionSummary?.createdAt ?? now,
);
const materialChanged =
!currentContentUnchanged ||
existing.displayName !== discovered.displayName ||
(existing.summary ?? undefined) !== (discovered.summary ?? undefined) ||
(existing.githubPath ?? undefined) !== discovered.path ||
!sameLatestVersionSummary(existing.latestVersionSummary, nextLatestVersionSummary);
const patch = {
displayName: discovered.displayName,
summary: discovered.summary,
ownerUserId,
...(ownerPublisherId ? { ownerPublisherId } : {}),
githubSourceId: sourceId,
githubPath: discovered.path,
githubHasSkillCard: Boolean(discovered.skillCardMarkdownPath),
githubCurrentCommit: snapshot.commit,
githubCurrentContentHash: discovered.contentHash,
githubCurrentStatus: "present",
githubCurrentCheckedAt: now,
githubScanStatus: scanStatus,
githubRemovedAt: undefined,
softDeletedAt: undefined,
...(materialChanged
? {
latestVersionSummary: latestVersionSummary(discovered.upstreamVersion, now),
updatedAt: now,
}
: {}),
...moderation,
};
skillPatches.push({ skillId: existing._id, slug: existing.slug, patch });
if (materialChanged) stats.changed += 1;
else stats.unchanged += 1;
}
for (const existing of existingSkills) {
if (matchedSkillIds.has(existing._id)) continue;
const removedAt = existing.githubRemovedAt ?? now;
const moderation = githubBackedSkillModeration(
existing.githubScanStatus ?? "pending",
removedAt,
);
const wasAlreadyRemoved =
existing.githubCurrentStatus === "missing" && existing.githubRemovedAt !== undefined;
skillPatches.push({
skillId: existing._id,
slug: existing.slug,
patch: {
githubCurrentCommit: snapshot.commit,
githubCurrentStatus: "missing",
githubCurrentCheckedAt: now,
githubRemovedAt: removedAt,
softDeletedAt: existing.softDeletedAt ?? removedAt,
...(wasAlreadyRemoved ? {} : { updatedAt: now }),
...moderation,
},
});
stats.removed += 1;
}
return { sourcePatch, skillPatches, skillInserts, stats };
}
function githubScanStatusForUnchangedContent(
status: GitHubSkillScanStatus | undefined,
): GitHubSkillScanStatus {
if (
status === "clean" ||
status === "failed" ||
status === "malicious" ||
status === "suspicious"
) {
return status;
}
return "pending";
}
export function githubBackedSkillModeration(
scanStatus: GitHubSkillScanStatus,
removedAt?: number,
): GitHubBackedSkillModeration {
if (typeof removedAt === "number") {
return {
moderationStatus: "hidden",
moderationReason: "github.upstream.removed",
moderationVerdict: undefined,
moderationFlags: [],
isSuspicious: false,
};
}
if (scanStatus === "pending") {
return {
moderationStatus: "active",
moderationReason: "pending.scan",
moderationVerdict: undefined,
moderationFlags: [],
isSuspicious: false,
};
}
if (scanStatus === "failed") {
return {
moderationStatus: "hidden",
moderationReason: "scanner.failed",
moderationVerdict: undefined,
moderationFlags: [],
isSuspicious: false,
};
}
if (scanStatus === "malicious") {
return {
moderationStatus: "hidden",
moderationReason: "scanner.llm.malicious",
moderationVerdict: "malicious",
moderationFlags: ["blocked.malware"],
isSuspicious: true,
};
}
if (scanStatus === "suspicious") {
return {
moderationStatus: "active",
moderationReason: "scanner.llm.suspicious",
moderationVerdict: "suspicious",
moderationFlags: ["flagged.suspicious"],
isSuspicious: true,
};
}
return {
moderationStatus: "active",
moderationReason: undefined,
moderationVerdict: "clean",
moderationFlags: [],
isSuspicious: false,
};
}
function latestVersionSummary(version: string | undefined, now: number) {
if (!version) return undefined;
return {
version,
createdAt: now,
changelog: "Synced from GitHub source.",
changelogSource: "auto" as const,
};
}
function sameLatestVersionSummary(
a: ExistingGitHubSkillForSync["latestVersionSummary"] | undefined,
b: ReturnType<typeof latestVersionSummary>,
) {
if (!a && !b) return true;
if (!a || !b) return false;
return a.version === b.version;
}
function assertStoredMarkdownSize(path: string, bytes: Uint8Array) {
if (bytes.byteLength > MAX_STORED_MARKDOWN_BYTES) {
throw new Error(`GitHub skill markdown file is too large to cache: ${path}`);
}
}
function assertStoredSkillContentSize(totalBytes: number) {
if (totalBytes > MAX_STORED_SKILL_CONTENT_BYTES) {
throw new Error("GitHub skill cached markdown is too large");
}
}
function assertUniqueDiscoveredSlugs(skills: DiscoveredGitHubSkill[]) {
const firstPathBySlug = new Map<string, string>();
for (const skill of skills) {
const firstPath = firstPathBySlug.get(skill.slug);
if (firstPath) {
throw duplicateSkillSlugError(skill.slug, firstPath, skill.path);
}
firstPathBySlug.set(skill.slug, skill.path);
}
}
function normalizeEntryMap(entries: Record<string, Uint8Array>) {
const out: Record<string, Uint8Array> = {};
for (const [rawPath, bytes] of Object.entries(entries)) {
const normalized = normalizeRepoPath(rawPath);
if (!normalized) continue;
out[normalized] = new Uint8Array(bytes);
}
return out;
}
function discoverSkillPaths(entries: Record<string, Uint8Array>) {
const candidates = Object.keys(entries)
.filter((path) => path.split("/").at(-1)?.toLowerCase() === SKILL_MARKDOWN_BASENAME)
.sort((a, b) => a.localeCompare(b));
const pathsBySlug = new Map<string, string[]>();
for (const skillMdPath of candidates) {
const path = parentPath(skillMdPath);
const folderName = path.split("/").filter(Boolean).at(-1) ?? "";
const slug = slugFromPathSegment(folderName);
if (!slug) continue;
const paths = pathsBySlug.get(slug) ?? [];
paths.push(skillMdPath);
pathsBySlug.set(slug, paths);
}
const selected: string[] = [];
for (const [slug, paths] of pathsBySlug) {
if (paths.length === 1) {
selected.push(paths[0] as string);
continue;
}
const canonicalPath = `skills/${slug}/${SKILL_MARKDOWN_BASENAME}`;
const exactTopLevelMatches = paths.filter((path) => path.toLowerCase() === canonicalPath);
const topLevelSkillMatches = paths.filter((path) => path.toLowerCase().startsWith("skills/"));
if (exactTopLevelMatches.length === 1 && topLevelSkillMatches.length === 1) {
selected.push(exactTopLevelMatches[0] as string);
continue;
}
throw duplicateSkillSlugError(
slug,
parentPath(paths[0] as string),
parentPath(paths[1] as string),
);
}
return selected.sort((a, b) => a.localeCompare(b));
}
function duplicateSkillSlugError(slug: string, firstPath: string, secondPath: string) {
return new Error(
`GitHub skill source has duplicate normalized slug "${slug}" at ${firstPath} and ${secondPath}`,
);
}
function findFolderFilePath(
entries: Record<string, Uint8Array>,
folderPath: string,
basename: string,
) {
const prefix = folderPath ? `${folderPath}/` : "";
return Object.keys(entries).find((entryPath) => {
if (prefix) {
if (!entryPath.startsWith(prefix)) return false;
const relativePath = entryPath.slice(prefix.length);
return !relativePath.includes("/") && relativePath.toLowerCase() === basename;
}
return !entryPath.includes("/") && entryPath.toLowerCase() === basename;
});
}
function normalizeRepoPath(path: string) {
if (path.includes("\u0000")) return "";
const normalized = path
.replaceAll("\\", "/")
.trim()
.replace(/^\.\/+/, "")
.replace(/^\/+/, "");
if (!normalized) return "";
const segments = normalized.split("/").filter(Boolean);
if (segments.some((segment) => segment === "." || segment === "..")) return "";
return segments.join("/");
}
function parentPath(path: string) {
return path.split("/").slice(0, -1).join("/");
}
function decodeUtf8(bytes: Uint8Array) {
return new TextDecoder().decode(bytes);
}
async function sha256Hex(bytes: Uint8Array) {
const safe = new Uint8Array(bytes);
const digest = await crypto.subtle.digest("SHA-256", safe);
return toHex(new Uint8Array(digest));
}
function toHex(bytes: Uint8Array) {
let out = "";
for (const byte of bytes) out += byte.toString(16).padStart(2, "0");
return out;
}
function slugFromPathSegment(value: string) {
return value
.trim()
.toLowerCase()
.replace(/[\s_]+/g, "-")
.replace(/[^a-z0-9-]+/g, "-")
.replace(/-+/g, "-")
.replace(/^-|-$/g, "");
}
function titleizeSlug(slug: string) {
return slug
.split("-")
.filter(Boolean)
.map((part) => part.charAt(0).toUpperCase() + part.slice(1))
.join(" ");
}
function firstMarkdownHeading(markdown: string) {
for (const line of markdown.split(/\r?\n/)) {
const match = /^#\s+(.+)$/.exec(line.trim());
if (match?.[1]) return match[1].trim();
}
return undefined;
}
+3 -1
View File
@@ -6,7 +6,8 @@ export const GLOBAL_STATS_KEY = "default";
type SkillVisibilityFields = Pick<
Doc<"skills">,
"softDeletedAt" | "moderationStatus" | "moderationFlags"
>;
> &
Partial<Pick<Doc<"skills">, "moderationVerdict">>;
type GlobalStatsReadCtx = Pick<MutationCtx | QueryCtx, "db">;
type GlobalStatsWriteCtx = Pick<MutationCtx, "db">;
@@ -16,6 +17,7 @@ export function isPublicSkillDoc<T extends SkillVisibilityFields>(
): skill is T {
if (!skill || skill.softDeletedAt) return false;
if (skill.moderationStatus && skill.moderationStatus !== "active") return false;
if (skill.moderationVerdict === "malicious") return false;
if (skill.moderationFlags?.includes("blocked.malware")) return false;
return true;
}
+214
View File
@@ -0,0 +1,214 @@
import { describe, expect, it } from "vitest";
import { buildSkillInstallResolution } from "./installResolver";
const baseSkill = {
slug: "aiq-deploy",
displayName: "AIQ Deploy",
latestVersionSummary: null,
installKind: "github" as const,
githubPath: "skills/aiq-deploy",
githubCurrentCommit: "1".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy",
githubCurrentStatus: "present" as const,
githubScanStatus: "clean" as const,
githubRemovedAt: undefined,
};
const source = {
repo: "NVIDIA/skills",
defaultBranch: "main",
};
describe("buildSkillInstallResolution", () => {
it("returns an archive descriptor for hosted direct uploads", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
slug: "direct-skill",
displayName: "Direct Skill",
latestVersionSummary: { version: "1.2.3" },
},
source: null,
});
expect(resolution).toEqual({
ok: true,
slug: "direct-skill",
installKind: "archive",
archive: {
version: "1.2.3",
downloadUrl: "https://clawhub.ai/api/v1/download?slug=direct-skill&version=1.2.3",
},
});
});
it("returns a pinned GitHub descriptor when current upstream state is scan-clean", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: baseSkill,
source,
});
expect(resolution).toEqual({
ok: true,
slug: "aiq-deploy",
installKind: "github",
github: {
repo: "NVIDIA/skills",
path: "skills/aiq-deploy",
commit: "1".repeat(40),
contentHash: "hash-aiq-deploy",
sourceUrl: `https://github.com/NVIDIA/skills/tree/${"1".repeat(40)}/skills/aiq-deploy`,
},
});
});
it("allows GitHub-backed installs when upstream content changed and the current hash is clean", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: "hash-aiq-deploy-v2",
},
source,
});
expect(resolution).toMatchObject({
ok: true,
installKind: "github",
github: {
commit: "2".repeat(40),
contentHash: "hash-aiq-deploy-v2",
sourceUrl: `https://github.com/NVIDIA/skills/tree/${"2".repeat(40)}/skills/aiq-deploy`,
},
});
});
it("allows GitHub-backed installs when only unrelated repository content changed", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubCurrentCommit: "2".repeat(40),
githubCurrentContentHash: baseSkill.githubCurrentContentHash,
},
source,
});
expect(resolution).toMatchObject({
ok: true,
installKind: "github",
github: {
commit: "2".repeat(40),
contentHash: "hash-aiq-deploy",
sourceUrl: `https://github.com/NVIDIA/skills/tree/${"2".repeat(40)}/skills/aiq-deploy`,
},
});
});
it.each([
{
name: "upstream path is missing",
patch: { githubCurrentStatus: "missing" as const },
reason: "github_upstream_missing",
status: 410,
},
{
name: "skill was pulled upstream",
patch: { githubRemovedAt: 456 },
reason: "github_upstream_removed",
status: 410,
},
{
name: "scan is pending",
patch: { githubScanStatus: "pending" as const },
reason: "github_verification_pending",
status: 423,
},
{
name: "scan failed",
patch: { githubScanStatus: "failed" as const },
reason: "github_scan_failed",
status: 403,
},
{
name: "scan is suspicious",
patch: { githubScanStatus: "suspicious" as const },
reason: "github_scan_failed",
status: 403,
},
])("blocks GitHub-backed installs when $name", ({ patch, reason, status }) => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: { ...baseSkill, ...patch },
source,
});
expect(resolution).toMatchObject({
ok: false,
slug: "aiq-deploy",
reason,
status,
});
});
it("explains pending GitHub-backed verification clearly", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubScanStatus: "pending",
},
source,
});
expect(resolution).toMatchObject({
ok: false,
slug: "aiq-deploy",
reason: "github_verification_pending",
status: 423,
message:
"GitHub-backed skill security scan is in progress. Try again shortly, or rerun with --force-install to install the unverified upstream commit.",
});
});
it("allows force-install for pending GitHub-backed verification", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubScanStatus: "pending",
},
source,
forceInstall: true,
});
expect(resolution).toMatchObject({
ok: true,
installKind: "github",
github: {
commit: "1".repeat(40),
contentHash: "hash-aiq-deploy",
},
});
});
it("does not force-install failed GitHub-backed scans", () => {
const resolution = buildSkillInstallResolution({
origin: "https://clawhub.ai",
skill: {
...baseSkill,
githubScanStatus: "failed",
},
source,
forceInstall: true,
});
expect(resolution).toMatchObject({
ok: false,
reason: "github_scan_failed",
status: 403,
});
});
});
+180
View File
@@ -0,0 +1,180 @@
export type GitHubSkillScanStatus = "clean" | "suspicious" | "malicious" | "pending" | "failed";
export type GitHubCurrentStatus = "present" | "missing" | "unknown";
export type InstallResolverSkill = {
slug: string;
displayName: string;
latestVersionSummary?: { version: string } | null;
installKind?: "github";
githubPath?: string;
githubCurrentCommit?: string;
githubCurrentContentHash?: string;
githubCurrentStatus?: GitHubCurrentStatus;
githubScanStatus?: GitHubSkillScanStatus;
githubRemovedAt?: number;
};
export type InstallResolverSource = {
repo: string;
defaultBranch?: string | null;
};
export type SkillInstallResolution =
| {
ok: true;
slug: string;
installKind: "archive";
archive: {
version: string;
downloadUrl: string;
};
}
| {
ok: true;
slug: string;
installKind: "github";
github: {
repo: string;
path: string;
commit: string;
contentHash: string;
sourceUrl: string;
};
}
| {
ok: false;
slug: string;
reason:
| "archive_version_missing"
| "github_source_missing"
| "github_upstream_removed"
| "github_upstream_missing"
| "github_upstream_unknown"
| "github_verification_pending"
| "github_scan_failed";
message: string;
status: 403 | 409 | 410 | 423;
};
export function buildSkillInstallResolution({
origin,
skill,
source,
forceInstall = false,
}: {
origin: string;
skill: InstallResolverSkill;
source: InstallResolverSource | null;
forceInstall?: boolean;
}): SkillInstallResolution {
if (skill.installKind !== "github") {
const version = skill.latestVersionSummary?.version;
if (!version) {
return block(skill.slug, "archive_version_missing", 409);
}
const url = new URL("/api/v1/download", origin);
url.searchParams.set("slug", skill.slug);
url.searchParams.set("version", version);
return {
ok: true,
slug: skill.slug,
installKind: "archive",
archive: {
version,
downloadUrl: url.toString(),
},
};
}
if (skill.githubRemovedAt) {
return block(skill.slug, "github_upstream_removed", 410);
}
if (skill.githubCurrentStatus === "missing") {
return block(skill.slug, "github_upstream_missing", 410);
}
if (
skill.githubScanStatus === "failed" ||
skill.githubScanStatus === "malicious" ||
skill.githubScanStatus === "suspicious"
) {
return block(skill.slug, "github_scan_failed", 403);
}
if (!source || !skill.githubPath) {
return block(skill.slug, "github_source_missing", 409);
}
if (
skill.githubCurrentStatus !== "present" ||
!skill.githubCurrentCommit ||
!skill.githubCurrentContentHash
) {
return block(skill.slug, "github_upstream_unknown", 423);
}
if (
skill.githubScanStatus !== "clean" &&
!(forceInstall && skill.githubScanStatus === "pending")
) {
return block(skill.slug, "github_verification_pending", 423);
}
return {
ok: true,
slug: skill.slug,
installKind: "github",
github: {
repo: source.repo,
path: skill.githubPath,
commit: skill.githubCurrentCommit,
contentHash: skill.githubCurrentContentHash,
sourceUrl: buildGitHubTreeUrl(source.repo, skill.githubCurrentCommit, skill.githubPath),
},
};
}
function block(
slug: string,
reason: Extract<SkillInstallResolution, { ok: false }>["reason"],
status: Extract<SkillInstallResolution, { ok: false }>["status"],
): SkillInstallResolution {
return {
ok: false,
slug,
reason,
status,
message: INSTALL_BLOCK_MESSAGES[reason],
};
}
const INSTALL_BLOCK_MESSAGES: Record<
Extract<SkillInstallResolution, { ok: false }>["reason"],
string
> = {
archive_version_missing: "Hosted skill has no downloadable version.",
github_source_missing: "GitHub-backed skill source metadata is incomplete.",
github_upstream_removed: "GitHub-backed skill has been removed upstream.",
github_upstream_missing: "GitHub-backed skill path is missing upstream.",
github_upstream_unknown: "GitHub-backed skill needs an upstream freshness check before install.",
github_verification_pending:
"GitHub-backed skill security scan is in progress. Try again shortly, or rerun with --force-install to install the unverified upstream commit.",
github_scan_failed: "GitHub-backed skill failed ClawHub security scanning.",
};
function buildGitHubTreeUrl(repo: string, commit: string, path: string) {
return `https://github.com/${encodeURIComponentRepo(repo)}/tree/${commit}/${encodeURIComponentPath(
path,
)}`;
}
function encodeURIComponentRepo(repo: string) {
return repo
.split("/")
.map((segment) => encodeURIComponent(segment))
.join("/");
}
function encodeURIComponentPath(path: string) {
return path
.split("/")
.map((segment) => encodeURIComponent(segment))
.join("/");
}
+18 -18
View File
@@ -223,7 +223,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("suspicious");
});
it("flags browser automation that puts passwords in argv", () => {
it("does not duplicate SkillSpector credential-browser automation analysis", () => {
const result = runStaticModerationScan({
slug: "email-daily-summary",
displayName: "Email Daily Summary",
@@ -242,11 +242,11 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("clean");
});
it("flags persisted browser-use eval against authenticated mail contexts", () => {
it("does not duplicate SkillSpector persisted browser eval analysis", () => {
const result = runStaticModerationScan({
slug: "email-daily-summary",
displayName: "Email Daily Summary",
@@ -266,8 +266,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.browser_credential_automation");
expect(result.status).toBe("clean");
});
it("does not flag ordinary browser-use navigation docs", () => {
@@ -290,7 +290,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("clean");
});
it("blocks stealth browser automation that advertises bot-protection bypass and persistent sessions", () => {
it("does not duplicate SkillSpector browser automation analysis in the static scanner", () => {
const result = runStaticModerationScan({
slug: "stealth-browser",
displayName: "Stealth Browser",
@@ -311,8 +311,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("malicious.stealth_browser_abuse");
expect(result.status).toBe("malicious");
expect(result.reasonCodes).not.toContain("malicious.stealth_browser_abuse");
expect(result.status).toBe("clean");
});
it("flags wallet mnemonics passed as CLI argv", () => {
@@ -497,7 +497,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("suspicious");
});
it("flags Playwright file URL renders of interpolated SVG", () => {
it("does not duplicate SkillSpector browser file-render analysis", () => {
const result = runStaticModerationScan({
slug: "office-quotes",
displayName: "Office Quotes",
@@ -520,8 +520,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.browser_file_render");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.browser_file_render");
expect(result.status).toBe("clean");
});
it("does not flag Playwright file renders with JavaScript disabled", () => {
@@ -1754,7 +1754,7 @@ describe("moderationEngine", () => {
expect(result.status).toBe("clean");
});
it("flags shell positional input passed directly to browser typing", () => {
it("does not duplicate SkillSpector browser typing analysis", () => {
const result = runStaticModerationScan({
slug: "wechat-helper",
displayName: "WeChat Helper",
@@ -1775,11 +1775,11 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("clean");
});
it("checks every positional shell assignment before browser typing", () => {
it("does not inspect shell positional assignments for browser typing", () => {
const result = runStaticModerationScan({
slug: "wechat-helper",
displayName: "WeChat Helper",
@@ -1799,8 +1799,8 @@ describe("moderationEngine", () => {
],
});
expect(result.reasonCodes).toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("suspicious");
expect(result.reasonCodes).not.toContain("suspicious.unsafe_browser_text_input");
expect(result.status).toBe("clean");
});
it("allows browser typing after basic shell input validation", () => {
-177
View File
@@ -94,8 +94,6 @@ const GOOGLE_SHEETS_SPREADSHEET_URL_PATTERN =
/https?:\/\/[^\s"'`]*\/spreadsheets\/([A-Za-z0-9_-]{20,})\/[^\s"'`]*/i;
const DESTRUCTIVE_DELETE_PATTERN =
/\brm\s+-[A-Za-z]*r[A-Za-z]*f[A-Za-z]*\s+(["']?)(\/root\/\.openclaw\/|\/home\/[^/\s"'`]+\/\.openclaw\/|\/Users\/[^/\s"'`]+\/\.openclaw\/|~\/\.openclaw\/|\$HOME\/\.openclaw\/|\$\{HOME\}\/\.openclaw\/|\/etc\/|\/usr\/|\/opt\/|\/Library\/|\/Applications\/)[^\s"'`;|&)]*\1/i;
const SHELL_POSITIONAL_ASSIGNMENT_PATTERN =
/^\s*([A-Z_][A-Z0-9_]*)=(["']?)\$(?:[1-9][0-9]*|@|\*)\2\s*(?:#.*)?$/gm;
const SECRET_ASSIGNMENT_PATTERN =
/\b(?:[A-Za-z0-9]+[_\s-]+)*(?:(?:api|client|consumer)[_\s-]?(?:secret|key|token)|secret[_\s-]?key|access[_\s-]?(?:token|key|secret|grant)|auth[_\s-]?token|bearer(?:[_\s-]?token)?|private[_\s-]?key|service[_\s-]?role[_\s-]?key|github[_\s-]?(?:pat|token)|(?:openrouter|supabase|storj)[_\s-]?(?:key|token|secret|access[_\s-]?grant)|password)\b\s*[:=]\s*["'`]?([A-Za-z0-9][A-Za-z0-9._~+/=-]{15,})["'`]?/i;
const AUTH_HEADER_SECRET_PATTERN =
@@ -111,12 +109,6 @@ const HOST_PLATFORM_SOURCE_CONTEXT_PATTERN =
const HOST_PLATFORM_PATCH_COMMAND_PATTERN =
/\b(?:sed\s+-i|perl\s+-0?pi|cp\s+|cat\s+>|python3?\b.{0,120}(?:write|replace))/i;
const HOST_PLATFORM_REBUILD_PATTERN = /\b(?:pnpm\s+build|npm\s+run\s+build|bun\s+run\s+build)\b/i;
const BROWSER_USE_PASSWORD_ARGV_PATTERN =
/\bbrowser-use\s+input\b[^\n]*(?:password|passwd|\$[A-Z_]*(?:PASSWORD|PASS|PWD)[A-Z0-9_]*|<password>|\{password\})/i;
const BROWSER_USE_AUTH_EVAL_PATTERN = /\bbrowser-use\s+(?:eval|python)\b/i;
const AUTHENTICATED_MAIL_CONTEXT_PATTERN = /\b(?:mail\.google\.com|gmail|webmail|mailbox|inbox)\b/i;
const PERSISTENCE_SCHEDULER_PATTERN =
/\b(?:launchctl\s+load|crontab\b|LaunchAgents\/|systemctl\s+(?:--user\s+)?enable)\b/i;
const SECRET_ARGV_WARNING_PATTERN =
/\b(?:do\s+not|don't|avoid|never|reject)\b[^\n]{0,120}\b(?:argv|argument|from-mnemonic|private[-_\s]?key|seed[-\s]?phrase|mnemonic)\b/i;
const FROM_MNEMONIC_ARGV_PATTERN =
@@ -135,17 +127,6 @@ const PYTHON_BASE64_FILE_READ_PATTERN =
/base64\.b64encode\s*\(\s*(?:[A-Za-z_][A-Za-z0-9_]*\.read_bytes\s*\(\s*\)|Path\s*\([^)]*\)\.read_bytes\s*\(\s*\)|open\s*\([^)]*["']rb["'][\s\S]{0,120}\.read\s*\(\s*\))/i;
const PYTHON_NETWORK_UPLOAD_PATTERN =
/\b(?:requests|session|self\.session|client|httpx\.(?:post|request))\.post\s*\([\s\S]{0,1600}(?:json\s*=|data\s*=|files\s*=|headers\s*=|Authorization)/i;
const PLAYWRIGHT_CHROMIUM_PATTERN = /\b(?:playwright\.)?chromium\.launch\s*\(/i;
const FILE_URL_BROWSER_NAVIGATION_PATTERN = /\bpage\.goto\s*\([^)]*file:\/\//i;
const SVG_HTML_INTERPOLATION_PATTERN =
/(?:<body>[\s\S]{0,240}\$\{[^}]*svg[^}]*\}|writeFile(?:Sync)?\s*\([^)]*\.html[^)]*\$\{[^}]*svg[^}]*\}|\$\{[^}]*svg[^}]*\}[\s\S]{0,240}<\/body>)/i;
const BROWSER_JS_DISABLED_PATTERN =
/javaScriptEnabled\s*:\s*false|Content-Security-Policy|script-src\s+['"]?none/i;
const STEALTH_BROWSER_CONTEXT_PATTERN =
/\b(?:stealth|anti[-\s]?detect|undetected|fingerprint spoof|navigator\.webdriver)\b/i;
const BOT_PROTECTION_BYPASS_PATTERN = /\b(?:captcha|cloudflare|turnstile|bot detection|waf)\b/i;
const BROWSER_SESSION_PERSISTENCE_PATTERN =
/\b(?:persistent_context|userDataDir|storageState|session persistence|persist(?:ed)? cookies?)\b/i;
const AGENT_OUTPUT_DIR_ARGUMENT_PATTERN =
/add_argument\s*\(\s*["']--outdir["']|args\.outdir|output_path\s*=\s*Path\s*\(\s*args\.outdir\s*\)/i;
const FFMPEG_FORCE_OUTPUT_PATTERN =
@@ -268,31 +249,6 @@ function findCredentialExposureInstruction(content: string) {
return null;
}
function findBrowserCredentialAutomation(content: string) {
const lines = content.split("\n");
for (let i = 0; i < lines.length; i += 1) {
const line = lines[i] ?? "";
if (BROWSER_USE_PASSWORD_ARGV_PATTERN.test(line)) {
return { line: i + 1, text: line };
}
}
if (
BROWSER_USE_AUTH_EVAL_PATTERN.test(content) &&
AUTHENTICATED_MAIL_CONTEXT_PATTERN.test(content) &&
PERSISTENCE_SCHEDULER_PATTERN.test(content)
) {
for (let i = 0; i < lines.length; i += 1) {
const line = lines[i] ?? "";
if (BROWSER_USE_AUTH_EVAL_PATTERN.test(line) || PERSISTENCE_SCHEDULER_PATTERN.test(line)) {
return { line: i + 1, text: line };
}
}
}
return null;
}
function redactSecretArgvEvidence(line: string) {
return line.replace(SECRET_ARGV_REDACTION_PATTERN, "$1$2[REDACTED]$2");
}
@@ -401,49 +357,6 @@ function isScopedOpenClawDelete(line: string, slug?: string) {
return false;
}
function hasShellVariableValidation(content: string, variable: string, useIndex: number) {
const escaped = variable.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const beforeUse = content.slice(0, useIndex);
const variableReference = String.raw`(?:\$\{${escaped}\}|\$${escaped})`;
const lengthCheck = new RegExp(
String.raw`\$\{#${escaped}\}\s*(?:-[a-z]\s+)?(?:[<>!=]=?|-[gl][te])`,
"m",
);
const controlCharStrip = new RegExp(
String.raw`(?:tr\s+-d\s+["']?\\(?:000|x00).{0,80}\\(?:037|x1[fF]|177|x7[fF])|${escaped}\s*=.*tr\s+-d)`,
"s",
);
const explicitValidation = new RegExp(
String.raw`(?:validate|sanitize|strip|clean)[A-Za-z0-9_ -]{0,60}${variableReference}|${variableReference}.{0,60}(?:validate|sanitize|strip|clean)`,
"is",
);
return (
lengthCheck.test(beforeUse) ||
controlCharStrip.test(beforeUse) ||
explicitValidation.test(beforeUse)
);
}
function findUnsafeBrowserTextInput(content: string) {
for (const assignment of content.matchAll(SHELL_POSITIONAL_ASSIGNMENT_PATTERN)) {
const variable = assignment[1];
if (!variable) continue;
const escaped = variable.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const browserTextPattern = new RegExp(
String.raw`\bbrowser\s+action=act\b[^\n]*\bkind=["']?type["']?[^\n]*\btext=(?:"\$${escaped}"|'\$${escaped}'|\$${escaped})(?![A-Za-z0-9_])`,
"i",
);
const match = content.match(browserTextPattern);
if (!match || match.index === undefined) continue;
if (hasShellVariableValidation(content, variable, match.index)) continue;
return findLineAtIndex(content, match.index);
}
return null;
}
function addFinding(
findings: ModerationFinding[],
finding: Omit<ModerationFinding, "evidence"> & { evidence: string },
@@ -602,21 +515,6 @@ function findJsSensitiveFileNetworkSend(content: string) {
return null;
}
function findUnsafeBrowserFileRender(content: string) {
if (!PLAYWRIGHT_CHROMIUM_PATTERN.test(content)) return null;
if (!FILE_URL_BROWSER_NAVIGATION_PATTERN.test(content)) return null;
if (!SVG_HTML_INTERPOLATION_PATTERN.test(content)) return null;
if (BROWSER_JS_DISABLED_PATTERN.test(content)) return null;
return findFirstLine(content, FILE_URL_BROWSER_NAVIGATION_PATTERN);
}
function findStealthBrowserAbuse(content: string) {
if (!STEALTH_BROWSER_CONTEXT_PATTERN.test(content)) return null;
if (!BOT_PROTECTION_BYPASS_PATTERN.test(content)) return null;
if (!BROWSER_SESSION_PERSISTENCE_PATTERN.test(content)) return null;
return findFirstLine(content, STEALTH_BROWSER_CONTEXT_PATTERN);
}
function findUnsafeAgentControlledFileWrite(content: string) {
if (!AGENT_OUTPUT_DIR_ARGUMENT_PATTERN.test(content)) return null;
if (!FFMPEG_FORCE_OUTPUT_PATTERN.test(content)) return null;
@@ -836,18 +734,6 @@ function scanCodeFile(
});
}
const unsafeBrowserTextInput = findUnsafeBrowserTextInput(content);
if (unsafeBrowserTextInput) {
addFinding(findings, {
code: REASON_CODES.UNSAFE_BROWSER_TEXT_INPUT,
severity: "warn",
file: path,
line: unsafeBrowserTextInput.line,
message: "Shell positional input is typed into browser automation without validation.",
evidence: unsafeBrowserTextInput.text,
});
}
const hostPlatformSourcePatch = findHostPlatformSourcePatch(content);
if (hostPlatformSourcePatch) {
addFinding(findings, {
@@ -860,32 +746,6 @@ function scanCodeFile(
});
}
const unsafeBrowserFileRender = findUnsafeBrowserFileRender(content);
if (unsafeBrowserFileRender) {
addFinding(findings, {
code: REASON_CODES.BROWSER_FILE_RENDER,
severity: "critical",
file: path,
line: unsafeBrowserFileRender.line,
message:
"Browser automation renders interpolated SVG/HTML from a file URL with JavaScript enabled.",
evidence: unsafeBrowserFileRender.text,
});
}
const stealthBrowserAbuse = findStealthBrowserAbuse(content);
if (stealthBrowserAbuse) {
addFinding(findings, {
code: REASON_CODES.STEALTH_BROWSER_ABUSE,
severity: "critical",
file: path,
line: stealthBrowserAbuse.line,
message:
"Browser automation advertises stealth/anti-detection behavior with bot-protection bypass and persistent sessions.",
evidence: stealthBrowserAbuse.text,
});
}
const unsafeAgentControlledFileWrite = findUnsafeAgentControlledFileWrite(content);
if (unsafeAgentControlledFileWrite) {
addFinding(findings, {
@@ -1085,18 +945,6 @@ function scanMarkdownFile(
});
}
const browserCredentialAutomation = findBrowserCredentialAutomation(content);
if (browserCredentialAutomation) {
addFinding(findings, {
code: REASON_CODES.BROWSER_CREDENTIAL_AUTOMATION,
severity: "critical",
file: path,
line: browserCredentialAutomation.line,
message: "Browser automation instructions expose credentials or persist authenticated eval.",
evidence: browserCredentialAutomation.text,
});
}
const secretArgvExposure = findSecretArgvExposure(content);
if (secretArgvExposure) {
addFinding(findings, {
@@ -1122,19 +970,6 @@ function scanMarkdownFile(
});
}
const stealthBrowserAbuse = findStealthBrowserAbuse(content);
if (stealthBrowserAbuse) {
addFinding(findings, {
code: REASON_CODES.STEALTH_BROWSER_ABUSE,
severity: "critical",
file: path,
line: stealthBrowserAbuse.line,
message:
"Browser automation advertises stealth/anti-detection behavior with bot-protection bypass and persistent sessions.",
evidence: stealthBrowserAbuse.text,
});
}
if (hasMaliciousInstallPrompt(content)) {
const match = findFirstLine(
content,
@@ -1163,18 +998,6 @@ function scanMarkdownFile(
});
}
const unsafeBrowserTextInput = findUnsafeBrowserTextInput(content);
if (unsafeBrowserTextInput) {
addFinding(findings, {
code: REASON_CODES.UNSAFE_BROWSER_TEXT_INPUT,
severity: "warn",
file: path,
line: unsafeBrowserTextInput.line,
message: "Shell positional input is typed into browser automation without validation.",
evidence: unsafeBrowserTextInput.text,
});
}
if (
/ignore\s+(all\s+)?previous\s+instructions/i.test(content) ||
/system\s*prompt\s*[:=]/i.test(content)
+1 -7
View File
@@ -12,7 +12,7 @@ export type ModerationFinding = {
evidence: string;
};
export const MODERATION_ENGINE_VERSION = "v2.4.24";
export const MODERATION_ENGINE_VERSION = "v2.4.25";
export const REASON_CODES = {
LLM_REVIEW: "review.llm_review",
@@ -21,13 +21,10 @@ export const REASON_CODES = {
GENERATED_SOURCE_TEMPLATE: "suspicious.generated_source_template_injection",
EXPOSED_RESOURCE_IDENTIFIER: "suspicious.exposed_resource_identifier",
DESTRUCTIVE_DELETE_COMMAND: "suspicious.destructive_delete_command",
UNSAFE_BROWSER_TEXT_INPUT: "suspicious.unsafe_browser_text_input",
EXPOSED_SECRET_LITERAL: "suspicious.exposed_secret_literal",
CREDENTIAL_EXPOSURE_INSTRUCTIONS: "suspicious.credential_exposure_instructions",
BROWSER_CREDENTIAL_AUTOMATION: "suspicious.browser_credential_automation",
SECRET_ARGV_EXPOSURE: "suspicious.secret_argv_exposure",
HOST_PLATFORM_SOURCE_PATCH: "suspicious.host_platform_source_patch",
BROWSER_FILE_RENDER: "suspicious.browser_file_render",
UNSAFE_FILE_WRITE: "suspicious.unsafe_file_write",
INSECURE_TLS_VERIFICATION: "suspicious.insecure_tls_verification",
AUTONOMOUS_CREDENTIAL_EGRESS: "suspicious.autonomous_credential_egress",
@@ -44,15 +41,12 @@ export const REASON_CODES = {
MANIFEST_PRIVILEGED_ALWAYS: "suspicious.privileged_always",
MALICIOUS_INSTALL_PROMPT: "malicious.install_terminal_payload",
KNOWN_BLOCKED_SIGNATURE: "malicious.known_blocked_signature",
STEALTH_BROWSER_ABUSE: "malicious.stealth_browser_abuse",
DEP_NOT_FOUND: "suspicious.dep_not_found_on_registry",
} as const;
const MALICIOUS_CODES = new Set<string>([
REASON_CODES.CRYPTO_MINING,
REASON_CODES.MALICIOUS_INSTALL_PROMPT,
REASON_CODES.KNOWN_BLOCKED_SIGNATURE,
REASON_CODES.STEALTH_BROWSER_ABUSE,
]);
const EXTERNALLY_CLEARABLE_SUSPICIOUS_CODES = new Set<string>([REASON_CODES.CREDENTIAL_HARVEST]);
+18
View File
@@ -0,0 +1,18 @@
export const Events = {
GitHubSkillSourceSyncStarted: "github_skill_source_sync.started",
GitHubSkillSourceSyncCompleted: "github_skill_source_sync.completed",
GitHubSkillSourceSyncSourceFailed: "github_skill_source_sync.source_failed",
GitHubSkillSourceSyncFailed: "github_skill_source_sync.failed",
} as const;
export type EventName = (typeof Events)[keyof typeof Events];
type EventPayload = Record<string, unknown>;
export function logEvent(event: EventName, payload: EventPayload = {}) {
console.log(JSON.stringify({ event, ...payload }));
}
export function logErrorEvent(event: EventName, payload: EventPayload = {}) {
console.error(JSON.stringify({ event, ...payload }));
}
+83 -38
View File
@@ -1,6 +1,6 @@
import { describe, expect, it, vi } from "vitest";
import type { Doc } from "../_generated/dataModel";
import { isOfficialPublisher } from "./officialPublishers";
import { hasOfficialPublisherRow, isOfficialPublisher } from "./officialPublishers";
function makePublisher(
overrides: Partial<Record<keyof Doc<"publishers">, unknown>>,
@@ -17,60 +17,105 @@ function makePublisher(
} as Doc<"publishers">;
}
function makeOfficialRow(publisherId: string) {
return {
_id: `officialPublishers:${publisherId}`,
_creationTime: 1,
publisherId,
createdAt: 1,
updatedAt: 1,
};
}
function makeCtx({ officialPublisherIds = [] }: { officialPublisherIds?: string[] } = {}) {
return {
db: {
query: vi.fn((table: string) => {
if (table !== "officialPublishers") {
throw new Error(`Unexpected table ${table}`);
}
return {
withIndex: vi.fn((_indexName: string, buildQuery: (q: unknown) => unknown) => {
let requestedPublisherId: string | undefined;
buildQuery({
eq: vi.fn((field: string, value: string) => {
if (field === "publisherId") requestedPublisherId = value;
return {};
}),
});
return {
unique: vi.fn(async () =>
requestedPublisherId && officialPublisherIds.includes(requestedPublisherId)
? makeOfficialRow(requestedPublisherId)
: null,
),
};
}),
};
}),
},
};
}
describe("isOfficialPublisher", () => {
it("treats the openclaw org publisher as official", async () => {
const ctx = { db: { query: vi.fn() } };
it("treats a publisher with an official row as official", async () => {
const ctx = makeCtx({ officialPublisherIds: ["publishers:acme"] });
await expect(
isOfficialPublisher(ctx as never, makePublisher({ handle: "openclaw" })),
isOfficialPublisher(ctx as never, makePublisher({ _id: "publishers:acme", handle: "acme" })),
).resolves.toBe(true);
});
it("treats the nvidia org publisher as official", async () => {
const ctx = { db: { query: vi.fn() } };
it("treats a personal publisher with an official row as official", async () => {
const ctx = makeCtx({ officialPublisherIds: ["publishers:alice"] });
await expect(
isOfficialPublisher(ctx as never, makePublisher({ handle: "nvidia" })),
isOfficialPublisher(
ctx as never,
makePublisher({
_id: "publishers:alice",
kind: "user",
handle: "alice",
linkedUserId: "users:alice",
}),
),
).resolves.toBe(true);
});
it("treats personal publishers for openclaw org members as official", async () => {
const openclaw = makePublisher({ _id: "publishers:openclaw", handle: "openclaw" });
it("does not treat legacy official handles as official without a row", async () => {
const ctx = makeCtx();
await expect(
isOfficialPublisher(
ctx as never,
makePublisher({ _id: "publishers:openclaw", handle: "openclaw" }),
),
).resolves.toBe(false);
});
it("does not inherit official status from org membership", async () => {
const personal = makePublisher({
_id: "publishers:alice",
kind: "user",
handle: "alice",
linkedUserId: "users:alice",
});
const ctx = {
db: {
query: vi.fn((table: string) => {
if (table === "publishers") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn(async () => openclaw),
})),
};
}
if (table === "publisherMembers") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn(async () => ({
_id: "publisherMembers:alice",
publisherId: "publishers:openclaw",
userId: "users:alice",
role: "publisher",
createdAt: 1,
updatedAt: 1,
})),
})),
};
}
throw new Error(`Unexpected table ${table}`);
}),
},
};
const ctx = makeCtx({ officialPublisherIds: ["publishers:openclaw"] });
await expect(isOfficialPublisher(ctx as never, personal)).resolves.toBe(true);
await expect(isOfficialPublisher(ctx as never, personal)).resolves.toBe(false);
});
it("can check raw official rows independently from active publisher state", async () => {
const ctx = makeCtx({ officialPublisherIds: ["publishers:acme"] });
await expect(
isOfficialPublisher(
ctx as never,
makePublisher({ _id: "publishers:acme", handle: "acme", deactivatedAt: 123 }),
),
).resolves.toBe(false);
await expect(hasOfficialPublisherRow(ctx as never, "publishers:acme" as never)).resolves.toBe(
true,
);
});
});
+12 -35
View File
@@ -1,51 +1,28 @@
import type { Doc } from "../_generated/dataModel";
import type { MutationCtx, QueryCtx } from "../_generated/server";
import { toPublicPublisher, type PublicPublisher } from "./public";
import {
getPublisherByHandle,
getPublisherMembership,
normalizePublisherHandle,
} from "./publishers";
const OFFICIAL_ORG_HANDLES = ["openclaw", "nvidia"] as const;
const OFFICIAL_ORG_HANDLE_SET = new Set<string>(OFFICIAL_ORG_HANDLES);
type DbCtx = Pick<QueryCtx | MutationCtx, "db">;
type OfficialPublisherCandidate = Pick<
Doc<"publishers">,
| "_id"
| "_creationTime"
| "kind"
| "handle"
| "displayName"
| "image"
| "bio"
| "linkedUserId"
| "deletedAt"
| "deactivatedAt"
>;
type OfficialPublisherCandidate = Pick<Doc<"publishers">, "_id" | "deletedAt" | "deactivatedAt">;
export async function isOfficialPublisher(
ctx: DbCtx,
publisher: OfficialPublisherCandidate | null | undefined,
): Promise<boolean> {
if (!publisher || publisher.deletedAt || publisher.deactivatedAt) return false;
if (publisher.kind === "org") {
const handle = normalizePublisherHandle(publisher.handle);
return Boolean(handle && OFFICIAL_ORG_HANDLE_SET.has(handle));
}
if (!publisher.linkedUserId) return false;
return await hasOfficialPublisherRow(ctx, publisher._id);
}
for (const officialOrgHandle of OFFICIAL_ORG_HANDLES) {
const officialOrg = await getPublisherByHandle(ctx, officialOrgHandle);
if (!officialOrg || officialOrg.deletedAt || officialOrg.deactivatedAt) continue;
const membership = await getPublisherMembership(ctx, officialOrg._id, publisher.linkedUserId);
if (membership) return true;
}
return false;
export async function hasOfficialPublisherRow(
ctx: DbCtx,
publisherId: Doc<"publishers">["_id"],
): Promise<boolean> {
const officialPublisher = await ctx.db
.query("officialPublishers")
.withIndex("by_publisher", (q) => q.eq("publisherId", publisherId))
.unique();
return Boolean(officialPublisher);
}
export async function toPublicPublisherWithOfficial(
+3
View File
@@ -20,6 +20,7 @@ const SHARED_KEYS = [
"summary",
"capabilityTags",
"executesCode",
"stats",
"runtimeId",
"scanStatus",
"softDeletedAt",
@@ -45,6 +46,7 @@ const CAPABILITY_SHARED_KEYS = [
"capabilityTags",
"executesCode",
"verificationTier",
"stats",
"scanStatus",
"softDeletedAt",
"createdAt",
@@ -70,6 +72,7 @@ const PLUGIN_CATEGORY_SHARED_KEYS = [
"pluginCategoryTags",
"executesCode",
"verificationTier",
"stats",
"scanStatus",
"softDeletedAt",
"createdAt",
+24
View File
@@ -66,6 +66,22 @@ describe("public skill mapping", () => {
});
});
it("exposes GitHub-backed skill source fields", () => {
const mapped = toPublicSkill(
makeSkill({
installKind: "github",
githubPath: "skills/demo",
githubCurrentCommit: "a".repeat(40),
}),
);
expect(mapped).toMatchObject({
installKind: "github",
githubPath: "skills/demo",
githubCurrentCommit: "a".repeat(40),
});
});
it("returns skill when moderationStatus is active", () => {
const skill = makeSkill({ moderationStatus: "active" });
expect(toPublicSkill(skill)).not.toBeNull();
@@ -93,6 +109,14 @@ describe("public skill mapping", () => {
});
expect(toPublicSkill(skill)).toBeNull();
});
it("filters out skills with a malicious moderation verdict", () => {
const skill = makeSkill({
moderationStatus: "active",
moderationVerdict: "malicious",
});
expect(toPublicSkill(skill)).toBeNull();
});
});
describe("public publisher mapping", () => {
+22 -2
View File
@@ -24,6 +24,12 @@ export type PublicSkill = Pick<
| "canonicalSkillId"
| "forkOf"
| "latestVersionId"
| "installKind"
| "githubPath"
| "githubCurrentCommit"
| "githubCurrentStatus"
| "githubScanStatus"
| "githubHasSkillCard"
| "tags"
| "capabilityTags"
| "badges"
@@ -31,7 +37,9 @@ export type PublicSkill = Pick<
| "isSuspicious"
| "createdAt"
| "updatedAt"
>;
> & {
githubSourceRepo?: string;
};
/**
* Minimum set of fields needed by `hydrateResults` to filter and convert
@@ -52,6 +60,10 @@ export type HydratableSkill = Pick<
| "canonicalSkillId"
| "forkOf"
| "latestVersionId"
| "installKind"
| "githubHasSkillCard"
| "githubCurrentStatus"
| "githubScanStatus"
| "latestVersionSummary"
| "tags"
| "capabilityTags"
@@ -64,11 +76,13 @@ export type HydratableSkill = Pick<
| "softDeletedAt"
| "moderationStatus"
| "moderationFlags"
| "moderationVerdict"
| "moderationReason"
| "isSuspicious"
| "createdAt"
| "updatedAt"
>;
> &
Partial<Pick<Doc<"skills">, "githubPath" | "githubCurrentCommit">>;
export type PublicSoul = Pick<
Doc<"souls">,
@@ -149,6 +163,12 @@ export function toPublicSkill(skill: HydratableSkill | null | undefined): Public
canonicalSkillId: skill.canonicalSkillId,
forkOf: skill.forkOf,
latestVersionId: skill.latestVersionId,
installKind: skill.installKind,
githubPath: skill.githubPath,
githubCurrentCommit: skill.githubCurrentCommit,
githubCurrentStatus: skill.githubCurrentStatus,
githubScanStatus: skill.githubScanStatus,
githubHasSkillCard: skill.githubHasSkillCard,
tags: skill.tags,
capabilityTags: skill.capabilityTags,
badges: skill.badges,
@@ -0,0 +1,15 @@
import { describe, expect, it } from "vitest";
import { isReservedPublicOwnerHandle } from "./publicRouteReservations";
describe("public route reservations", () => {
it.each(["admin", "clawhub", "plugins", "skills"])(
"reserves @%s as a public owner handle",
(handle) => {
expect(isReservedPublicOwnerHandle(handle)).toBe(true);
},
);
it("does not normalize at-sign prefixes", () => {
expect(isReservedPublicOwnerHandle("@clawhub")).toBe(false);
});
});
+1 -1
View File
@@ -1,4 +1,4 @@
const RESERVED_PUBLIC_OWNER_HANDLES = new Set(["plugins", "skills"]);
const RESERVED_PUBLIC_OWNER_HANDLES = new Set(["admin", "clawhub", "plugins", "skills"]);
const RESERVED_UNSCOPED_PACKAGE_NAMES = new Set(["publish"]);
export function isReservedPublicOwnerHandle(handle: string | undefined | null) {
+241
View File
@@ -2,8 +2,13 @@
import { describe, expect, it } from "vitest";
import {
computeCurrentSkillTemporalAbuseScore,
computeHistoricalSkillTemporalAbuseScore,
computePublisherAbuseRawScore,
computeTemporalAbuseCohortBenchmark,
computeTemporalPublisherAbuseZScore,
DEFAULT_PUBLISHER_ABUSE_MODEL_CONFIG,
labelForTemporalPublisherAbuse,
labelForPublisherAbuseZScore,
scorePublisherAbuseCohort,
} from "./publisherAbuseScoring";
@@ -18,6 +23,40 @@ describe("publisher abuse scoring", () => {
);
});
it("maps temporal labels to review-compatible z-scores", () => {
const review = computeTemporalPublisherAbuseZScore({
label: "review",
highTemporalSkillCount: 1,
maxTemporalPressure: 20,
});
const potentialBan = computeTemporalPublisherAbuseZScore({
label: "potential_ban_candidate",
highTemporalSkillCount: 2,
maxTemporalPressure: 20,
});
expect(
computeTemporalPublisherAbuseZScore({
label: "pass",
highTemporalSkillCount: 0,
maxTemporalPressure: 0,
}),
).toBe(0);
expect(review).toBeGreaterThanOrEqual(1.5);
expect(review).toBeLessThan(2.5);
expect(potentialBan).toBeGreaterThanOrEqual(2.5);
expect(potentialBan).toBeGreaterThan(review);
});
it("escalates one P99 temporal hit as a potential ban candidate", () => {
expect(
labelForTemporalPublisherAbuse({ highTemporalSkillCount: 1, p99TemporalSkillCount: 1 }),
).toBe("potential_ban_candidate");
expect(
labelForTemporalPublisherAbuse({ highTemporalSkillCount: 1, p99TemporalSkillCount: 0 }),
).toBe("review");
});
it("keeps a high-volume publisher with strong usage below low-engagement publishers", () => {
const scored = scorePublisherAbuseCohort([
publisher("byungkyu", {
@@ -125,8 +164,198 @@ describe("publisher abuse scoring", () => {
"pass",
);
});
it("flags a current 7-day download spike with flat installs", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
benchmark: temporalBenchmark({
downloads30dP95: 2_000,
downloads30dP99: 5_000,
spikeMultiplier7dP95: 5,
spikeMultiplier7dP99: 20,
}),
dailyStats: [
...dailyRange(64, 30, { downloads: 5, installs: 0 }),
...dailyRange(94, 7, { downloads: 200, installs: 0 }),
],
});
expect(score.spike).toBe(true);
expect(score.sustained).toBe(false);
expect(score.recent7Downloads).toBe(1_400);
expect(score.recent7Installs).toBe(0);
expect(score.previous30Downloads).toBe(150);
expect(score.spikeMultiplier).toBeCloseTo(14);
expect(score.spikeMultiplierCohortBand).toBe("p95");
expect(score.reasonCodes).toContain("temporal_download_spike_flat_installs");
});
it("flags sustained high downloads with flat installs", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
benchmark: temporalBenchmark({
downloads30dP95: 3_000,
downloads30dP99: 6_000,
spikeMultiplier7dP95: 20,
spikeMultiplier7dP99: 50,
}),
dailyStats: dailyRange(71, 30, { downloads: 120, installs: 0 }),
});
expect(score.spike).toBe(false);
expect(score.sustained).toBe(true);
expect(score.recent30Downloads).toBe(3_600);
expect(score.recent30Installs).toBe(0);
expect(score.downloadInstallRatio30).toBe(3_600);
expect(score.downloads30dCohortBand).toBe("p95");
expect(score.reasonCodes).toContain("temporal_sustained_downloads_flat_installs");
});
it("flags high-volume installs that track downloads too closely", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
dailyStats: dailyRange(94, 7, { downloads: 200, installs: 180 }),
});
expect(score.nearConversion).toBe(true);
expect(score.recent7Downloads).toBe(1_400);
expect(score.recent7Installs).toBe(1_260);
expect(score.installDownloadRatio7).toBeCloseTo(0.9);
expect(score.reasonCodes).toContain("temporal_installs_track_downloads");
});
it("keeps low-volume one-to-one install traffic below close-ratio thresholds", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
dailyStats: dailyRange(94, 7, { downloads: 1, installs: 1 }),
});
expect(score.nearConversion).toBe(false);
expect(score.reasonCodes).not.toContain("temporal_installs_track_downloads");
});
it("keeps observed high-end install ratios below close-ratio thresholds", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
dailyStats: dailyRange(94, 7, { downloads: 20, installs: 1 }),
});
expect(score.recent7Downloads).toBe(140);
expect(score.recent7Installs).toBe(7);
expect(score.installDownloadRatio7).toBeCloseTo(0.05);
expect(score.nearConversion).toBe(false);
expect(score.reasonCodes).not.toContain("temporal_installs_track_downloads");
});
it("requires installs to be close to downloads, not just statistically elevated", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
dailyStats: dailyRange(94, 7, { downloads: 300, installs: 15 }),
});
expect(score.recent7Downloads).toBe(2_100);
expect(score.recent7Installs).toBe(105);
expect(score.installDownloadRatio7).toBeCloseTo(0.05);
expect(score.installDownloadExcessZScore7).toBeGreaterThan(10);
expect(score.nearConversion).toBe(false);
expect(score.reasonCodes).not.toContain("temporal_installs_track_downloads");
});
it("reports a 30-day close-ratio window when the 7-day threshold is not met", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
dailyStats: dailyRange(71, 30, { downloads: 100, installs: 80 }),
});
expect(score.nearConversion).toBe(true);
expect(score.installDownloadRatio7).toBeCloseTo(0.8);
expect(score.installDownloadRatio30).toBeCloseTo(0.8);
expect(score.nearConversionWindowStartDay).toBe(71);
expect(score.nearConversionWindowEndDay).toBe(100);
});
it("keeps ordinary steady download traffic below temporal thresholds", () => {
const todayDay = 100;
const score = computeCurrentSkillTemporalAbuseScore({
todayDay,
benchmark: temporalBenchmark({
downloads30dP95: 4_000,
downloads30dP99: 8_000,
spikeMultiplier7dP95: 20,
spikeMultiplier7dP99: 50,
}),
dailyStats: [
...dailyRange(64, 30, { downloads: 80, installs: 1 }),
...dailyRange(94, 7, { downloads: 85, installs: 1 }),
],
});
expect(score.spike).toBe(false);
expect(score.sustained).toBe(false);
expect(score.pressure).toBe(0);
expect(score.reasonCodes).toEqual([]);
});
it("finds historical spike and sustained windows for backfill scans", () => {
const score = computeHistoricalSkillTemporalAbuseScore({
benchmark: temporalBenchmark({
downloads30dP95: 3_000,
downloads30dP99: 10_000,
spikeMultiplier7dP95: 5,
spikeMultiplier7dP99: 25,
}),
dailyStats: [
...dailyRange(10, 30, { downloads: 3, installs: 0 }),
...dailyRange(40, 7, { downloads: 220, installs: 0 }),
...dailyRange(80, 30, { downloads: 150, installs: 0 }),
],
});
expect(score.spike).toBe(true);
expect(score.sustained).toBe(true);
expect(score.spikeWindowStartDay).toBe(40);
expect(score.sustainedWindowStartDay).toBe(80);
expect(score.reasonCodes).toEqual([
"temporal_download_spike_flat_installs",
"temporal_sustained_downloads_flat_installs",
]);
});
it("computes cohort benchmark percentiles from scanned skill windows", () => {
const benchmark = computeTemporalAbuseCohortBenchmark([
...Array.from({ length: 95 }, () => ({ recent30Downloads: 100, spikeMultiplier: 1 })),
...Array.from({ length: 4 }, () => ({ recent30Downloads: 500, spikeMultiplier: 2 })),
{ recent30Downloads: 10_000, spikeMultiplier: 30 },
]);
expect(benchmark.sampleSize).toBe(100);
expect(benchmark.downloads30dMedian).toBe(100);
expect(benchmark.downloads30dP95).toBe(100);
expect(benchmark.downloads30dP99).toBe(500);
expect(benchmark.spikeMultiplier7dP99).toBe(2);
});
});
function temporalBenchmark(overrides = {}) {
return {
sampleSize: 100,
downloads30dAverage: 500,
downloads30dMedian: 100,
downloads30dP95: 1_000,
downloads30dP99: 5_000,
spikeMultiplier7dP95: 5,
spikeMultiplier7dP99: 25,
...overrides,
};
}
function publisher(
handleSnapshot: string,
stats: {
@@ -143,3 +372,15 @@ function publisher(
...stats,
};
}
function dailyRange(
startDay: number,
length: number,
stats: { downloads: number; installs: number },
) {
return Array.from({ length }, (_, index) => ({
day: startDay + index,
downloads: stats.downloads,
installs: stats.installs,
}));
}
+461
View File
@@ -1,4 +1,5 @@
export const PUBLISHER_ABUSE_MODEL_VERSION = "publisher-abuse-pressure.v1";
export const PUBLISHER_TEMPORAL_ABUSE_MODEL_VERSION = "publisher-abuse-temporal.v1";
export type PublisherAbuseLabel = "pass" | "review" | "potential_ban_candidate";
@@ -50,6 +51,52 @@ export type PublisherAbuseScore = PublisherAbuseRawScore & {
zScore: number;
};
export type SkillTemporalAbuseDailyStat = {
day: number;
downloads: number;
installs: number;
};
export type SkillTemporalAbuseScore = {
spike: boolean;
sustained: boolean;
nearConversion: boolean;
pressure: number;
recent7Downloads: number;
recent7Installs: number;
previous30Downloads: number;
baseline7Downloads: number;
spikeMultiplier: number;
recent30Downloads: number;
recent30Installs: number;
downloadInstallRatio30: number;
downloads30dCohortBand?: "p95" | "p99";
spikeMultiplierCohortBand?: "p95" | "p99";
downloads30dVsPeerP95?: number;
spikeMultiplierVsPeerP95?: number;
installDownloadRatio7: number;
installDownloadRatio30: number;
installDownloadExcessZScore7: number;
installDownloadExcessZScore30: number;
spikeWindowStartDay?: number;
spikeWindowEndDay?: number;
sustainedWindowStartDay?: number;
sustainedWindowEndDay?: number;
nearConversionWindowStartDay?: number;
nearConversionWindowEndDay?: number;
reasonCodes: string[];
};
export type TemporalAbuseCohortBenchmark = {
sampleSize: number;
downloads30dAverage: number;
downloads30dMedian: number;
downloads30dP95: number;
downloads30dP99: number;
spikeMultiplier7dP95: number;
spikeMultiplier7dP99: number;
};
export const DEFAULT_PUBLISHER_ABUSE_MODEL_CONFIG = {
modelVersion: PUBLISHER_ABUSE_MODEL_VERSION,
skillPivot: 100,
@@ -70,6 +117,18 @@ export const DEFAULT_PUBLISHER_ABUSE_MODEL_CONFIG = {
} satisfies PublisherAbuseModelConfig;
const MIN_PRESSURE_FOR_LOG = 1e-9;
const TEMPORAL_SPIKE_RECENT_DAYS = 7;
const TEMPORAL_SPIKE_BASELINE_DAYS = 30;
const TEMPORAL_SUSTAINED_DAYS = 30;
const TEMPORAL_MAX_SPIKE_INSTALLS = 2;
const TEMPORAL_MAX_SUSTAINED_INSTALLS = 5;
const TEMPORAL_MIN_BASELINE_7_DOWNLOADS = 100;
const TEMPORAL_MIN_NEAR_CONVERSION_7_DOWNLOADS = 1_000;
const TEMPORAL_MIN_NEAR_CONVERSION_30_DOWNLOADS = 2_000;
const TEMPORAL_MIN_NEAR_CONVERSION_INSTALLS = 500;
const TEMPORAL_EXPECTED_INSTALL_DOWNLOAD_RATIO = 0.012;
const TEMPORAL_MIN_INSTALL_DOWNLOAD_RATIO = 0.5;
const TEMPORAL_MIN_INSTALL_DOWNLOAD_EXCESS_Z_SCORE = 50;
export function labelForPublisherAbuseZScore(
zScore: number,
@@ -80,6 +139,21 @@ export function labelForPublisherAbuseZScore(
return "pass";
}
export function computeTemporalPublisherAbuseZScore(input: {
label: PublisherAbuseLabel;
highTemporalSkillCount: number;
maxTemporalPressure: number;
}): number {
if (input.label === "pass") return 0;
const pressureBoost = Math.log10(Math.max(input.maxTemporalPressure, 1) + 1) / 2;
const skillCountBoost = Math.max(0, input.highTemporalSkillCount - 2) * 0.2;
if (input.label === "potential_ban_candidate") {
return 2.5 + Math.min(2, pressureBoost + skillCountBoost);
}
return 1.5 + Math.min(0.99, pressureBoost);
}
export function computePublisherAbuseRawScore(
input: PublisherAbuseInput,
config: PublisherAbuseModelConfig = DEFAULT_PUBLISHER_ABUSE_MODEL_CONFIG,
@@ -204,6 +278,168 @@ export function summarizePublisherAbuseLogPressure(
};
}
export function computeCurrentSkillTemporalAbuseScore(input: {
todayDay: number;
dailyStats: SkillTemporalAbuseDailyStat[];
benchmark?: TemporalAbuseCohortBenchmark;
}): SkillTemporalAbuseScore {
const statsByDay = aggregateSkillTemporalDailyStats(input.dailyStats);
const score = computeSkillTemporalAbuseScoreForWindows({
statsByDay,
spikeStartDay: input.todayDay - TEMPORAL_SPIKE_RECENT_DAYS + 1,
sustainedStartDay: input.todayDay - TEMPORAL_SUSTAINED_DAYS + 1,
});
return classifySkillTemporalAbuseScore(score, input.benchmark);
}
export function computeHistoricalSkillTemporalAbuseScore(input: {
dailyStats: SkillTemporalAbuseDailyStat[];
benchmark?: TemporalAbuseCohortBenchmark;
}): SkillTemporalAbuseScore {
const statsByDay = aggregateSkillTemporalDailyStats(input.dailyStats);
const days = [...statsByDay.keys()];
if (days.length === 0) return emptySkillTemporalAbuseScore();
const minDay = Math.min(...days);
const maxDay = Math.max(...days);
let bestSpike = emptySkillTemporalAbuseScore();
let bestSustained = emptySkillTemporalAbuseScore();
let bestNearConversion = emptySkillTemporalAbuseScore();
for (let startDay = minDay; startDay <= maxDay; startDay += 1) {
if (startDay + TEMPORAL_SPIKE_RECENT_DAYS - 1 <= maxDay) {
const score = classifySkillTemporalAbuseScore(
computeSkillTemporalAbuseScoreForWindows({
statsByDay,
spikeStartDay: startDay,
sustainedStartDay: startDay,
}),
input.benchmark,
);
if (score.spike && score.spikeMultiplier > bestSpike.spikeMultiplier) {
bestSpike = score;
}
if (
score.nearConversion &&
score.nearConversionWindowEndDay === startDay + TEMPORAL_SPIKE_RECENT_DAYS - 1 &&
score.installDownloadRatio7 > bestNearConversion.installDownloadRatio7
) {
bestNearConversion = score;
}
}
if (startDay + TEMPORAL_SUSTAINED_DAYS - 1 <= maxDay) {
const score = classifySkillTemporalAbuseScore(
computeSkillTemporalAbuseScoreForWindows({
statsByDay,
spikeStartDay: startDay,
sustainedStartDay: startDay,
}),
input.benchmark,
);
if (score.sustained && score.recent30Downloads > bestSustained.recent30Downloads) {
bestSustained = score;
}
if (
score.nearConversion &&
score.nearConversionWindowEndDay === startDay + TEMPORAL_SUSTAINED_DAYS - 1 &&
score.installDownloadRatio30 > bestNearConversion.installDownloadRatio30
) {
bestNearConversion = score;
}
}
}
return mergeTemporalAbuseWindowScores(bestSpike, bestSustained, bestNearConversion);
}
export function labelForTemporalPublisherAbuse(input: {
highTemporalSkillCount: number;
p99TemporalSkillCount?: number;
}): PublisherAbuseLabel {
if ((input.p99TemporalSkillCount ?? 0) >= 1 || input.highTemporalSkillCount >= 2) {
return "potential_ban_candidate";
}
if (input.highTemporalSkillCount >= 1) return "review";
return "pass";
}
export function computeTemporalAbuseCohortBenchmark(
scores: Pick<SkillTemporalAbuseScore, "recent30Downloads" | "spikeMultiplier">[],
): TemporalAbuseCohortBenchmark {
const downloads30d = scores.map((score) => nonNegative(score.recent30Downloads));
const spikeMultipliers = scores.map((score) => nonNegative(score.spikeMultiplier));
return {
sampleSize: scores.length,
downloads30dAverage: average(downloads30d),
downloads30dMedian: percentile(downloads30d, 0.5),
downloads30dP95: percentile(downloads30d, 0.95),
downloads30dP99: percentile(downloads30d, 0.99),
spikeMultiplier7dP95: percentile(spikeMultipliers, 0.95),
spikeMultiplier7dP99: percentile(spikeMultipliers, 0.99),
};
}
export function classifySkillTemporalAbuseScore(
score: SkillTemporalAbuseScore,
benchmark: TemporalAbuseCohortBenchmark | undefined,
): SkillTemporalAbuseScore {
if (!benchmark || benchmark.sampleSize <= 0) return score;
const downloads30dVsPeerP95 = score.recent30Downloads / Math.max(1, benchmark.downloads30dP95);
const spikeMultiplierVsPeerP95 =
score.spikeMultiplier / Math.max(1, benchmark.spikeMultiplier7dP95);
const downloads30dCohortBand =
score.recent30Installs <= TEMPORAL_MAX_SUSTAINED_INSTALLS
? percentileBand({
value: score.recent30Downloads,
p95: benchmark.downloads30dP95,
p99: benchmark.downloads30dP99,
})
: undefined;
const spikeMultiplierCohortBand =
score.recent7Installs <= TEMPORAL_MAX_SPIKE_INSTALLS && score.recent7Downloads > 0
? percentileBand({
value: score.spikeMultiplier,
p95: benchmark.spikeMultiplier7dP95,
p99: benchmark.spikeMultiplier7dP99,
})
: undefined;
const spike = Boolean(spikeMultiplierCohortBand);
const sustained = Boolean(downloads30dCohortBand);
const nearConversion = score.nearConversion;
const nearConversionPressure = nearConversion
? Math.max(score.installDownloadExcessZScore7, score.installDownloadExcessZScore30)
: 0;
const reasonCodes: string[] = [];
if (spike) reasonCodes.push("temporal_download_spike_flat_installs");
if (sustained) reasonCodes.push("temporal_sustained_downloads_flat_installs");
if (nearConversion) reasonCodes.push("temporal_installs_track_downloads");
return {
...score,
spike,
sustained,
nearConversion,
pressure: Math.max(
spike ? spikeMultiplierVsPeerP95 : 0,
sustained ? downloads30dVsPeerP95 : 0,
nearConversionPressure,
),
downloads30dCohortBand,
spikeMultiplierCohortBand,
downloads30dVsPeerP95,
spikeMultiplierVsPeerP95,
spikeWindowStartDay: spike ? score.spikeWindowStartDay : undefined,
spikeWindowEndDay: spike ? score.spikeWindowEndDay : undefined,
sustainedWindowStartDay: sustained ? score.sustainedWindowStartDay : undefined,
sustainedWindowEndDay: sustained ? score.sustainedWindowEndDay : undefined,
nearConversionWindowStartDay: nearConversion ? score.nearConversionWindowStartDay : undefined,
nearConversionWindowEndDay: nearConversion ? score.nearConversionWindowEndDay : undefined,
reasonCodes,
};
}
function reasonCodesForPublisher(input: {
publishedSkills: number;
installsPerSkill: number;
@@ -229,6 +465,213 @@ function reasonCodesForPublisher(input: {
return codes;
}
function computeSkillTemporalAbuseScoreForWindows(input: {
statsByDay: Map<number, { downloads: number; installs: number }>;
spikeStartDay: number;
sustainedStartDay: number;
}): SkillTemporalAbuseScore {
const spikeEndDay = input.spikeStartDay + TEMPORAL_SPIKE_RECENT_DAYS - 1;
const sustainedEndDay = input.sustainedStartDay + TEMPORAL_SUSTAINED_DAYS - 1;
const recent7 = sumTemporalStatsRange(input.statsByDay, input.spikeStartDay, spikeEndDay);
const previous30 = sumTemporalStatsRange(
input.statsByDay,
input.spikeStartDay - TEMPORAL_SPIKE_BASELINE_DAYS,
input.spikeStartDay - 1,
);
const recent30 = sumTemporalStatsRange(
input.statsByDay,
input.sustainedStartDay,
sustainedEndDay,
);
const baseline7Downloads = Math.max(
TEMPORAL_MIN_BASELINE_7_DOWNLOADS,
(previous30.downloads / TEMPORAL_SPIKE_BASELINE_DAYS) * TEMPORAL_SPIKE_RECENT_DAYS,
);
const spikeMultiplier = baseline7Downloads > 0 ? recent7.downloads / baseline7Downloads : 0;
const downloadInstallRatio30 = recent30.downloads / Math.max(1, recent30.installs);
const installDownloadRatio7 = recent7.installs / Math.max(1, recent7.downloads);
const installDownloadRatio30 = recent30.installs / Math.max(1, recent30.downloads);
const installDownloadExcessZScore7 = installDownloadExcessZScore({
downloads: recent7.downloads,
installs: recent7.installs,
});
const installDownloadExcessZScore30 = installDownloadExcessZScore({
downloads: recent30.downloads,
installs: recent30.installs,
});
const nearConversion7 =
recent7.downloads >= TEMPORAL_MIN_NEAR_CONVERSION_7_DOWNLOADS &&
recent7.installs >= TEMPORAL_MIN_NEAR_CONVERSION_INSTALLS &&
installDownloadRatio7 >= TEMPORAL_MIN_INSTALL_DOWNLOAD_RATIO &&
installDownloadExcessZScore7 >= TEMPORAL_MIN_INSTALL_DOWNLOAD_EXCESS_Z_SCORE;
const nearConversion30 =
recent30.downloads >= TEMPORAL_MIN_NEAR_CONVERSION_30_DOWNLOADS &&
recent30.installs >= TEMPORAL_MIN_NEAR_CONVERSION_INSTALLS &&
installDownloadRatio30 >= TEMPORAL_MIN_INSTALL_DOWNLOAD_RATIO &&
installDownloadExcessZScore30 >= TEMPORAL_MIN_INSTALL_DOWNLOAD_EXCESS_Z_SCORE;
const nearConversion = nearConversion7 || nearConversion30;
const reasonCodes: string[] = [];
if (nearConversion) reasonCodes.push("temporal_installs_track_downloads");
return {
spike: false,
sustained: false,
nearConversion,
pressure: nearConversion
? Math.max(installDownloadExcessZScore7, installDownloadExcessZScore30)
: 0,
recent7Downloads: recent7.downloads,
recent7Installs: recent7.installs,
previous30Downloads: previous30.downloads,
baseline7Downloads,
spikeMultiplier,
recent30Downloads: recent30.downloads,
recent30Installs: recent30.installs,
downloadInstallRatio30,
installDownloadRatio7,
installDownloadRatio30,
installDownloadExcessZScore7,
installDownloadExcessZScore30,
spikeWindowStartDay: input.spikeStartDay,
spikeWindowEndDay: spikeEndDay,
sustainedWindowStartDay: input.sustainedStartDay,
sustainedWindowEndDay: sustainedEndDay,
nearConversionWindowStartDay: nearConversion7
? input.spikeStartDay
: nearConversion30
? input.sustainedStartDay
: undefined,
nearConversionWindowEndDay: nearConversion7
? spikeEndDay
: nearConversion30
? sustainedEndDay
: undefined,
reasonCodes,
};
}
function mergeTemporalAbuseWindowScores(
bestSpike: SkillTemporalAbuseScore,
bestSustained: SkillTemporalAbuseScore,
bestNearConversion: SkillTemporalAbuseScore,
): SkillTemporalAbuseScore {
if (!bestSpike.spike && !bestSustained.sustained && !bestNearConversion.nearConversion) {
return emptySkillTemporalAbuseScore();
}
const reasonCodes: string[] = [];
if (bestSpike.spike) reasonCodes.push("temporal_download_spike_flat_installs");
if (bestSustained.sustained) reasonCodes.push("temporal_sustained_downloads_flat_installs");
if (bestNearConversion.nearConversion) reasonCodes.push("temporal_installs_track_downloads");
return {
spike: bestSpike.spike,
sustained: bestSustained.sustained,
nearConversion: bestNearConversion.nearConversion,
pressure: Math.max(bestSpike.pressure, bestSustained.pressure, bestNearConversion.pressure),
recent7Downloads: bestSpike.spike
? bestSpike.recent7Downloads
: bestNearConversion.recent7Downloads,
recent7Installs: bestSpike.spike
? bestSpike.recent7Installs
: bestNearConversion.recent7Installs,
previous30Downloads: bestSpike.spike
? bestSpike.previous30Downloads
: bestNearConversion.previous30Downloads,
baseline7Downloads: bestSpike.spike
? bestSpike.baseline7Downloads
: bestNearConversion.baseline7Downloads,
spikeMultiplier: bestSpike.spike
? bestSpike.spikeMultiplier
: bestNearConversion.spikeMultiplier,
recent30Downloads: bestSustained.sustained
? bestSustained.recent30Downloads
: bestNearConversion.recent30Downloads,
recent30Installs: bestSustained.sustained
? bestSustained.recent30Installs
: bestNearConversion.recent30Installs,
downloadInstallRatio30: bestSustained.sustained
? bestSustained.downloadInstallRatio30
: bestNearConversion.downloadInstallRatio30,
installDownloadRatio7: bestNearConversion.installDownloadRatio7,
installDownloadRatio30: bestNearConversion.installDownloadRatio30,
installDownloadExcessZScore7: bestNearConversion.installDownloadExcessZScore7,
installDownloadExcessZScore30: bestNearConversion.installDownloadExcessZScore30,
downloads30dCohortBand: bestSustained.downloads30dCohortBand,
spikeMultiplierCohortBand: bestSpike.spikeMultiplierCohortBand,
downloads30dVsPeerP95: bestSustained.downloads30dVsPeerP95,
spikeMultiplierVsPeerP95: bestSpike.spikeMultiplierVsPeerP95,
spikeWindowStartDay: bestSpike.spikeWindowStartDay,
spikeWindowEndDay: bestSpike.spikeWindowEndDay,
sustainedWindowStartDay: bestSustained.sustainedWindowStartDay,
sustainedWindowEndDay: bestSustained.sustainedWindowEndDay,
nearConversionWindowStartDay: bestNearConversion.nearConversionWindowStartDay,
nearConversionWindowEndDay: bestNearConversion.nearConversionWindowEndDay,
reasonCodes,
};
}
function aggregateSkillTemporalDailyStats(dailyStats: SkillTemporalAbuseDailyStat[]) {
const byDay = new Map<number, { downloads: number; installs: number }>();
for (const point of dailyStats) {
if (!Number.isFinite(point.day)) continue;
const day = Math.trunc(point.day);
const existing = byDay.get(day) ?? { downloads: 0, installs: 0 };
existing.downloads += nonNegative(point.downloads);
existing.installs += nonNegative(point.installs);
byDay.set(day, existing);
}
return byDay;
}
function sumTemporalStatsRange(
statsByDay: Map<number, { downloads: number; installs: number }>,
startDay: number,
endDay: number,
) {
let downloads = 0;
let installs = 0;
for (let day = startDay; day <= endDay; day += 1) {
const point = statsByDay.get(day);
if (!point) continue;
downloads += point.downloads;
installs += point.installs;
}
return { downloads, installs };
}
function emptySkillTemporalAbuseScore(): SkillTemporalAbuseScore {
return {
spike: false,
sustained: false,
nearConversion: false,
pressure: 0,
recent7Downloads: 0,
recent7Installs: 0,
previous30Downloads: 0,
baseline7Downloads: TEMPORAL_MIN_BASELINE_7_DOWNLOADS,
spikeMultiplier: 0,
recent30Downloads: 0,
recent30Installs: 0,
downloadInstallRatio30: 0,
installDownloadRatio7: 0,
installDownloadRatio30: 0,
installDownloadExcessZScore7: 0,
installDownloadExcessZScore30: 0,
reasonCodes: [],
};
}
function installDownloadExcessZScore(input: { downloads: number; installs: number }) {
if (input.downloads <= 0) return 0;
const expected = input.downloads * TEMPORAL_EXPECTED_INSTALL_DOWNLOAD_RATIO;
const variance =
input.downloads *
TEMPORAL_EXPECTED_INSTALL_DOWNLOAD_RATIO *
(1 - TEMPORAL_EXPECTED_INSTALL_DOWNLOAD_RATIO);
const stdDev = Math.sqrt(Math.max(variance, 1));
return (input.installs - expected) / stdDev;
}
function nonNegative(value: number) {
return Number.isFinite(value) ? Math.max(0, value) : 0;
}
@@ -238,6 +681,24 @@ function average(values: number[]) {
return values.reduce((sum, value) => sum + value, 0) / values.length;
}
function percentile(values: number[], quantile: number) {
if (values.length === 0) return 0;
const sorted = [...values].sort((left, right) => left - right);
const index = Math.max(0, Math.min(sorted.length - 1, Math.ceil(quantile * sorted.length) - 1));
return sorted[index] ?? 0;
}
function percentileBand(input: {
value: number;
p95: number;
p99: number;
}): "p95" | "p99" | undefined {
if (input.value <= 0) return undefined;
if (input.p99 > 0 && input.value > input.p99) return "p99";
if (input.p95 > 0 && input.value > input.p95) return "p95";
return undefined;
}
function standardDeviation(values: number[], mean: number) {
if (values.length === 0) return 0;
const variance = values.reduce((sum, value) => sum + (value - mean) ** 2, 0) / values.length;
+196
View File
@@ -0,0 +1,196 @@
import { describe, expect, it } from "vitest";
import { buildGitHubSkillCatalogDisplay } from "./publisherCatalogDisplay";
const baseItem = {
kind: "skill" as const,
summary: null,
icon: null,
href: "/nvidia/example",
downloads: 0,
stars: 0,
isOfficial: true,
updatedAt: 1,
sourceBacked: true,
sourceRepo: "NVIDIA/skills",
sourcePath: null,
sourceVerifiedCommit: null,
};
describe("buildGitHubSkillCatalogDisplay", () => {
it("groups source-backed skills by manifest entries and ignores missing entries", () => {
const display = buildGitHubSkillCatalogDisplay({
sources: [
{
_id: "githubSkillSources:nvidia",
repo: "NVIDIA/skills",
displayManifestStatus: "ok",
displayManifest: {
notGrouped: "bottom",
groupings: [
{
title: "Agentic AI",
description: "Agentic AI skills.",
skills: ["aiq-deploy", "missing-upstream-entry"],
},
{
title: "Vision AI",
skills: ["vision-helper"],
},
],
},
},
],
items: [
{
...baseItem,
_id: "skills:aiq-deploy",
displayName: "AIQ Deploy",
slug: "aiq-deploy",
sourceId: "githubSkillSources:nvidia",
},
{
...baseItem,
_id: "skills:vision-helper",
displayName: "Vision Helper",
slug: "vision-helper",
sourceId: "githubSkillSources:nvidia",
},
],
});
expect(display).toMatchObject({
mode: "grouped",
sourceRepos: ["NVIDIA/skills"],
sections: [
{
title: "Agentic AI",
description: "Agentic AI skills.",
sourceRepo: "NVIDIA/skills",
items: [{ displayName: "AIQ Deploy" }],
},
{
title: "Vision AI",
sourceRepo: "NVIDIA/skills",
items: [{ displayName: "Vision Helper" }],
},
],
});
});
it("matches manifest entries by normalized display name and places unlisted skills at the requested edge", () => {
const display = buildGitHubSkillCatalogDisplay({
sources: [
{
_id: "githubSkillSources:nvidia",
repo: "NVIDIA/skills",
displayManifestStatus: "ok",
displayManifest: {
notGrouped: "top",
groupings: [
{
title: "Physical AI",
skills: ["Isaac Sim Helper"],
},
],
},
},
],
items: [
{
...baseItem,
_id: "skills:isaac-sim-helper",
displayName: "Isaac Sim Helper",
slug: "isaac-sim-helper",
sourceId: "githubSkillSources:nvidia",
},
{
...baseItem,
_id: "skills:extra",
displayName: "Extra Skill",
slug: "extra",
sourceId: "githubSkillSources:nvidia",
},
],
});
expect(display?.sections.map((section) => section.title)).toEqual([
"Other skills",
"Physical AI",
]);
expect(display?.sections[0]?.items.map((item) => item.displayName)).toEqual(["Extra Skill"]);
});
it("falls back to the normal catalog when the source manifest is missing or invalid", () => {
const display = buildGitHubSkillCatalogDisplay({
sources: [
{
_id: "githubSkillSources:nvidia",
repo: "NVIDIA/skills",
displayManifestStatus: "invalid",
},
],
items: [
{
...baseItem,
_id: "skills:aiq-deploy",
displayName: "AIQ Deploy",
slug: "aiq-deploy",
sourceId: "githubSkillSources:nvidia",
},
],
});
expect(display).toBeNull();
});
it("keeps source-backed skills from non-renderable sources in other skills", () => {
const display = buildGitHubSkillCatalogDisplay({
sources: [
{
_id: "githubSkillSources:nvidia",
repo: "NVIDIA/skills",
displayManifestStatus: "ok",
displayManifest: {
notGrouped: "bottom",
groupings: [
{
title: "Agentic AI",
skills: ["aiq-deploy"],
},
],
},
},
{
_id: "githubSkillSources:invalid",
repo: "example/skills",
displayManifestStatus: "invalid",
},
],
items: [
{
...baseItem,
_id: "skills:aiq-deploy",
displayName: "AIQ Deploy",
slug: "aiq-deploy",
sourceId: "githubSkillSources:nvidia",
},
{
...baseItem,
_id: "skills:unlisted",
displayName: "Unlisted Source Skill",
slug: "unlisted-source-skill",
sourceRepo: "example/skills",
sourceId: "githubSkillSources:invalid",
},
],
});
expect(display?.sections.map((section) => section.title)).toEqual([
"Agentic AI",
"Other skills",
]);
expect(display?.sections.at(-1)?.items.map((item) => item.displayName)).toEqual([
"Unlisted Source Skill",
]);
});
});
+157
View File
@@ -0,0 +1,157 @@
type DisplayManifest = {
notGrouped?: "top" | "bottom";
groupings: Array<{
title: string;
description?: string;
skills: string[];
}>;
};
export type GitHubSkillCatalogSource = {
_id: string;
repo: string;
displayManifestStatus?: "ok" | "missing" | "invalid" | "failed";
displayManifest?: DisplayManifest;
};
export type GitHubSkillCatalogItem = {
_id: string;
kind: "skill" | "plugin";
displayName: string;
slug?: string | null;
sourceBacked?: boolean;
sourceId?: string | null;
sourceRepo?: string | null;
sourcePath?: string | null;
sourceVerifiedCommit?: string | null;
summary: string | null;
icon: string | null;
href: string;
downloads: number;
stars: number;
isOfficial: boolean;
updatedAt: number;
};
export type GitHubSkillCatalogSection = {
key: string;
title: string;
description: string | null;
sourceRepo: string | null;
items: GitHubSkillCatalogItem[];
};
export type GitHubSkillCatalogDisplay = {
mode: "grouped";
sourceRepos: string[];
sections: GitHubSkillCatalogSection[];
};
function normalizeManifestSkillKey(value: string) {
return value
.trim()
.toLowerCase()
.replace(/[\s_]+/g, "-")
.replace(/[^a-z0-9-]+/g, "-")
.replace(/-+/g, "-")
.replace(/^-|-$/g, "");
}
function getItemKeys(item: GitHubSkillCatalogItem) {
const keys = new Set<string>();
if (item.slug) keys.add(normalizeManifestSkillKey(item.slug));
keys.add(normalizeManifestSkillKey(item.displayName));
const sourcePathName = item.sourcePath?.split("/").filter(Boolean).at(-1);
if (sourcePathName) keys.add(normalizeManifestSkillKey(sourcePathName));
return keys;
}
function findManifestItem(
candidates: GitHubSkillCatalogItem[],
manifestEntry: string,
usedItemIds: Set<string>,
) {
const key = normalizeManifestSkillKey(manifestEntry);
if (!key) return null;
return (
candidates.find((item) => !usedItemIds.has(item._id) && getItemKeys(item).has(key)) ?? null
);
}
function isRenderableSource(source: GitHubSkillCatalogSource) {
return (
source.displayManifestStatus === "ok" &&
Boolean(source.displayManifest) &&
source.displayManifest!.groupings.length > 0
);
}
export function buildGitHubSkillCatalogDisplay({
sources,
items,
}: {
sources: GitHubSkillCatalogSource[];
items: GitHubSkillCatalogItem[];
}): GitHubSkillCatalogDisplay | null {
const renderableSources = sources.filter(isRenderableSource);
if (renderableSources.length === 0) return null;
const sourceRepos = Array.from(new Set(renderableSources.map((source) => source.repo)));
const usedItemIds = new Set<string>();
const sections: GitHubSkillCatalogSection[] = [];
const otherPosition = renderableSources.some(
(source) => source.displayManifest?.notGrouped === "top",
)
? "top"
: "bottom";
for (const source of renderableSources) {
const sourceItems = items.filter(
(item) => item.kind === "skill" && item.sourceId === source._id,
);
if (sourceItems.length === 0) continue;
for (const [groupIndex, group] of source.displayManifest!.groupings.entries()) {
const groupItems = group.skills
.map((entry) => findManifestItem(sourceItems, entry, usedItemIds))
.filter((item): item is GitHubSkillCatalogItem => Boolean(item));
if (groupItems.length === 0) continue;
for (const item of groupItems) usedItemIds.add(item._id);
sections.push({
key: `${source._id}:${groupIndex}:${group.title}`,
title: group.title,
description: group.description ?? null,
sourceRepo: source.repo,
items: groupItems,
});
}
}
const otherItems = items.filter((item) => item.kind === "skill" && !usedItemIds.has(item._id));
const otherSection =
otherItems.length > 0
? {
key: "other-skills",
title: "Other skills",
description: null,
sourceRepo: null,
items: otherItems,
}
: null;
const orderedSections =
otherPosition === "top" && otherSection
? [otherSection, ...sections]
: [...sections, ...(otherSection ? [otherSection] : [])];
if (orderedSections.length === 0) return null;
return {
mode: "grouped",
sourceRepos,
sections: orderedSections,
};
}
+27
View File
@@ -52,6 +52,13 @@ describe("publisher stat maintenance", () => {
return {
collect: vi.fn(async () => [
makeSkill({ statsDownloads: 11, statsStars: 2, statsInstallsAllTime: 5 }),
makeSkill({
_id: "skills:hidden",
moderationStatus: "hidden",
statsDownloads: 100,
statsStars: 100,
statsInstallsAllTime: 100,
}),
]),
};
}
@@ -136,6 +143,26 @@ describe("publisher stat maintenance", () => {
expect(ctx.db.query).not.toHaveBeenCalled();
});
it("does not count hidden skills in public publisher aggregates", async () => {
const ctx = {
db: {
get: vi.fn(),
patch: vi.fn(),
query: vi.fn(),
},
};
await adjustPublisherStatsForSkillChange(
ctx as never,
null,
makeSkill({ moderationStatus: "hidden", moderationReason: "pending.scan" }),
);
expect(ctx.db.get).not.toHaveBeenCalled();
expect(ctx.db.patch).not.toHaveBeenCalled();
expect(ctx.db.query).not.toHaveBeenCalled();
});
it("keeps legacy aggregate updates bounded when skill-only aggregates are missing", async () => {
const patch = vi.fn();
const ctx = {
+3 -2
View File
@@ -1,5 +1,6 @@
import type { Doc, Id } from "../_generated/dataModel";
import type { MutationCtx } from "../_generated/server";
import { isPublicSkillDoc } from "./globalStats";
import { readCanonicalStat } from "./skillStats";
export type PublisherStatsContribution = {
@@ -27,7 +28,7 @@ export function emptyPublisherStatsContribution(): PublisherStatsContribution {
}
export function getSkillPublisherContribution(skill: Doc<"skills">): PublisherStatsContribution {
if (skill.softDeletedAt) return emptyPublisherStatsContribution();
if (!isPublicSkillDoc(skill)) return emptyPublisherStatsContribution();
const totalInstalls = readCanonicalStat(skill, "installsAllTime");
const totalDownloads = readCanonicalStat(skill, "downloads");
const totalStars = readCanonicalStat(skill, "stars");
@@ -91,7 +92,7 @@ function publisherHasSkillTotalStats(
);
}
async function recomputePublisherStats(
export async function recomputePublisherStats(
ctx: Pick<MutationCtx, "db">,
publisherId: Id<"publishers">,
): Promise<PublisherStatsContribution> {
+1 -1
View File
@@ -32,7 +32,7 @@ function normalizeGeneratedPublisherHandle(handle: string | undefined | null) {
return sanitized || undefined;
}
function derivePersonalPublisherHandle(user: Doc<"users">) {
export function derivePersonalPublisherHandle(user: Doc<"users">) {
const emailLocalPart = user.email?.split("@")[0];
const userIdSuffix = String(user._id).split(":").pop();
return (
+22
View File
@@ -0,0 +1,22 @@
import { describe, expect, it } from "vitest";
import {
getPublicSkillFileAccessBlock,
getSkillFileModerationInfoFromSkill,
} from "./skillFileAccess";
describe("skill file moderation access", () => {
it("blocks skills whose current moderation verdict is malicious", () => {
const moderationInfo = getSkillFileModerationInfoFromSkill({
moderationStatus: "hidden",
moderationReason: "scanner.llm.malicious",
moderationFlags: [],
moderationVerdict: "malicious",
});
expect(moderationInfo.isMalwareBlocked).toBe(true);
expect(getPublicSkillFileAccessBlock(moderationInfo)).toMatchObject({
status: 403,
message: expect.stringContaining("malicious"),
});
});
});
+50 -1
View File
@@ -1,10 +1,19 @@
import type { Id } from "../_generated/dataModel";
type SkillFileModerationInfo = {
export type SkillFileModerationInfo = {
isPendingScan?: boolean | null;
isMalwareBlocked?: boolean | null;
isHiddenByMod?: boolean | null;
isRemoved?: boolean | null;
sourceVersionId?: Id<"skillVersions"> | string | null;
};
type SkillModerationSource = {
moderationStatus?: string | null;
moderationReason?: string | null;
moderationFlags?: string[] | null;
moderationVerdict?: string | null;
moderationSourceVersionId?: Id<"skillVersions"> | string | null;
};
type SkillFileAccessBlock = {
@@ -12,6 +21,33 @@ type SkillFileAccessBlock = {
message: string;
};
function isPendingSkillModerationReason(reason: string | null | undefined) {
const normalized = reason?.trim().toLowerCase();
return (
normalized === "pending.scan" ||
normalized === "pending.scan.stale" ||
normalized === "scanner.vt.pending" ||
normalized === "scanner.llm.pending"
);
}
export function getSkillFileModerationInfoFromSkill(
skill: SkillModerationSource,
): SkillFileModerationInfo {
const isPendingScan =
skill.moderationStatus === "hidden" && isPendingSkillModerationReason(skill.moderationReason);
const isMalwareBlocked =
skill.moderationVerdict === "malicious" ||
(skill.moderationFlags?.includes("blocked.malware") ?? false);
return {
isPendingScan,
isMalwareBlocked,
isHiddenByMod: skill.moderationStatus === "hidden" && !isPendingScan && !isMalwareBlocked,
isRemoved: skill.moderationStatus === "removed",
sourceVersionId: skill.moderationSourceVersionId ?? null,
};
}
export function getPublicSkillFileAccessBlock(
moderationInfo: SkillFileModerationInfo | null | undefined,
): SkillFileAccessBlock | null {
@@ -38,6 +74,19 @@ export function getPublicSkillFileAccessBlock(
return null;
}
export function getPublicSkillVersionAccessBlock(
moderationInfo: SkillFileModerationInfo | null | undefined,
versionId: Id<"skillVersions"> | string,
fallbackModeratedVersionId?: Id<"skillVersions"> | string | null,
): SkillFileAccessBlock | null {
const block = getPublicSkillFileAccessBlock(moderationInfo);
if (!block) return null;
if (moderationInfo?.isRemoved || moderationInfo?.isHiddenByMod) return block;
const moderatedVersionId = moderationInfo?.sourceVersionId ?? fallbackModeratedVersionId;
return moderatedVersionId === versionId ? block : null;
}
export function isSkillVersionForSkill(
version: { skillId?: Id<"skills"> | string | null } | null | undefined,
skillId: Id<"skills"> | string,
-4
View File
@@ -365,10 +365,6 @@ export async function publishVersionForUser(
source: "publish",
});
await ctx.scheduler.runAfter(0, internal.depRegistryScan.checkDependencyRegistries, {
versionId: publishResult.versionId,
});
// Schedule the async "API key required?" analyser; non-fatal on failure
// (UI treats `apiKeyRequired === undefined` as "no badge"). Mirrors the
// `backupSkillForPublishInternal` pattern below: `void runAfter(...).catch(...)`
+10
View File
@@ -1,6 +1,7 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
import { toPublicSkill } from "./public";
import {
digestToHydratableSkill,
extractDigestFields,
@@ -90,6 +91,7 @@ describe("extractDigestFields", () => {
comments: 1,
});
expect(digest.moderationFlags).toEqual(["flagged.test"]);
expect(digest.moderationVerdict).toBeUndefined();
expect(digest.isSuspicious).toBe(false);
expect(digest.createdAt).toBe(1000);
expect(digest.updatedAt).toBe(2000);
@@ -177,6 +179,14 @@ describe("extractDigestFields", () => {
expect(hydratable.isSuspicious).toBe(true);
});
it("preserves malicious moderation verdicts through digest hydration", () => {
const digest = extractDigestFields(makeSkillDoc({ moderationVerdict: "malicious" }) as never);
const hydratable = digestToHydratableSkill(digest as never);
expect(hydratable.moderationVerdict).toBe("malicious");
expect(toPublicSkill(hydratable)).toBeNull();
});
});
describe("extractValidatedDigestFields", () => {
+26
View File
@@ -1,6 +1,7 @@
import type { Doc, Id } from "../_generated/dataModel";
import type { MutationCtx } from "../_generated/server";
import type { HydratableSkill, PublicPublisher } from "./public";
import { getOwnerPublisher } from "./publishers";
import { tokenize } from "./searchText";
import { readCanonicalStat } from "./skillStats";
@@ -25,6 +26,10 @@ const SHARED_KEYS = [
"canonicalSkillId",
"forkOf",
"latestVersionId",
"installKind",
"githubHasSkillCard",
"githubCurrentStatus",
"githubScanStatus",
"latestVersionSummary",
"tags",
"capabilityTags",
@@ -37,6 +42,7 @@ const SHARED_KEYS = [
"softDeletedAt",
"moderationStatus",
"moderationFlags",
"moderationVerdict",
"moderationReason",
"isSuspicious",
"createdAt",
@@ -130,6 +136,26 @@ export async function upsertSkillSearchDigest(
}
}
export async function syncSkillSearchDigestForSkill(
ctx: Pick<MutationCtx, "db">,
skill: Doc<"skills"> | null | undefined,
) {
if (!skill) return;
const fields = await extractValidatedDigestFields(ctx, skill);
const owner = await getOwnerPublisher(ctx, {
ownerPublisherId: skill.ownerPublisherId,
ownerUserId: skill.ownerUserId,
});
await upsertSkillSearchDigest(ctx, {
...fields,
ownerHandle: owner?.handle ?? "",
ownerKind: owner?.kind,
ownerName: owner?.linkedUserId ? owner.handle : undefined,
ownerDisplayName: owner?.displayName,
ownerImage: owner?.image,
});
}
/** Compare new fields against existing row. Returns true if any field differs. */
function hasDigestChanged(
existing: Doc<"skillSearchDigest">,
+15 -8
View File
@@ -73,14 +73,21 @@ describe("assertValidSkillSlug", () => {
expect(() => assertValidSkillSlug(slug)).toThrow(new RegExp(hint, "i"));
});
it.each(["admin", "settings", "api", "openclaw", "clawhub", "souls", "packages", "publishers"])(
"rejects reserved slug %s",
(slug) => {
// Some short reserved entries (e.g. "u") are also blocked by the
// length rule; we only assert that a throw happens for every entry.
expect(() => assertValidSkillSlug(slug)).toThrow();
},
);
it.each([
"account-banned",
"admin",
"settings",
"api",
"openclaw",
"clawhub",
"souls",
"packages",
"publishers",
])("rejects reserved slug %s", (slug) => {
// Some short reserved entries (e.g. "u") are also blocked by the
// length rule; we only assert that a throw happens for every entry.
expect(() => assertValidSkillSlug(slug)).toThrow();
});
it.each(["openclaw", "publishers"])(
"emits the reserved-specific error for long reserved slug %s",
+3 -2
View File
@@ -4,7 +4,7 @@ import { ConvexError } from "convex/values";
// - Lowercase letters, digits, and single hyphens only.
// - Must start and end with a letter or digit.
// - No consecutive hyphens ("--", "---", ...).
// - Length 3..48 (URL/SEO friendly, aligned with publisher handle).
// - Length 3..96 (URL-friendly, but long enough for source-backed upstream slugs).
//
// The pattern enforces first/last char class and forbids consecutive hyphens
// via a negative lookahead. Length bounds are checked separately so we can
@@ -12,7 +12,7 @@ import { ConvexError } from "convex/values";
const SLUG_PATTERN = /^[a-z0-9](?:(?!--)[a-z0-9-])*[a-z0-9]$/;
const MIN_SLUG_LENGTH = 3;
const MAX_SLUG_LENGTH = 48;
const MAX_SLUG_LENGTH = 96;
// Reserved slugs. These are blocked because they would:
// 1. Clash semantically with top-level routes under src/routes/*.
@@ -25,6 +25,7 @@ const MAX_SLUG_LENGTH = 48;
const RESERVED_SKILL_SLUGS: ReadonlySet<string> = new Set([
// Current top-level route segments under src/routes/.
"about",
"account-banned",
"admin",
"cli",
"dashboard",
+647 -179
View File
@@ -1,5 +1,9 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from "vitest";
import { beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("@convex-dev/auth/server", () => ({
getAuthUserId: vi.fn(),
}));
vi.mock("./_generated/api", () => ({
internal: {
@@ -11,6 +15,9 @@ vi.mock("./_generated/api", () => ({
getUserOwnedSkillsBackfillPageInternal: Symbol("getUserOwnedSkillsBackfillPageInternal"),
applyUserStatsBackfillPatchInternal: Symbol("applyUserStatsBackfillPatchInternal"),
backfillUserStatsInternal: Symbol("backfillUserStatsInternal"),
getPublisherStatsBackfillPageInternal: Symbol("getPublisherStatsBackfillPageInternal"),
recomputePublisherStatsInternal: Symbol("recomputePublisherStatsInternal"),
backfillPublisherStatsInternal: Symbol("backfillPublisherStatsInternal"),
getSkillFingerprintBackfillPageInternal: Symbol("getSkillFingerprintBackfillPageInternal"),
applySkillFingerprintBackfillPatchInternal: Symbol(
"applySkillFingerprintBackfillPatchInternal",
@@ -18,7 +25,9 @@ vi.mock("./_generated/api", () => ({
backfillSkillFingerprintsInternal: Symbol("backfillSkillFingerprintsInternal"),
applySkillCapabilityTagsInternal: Symbol("applySkillCapabilityTagsInternal"),
backfillSkillCapabilityTagsInternal: Symbol("backfillSkillCapabilityTagsInternal"),
backfillDigestVersionSummary: Symbol("backfillDigestVersionSummary"),
backfillSkillSearchDigestModerationVerdictsInternal: Symbol(
"backfillSkillSearchDigestModerationVerdictsInternal",
),
getEmptySkillCleanupPageInternal: Symbol("getEmptySkillCleanupPageInternal"),
applyEmptySkillCleanupInternal: Symbol("applyEmptySkillCleanupInternal"),
nominateUserForEmptySkillSpamInternal: Symbol("nominateUserForEmptySkillSpamInternal"),
@@ -42,131 +51,497 @@ vi.mock("./lib/skillSummary", () => ({
const {
applySkillCapabilityTagsInternal,
backfillDigestVersionSummary,
backfillLatestVersionSummaryInternal,
backfillSkillSearchDigestInternal,
backfillSkillSearchDigestModerationVerdictsInternal,
backfillPublisherStatsInternalHandler,
backfillSkillFingerprintsInternalHandler,
backfillSkillSummariesInternalHandler,
backfillUserStatsInternalHandler,
cleanupEmptySkillsInternalHandler,
nominateEmptySkillSpammersInternalHandler,
repairLegacyPublisherOwnershipForUserHandler,
upsertSkillBadgeRecordInternal,
} = await import("./maintenance");
const { internal } = await import("./_generated/api");
const { generateSkillSummary } = await import("./lib/skillSummary");
const { getAuthUserId } = await import("@convex-dev/auth/server");
beforeEach(() => {
vi.mocked(getAuthUserId).mockReset();
vi.mocked(getAuthUserId).mockResolvedValue(null);
});
function makeBlob(text: string) {
return { text: () => Promise.resolve(text) } as unknown as Blob;
}
describe("maintenance backfill", () => {
it("patches stale skill search digest rank stats from legacy skill stats", async () => {
const existingDigest = {
_id: "skillSearchDigest:1",
skillId: "skills:1",
slug: "demo",
displayName: "Demo",
summary: "Old summary",
ownerUserId: "users:owner",
tags: {},
stats: {
downloads: 3,
stars: 2,
installsCurrent: 4,
installsAllTime: 5,
versions: 1,
comments: 0,
},
softDeletedAt: undefined,
createdAt: 100,
updatedAt: 200,
};
const skill = {
_id: "skills:1",
slug: "demo",
displayName: "Demo",
summary: "New summary",
ownerUserId: "users:owner",
tags: {},
stats: {
downloads: 42,
stars: 7,
installsCurrent: 9,
installsAllTime: 100,
versions: 1,
comments: 0,
},
softDeletedAt: undefined,
createdAt: 100,
updatedAt: 300,
};
const paginate = vi.fn().mockResolvedValue({
page: [skill],
continueCursor: null,
isDone: true,
});
const unique = vi.fn().mockResolvedValue(existingDigest);
class TestEqBuilder {
eq(_field: string, _value: unknown) {
return this;
}
}
const withIndex = vi.fn((_indexName: string, build: (q: TestEqBuilder) => unknown) => {
build(new TestEqBuilder());
return { unique };
});
const query = vi.fn((table: string) => {
if (table === "skills") return { paginate };
if (table === "skillSearchDigest") return { withIndex };
throw new Error(`unexpected table ${table}`);
});
const patch = vi.fn().mockResolvedValue(undefined);
const insert = vi.fn().mockResolvedValue("skillSearchDigest:inserted");
const replace = vi.fn().mockResolvedValue(undefined);
const deleteDoc = vi.fn().mockResolvedValue(undefined);
type QueryEq = {
eq: (field: string, value: unknown) => QueryEq;
};
const result = await (
backfillSkillSearchDigestInternal as unknown as { _handler: Function }
)._handler(
function makeLegacyPublisherOwnershipDb() {
const now = 1_717_456_000_000;
let nextPublisherId = 2;
let nextMemberId = 1;
const users = new Map<string, Record<string, unknown>>([
[
"users:legacy",
{
db: {
get: vi.fn(),
query,
patch,
insert,
replace,
delete: deleteDoc,
normalizeId: vi.fn(),
_id: "users:legacy",
_creationTime: now - 1000,
handle: "legacy-owner",
name: "Legacy Owner",
displayName: "Legacy Owner",
deletedAt: undefined,
deactivatedAt: undefined,
purgedAt: undefined,
},
],
[
"users:deleted",
{
_id: "users:deleted",
_creationTime: now - 1000,
handle: "deleted-owner",
deletedAt: now - 10,
deactivatedAt: undefined,
purgedAt: undefined,
},
],
]);
const publishers = new Map<string, Record<string, unknown>>([
[
"publishers:existing",
{
_id: "publishers:existing",
_creationTime: now - 500,
kind: "user",
handle: "existing-owner",
displayName: "Existing Owner",
linkedUserId: "users:existing",
publishedSkills: 0,
publishedPackages: 0,
totalInstalls: 0,
totalDownloads: 0,
totalStars: 0,
skillTotalInstalls: 0,
skillTotalDownloads: 0,
skillTotalStars: 0,
createdAt: now - 500,
updatedAt: now - 500,
},
],
]);
const publisherMembers = new Map<string, Record<string, unknown>>();
const skills = new Map<string, Record<string, unknown>>([
[
"skills:legacy",
{
_id: "skills:legacy",
_creationTime: now - 400,
slug: "legacy-skill",
displayName: "Legacy Skill",
ownerUserId: "users:legacy",
ownerPublisherId: undefined,
latestVersionId: "skillVersions:legacy",
tags: { latest: "skillVersions:legacy" },
stats: {
downloads: 10,
stars: 3,
installsCurrent: 2,
installsAllTime: 5,
comments: 0,
versions: 1,
},
scheduler: {
runAfter: vi.fn(),
statsDownloads: 10,
statsStars: 3,
statsInstallsCurrent: 2,
statsInstallsAllTime: 5,
softDeletedAt: undefined,
moderationStatus: "active",
createdAt: now - 300,
updatedAt: now - 200,
},
],
[
"skills:deleted-owner",
{
_id: "skills:deleted-owner",
_creationTime: now - 400,
slug: "deleted-owner-skill",
displayName: "Deleted Owner Skill",
ownerUserId: "users:deleted",
ownerPublisherId: undefined,
latestVersionId: "skillVersions:deleted-owner",
tags: { latest: "skillVersions:deleted-owner" },
stats: {
downloads: 1,
stars: 0,
installsCurrent: 0,
installsAllTime: 0,
comments: 0,
versions: 1,
},
} as never,
{ batchSize: 10 },
);
softDeletedAt: undefined,
moderationStatus: "active",
createdAt: now - 300,
updatedAt: now - 200,
},
],
]);
const skillVersions = new Map<string, Record<string, unknown>>([
[
"skillVersions:legacy",
{
_id: "skillVersions:legacy",
skillId: "skills:legacy",
version: "1.0.0",
softDeletedAt: undefined,
},
],
[
"skillVersions:deleted-owner",
{
_id: "skillVersions:deleted-owner",
skillId: "skills:deleted-owner",
version: "1.0.0",
softDeletedAt: undefined,
},
],
]);
const skillSlugAliases = new Map<string, Record<string, unknown>>([
[
"skillSlugAliases:legacy",
{
_id: "skillSlugAliases:legacy",
slug: "old-legacy-skill",
skillId: "skills:legacy",
ownerUserId: "users:legacy",
ownerPublisherId: undefined,
createdAt: now - 250,
updatedAt: now - 250,
},
],
]);
const skillEmbeddings = new Map<string, Record<string, unknown>>([
[
"skillEmbeddings:legacy",
{
_id: "skillEmbeddings:legacy",
skillId: "skills:legacy",
versionId: "skillVersions:legacy",
ownerId: "users:legacy",
ownerPublisherId: undefined,
embedding: [0.1, 0.2],
isLatest: true,
isApproved: true,
visibility: "public",
updatedAt: now - 200,
},
],
]);
const skillSearchDigest = new Map<string, Record<string, unknown>>([
[
"skillSearchDigest:legacy",
{
_id: "skillSearchDigest:legacy",
skillId: "skills:legacy",
slug: "legacy-skill",
displayName: "Legacy Skill",
ownerUserId: "users:legacy",
ownerPublisherId: undefined,
ownerHandle: "legacy-owner",
ownerKind: "user",
stats: {
downloads: 10,
stars: 3,
installsCurrent: 2,
installsAllTime: 5,
comments: 0,
versions: 1,
},
statsDownloads: 10,
statsStars: 3,
statsInstallsCurrent: 2,
statsInstallsAllTime: 5,
softDeletedAt: undefined,
moderationStatus: "active",
createdAt: now - 300,
updatedAt: now - 200,
},
],
]);
const packages = new Map<string, Record<string, unknown>>([
[
"packages:legacy",
{
_id: "packages:legacy",
_creationTime: now - 400,
name: "@legacy-owner/demo-plugin",
normalizedName: "@legacy-owner/demo-plugin",
displayName: "Demo Plugin",
family: "bundle-plugin",
channel: "community",
isOfficial: false,
ownerUserId: "users:legacy",
ownerPublisherId: undefined,
summary: "Demo package",
latestReleaseId: undefined,
tags: {},
compatibility: undefined,
capabilities: undefined,
verification: undefined,
scanStatus: "clean",
stats: { downloads: 7, installs: 4, stars: 2, versions: 1 },
softDeletedAt: undefined,
createdAt: now - 300,
updatedAt: now - 200,
},
],
]);
const packageSearchDigest = new Map<string, Record<string, unknown>>([
[
"packageSearchDigest:legacy",
{
_id: "packageSearchDigest:legacy",
packageId: "packages:legacy",
name: "@legacy-owner/demo-plugin",
normalizedName: "@legacy-owner/demo-plugin",
displayName: "Demo Plugin",
family: "bundle-plugin",
channel: "community",
isOfficial: false,
ownerUserId: "users:legacy",
ownerPublisherId: undefined,
ownerHandle: "legacy-owner",
ownerKind: "user",
summary: "Demo package",
scanStatus: "clean",
softDeletedAt: undefined,
createdAt: now - 300,
updatedAt: now - 200,
},
],
]);
const packageCapabilitySearchDigest = new Map<string, Record<string, unknown>>();
const packagePluginCategorySearchDigest = new Map<string, Record<string, unknown>>();
expect(result).toEqual({ upserted: 1, isDone: true, scanned: 1 });
expect(paginate).toHaveBeenCalledWith({ cursor: null, numItems: 10 });
expect(withIndex).toHaveBeenCalledWith("by_skill", expect.any(Function));
expect(insert).not.toHaveBeenCalled();
expect(patch).toHaveBeenCalledWith(
"skillSearchDigest:1",
expect.objectContaining({
summary: "New summary",
statsDownloads: 42,
statsStars: 7,
statsInstallsCurrent: 9,
statsInstallsAllTime: 100,
stats: expect.objectContaining({
downloads: 42,
stars: 7,
installsCurrent: 9,
installsAllTime: 100,
}),
}),
);
const tableMap: Record<string, Map<string, Record<string, unknown>>> = {
users,
publishers,
publisherMembers,
skills,
skillVersions,
skillSlugAliases,
skillEmbeddings,
skillSearchDigest,
packages,
packageSearchDigest,
packageCapabilitySearchDigest,
packagePluginCategorySearchDigest,
};
const patchCalls: Array<{ id: string; patch: Record<string, unknown> }> = [];
const insertCalls: Array<{ table: string; value: Record<string, unknown> }> = [];
const getRows = (table: string) => Array.from(tableMap[table]?.values() ?? []);
const getTableForId = (id: string) => id.split(":")[0];
const readField = (row: Record<string, unknown>, field: string) =>
field.split(".").reduce<unknown>((value, part) => {
if (!value || typeof value !== "object") return undefined;
return (value as Record<string, unknown>)[part];
}, row);
const makeQuery = (table: string, rows: Record<string, unknown>[]) => ({
collect: vi.fn(async () => rows),
unique: vi.fn(async () => rows[0] ?? null),
take: vi.fn(async (limit: number) => rows.slice(0, limit)),
order: vi.fn(() => ({
take: vi.fn(async (limit: number) => rows.slice(0, limit)),
paginate: vi.fn(async ({ cursor, numItems }: { cursor: string | null; numItems: number }) =>
paginateRows(rows, cursor, numItems),
),
})),
paginate: vi.fn(async ({ cursor, numItems }: { cursor: string | null; numItems: number }) =>
paginateRows(rows, cursor, numItems),
),
withIndex: vi.fn((indexName: string, build?: (q: QueryEq) => unknown) => {
const filters: Array<{ field: string; value: unknown }> = [];
const q: QueryEq = {
eq: (field, value) => {
filters.push({ field, value });
return q;
},
};
build?.(q);
let indexedRows = getRows(table).filter((row) =>
filters.every((filter) => readField(row, filter.field) === filter.value),
);
if (table === "users" && indexName === "by_active_handle") {
indexedRows = indexedRows.filter(
(row) => row.deletedAt === undefined && row.deactivatedAt === undefined,
);
}
return makeQuery(table, indexedRows);
}),
});
const db = {
get: vi.fn(async (id: string) => tableMap[getTableForId(id)]?.get(id) ?? null),
query: vi.fn((table: string) => makeQuery(table, getRows(table))),
patch: vi.fn(async (id: string, patch: Record<string, unknown>) => {
patchCalls.push({ id, patch });
const row = tableMap[getTableForId(id)]?.get(id);
if (row) Object.assign(row, patch);
}),
insert: vi.fn(async (table: string, value: Record<string, unknown>) => {
const id =
table === "publishers"
? `publishers:created${nextPublisherId++}`
: table === "publisherMembers"
? `publisherMembers:created${nextMemberId++}`
: `${table}:created`;
insertCalls.push({ table, value });
tableMap[table].set(id, { _id: id, _creationTime: now, ...value });
return id;
}),
delete: vi.fn(async (id: string) => {
tableMap[getTableForId(id)]?.delete(id);
}),
normalizeId: vi.fn(),
};
return {
db,
patchCalls,
insertCalls,
tableMap,
};
}
function paginateRows(rows: Record<string, unknown>[], cursor: string | null, numItems: number) {
const start = cursor ? Number(cursor) : 0;
const page = rows.slice(start, start + numItems);
const next = start + page.length;
return {
page,
continueCursor: next >= rows.length ? null : String(next),
isDone: next >= rows.length,
};
}
describe("maintenance legacy publisher ownership repair", () => {
it("repairs legacy owner projections for one targeted user by handle", async () => {
const { db, tableMap } = makeLegacyPublisherOwnershipDb();
const scheduler = { runAfter: vi.fn() };
const skillsResult = await repairLegacyPublisherOwnershipForUserHandler(
{ db, scheduler } as never,
{
handle: "legacy-owner",
phase: "skills",
dryRun: false,
batchSize: 10,
scheduleNext: false,
},
);
const createdPublisher = Array.from(tableMap.publishers.values()).find(
(publisher) => publisher.handle === "legacy-owner",
);
expect(skillsResult).toMatchObject({
phase: "skills",
dryRun: false,
userId: "users:legacy",
publisherId: createdPublisher?._id,
scanned: 1,
repaired: 1,
skipped: 0,
isDone: true,
});
expect(tableMap.skills.get("skills:legacy")).toMatchObject({
ownerPublisherId: createdPublisher?._id,
});
expect(tableMap.skills.get("skills:deleted-owner")).toMatchObject({
ownerPublisherId: undefined,
});
expect(tableMap.skillSlugAliases.get("skillSlugAliases:legacy")).toMatchObject({
ownerPublisherId: createdPublisher?._id,
});
expect(tableMap.skillEmbeddings.get("skillEmbeddings:legacy")).not.toHaveProperty(
"ownerPublisherId",
createdPublisher?._id,
);
const packagesResult = await repairLegacyPublisherOwnershipForUserHandler(
{ db, scheduler } as never,
{
handle: "legacy-owner",
phase: "packages",
dryRun: false,
batchSize: 10,
scheduleNext: false,
},
);
expect(packagesResult).toMatchObject({
phase: "packages",
dryRun: false,
userId: "users:legacy",
publisherId: createdPublisher?._id,
scanned: 1,
repaired: 1,
skipped: 0,
isDone: true,
});
expect(tableMap.packages.get("packages:legacy")).toMatchObject({
ownerPublisherId: createdPublisher?._id,
});
});
it("does not touch skill embeddings during apply-mode skill repair", async () => {
const { db } = makeLegacyPublisherOwnershipDb();
const scheduler = { runAfter: vi.fn() };
const patch = db.patch;
db.patch = vi.fn(async (id: string, value: Record<string, unknown>) => {
if (id === "skillEmbeddings:legacy") throw new Error("embedding sync failed");
await patch(id, value);
});
await expect(
repairLegacyPublisherOwnershipForUserHandler({ db, scheduler } as never, {
handle: "legacy-owner",
phase: "skills",
dryRun: false,
batchSize: 10,
scheduleNext: false,
}),
).resolves.toMatchObject({
phase: "skills",
repaired: 1,
});
});
it("propagates apply-mode package patch failures", async () => {
const { db } = makeLegacyPublisherOwnershipDb();
const scheduler = { runAfter: vi.fn() };
const patch = db.patch;
db.patch = vi.fn(async (id: string, value: Record<string, unknown>) => {
if (id === "packages:legacy") throw new Error("package patch failed");
await patch(id, value);
});
await expect(
repairLegacyPublisherOwnershipForUserHandler({ db, scheduler } as never, {
handle: "legacy-owner",
phase: "packages",
dryRun: false,
batchSize: 10,
scheduleNext: false,
}),
).rejects.toThrow("package patch failed");
});
});
describe("maintenance backfill", () => {
it("repairs summary + parsed by reparsing SKILL.md", async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [
@@ -379,80 +754,6 @@ describe("maintenance backfill", () => {
expect(runAfter).not.toHaveBeenCalled();
});
it("backfills digest capability tags even when version summary already matches", async () => {
const digest = {
_id: "skillSearchDigest:1",
skillId: "skills:1",
latestVersionId: "skillVersions:1",
latestVersionSkillId: "skills:1",
latestVersionSummary: {
version: "1.0.0",
createdAt: 123,
changelog: "Same changelog",
changelogSource: "user",
clawdis: undefined,
},
capabilityTags: ["old"],
};
const skill = {
_id: "skills:1",
slug: "demo",
displayName: "Demo",
stats: {
downloads: 0,
stars: 0,
installsCurrent: 0,
installsAllTime: 0,
versions: 1,
comments: 0,
},
latestVersionId: "skillVersions:1",
latestVersionSummary: digest.latestVersionSummary,
capabilityTags: ["read-files"],
};
const version = {
_id: "skillVersions:1",
skillId: "skills:1",
softDeletedAt: undefined,
version: "1.0.0",
};
const paginate = vi.fn().mockResolvedValue({
page: [digest],
continueCursor: null,
isDone: true,
});
const patch = vi.fn().mockResolvedValue(undefined);
const ctx = {
db: {
query: vi.fn(() => ({ paginate })),
get: vi.fn(async (id: string) => {
if (id === "skills:1") return skill;
if (id === "skillVersions:1") return version;
return null;
}),
patch,
normalizeId: vi.fn(),
},
scheduler: {
runAfter: vi.fn(),
},
} as never;
const result = await (
backfillDigestVersionSummary as unknown as { _handler: Function }
)._handler(ctx, {
batchSize: 10,
});
expect(result).toEqual({ patched: 1, isDone: true, scanned: 1 });
expect(patch).toHaveBeenCalledWith("skillSearchDigest:1", {
latestVersionId: "skillVersions:1",
latestVersionSkillId: "skills:1",
latestVersionSummary: digest.latestVersionSummary,
capabilityTags: ["read-files"],
});
});
it("backfills denormalized user hover stats from indexed owner pages", async () => {
const runQuery = vi
.fn()
@@ -514,6 +815,54 @@ describe("maintenance backfill", () => {
},
);
});
it("backfills denormalized publisher stats through the recompute mutation", async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [{ _id: "publishers:1" }, { _id: "publishers:2" }],
cursor: "next",
isDone: false,
});
const runMutation = vi.fn().mockResolvedValue({ ok: true });
const result = await backfillPublisherStatsInternalHandler({ runQuery, runMutation } as never, {
dryRun: true,
batchSize: 2,
maxBatches: 1,
});
expect(result).toEqual({
ok: true,
stats: {
publishersScanned: 2,
publishersPatched: 0,
},
isDone: false,
cursor: "next",
});
expect(runQuery).toHaveBeenCalledWith(
internal.maintenance.getPublisherStatsBackfillPageInternal,
{
cursor: undefined,
batchSize: 2,
},
);
expect(runMutation).toHaveBeenNthCalledWith(
1,
internal.maintenance.recomputePublisherStatsInternal,
{
publisherId: "publishers:1",
dryRun: true,
},
);
expect(runMutation).toHaveBeenNthCalledWith(
2,
internal.maintenance.recomputePublisherStatsInternal,
{
publisherId: "publishers:2",
dryRun: true,
},
);
});
});
describe("maintenance badge denormalization", () => {
@@ -779,6 +1128,125 @@ describe("maintenance capability tag backfill", () => {
});
});
describe("skill search digest moderation verdict backfill", () => {
it("patches digest moderation verdicts from canonical skill rows and schedules the next page", async () => {
const paginate = vi.fn().mockResolvedValue({
page: [
{
_id: "skillSearchDigest:malicious",
skillId: "skills:malicious",
moderationVerdict: undefined,
},
{
_id: "skillSearchDigest:clean",
skillId: "skills:clean",
moderationVerdict: "clean",
},
{
_id: "skillSearchDigest:missing",
skillId: "skills:missing",
moderationVerdict: undefined,
},
],
continueCursor: "next-page",
isDone: false,
});
const query = vi.fn().mockReturnValue({ paginate });
const get = vi
.fn()
.mockResolvedValueOnce({
_id: "skills:malicious",
moderationVerdict: "malicious",
updatedAt: 123,
})
.mockResolvedValueOnce({
_id: "skills:clean",
moderationVerdict: "clean",
updatedAt: 456,
})
.mockResolvedValueOnce(null);
const patch = vi.fn().mockResolvedValue(undefined);
const runAfter = vi.fn().mockResolvedValue(undefined);
const result = await (
backfillSkillSearchDigestModerationVerdictsInternal as unknown as { _handler: Function }
)._handler(
{
db: { query, get, patch, normalizeId: vi.fn() },
scheduler: { runAfter },
} as never,
{ cursor: "start", batchSize: 25 },
);
expect(result).toEqual({
scanned: 3,
patched: 1,
missingSkills: 1,
cursor: "next-page",
isDone: false,
dryRun: false,
});
expect(query).toHaveBeenCalledWith("skillSearchDigest");
expect(paginate).toHaveBeenCalledWith({ cursor: "start", numItems: 25 });
expect(patch).toHaveBeenCalledWith("skillSearchDigest:malicious", {
moderationVerdict: "malicious",
updatedAt: 123,
});
expect(runAfter).toHaveBeenCalledWith(
0,
internal.maintenance.backfillSkillSearchDigestModerationVerdictsInternal,
{
cursor: "next-page",
batchSize: 25,
dryRun: false,
},
);
});
it("reports would-be patches without writing or scheduling in dry run mode", async () => {
const paginate = vi.fn().mockResolvedValue({
page: [
{
_id: "skillSearchDigest:malicious",
skillId: "skills:malicious",
moderationVerdict: undefined,
},
],
continueCursor: "next-page",
isDone: false,
});
const query = vi.fn().mockReturnValue({ paginate });
const get = vi.fn().mockResolvedValue({
_id: "skills:malicious",
moderationVerdict: "malicious",
updatedAt: 123,
});
const patch = vi.fn().mockResolvedValue(undefined);
const runAfter = vi.fn().mockResolvedValue(undefined);
const result = await (
backfillSkillSearchDigestModerationVerdictsInternal as unknown as { _handler: Function }
)._handler(
{
db: { query, get, patch, normalizeId: vi.fn() },
scheduler: { runAfter },
} as never,
{ batchSize: 25, dryRun: true },
);
expect(result).toEqual({
scanned: 1,
patched: 1,
missingSkills: 0,
cursor: "next-page",
isDone: false,
dryRun: true,
});
expect(patch).not.toHaveBeenCalled();
expect(runAfter).not.toHaveBeenCalled();
});
});
describe("maintenance fingerprint backfill", () => {
it("backfills fingerprint field and inserts index entry", async () => {
const { hashSkillFiles } = await import("./lib/skills");
+402 -364
View File
@@ -1,10 +1,18 @@
import { ConvexError, v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc, Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import type { ActionCtx, MutationCtx } from "./_generated/server";
import { action, internalAction, internalMutation, internalQuery } from "./functions";
import { assertRole, requireUserFromAction } from "./lib/access";
import { extractPackageDigestFields, upsertPackageSearchDigest } from "./lib/packageSearchDigest";
import {
derivePersonalPublisherHandle,
ensurePersonalPublisherForUser,
getPersonalPublisherForUser,
getPublisherByHandle,
getUserByHandleOrPersonalPublisher,
isPublisherActive,
} from "./lib/publishers";
import { recomputePublisherStats } from "./lib/publisherStats";
import { buildSkillSummaryBackfillPatch, type ParsedSkillData } from "./lib/skillBackfill";
import { deriveSkillCapabilityTags } from "./lib/skillCapabilityTags";
import { isSkillCardPath } from "./lib/skillCards";
@@ -16,12 +24,6 @@ import {
} from "./lib/skillQuality";
import { hashSkillFiles, isTextFile } from "./lib/skills";
import { computeIsSuspicious } from "./lib/skillSafety";
import {
extractValidatedDigestFields,
getFirstSearchToken,
normalizeSkillSearchText,
upsertSkillSearchDigest,
} from "./lib/skillSearchDigest";
import { generateSkillSummary } from "./lib/skillSummary";
const DEFAULT_BATCH_SIZE = 50;
@@ -48,6 +50,11 @@ type UserStatsBackfillStats = {
usersPatched: number;
};
type PublisherStatsBackfillStats = {
publishersScanned: number;
publishersPatched: number;
};
type BackfillPageItem =
| {
kind: "ok";
@@ -75,12 +82,35 @@ type UserStatsBackfillPageResult = {
isDone: boolean;
};
type PublisherStatsBackfillPageResult = {
items: Array<Pick<Doc<"publishers">, "_id">>;
cursor: string | null;
isDone: boolean;
};
type UserOwnedSkillsBackfillPageResult = {
items: Array<Pick<Doc<"skills">, "stats" | "softDeletedAt">>;
cursor: string | null;
isDone: boolean;
};
type LegacyPublisherOwnershipTargetPhase = "skills" | "packages";
type LegacyPublisherOwnershipForUserRepairResult = {
phase: LegacyPublisherOwnershipTargetPhase;
dryRun: boolean;
userId: Id<"users">;
handle?: string;
publisherId: Id<"publishers"> | null;
scanned: number;
repaired: number;
skipped: number;
errors: string[];
cursor: string | null;
isDone: boolean;
nextPhase?: LegacyPublisherOwnershipTargetPhase;
};
export const getSkillBackfillPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
@@ -179,6 +209,25 @@ export const getUserStatsBackfillPageInternal = internalQuery({
},
});
export const getPublisherStatsBackfillPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<PublisherStatsBackfillPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const { page, isDone, continueCursor } = await ctx.db
.query("publishers")
.order("asc")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
return {
items: page.map((publisher) => ({ _id: publisher._id })),
cursor: continueCursor,
isDone,
};
},
});
export const getUserOwnedSkillsBackfillPageInternal = internalQuery({
args: {
ownerUserId: v.id("users"),
@@ -219,6 +268,20 @@ export const applyUserStatsBackfillPatchInternal = internalMutation({
},
});
export const recomputePublisherStatsInternal = internalMutation({
args: {
publisherId: v.id("publishers"),
dryRun: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const stats = await recomputePublisherStats(ctx, args.publisherId);
if (!args.dryRun) {
await ctx.db.patch(args.publisherId, stats);
}
return { ok: true as const, stats };
},
});
export type BackfillActionArgs = {
dryRun?: boolean;
batchSize?: number;
@@ -248,6 +311,20 @@ export type UserStatsBackfillActionResult = {
cursor: string | null;
};
export type PublisherStatsBackfillActionArgs = {
dryRun?: boolean;
batchSize?: number;
maxBatches?: number;
cursor?: string;
};
export type PublisherStatsBackfillActionResult = {
ok: true;
stats: PublisherStatsBackfillStats;
isDone: boolean;
cursor: string | null;
};
export async function backfillSkillSummariesInternalHandler(
ctx: ActionCtx,
args: BackfillActionArgs,
@@ -410,6 +487,45 @@ export async function backfillUserStatsInternalHandler(
return { ok: true as const, stats: totals, isDone, cursor };
}
export async function backfillPublisherStatsInternalHandler(
ctx: ActionCtx,
args: PublisherStatsBackfillActionArgs,
): Promise<PublisherStatsBackfillActionResult> {
const dryRun = Boolean(args.dryRun);
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES);
const totals: PublisherStatsBackfillStats = {
publishersScanned: 0,
publishersPatched: 0,
};
let cursor: string | null = args.cursor ?? null;
let isDone = false;
for (let i = 0; i < maxBatches; i++) {
const page = (await ctx.runQuery(internal.maintenance.getPublisherStatsBackfillPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as PublisherStatsBackfillPageResult;
cursor = page.cursor;
isDone = page.isDone;
for (const publisher of page.items) {
totals.publishersScanned++;
await ctx.runMutation(internal.maintenance.recomputePublisherStatsInternal, {
publisherId: publisher._id,
dryRun,
});
if (!dryRun) totals.publishersPatched++;
}
if (isDone) break;
}
return { ok: true as const, stats: totals, isDone, cursor };
}
export const backfillSkillSummariesInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
@@ -431,6 +547,16 @@ export const backfillUserStatsInternal = internalAction({
handler: backfillUserStatsInternalHandler,
});
export const backfillPublisherStatsInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
cursor: v.optional(v.string()),
},
handler: backfillPublisherStatsInternalHandler,
});
export const backfillSkillSummaries: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
@@ -449,6 +575,37 @@ export const backfillSkillSummaries: ReturnType<typeof action> = action({
},
});
export const backfillPublisherStats: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
cursor: v.optional(v.string()),
},
handler: async (ctx, args): Promise<PublisherStatsBackfillActionResult> => {
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin"]);
return ctx.runAction(
internal.maintenance.backfillPublisherStatsInternal,
args,
) as Promise<PublisherStatsBackfillActionResult>;
},
});
export const scheduleBackfillPublisherStats: ReturnType<typeof action> = action({
args: { dryRun: v.optional(v.boolean()) },
handler: async (ctx, args) => {
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin"]);
await ctx.scheduler.runAfter(0, internal.maintenance.backfillPublisherStatsInternal, {
dryRun: Boolean(args.dryRun),
batchSize: DEFAULT_BATCH_SIZE,
maxBatches: DEFAULT_MAX_BATCHES,
});
return { ok: true as const };
},
});
export const scheduleBackfillSkillSummaries: ReturnType<typeof action> = action({
args: { dryRun: v.optional(v.boolean()), useAi: v.optional(v.boolean()) },
handler: async (ctx, args) => {
@@ -1859,46 +2016,6 @@ export const nominateEmptySkillSpammers: ReturnType<typeof action> = action({
},
});
// Backfill embeddingSkillMap from existing skillEmbeddings.
// Run once after deploying the schema change:
// npx convex run maintenance:backfillEmbeddingSkillMapInternal --prod
export const backfillEmbeddingSkillMapInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 200, 10, 500);
const { page, continueCursor, isDone } = await ctx.db
.query("skillEmbeddings")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
let inserted = 0;
for (const embedding of page) {
const existing = await ctx.db
.query("embeddingSkillMap")
.withIndex("by_embedding", (q) => q.eq("embeddingId", embedding._id))
.unique();
if (!existing) {
await ctx.db.insert("embeddingSkillMap", {
embeddingId: embedding._id,
skillId: embedding.skillId,
});
inserted++;
}
}
if (!isDone) {
await ctx.scheduler.runAfter(0, internal.maintenance.backfillEmbeddingSkillMapInternal, {
cursor: continueCursor,
batchSize: args.batchSize,
});
}
return { inserted, isDone, scanned: page.length };
},
});
// Sync skillBadges table → denormalized skill.badges field.
// Run after deploying the badge-read removal to ensure all skills
// have up-to-date badges on the skill doc itself.
@@ -2020,6 +2137,77 @@ export const backfillLatestVersionSummaryInternal = internalMutation({
},
});
export const backfillSkillSearchDigestModerationVerdictsInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
dryRun: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 100, 10, 200);
const dryRun = args.dryRun ?? false;
const { page, continueCursor, isDone } = await ctx.db
.query("skillSearchDigest")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
let patched = 0;
let missingSkills = 0;
for (const digest of page) {
const skill = await ctx.db.get(digest.skillId);
if (!skill) {
missingSkills++;
continue;
}
if (digest.moderationVerdict === skill.moderationVerdict) continue;
patched++;
if (!dryRun) {
await ctx.db.patch(digest._id, {
moderationVerdict: skill.moderationVerdict,
updatedAt: skill.updatedAt,
});
}
}
if (!dryRun && !isDone) {
await ctx.scheduler.runAfter(
0,
internal.maintenance.backfillSkillSearchDigestModerationVerdictsInternal,
{
cursor: continueCursor,
batchSize: args.batchSize,
dryRun,
},
);
}
return {
scanned: page.length,
patched,
missingSkills,
cursor: continueCursor,
isDone,
dryRun,
};
},
});
export const backfillSkillSearchDigestModerationVerdicts: ReturnType<typeof action> = action({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
dryRun: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin"]);
return await ctx.runMutation(
internal.maintenance.backfillSkillSearchDigestModerationVerdictsInternal,
args,
);
},
});
// Repair stale skill-level moderation that was sourced from a non-latest version.
// Run once after deploying the latest-version moderation fix:
// npx convex run maintenance:backfillLatestSkillModeration --prod
@@ -2070,340 +2258,190 @@ export const backfillIsSuspiciousInternal = internalMutation({
},
});
// Backfill skillSearchDigest from existing skills.
// Run once after deploying the schema change:
// npx convex run maintenance:backfillSkillSearchDigestInternal --prod
export const backfillSkillSearchDigestInternal = internalMutation({
function isActiveLegacyPublisherRepairUser(
user: Doc<"users"> | null | undefined,
): user is Doc<"users"> {
return Boolean(user && !user.deletedAt && !user.deactivatedAt && !user.purgedAt);
}
function nextLegacyPublisherOwnershipTargetPhase(
phase: LegacyPublisherOwnershipTargetPhase,
): LegacyPublisherOwnershipTargetPhase | undefined {
return phase === "skills" ? "packages" : undefined;
}
async function getExistingActivePersonalPublisher(
ctx: Pick<MutationCtx, "db">,
user: Doc<"users">,
) {
if (user.personalPublisherId) {
const publisher = await ctx.db.get(user.personalPublisherId);
if (isPublisherActive(publisher)) return publisher;
}
const publisher = await getPersonalPublisherForUser(ctx, user._id);
return isPublisherActive(publisher) ? publisher : null;
}
async function resolvePersonalPublisherForOwnershipRepair(
ctx: Pick<MutationCtx, "db">,
user: Doc<"users">,
dryRun: boolean,
) {
if (dryRun) {
const existing = await getExistingActivePersonalPublisher(ctx, user);
if (existing) return existing;
const handle = derivePersonalPublisherHandle(user);
const conflict = await getPublisherByHandle(ctx, handle);
if (conflict && conflict.linkedUserId !== user._id) {
throw new ConvexError(`Publisher handle "@${handle}" is already claimed`);
}
return null;
}
return await ensurePersonalPublisherForUser(ctx, user, {
source: "maintenance.legacy_publisher_ownership",
});
}
async function resolveLegacyPublisherOwnershipTargetUser(
ctx: Pick<MutationCtx, "db">,
args: { userId?: Id<"users">; handle?: string },
) {
const user = args.userId
? await ctx.db.get(args.userId)
: await getUserByHandleOrPersonalPublisher(ctx, args.handle);
if (!user) throw new ConvexError("Target user not found");
if (!isActiveLegacyPublisherRepairUser(user)) throw new ConvexError("Target user is inactive");
return user;
}
async function patchLegacySkillOwnerPublisher(
ctx: Pick<MutationCtx, "db">,
skill: Doc<"skills">,
publisherId: Id<"publishers">,
) {
await ctx.db.patch(skill._id, { ownerPublisherId: publisherId });
const aliases = await ctx.db
.query("skillSlugAliases")
.withIndex("by_skill", (q) => q.eq("skillId", skill._id))
.collect();
for (const alias of aliases) {
if (alias.ownerPublisherId === publisherId) continue;
await ctx.db.patch(alias._id, { ownerPublisherId: publisherId });
}
}
async function patchLegacyPackageOwnerPublisher(
ctx: Pick<MutationCtx, "db">,
pkg: Doc<"packages">,
publisherId: Id<"publishers">,
) {
await ctx.db.patch(pkg._id, { ownerPublisherId: publisherId });
}
export async function repairLegacyPublisherOwnershipForUserHandler(
ctx: MutationCtx,
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
userId?: Id<"users">;
handle?: string;
phase?: LegacyPublisherOwnershipTargetPhase;
cursor?: string;
batchSize?: number;
delayMs?: number;
dryRun?: boolean;
scheduleNext?: boolean;
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 200, 10, 500);
const { page, continueCursor, isDone } = await ctx.db
.query("skills")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
): Promise<LegacyPublisherOwnershipForUserRepairResult> {
const phase = args.phase ?? "skills";
const dryRun = args.dryRun === true;
const batchSize = clampInt(args.batchSize ?? 50, 1, 200);
const delayMs = clampInt(args.delayMs ?? 500, 0, 60_000);
const user = await resolveLegacyPublisherOwnershipTargetUser(ctx, args);
const publisher = await resolvePersonalPublisherForOwnershipRepair(ctx, user, dryRun);
if (!dryRun && !isPublisherActive(publisher)) {
throw new ConvexError("Target personal publisher could not be repaired");
}
let upserted = 0;
for (const skill of page) {
await upsertSkillSearchDigest(ctx, await extractValidatedDigestFields(ctx, skill));
upserted++;
let scanned = 0;
let repaired = 0;
let skipped = 0;
const page =
phase === "skills"
? await ctx.db
.query("skills")
.withIndex("by_owner", (q) => q.eq("ownerUserId", user._id))
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
: await ctx.db
.query("packages")
.withIndex("by_owner", (q) => q.eq("ownerUserId", user._id))
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
for (const item of page.page) {
scanned++;
if (item.ownerPublisherId) {
skipped++;
continue;
}
if (!isDone) {
await ctx.scheduler.runAfter(0, internal.maintenance.backfillSkillSearchDigestInternal, {
cursor: continueCursor,
batchSize: args.batchSize,
});
if (dryRun) {
repaired++;
continue;
}
return { upserted, isDone, scanned: page.length };
},
});
// Backfill plugin category digest rows for existing active packages.
// Run once after deploying the schema change:
// npx convex run maintenance:backfillPackagePluginCategoryDigestsInternal --prod
export const backfillPackagePluginCategoryDigestsInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 200, 10, 500);
const { page, continueCursor, isDone } = await ctx.db
.query("packages")
.withIndex("by_active_updated", (q) => q.eq("softDeletedAt", undefined))
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
let synced = 0;
for (const pkg of page) {
await upsertPackageSearchDigest(ctx, extractPackageDigestFields(pkg));
synced++;
}
if (!isDone) {
await ctx.scheduler.runAfter(
0,
internal.maintenance.backfillPackagePluginCategoryDigestsInternal,
{
cursor: continueCursor,
batchSize: args.batchSize,
},
);
}
return { synced, isDone, scanned: page.length };
},
});
const DIGEST_OWNER_BACKFILL_KEY = "digest-owner-backfill";
// Start/resume backfill:
// npx convex run maintenance:backfillDigestOwnerFields '{"batchSize":50,"delayMs":5000}' --prod
// Stop:
// npx convex run maintenance:stopBackfillDigestOwnerFields --prod
// Check status:
// npx convex run maintenance:backfillDigestOwnerFieldsStatus --prod
export const backfillDigestOwnerFields = internalMutation({
args: {
batchSize: v.optional(v.number()),
delayMs: v.optional(v.number()),
},
handler: async (ctx, args) => {
// Clear any previous stop flag and store config
const existing = await ctx.db
.query("skillStatBackfillState")
.withIndex("by_key", (q) => q.eq("key", DIGEST_OWNER_BACKFILL_KEY))
.unique();
if (existing) {
await ctx.db.patch(existing._id, {
cursor: undefined,
doneAt: undefined,
updatedAt: Date.now(),
});
if (phase === "skills") {
await patchLegacySkillOwnerPublisher(ctx, item as Doc<"skills">, publisher!._id);
} else {
await ctx.db.insert("skillStatBackfillState", {
key: DIGEST_OWNER_BACKFILL_KEY,
updatedAt: Date.now(),
});
await patchLegacyPackageOwnerPublisher(ctx, item as Doc<"packages">, publisher!._id);
}
// Kick off first batch
await ctx.scheduler.runAfter(0, internal.maintenance.backfillDigestOwnerFieldsInternal, {
batchSize: args.batchSize,
delayMs: args.delayMs,
});
return { started: true };
},
});
repaired++;
}
export const stopBackfillDigestOwnerFields = internalMutation({
args: {},
handler: async (ctx) => {
const state = await ctx.db
.query("skillStatBackfillState")
.withIndex("by_key", (q) => q.eq("key", DIGEST_OWNER_BACKFILL_KEY))
.unique();
if (state) {
await ctx.db.patch(state._id, { doneAt: Date.now(), updatedAt: Date.now() });
}
return { stopped: true };
},
});
export const backfillDigestOwnerFieldsStatus = internalQuery({
args: {},
handler: async (ctx) => {
const state = await ctx.db
.query("skillStatBackfillState")
.withIndex("by_key", (q) => q.eq("key", DIGEST_OWNER_BACKFILL_KEY))
.unique();
if (!state) return { status: "never_started" };
if (state.doneAt) return { status: "stopped", cursor: state.cursor, stoppedAt: state.doneAt };
return { status: "running", cursor: state.cursor };
},
});
export const backfillDigestOwnerFieldsInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
delayMs: v.optional(v.number()),
},
handler: async (ctx, args) => {
// Check stop flag
const state = await ctx.db
.query("skillStatBackfillState")
.withIndex("by_key", (q) => q.eq("key", DIGEST_OWNER_BACKFILL_KEY))
.unique();
if (state?.doneAt) {
return { patched: 0, isDone: false, scanned: 0, stopped: true };
}
const batchSize = clampInt(args.batchSize ?? 200, 10, 500);
const delayMs = clampInt(args.delayMs ?? 0, 0, 60_000);
const { page, continueCursor, isDone } = await ctx.db
.query("skillSearchDigest")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
let patched = 0;
for (const digest of page) {
if (digest.ownerHandle !== undefined) continue;
const owner = await ctx.db.get(digest.ownerUserId);
const isOwnerVisible = owner && !owner.deletedAt && !owner.deactivatedAt;
await ctx.db.patch(digest._id, {
ownerHandle: isOwnerVisible ? (owner.handle ?? "") : "",
ownerName: isOwnerVisible ? owner.name : undefined,
ownerDisplayName: isOwnerVisible ? owner.displayName : undefined,
ownerImage: isOwnerVisible ? owner.image : undefined,
});
patched++;
}
// Save cursor progress
if (state) {
await ctx.db.patch(state._id, {
cursor: continueCursor,
doneAt: isDone ? Date.now() : undefined,
updatedAt: Date.now(),
});
}
if (!isDone) {
await ctx.scheduler.runAfter(
delayMs,
internal.maintenance.backfillDigestOwnerFieldsInternal,
{
cursor: continueCursor,
batchSize: args.batchSize,
delayMs: args.delayMs,
},
);
}
return { patched, isDone, scanned: page.length, stopped: false };
},
});
// Backfill latestVersionSummary from skills into existing skillSearchDigest rows.
// Run:
// npx convex run maintenance:backfillDigestVersionSummary '{"batchSize":100}' --prod
export const backfillDigestVersionSummary = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 200, 10, 500);
const { page, continueCursor, isDone } = await ctx.db
.query("skillSearchDigest")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
let patched = 0;
for (const digest of page) {
const skill = await ctx.db.get(digest.skillId);
if (!skill) continue;
const fields = await extractValidatedDigestFields(ctx, skill);
const patch = {
latestVersionId: fields.latestVersionId,
latestVersionSkillId: fields.latestVersionSkillId,
latestVersionSummary: fields.latestVersionSummary,
capabilityTags: fields.capabilityTags,
};
if (
digest.latestVersionId === patch.latestVersionId &&
digest.latestVersionSkillId === patch.latestVersionSkillId &&
JSON.stringify(digest.latestVersionSummary) ===
JSON.stringify(patch.latestVersionSummary) &&
JSON.stringify(digest.capabilityTags ?? []) === JSON.stringify(patch.capabilityTags ?? [])
) {
continue;
}
await ctx.db.patch(digest._id, patch);
patched++;
}
if (!isDone) {
await ctx.scheduler.runAfter(0, internal.maintenance.backfillDigestVersionSummary, {
cursor: continueCursor,
batchSize: args.batchSize,
});
}
return { patched, isDone, scanned: page.length };
},
});
// Backfill isSuspicious on skillSearchDigest rows where it's undefined.
// Computes from digest's own moderationFlags/moderationReason — no skills table read.
// Run: npx convex run maintenance:backfillDigestIsSuspicious --prod
export const backfillDigestIsSuspicious = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
delayMs: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 100, 10, 200);
const delayMs = args.delayMs ?? 500;
const { page, continueCursor, isDone } = await ctx.db
.query("skillSearchDigest")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
let patched = 0;
for (const digest of page) {
if (digest.isSuspicious !== undefined) continue;
const isSuspicious = computeIsSuspicious(digest);
await ctx.db.patch(digest._id, { isSuspicious });
patched++;
}
if (!isDone) {
await ctx.scheduler.runAfter(delayMs, internal.maintenance.backfillDigestIsSuspicious, {
cursor: continueCursor,
const nextPhase = page.isDone ? nextLegacyPublisherOwnershipTargetPhase(phase) : phase;
if (!dryRun && args.scheduleNext !== false && nextPhase) {
await ctx.scheduler.runAfter(
delayMs,
internal.maintenance.repairLegacyPublisherOwnershipForUser,
{
userId: user._id,
phase: nextPhase,
cursor: page.isDone ? undefined : (page.continueCursor ?? undefined),
batchSize: args.batchSize,
delayMs: args.delayMs,
});
}
scheduleNext: args.scheduleNext,
},
);
}
return { patched, isDone, scanned: page.length };
},
});
return {
phase,
dryRun,
userId: user._id,
handle: user.handle,
publisherId: publisher?._id ?? null,
scanned,
repaired,
skipped,
errors: [],
cursor: page.continueCursor,
isDone: page.isDone,
...(nextPhase ? { nextPhase } : {}),
};
}
// Backfill normalized search fields on skillSearchDigest for indexed prefix search.
// Run: npx convex run maintenance:backfillDigestNormalizedSearchFields --prod
export const backfillDigestNormalizedSearchFields = internalMutation({
// Targeted variant for production canaries and one-off account repair.
// Example:
// npx convex run maintenance:repairLegacyPublisherOwnershipForUser '{"handle":"harrylabsj","dryRun":true,"scheduleNext":false}' --prod
export const repairLegacyPublisherOwnershipForUser = internalMutation({
args: {
userId: v.optional(v.id("users")),
handle: v.optional(v.string()),
phase: v.optional(v.union(v.literal("skills"), v.literal("packages"))),
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
delayMs: v.optional(v.number()),
dryRun: v.optional(v.boolean()),
scheduleNext: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 100, 10, 200);
const delayMs = args.delayMs ?? 500;
const { page, continueCursor, isDone } = await ctx.db
.query("skillSearchDigest")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
let patched = 0;
for (const digest of page) {
const normalizedSlug = normalizeSkillSearchText(digest.slug);
const normalizedSlugFirstToken = getFirstSearchToken(digest.slug);
const normalizedDisplayName = normalizeSkillSearchText(digest.displayName);
const normalizedDisplayNameFirstToken = getFirstSearchToken(digest.displayName);
if (
digest.normalizedSlug === normalizedSlug &&
digest.normalizedSlugFirstToken === normalizedSlugFirstToken &&
digest.normalizedDisplayName === normalizedDisplayName &&
digest.normalizedDisplayNameFirstToken === normalizedDisplayNameFirstToken
) {
continue;
}
await ctx.db.patch(digest._id, {
normalizedSlug,
normalizedSlugFirstToken,
normalizedDisplayName,
normalizedDisplayNameFirstToken,
});
patched++;
}
if (!isDone && args.scheduleNext !== false) {
await ctx.scheduler.runAfter(
delayMs,
internal.maintenance.backfillDigestNormalizedSearchFields,
{
cursor: continueCursor,
batchSize: args.batchSize,
delayMs: args.delayMs,
scheduleNext: args.scheduleNext,
},
);
}
return { patched, isDone, scanned: page.length, cursor: continueCursor };
},
handler: repairLegacyPublisherOwnershipForUserHandler,
});
function clampInt(value: number, min: number, max: number) {
+360
View File
@@ -0,0 +1,360 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from "vitest";
vi.mock("./_generated/server", () => ({
internalMutation: (def: { handler: unknown }) => ({ _handler: def.handler }),
}));
const managementDevSeed = await import("./managementDevSeed");
type Handler<TArgs, TResult> = (ctx: unknown, args: TArgs) => Promise<TResult>;
type Wrapped<TArgs, TResult> = { _handler: Handler<TArgs, TResult> };
type TestDoc = Record<string, unknown> & { _id: string };
const seedManagementQueuesHandler = (
managementDevSeed.seedManagementQueues as unknown as Wrapped<
Record<string, never>,
{ reportsInserted: number; reportedSkills: number; duplicatePair: number }
>
)._handler;
const clearManagementQueuesHandler = (
managementDevSeed.clearManagementQueues as unknown as Wrapped<
Record<string, never>,
{ reportsDeleted: number; fingerprintsDeleted: number }
>
)._handler;
const DEMO_REPORT_MARKER = "managementDevSeed:report";
const DEMO_FINGERPRINT = "9f8c2a1b7e4d6c30a5b2f1d089c4e76b";
function chainEq(constraints: Record<string, unknown>) {
return {
eq(field: string, value: unknown) {
constraints[field] = value;
return chainEq(constraints);
},
};
}
function matches(doc: TestDoc, constraints: Record<string, unknown>) {
return Object.entries(constraints).every(([key, value]) => doc[key] === value);
}
function createDb(seedTables: Record<string, TestDoc[]>) {
const tables = Object.fromEntries(
Object.entries(seedTables).map(([name, docs]) => [name, docs.map((doc) => ({ ...doc }))]),
);
const queryCalls: Array<{
table: string;
indexName: string;
constraints: Record<string, unknown>;
}> = [];
const inserts: Array<{ table: string; doc: TestDoc }> = [];
let insertCounter = 0;
const list = (table: string) => {
tables[table] ??= [];
return tables[table];
};
const takeRows = (table: string, numItems: number, constraints?: Record<string, unknown>) => {
const rows = constraints ? list(table).filter((doc) => matches(doc, constraints)) : list(table);
return rows.slice(0, numItems);
};
return {
inserts,
queryCalls,
tables,
db: {
delete: async (id: string) => {
const table = id.split(":")[0] ?? "";
const rows = list(table);
const index = rows.findIndex((doc) => doc._id === id);
if (index !== -1) rows.splice(index, 1);
},
get: async (id: string) => {
const table = id.split(":")[0] ?? "";
return list(table).find((doc) => doc._id === id) ?? null;
},
insert: async (table: string, doc: Record<string, unknown>) => {
const inserted = { ...doc, _id: `${table}:inserted-${insertCounter}` };
insertCounter += 1;
list(table).push(inserted);
inserts.push({ table, doc: inserted });
return inserted._id;
},
patch: async (id: string, patch: Record<string, unknown>) => {
const table = id.split(":")[0] ?? "";
const row = list(table).find((doc) => doc._id === id);
if (row) Object.assign(row, patch);
},
query: (table: string) => ({
order: () => ({
take: async (numItems: number) => takeRows(table, numItems),
}),
take: async (numItems: number) => takeRows(table, numItems),
withIndex: (indexName: string, build: (q: ReturnType<typeof chainEq>) => unknown) => {
const constraints: Record<string, unknown> = {};
build(chainEq(constraints));
queryCalls.push({ table, indexName, constraints });
return {
order: () => ({
take: async (numItems: number) => takeRows(table, numItems, constraints),
}),
take: async (numItems: number) => takeRows(table, numItems, constraints),
};
},
}),
},
};
}
function restoreEnv(name: string, value: string | undefined) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
describe("managementDevSeed", () => {
const previousDeployment = process.env.CONVEX_DEPLOYMENT;
const previousDevAuthDeployment = process.env.DEV_AUTH_CONVEX_DEPLOYMENT;
const previousDevAuthEnabled = process.env.DEV_AUTH_ENABLED;
const previousDevImpersonation = process.env.CLAW_HUB_ENABLE_DEV_IMPERSONATION;
afterEach(() => {
restoreEnv("CONVEX_DEPLOYMENT", previousDeployment);
restoreEnv("DEV_AUTH_CONVEX_DEPLOYMENT", previousDevAuthDeployment);
restoreEnv("DEV_AUTH_ENABLED", previousDevAuthEnabled);
restoreEnv("CLAW_HUB_ENABLE_DEV_IMPERSONATION", previousDevImpersonation);
});
it("rejects production deployments before reading tables", async () => {
process.env.CONVEX_DEPLOYMENT = "prod:wry-manatee-359";
const query = vi.fn();
await expect(seedManagementQueuesHandler({ db: { query } }, {})).rejects.toThrow(
"disabled outside local/dev deployments",
);
await expect(clearManagementQueuesHandler({ db: { query } }, {})).rejects.toThrow(
"disabled outside local/dev deployments",
);
expect(query).not.toHaveBeenCalled();
});
it("honors the explicit fallback deployment when the primary marker is blank", async () => {
process.env.CONVEX_DEPLOYMENT = "";
process.env.DEV_AUTH_CONVEX_DEPLOYMENT = "prod:wry-manatee-359";
process.env.DEV_AUTH_ENABLED = "1";
const query = vi.fn();
await expect(seedManagementQueuesHandler({ db: { query } }, {})).rejects.toThrow(
"disabled outside local/dev deployments",
);
await expect(clearManagementQueuesHandler({ db: { query } }, {})).rejects.toThrow(
"disabled outside local/dev deployments",
);
expect(query).not.toHaveBeenCalled();
});
it("seeds content report and duplicate candidate rows for local dashboards", async () => {
process.env.CONVEX_DEPLOYMENT = "";
process.env.DEV_AUTH_CONVEX_DEPLOYMENT = "dev:admired-dodo-615";
const { db, inserts, tables } = createDb({
users: [{ _id: "users:reporter", handle: "local-admin" }],
skills: [
{ _id: "skills:one", latestVersionId: "skillVersions:one" },
{ _id: "skills:two", latestVersionId: "skillVersions:two" },
{ _id: "skills:three", latestVersionId: "skillVersions:three" },
{ _id: "skills:hidden", latestVersionId: "skillVersions:hidden", softDeletedAt: 1 },
],
skillVersions: [
{ _id: "skillVersions:one" },
{ _id: "skillVersions:two" },
{ _id: "skillVersions:three" },
{ _id: "skillVersions:hidden" },
],
skillReports: [],
skillVersionFingerprints: [],
});
await expect(seedManagementQueuesHandler({ db }, {})).resolves.toEqual({
reportsInserted: 6,
reportedSkills: 3,
duplicatePair: 2,
});
expect(tables.skillReports).toHaveLength(6);
expect(tables.skillReports.every((report) => report.triageNote === DEMO_REPORT_MARKER)).toBe(
true,
);
expect(tables.skills.find((skill) => skill._id === "skills:one")).toEqual(
expect.objectContaining({ reportCount: 1 }),
);
expect(tables.skills.find((skill) => skill._id === "skills:two")).toEqual(
expect.objectContaining({ reportCount: 2 }),
);
expect(tables.skills.find((skill) => skill._id === "skills:three")).toEqual(
expect.objectContaining({ reportCount: 3 }),
);
expect(tables.skillVersions.find((version) => version._id === "skillVersions:one")).toEqual(
expect.objectContaining({ fingerprint: DEMO_FINGERPRINT }),
);
expect(tables.skillVersions.find((version) => version._id === "skillVersions:two")).toEqual(
expect.objectContaining({ fingerprint: DEMO_FINGERPRINT }),
);
expect(inserts.filter((insert) => insert.table === "skillVersionFingerprints")).toHaveLength(2);
});
it("does not overwrite existing latest-version fingerprints", async () => {
process.env.CONVEX_DEPLOYMENT = "";
process.env.DEV_AUTH_CONVEX_DEPLOYMENT = "dev:admired-dodo-615";
const { db, inserts, tables } = createDb({
users: [{ _id: "users:reporter", handle: "local-admin" }],
skills: [
{ _id: "skills:one", latestVersionId: "skillVersions:one" },
{ _id: "skills:two", latestVersionId: "skillVersions:two" },
{ _id: "skills:three", latestVersionId: "skillVersions:three" },
{ _id: "skills:four", latestVersionId: "skillVersions:four" },
],
skillVersions: [
{ _id: "skillVersions:one", fingerprint: "real-fingerprint-one" },
{ _id: "skillVersions:two", fingerprint: "real-fingerprint-two" },
{ _id: "skillVersions:three" },
{ _id: "skillVersions:four" },
],
skillReports: [],
skillVersionFingerprints: [
{
_id: "skillVersionFingerprints:real-one",
skillId: "skills:one",
versionId: "skillVersions:one",
fingerprint: "real-fingerprint-one",
},
{
_id: "skillVersionFingerprints:real-two",
skillId: "skills:two",
versionId: "skillVersions:two",
fingerprint: "real-fingerprint-two",
},
],
});
await expect(seedManagementQueuesHandler({ db }, {})).resolves.toEqual({
reportsInserted: 6,
reportedSkills: 3,
duplicatePair: 2,
});
expect(tables.skillVersions.find((version) => version._id === "skillVersions:one")).toEqual(
expect.objectContaining({ fingerprint: "real-fingerprint-one" }),
);
expect(tables.skillVersions.find((version) => version._id === "skillVersions:two")).toEqual(
expect.objectContaining({ fingerprint: "real-fingerprint-two" }),
);
expect(tables.skillVersions.find((version) => version._id === "skillVersions:three")).toEqual(
expect.objectContaining({ fingerprint: DEMO_FINGERPRINT }),
);
expect(tables.skillVersions.find((version) => version._id === "skillVersions:four")).toEqual(
expect.objectContaining({ fingerprint: DEMO_FINGERPRINT }),
);
expect(
inserts
.filter((insert) => insert.table === "skillVersionFingerprints")
.map((insert) => insert.doc.versionId),
).toEqual(["skillVersions:three", "skillVersions:four"]);
});
it("clears only marked demo management rows", async () => {
process.env.CONVEX_DEPLOYMENT = "dev:admired-dodo-615";
const { db, queryCalls, tables } = createDb({
skills: [
{
_id: "skills:demo",
reportCount: 2,
lastReportedAt: 200,
},
{
_id: "skills:real",
reportCount: 1,
lastReportedAt: 200,
},
],
skillReports: [
{
_id: "skillReports:demo",
skillId: "skills:demo",
triageNote: DEMO_REPORT_MARKER,
status: "open",
createdAt: 100,
},
{
_id: "skillReports:demo-real",
skillId: "skills:demo",
triageNote: "real-user-report",
status: "open",
createdAt: 200,
},
{
_id: "skillReports:real",
skillId: "skills:real",
triageNote: "real-user-report",
status: "open",
createdAt: 200,
},
],
skillVersions: [
{ _id: "skillVersions:demo", fingerprint: DEMO_FINGERPRINT },
{ _id: "skillVersions:real", fingerprint: "real-fingerprint" },
],
skillVersionFingerprints: [
{
_id: "skillVersionFingerprints:demo",
versionId: "skillVersions:demo",
fingerprint: DEMO_FINGERPRINT,
},
{
_id: "skillVersionFingerprints:real",
versionId: "skillVersions:real",
fingerprint: "real-fingerprint",
},
],
});
await expect(clearManagementQueuesHandler({ db }, {})).resolves.toEqual({
reportsDeleted: 1,
fingerprintsDeleted: 1,
});
expect(tables.skillReports.map((report) => report._id)).toEqual([
"skillReports:demo-real",
"skillReports:real",
]);
expect(tables.skillVersionFingerprints.map((fingerprint) => fingerprint._id)).toEqual([
"skillVersionFingerprints:real",
]);
expect(tables.skillVersions.find((version) => version._id === "skillVersions:demo")).toEqual(
expect.objectContaining({ fingerprint: undefined }),
);
expect(tables.skillVersions.find((version) => version._id === "skillVersions:real")).toEqual(
expect.objectContaining({ fingerprint: "real-fingerprint" }),
);
expect(tables.skills.find((skill) => skill._id === "skills:demo")).toEqual(
expect.objectContaining({ reportCount: 1, lastReportedAt: 200 }),
);
expect(tables.skills.find((skill) => skill._id === "skills:real")).toEqual(
expect.objectContaining({ reportCount: 1, lastReportedAt: 200 }),
);
expect(queryCalls).toContainEqual({
table: "skillVersionFingerprints",
indexName: "by_fingerprint",
constraints: { fingerprint: DEMO_FINGERPRINT },
});
expect(queryCalls).toContainEqual({
table: "skillReports",
indexName: "by_skill_createdAt",
constraints: { skillId: "skills:demo" },
});
});
});
+181
View File
@@ -0,0 +1,181 @@
import type { Doc, Id } from "./_generated/dataModel";
import type { MutationCtx } from "./_generated/server";
// DEV-ONLY seed for the management Content-reports and Duplicate-candidates queues.
// Uses the un-wrapped mutation builder (not convex/functions.ts) so patching skills
// / versions and inserting report + fingerprint rows does NOT fire table triggers.
// It operates on existing seeded skills rather than creating new ones, so the base
// dev seed must have run first. All demo rows carry a marker so clearDemo can remove
// them precisely.
import { internalMutation } from "./_generated/server";
import { assertLocalDevSeedAllowed } from "./lib/devSeed";
const DEMO_REPORT_MARKER = "managementDevSeed:report";
// Hash-like so the dashboard's fingerprint chip reads like real data; still a
// fixed constant so clearDemo can find and remove the seeded rows.
const DEMO_FINGERPRINT = "9f8c2a1b7e4d6c30a5b2f1d089c4e76b";
const DEMO_REPORT_REASONS = [
"Possible prompt-injection hidden in the skill instructions.",
"Looks like a copy of another publisher's skill.",
"Requests credentials it does not appear to need.",
"Spammy catalog filler with no real functionality.",
];
const HOUR_MS = 60 * 60 * 1000;
const REPORT_SCAN_LIMIT = 500;
const SKILL_SCAN_LIMIT = 50;
type DuplicateDemoTarget = {
skill: Doc<"skills">;
versionId: Id<"skillVersions">;
};
// Remove previously seeded demo reports + duplicate fingerprints so the seed is
// idempotent and the dashboard can be reset.
async function clearDemo(ctx: Pick<MutationCtx, "db">): Promise<{
reportsDeleted: number;
fingerprintsDeleted: number;
}> {
let reportsDeleted = 0;
let fingerprintsDeleted = 0;
const affectedSkillIds = new Set<Id<"skills">>();
const reports = await ctx.db.query("skillReports").order("desc").take(REPORT_SCAN_LIMIT);
for (const report of reports) {
if (report.triageNote !== DEMO_REPORT_MARKER) continue;
affectedSkillIds.add(report.skillId);
await ctx.db.delete(report._id);
reportsDeleted += 1;
}
for (const skillId of affectedSkillIds) {
const skill = await ctx.db.get(skillId);
if (!skill) continue;
await restoreSkillReportSummary(ctx, skillId);
}
const fingerprints = await ctx.db
.query("skillVersionFingerprints")
.withIndex("by_fingerprint", (q) => q.eq("fingerprint", DEMO_FINGERPRINT))
.take(100);
for (const fingerprint of fingerprints) {
const version = await ctx.db.get(fingerprint.versionId);
if (version && version.fingerprint === DEMO_FINGERPRINT) {
await ctx.db.patch(fingerprint.versionId, { fingerprint: undefined });
}
await ctx.db.delete(fingerprint._id);
fingerprintsDeleted += 1;
}
return { reportsDeleted, fingerprintsDeleted };
}
async function restoreSkillReportSummary(
ctx: Pick<MutationCtx, "db">,
skillId: Id<"skills">,
): Promise<void> {
const reports = await ctx.db
.query("skillReports")
.withIndex("by_skill_createdAt", (q) => q.eq("skillId", skillId))
.order("desc")
.take(REPORT_SCAN_LIMIT);
const openReports = reports.filter((report) => (report.status ?? "open") === "open");
await ctx.db.patch(skillId, {
reportCount: openReports.length > 0 ? openReports.length : undefined,
lastReportedAt: openReports[0]?.createdAt,
});
}
async function findDuplicateDemoTargets(
ctx: Pick<MutationCtx, "db">,
skills: Doc<"skills">[],
): Promise<DuplicateDemoTarget[]> {
const targets: DuplicateDemoTarget[] = [];
for (const skill of skills) {
const versionId = skill.latestVersionId;
if (!versionId) continue;
const version = await ctx.db.get(versionId);
if (!version || version.fingerprint) continue;
targets.push({ skill, versionId });
if (targets.length === 2) break;
}
return targets;
}
export const seedManagementQueues = internalMutation({
args: {},
handler: async (
ctx,
): Promise<{
reportsInserted: number;
reportedSkills: number;
duplicatePair: number;
}> => {
assertLocalDevSeedAllowed("Management");
await clearDemo(ctx);
const now = Date.now();
const reporter = (await ctx.db.query("users").take(1))[0];
if (!reporter) {
throw new Error("No users found to attribute demo reports to; run the base dev seed first.");
}
const skills = (await ctx.db.query("skills").order("desc").take(SKILL_SCAN_LIMIT)).filter(
(skill) => !skill.softDeletedAt && skill.latestVersionId,
);
if (skills.length < 2) {
throw new Error("Need at least 2 seeded skills; run the base dev seed first.");
}
// Content reports: flag the first few skills with 1-3 reports each.
const reportTargets = skills.slice(0, Math.min(3, skills.length));
let reportsInserted = 0;
for (let i = 0; i < reportTargets.length; i += 1) {
const skill = reportTargets[i];
const count = 1 + (i % 3);
for (let r = 0; r < count; r += 1) {
await ctx.db.insert("skillReports", {
skillId: skill._id,
userId: reporter._id,
reason: DEMO_REPORT_REASONS[(i + r) % DEMO_REPORT_REASONS.length],
status: "open",
triageNote: DEMO_REPORT_MARKER,
createdAt: now - (i * 3 + r) * HOUR_MS,
});
reportsInserted += 1;
}
await ctx.db.patch(skill._id, {
reportCount: count,
lastReportedAt: now - i * HOUR_MS,
});
}
// Duplicate candidates: give a pair of skills the same latest-version fingerprint
// so each surfaces the other as a near-duplicate.
const duplicatePair = await findDuplicateDemoTargets(ctx, skills);
for (const { skill, versionId } of duplicatePair) {
await ctx.db.patch(versionId, { fingerprint: DEMO_FINGERPRINT });
await ctx.db.insert("skillVersionFingerprints", {
skillId: skill._id,
versionId,
fingerprint: DEMO_FINGERPRINT,
kind: "source",
createdAt: now,
});
}
return {
reportsInserted,
reportedSkills: reportTargets.length,
duplicatePair: duplicatePair.length,
};
},
});
export const clearManagementQueues = internalMutation({
args: {},
handler: async (ctx): Promise<{ reportsDeleted: number; fingerprintsDeleted: number }> => {
assertLocalDevSeedAllowed("Management");
return clearDemo(ctx);
},
});
+42
View File
@@ -0,0 +1,42 @@
declare module "@openclaw/plugin-inspector" {
export type PluginInspectorReport = {
status?: string;
summary?: {
breakageCount?: number;
[key: string]: unknown;
};
[key: string]: unknown;
};
export type PluginInspectorPaths = {
jsonPath: string;
markdownPath?: string;
issuesPath?: string;
[key: string]: unknown;
};
export const pluginRoot: {
runCheck(options?: {
pluginRoot?: string;
openclawPath?: string | false;
outDir?: string;
capture?: boolean;
mockSdk?: boolean;
allowExecution?: boolean;
configPath?: string;
generatedAt?: string;
}): Promise<{ report: PluginInspectorReport; paths: PluginInspectorPaths }>;
};
export const reports: {
renderTextSummary(report: PluginInspectorReport, options?: Record<string, unknown>): string;
sanitizeArtifact(report: PluginInspectorReport): unknown;
};
export const ci: {
writeOutputs(
report: PluginInspectorReport,
options?: Record<string, unknown>,
): Promise<unknown>;
};
}
+14
View File
@@ -0,0 +1,14 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
import { absolutePackageArtifactUrl } from "./packageInspectorHttp";
describe("package inspector HTTP helpers", () => {
it("returns the protected artifact route for scan claims", () => {
const request = new Request("https://example.com/api/v1/package-inspector/claim");
expect(absolutePackageArtifactUrl(request, "packageReleases:demo-1")).toBe(
"https://example.com/api/v1/package-inspector/artifact?releaseId=packageReleases%3Ademo-1",
);
});
});

Some files were not shown because too many files have changed in this diff Show More