Compare commits

...
177 Commits
Author SHA1 Message Date
Peter Steinberger ffa25f47da chore: rebrand user-facing to OpenClaw 2026-01-30 07:02:35 +01:00
Peter Steinberger f866dc05d1 style: format moderation flags 2026-01-30 05:27:19 +01:00
Peter Steinberger 69436fd79b fix: allow moltbot parsed data 2026-01-30 05:26:09 +01:00
Peter Steinberger f185ca6f55 feat: release 0.4.0 2026-01-30 05:23:12 +01:00
Peter Steinberger f3fc8d62b6 Revert "chore: rename molthub branding"
This reverts commit 8d68b55333.
2026-01-30 05:23:12 +01:00
Shakker 71f94a774f Merge pull request #70 from moltbot/remove-clawd-authenticator-tool
chore(moderation): block ClawdAuthenticatorTool (suspected malware)
2026-01-29 17:18:09 +00:00
vignesh07 07349e6107 chore(moderation): block ClawdAuthenticatorTool listing 2026-01-29 09:14:41 -08:00
Vignesh 5ef00e8c6a chore(security): harden file endpoints CSP + XFO + svg detection (#67) 2026-01-28 23:21:16 -06:00
Jamieson O'Reillyandtheonejvo c5e5e657dd fix: add CSP headers and Content-Disposition to prevent SVG XSS (#61)
Co-authored-by: theonejvo <theonejvo@users.noreply.github.com>
2026-01-28 22:21:39 -06:00
Jamie Turner 69e1e5c507 A few fixes for search. (#64) 2026-01-28 22:21:26 -06:00
Josh Palmer 481f1b9188 Merge pull request #66 from moltbot/fix/ci-lint
fix: restore lint compliance
2026-01-28 21:27:21 +01:00
Josh Palmer d4f5832554 🤖 chore: merge main into fix/ci-lint
- resolve conflicts in search/skill publish files
- apply Biome formatting updates from main

Tests: bun run lint:biome; bun run lint:oxlint
2026-01-28 21:23:50 +01:00
Josh Palmer ca3275ba92 🤖 fix: restore lint compliance
- apply Biome formatting and import ordering across linted files
- fix management useEffect dependencies flagged by Biome

Tests: bun run lint:biome; bun run lint:oxlint
2026-01-28 21:18:19 +01:00
Shadow d577add8a0 fix: unblock convex deploy typecheck 2026-01-28 13:15:18 -06:00
Shadow 90bd065f7e fix: correct public skill entries build 2026-01-28 13:12:41 -06:00
Shadow 219f05b160 fix: sanitize public skill and soul data 2026-01-28 13:03:19 -06:00
Shadow c8091ee8a8 chore: remove unauthenticated badge backfill 2026-01-28 08:33:14 -06:00
Shadow 63f367e1d6 chore: add unauthenticated badge backfill 2026-01-28 08:31:04 -06:00
Shadow e701d7b713 feat: add skill badges table 2026-01-28 01:11:46 -06:00
Shadow 79bd1f1d6c fix: query highlighted skills by batch index 2026-01-28 00:34:33 -06:00
Shadow ac51cc0236 fix: rely on highlighted badge 2026-01-27 23:27:36 -06:00
Shadow b9c23dc00e feat: add reports dashboard for moderation 2026-01-27 22:27:20 -06:00
Shadow f4e96995bc fix: allow clawdis parsed metadata 2026-01-27 21:39:23 -06:00
Shadow 33165db598 feat: unify skill routes with owner slugs 2026-01-27 21:28:46 -06:00
Shadow 4c10e4847b feat: add moderation management and backfill 2026-01-27 21:11:37 -06:00
Shadow 8d68b55333 chore: rename molthub branding 2026-01-27 18:25:02 -06:00
Jamie Turner 251de1f540 Performance optimizations for /skills page 2026-01-27 15:18:21 -06:00
Shadow 123f60fa93 Revert "Performance optimizations for /skills page" temporarily until we can deploy
This reverts commit 643faf71c8.
2026-01-27 12:20:15 -06:00
Jamie Turner 643faf71c8 Performance optimizations for /skills page 2026-01-27 12:09:57 -06:00
Aaron NgandPeter Steinberger a2c46fbb5d Search Fixes (#30)
* more search fixes

* update tests

* comments

* fix: tune search filters and limits (#30) (thanks @aaronn)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-01-25 00:08:29 +00:00
Peter Steinberger f51e0a087d test: fix lockfile mock version 2026-01-24 22:56:39 +00:00
emilianoandPeter Steinberger d9108b0948 feat: show published skills on user profile (#20)
* fix: resolve typecheck and lint errors

* fix: stabilize publish paths and token types

* feat: show published skills on user profile

* fix: document profile published skills (#20) (thanks @njoylab)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-01-24 22:55:18 +00:00
Peter Steinberger d7a017e1c3 fix: add update lookup test (#22) (thanks @daveonkels) 2026-01-24 22:30:09 +00:00
Dave OnkelsandClaude Opus 4.5 fffdf82540 fix: use path instead of url for skill metadata API call (#22)
The `cmdUpdate` function was passing a relative path to `apiRequest`
using the `url` property, but `url` expects a full URL. When `url` is
provided, it's used as-is without combining with the registry base URL.

This caused "Failed to parse URL from /api/v1/skills/<slug>" errors
when updating skills that don't have a local fingerprint match.

Changed to use `path` property which correctly combines with the
registry base URL via `new URL(args.path, registry)`.

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 22:28:54 +00:00
Ahmed Fuad MireClaude Opus 4.5vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>Ahmed
a16e624766 fix: relax search token matching to require at least one match (#27)
* fix: relax search token matching to require at least one match

The search was requiring ALL query tokens to exist in the skill's
displayName, slug, or summary. This was too strict and caused valid
results to be filtered out. For example, searching "HTTP API client"
would fail to match skills about "HTTP API" that didn't mention "client".

Changed from `.every()` to `.some()` so at least one token must match,
allowing the vector similarity to determine relevance for the rest.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: update matchesExactTokens to require prefix matching for query tokens

* more inclusive token check

* Update convex/lib/searchText.ts

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>

---------

Co-authored-by: Ahmed <ahmed.mire@kaluza.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
2026-01-24 21:23:03 +00:00
Peter Steinberger decce1d35c fix: skip missing skills in search hydration (#28) (thanks @aaronn) 2026-01-24 21:11:46 +00:00
Aaron Ng 468832af3f fix search (#28) 2026-01-24 21:11:06 +00:00
Shadow 54c793a660 fix: handle search embedding errors 2026-01-23 15:48:53 -06:00
Shadow 5d9a89a885 fix search 2026-01-23 15:24:26 -06:00
Peter Steinberger 31e9a57678 feat: add installs/trending sorts 2026-01-19 07:06:46 +00:00
Peter Steinberger 7680cc4ce8 feat: add idempotent star endpoints 2026-01-19 03:09:27 +00:00
Peter Steinberger de56255b95 fix: normalize monaco surface color 2026-01-19 02:21:51 +00:00
Peter Steinberger eaaa5e4423 Merge pull request #12 from NACC96/fix/search-mode-navigation
fix: search mode navigation and state management
2026-01-18 23:54:17 +00:00
Peter Steinberger 49e9c3c071 fix: stabilize search mode routing (#12) (thanks @NACC96) 2026-01-18 23:53:42 +00:00
NACC96 ef96fbee84 fix: auto-focus search input when search mode activates 2026-01-18 23:51:49 +00:00
NACC96 9fc032216e fix: update Header search links to use URL params instead of /search redirect 2026-01-18 23:51:49 +00:00
NACC96 236058b1b5 fix: preserve search flag in OnlyCrabsHome URL sync 2026-01-18 23:51:49 +00:00
NACC96 595bd2cf05 fix: search mode navigation and state management
- Fix "Explore search" button causing page refresh by using URL params
- Enable /search URL deep linking via beforeLoad redirect
- Fix logo click not closing search mode by properly syncing state with URL
2026-01-18 23:51:49 +00:00
Peter Steinberger 918b5528df chore: format peer check script 2026-01-18 23:42:04 +00:00
Peter Steinberger 2f6f11af3f ci: add peer dependency check 2026-01-18 23:40:55 +00:00
Peter Steinberger c90f92dbc7 fix: align auth core with convex auth 2026-01-18 23:33:51 +00:00
Peter Steinberger 8db8c89206 chore: update deps and adjust vite convex resolution 2026-01-18 23:29:59 +00:00
Peter Steinberger 00c31dcdd4 fix: derive auth from user query 2026-01-18 23:13:29 +00:00
Peter Steinberger 47646937b6 fix: dedupe convex auth modules 2026-01-18 22:33:09 +00:00
Shadow 10d250ea32 fix: keep ConvexAuthProvider during SSR 2026-01-18 14:43:24 -06:00
Peter Steinberger e7fa7afdf7 chore: bump clawdhub to 0.2.1 2026-01-18 16:28:40 +00:00
Peter Steinberger aa97727be8 fix: harden explore limit + tests/docs (#14) (thanks @jdrhyne) 2026-01-18 16:26:28 +00:00
Peter Steinberger d375496c17 Merge pull request #14 from jdrhyne/feat/explore-command
feat(cli): add explore command to browse latest updated skills
2026-01-18 16:25:48 +00:00
Peter Steinberger 9a912ee5eb chore: update dependencies 2026-01-18 14:10:09 +00:00
Peter Steinberger 11b257a062 fix: harden search and cli http 2026-01-18 14:04:35 +00:00
Peter Steinberger 02e509404a chore: update convex api types 2026-01-18 09:12:52 +00:00
Peter Steinberger 2cf6182991 fix: tighten search matching 2026-01-18 09:11:16 +00:00
Jonathan Rhyne e108789ab1 feat(cli): add explore command to browse latest updated skills
Adds a new `clawdhub explore` command that fetches the most recently
updated skills from the registry, sorted by updatedAt descending.

Usage:
  clawdhub explore           # Show latest 25 skills
  clawdhub explore --limit 10

Output includes slug, version, relative time since update, and summary.

The API endpoint already exists and returns skills sorted by updatedAt,
this just exposes it via the CLI.
2026-01-17 23:07:54 -05:00
Peter Steinberger 18bfea5035 fix: enable explore search button 2026-01-17 21:54:29 +00:00
Peter Steinberger 5f93ddb390 fix: rename ClawdBot to Clawdbot 2026-01-16 01:14:24 +00:00
Peter Steinberger 2b552c4803 feat: default workdir from clawdbot config 2026-01-13 06:04:38 +00:00
Peter Steinberger ece0b830fb test: raise branch coverage 2026-01-13 01:25:06 +00:00
Peter Steinberger 57afea9de5 chore: ignore test-results in biome 2026-01-13 01:24:56 +00:00
Peter Steinberger 1e788b2e0c test: add playwright smoke suite 2026-01-13 00:46:13 +00:00
Peter Steinberger f025721261 fix: prevent skills index crash 2026-01-13 00:40:04 +00:00
Shadow c2cc61df24 feat: add skills lazy loading 2026-01-12 15:52:52 -06:00
Shadow 7a8a3f75ce fix: paginate skills index 2026-01-12 13:36:28 -06:00
Peter Steinberger ac0bdf0375 fix: hide onlycrabs branding 2026-01-11 05:20:20 +01:00
Peter Steinberger 6c7dd2ec6d fix: hide onlycrabs link and code pill 2026-01-11 05:15:43 +01:00
Peter Steinberger c72e3007a1 test: guard skills list query limit 2026-01-10 22:04:42 +01:00
Peter Steinberger b66d28a184 fix: lower skills list query limit 2026-01-10 22:01:06 +01:00
Shadow dddee828af Change ClawdBot link to new URL 2026-01-10 14:01:27 -06:00
Peter Steinberger 106cb1896a Merge pull request #1 from clawdbot/nix-plugin-metadata
Add nix-clawdbot plugin pointers to skill metadata
2026-01-10 19:33:11 +00:00
Peter Steinberger e29ec88afd fix: restore backups + fork lineage (#1) (thanks @joshp123) 2026-01-10 20:32:47 +01:00
Josh Palmer 0eb3047ca6 feat: add nix plugin bundles
- include nix plugin metadata, config requirements, and CLI help
- add config examples and format bundle code blocks
- refresh bundle UI styling and layout
2026-01-10 20:27:22 +01:00
Peter Steinberger b34c7261bd feat: add v1 public api 2026-01-10 20:25:50 +01:00
DB Hurley e0d553602a feat: refresh skill detail layout and dashboard
- add dashboard with skill management and upload prefill
- redesign skill detail layout with full-width panels
- refactor modules and format dashboard/upload routes
2026-01-10 20:23:01 +01:00
Peter Steinberger f80fa90e01 fix(seed): harden SoulHub auto-seed 2026-01-10 20:18:19 +01:00
Josh Palmer 0cc0bdcd50 feat: SoulHub registry + auto-seed
SoulHub SOUL.md registry (souls table, versions, search, OG) + first-run auto-seed; fixes seed concurrency and GitHub backup owner handle.
2026-01-10 18:25:11 +00:00
Peter Steinberger cc0027a094 test(og): update OG layout version 2026-01-09 19:16:39 +01:00
Peter Steinberger dddbd3a78e fix(og): prevent OG title clipping 2026-01-09 19:13:19 +01:00
Peter Steinberger f350442002 feat: import skills from public GitHub 2026-01-09 09:10:49 +01:00
Peter Steinberger 826c60f4da test(cli): expand clawdbot sync coverage 2026-01-09 02:34:09 +01:00
Peter Steinberger a679c3a999 docs: note clawdbot sync roots 2026-01-09 02:05:34 +01:00
Peter Steinberger d5d8e6ae5b feat(cli): auto-scan clawdbot skill roots 2026-01-09 01:57:56 +01:00
Peter Steinberger f0772e7215 test: cover OG text clamping 2026-01-08 23:07:10 +01:00
Peter Steinberger 770bb3aeb8 fix: clamp OG description width 2026-01-08 23:02:33 +01:00
Peter Steinberger 6811691055 fix: prevent OG text bleed 2026-01-08 22:58:53 +01:00
Peter Steinberger 26b46d9f6e docs: note OG image runtime fix 2026-01-08 06:15:09 +01:00
Peter Steinberger d145c186a7 fix: resolve OG api base on all runtimes 2026-01-08 06:12:01 +01:00
Peter Steinberger 57af81d054 refactor: modularize skill OG images 2026-01-08 06:07:36 +01:00
Peter Steinberger 0131229843 fix: embed fonts in OG images 2026-01-08 05:54:49 +01:00
Peter Steinberger d7650583dc feat: dynamic skill OG images 2026-01-08 05:47:27 +01:00
Peter Steinberger cf2ad58e86 chore: remove docs page link 2026-01-08 04:20:25 +01:00
Peter Steinberger 153c3f5b9e style: soften markdown block styling 2026-01-07 22:41:57 +01:00
Peter Steinberger 243ca9ca2b feat: link docs and clarify cli usage 2026-01-07 21:11:04 +01:00
Peter Steinberger 8216c73c9b test: stabilize upload route mocks 2026-01-07 20:18:32 +01:00
Peter Steinberger 860902a574 fix: harden upload utils 2026-01-07 20:14:18 +01:00
Peter Steinberger 18af63b630 fix: silence lint/build warnings 2026-01-07 20:08:13 +01:00
Peter Steinberger 96ac7567ba chore: prepare 0.1.0 release notes 2026-01-07 20:07:06 +01:00
Peter Steinberger b55e266457 fix: harden GitHub backups 2026-01-07 18:48:09 +00:00
Shadowandvercel[bot] <35613825+vercel[bot]@users.noreply.github.com> 2492a52ca3 Update skillPublish.ts
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
2026-01-07 18:48:09 +00:00
Shadow 9b6a4c7b6f chore: format lint fixes 2026-01-07 18:48:09 +00:00
Shadow 232d06debd fix: preserve previous commit in github history 2026-01-07 18:48:09 +00:00
Shadow e6ba699b18 feat: back up skills to github 2026-01-07 18:48:09 +00:00
Peter Steinberger 7444c22d57 chore: bump cli version to 0.1.0 2026-01-07 19:13:00 +01:00
Peter Steinberger cf26cd143a fix: address deploy typecheck 2026-01-07 18:49:49 +01:00
Peter Steinberger 82b426cc8e fix: harden multipart publish parsing 2026-01-07 18:42:44 +01:00
Peter Steinberger ba7e82ba02 feat: add v1 public api 2026-01-07 18:28:51 +01:00
Peter Steinberger 9f83114dee chore: add docs:list helper 2026-01-07 17:58:00 +01:00
Peter Steinberger a98b51a2a1 test: fix upload route mocks 2026-01-07 17:53:01 +01:00
Peter Steinberger 4d6987097d style: format dashboard and upload routes 2026-01-07 17:52:57 +01:00
Peter Steinberger 1bf3bcf152 docs: add Mintlify-ready docs set 2026-01-07 17:52:55 +01:00
Peter Steinberger 974338fb97 docs: thank dbhurley for dashboard 2026-01-07 10:08:20 +01:00
Peter Steinberger 927cd4d425 Merge pull request #2 from dbhurley/feat/user-dashboard
feat: Add user dashboard with skill management
2026-01-07 09:07:42 +00:00
Peter Steinberger bfa59cabf5 merge main into feat/user-dashboard 2026-01-07 10:06:14 +01:00
Peter Steinberger 2a28f75fc6 docs: thank dbhurley in changelog 2026-01-07 09:50:31 +01:00
Peter Steinberger fff263ee68 Merge pull request #4 from dbhurley/fix/token-color-error
fix: handle shorthand hex colors in SkillDiffCard
2026-01-07 08:50:02 +00:00
Peter Steinberger af40c53bc1 style: pad diff file list 2026-01-07 07:52:37 +01:00
Peter Steinberger 74a0ca9bf6 style: remove diff hover lift 2026-01-07 07:42:10 +01:00
Peter Steinberger 3f85eccd1a refactor: make skill detail full width 2026-01-07 07:17:14 +01:00
Peter Steinberger 9513bd9f44 style: remove divider before versions 2026-01-07 07:04:55 +01:00
Peter Steinberger 27958bbb69 fix: type skill origin in tests 2026-01-07 06:18:41 +01:00
Peter Steinberger 7a62bec162 style: format skill tests 2026-01-07 06:16:21 +01:00
Peter Steinberger e895840411 test: cover skill utilities 2026-01-07 06:13:42 +01:00
Peter Steinberger 9c574a7709 feat: redesign skill detail layout 2026-01-07 05:56:12 +01:00
Peter Steinberger 6af5cf11ec docs: note Convex run --env-file auth gotcha 2026-01-07 05:50:42 +01:00
Peter Steinberger 237a965673 feat: dedupe skills via canonical forks 2026-01-07 05:20:19 +01:00
DB Hurley 219e3257ec fix: normalize hex colors in SkillDiffCard to prevent Monaco crash 2026-01-06 21:44:36 -05:00
Peter Steinberger 5e1598954f feat: add Vercel Analytics 2026-01-07 03:23:12 +01:00
Peter Steinberger 0a059e337f style: align version rows 2026-01-07 02:21:13 +01:00
Peter Steinberger 3d24969f1e style: refine comment form sizing 2026-01-07 02:20:00 +01:00
Peter Steinberger cc530d8a69 test: raise coverage for diffing 2026-01-07 01:46:03 +01:00
Peter Steinberger 6049863b8f feat: add skill diff viewer 2026-01-06 23:29:03 +01:00
Peter Steinberger 76ad34eb89 chore: release 0.0.5 2026-01-06 17:29:28 +01:00
Peter Steinberger 6be9f93eeb refactor: split large modules 2026-01-06 17:24:17 +01:00
Peter Steinberger 1f0d02019b fix: yaml frontmatter + summary backfill 2026-01-06 05:19:00 +01:00
Peter Steinberger 7ef25343ec docs: update changelog 2026-01-06 04:20:35 +01:00
Peter Steinberger 5365827416 fix: ignore plural skills.md markers 2026-01-06 04:03:54 +01:00
Peter Steinberger baee5aee2e fix: show wordmark on mobile 2026-01-06 03:07:36 +01:00
Peter Steinberger 1eebbd4c2d chore: log webhook sends 2026-01-06 02:53:08 +01:00
Peter Steinberger f9ea6ab355 feat: improve Open Graph preview 2026-01-06 02:46:47 +01:00
Peter Steinberger dfe6150474 feat: move theme picker into mobile menu 2026-01-06 02:03:43 +01:00
Peter Steinberger 860b3a369c feat: track installs via sync telemetry 2026-01-06 01:38:49 +01:00
Peter Steinberger 7eb8286f4c feat: add discord webhooks 2026-01-06 00:53:58 +01:00
Peter Steinberger 721dcea079 fix(web): improve mobile responsiveness 2026-01-06 00:26:32 +01:00
Peter Steinberger d067f8f5ed feat: auto-generate changelogs 2026-01-06 00:16:36 +01:00
Peter Steinberger 9a216acb4c fix: add skill og cards 2026-01-05 23:43:20 +01:00
DB Hurley d4b947af47 feat: Add user dashboard with skill management
- Add /dashboard route showing user's published skills
- Add 'Dashboard' link to user dropdown menu in header
- Skills display name, slug, description, stats (downloads, stars, versions)
- 'New Version' button links to upload with pre-populated slug
- Upload route accepts ?updateSlug param to pre-fill form for updates
- Auto-bumps version number when updating existing skill
- Responsive design for mobile
- Empty state with call-to-action for new users
2026-01-05 17:34:54 -05:00
Peter Steinberger bf1a2d6966 chore: release 0.0.4 2026-01-05 23:28:14 +01:00
Peter Steinberger 17ecc9d648 fix: reduce embedding payload size 2026-01-05 23:13:59 +01:00
Peter Steinberger 9f1e003401 fix: prefer discovered registry 2026-01-05 22:52:41 +01:00
Peter Steinberger 709f2f30de fix: cap embedding input size 2026-01-05 22:03:31 +01:00
Peter Steinberger 79c3baec13 docs(web): fix ClawdBot casing 2026-01-05 21:53:33 +01:00
Peter Steinberger ab1e3f0185 docs(web): adjust footer copy 2026-01-05 21:51:41 +01:00
Peter Steinberger 1d030b9c78 fix(cli): make bin executable 2026-01-05 21:50:30 +01:00
Peter Steinberger 9f4d111892 style(web): soften footer 2026-01-05 18:48:09 +01:00
Peter Steinberger bebee5e124 feat(web): add global footer + see-all 2026-01-05 18:42:08 +01:00
Peter Steinberger e8653dc793 fix(web): smooth hero search transition 2026-01-05 06:37:31 +01:00
Peter Steinberger d8dd6542ca docs: rewrite README 2026-01-05 05:51:16 +01:00
Peter Steinberger 2303457da6 test(web): cover skill detail loading 2026-01-05 04:47:05 +01:00
Peter Steinberger d7862a8d35 docs(changelog): note folder upload unwrap 2026-01-05 04:40:09 +01:00
Peter Steinberger c91b7069d8 fix(web): accept folder uploads with SKILL.md 2026-01-05 04:39:52 +01:00
Peter Steinberger 73ae10b023 feat(web): canonical skill urls 2026-01-05 04:22:37 +01:00
Peter Steinberger d32fb65207 feat(web): admin highlight toggle 2026-01-05 04:14:05 +01:00
Peter Steinberger d6efc003e1 fix(web): user profile avatar + loading 2026-01-05 04:10:59 +01:00
Peter Steinberger e767392fd6 fix(web): show loading state on skills list 2026-01-05 04:08:22 +01:00
Peter Steinberger 47603e4ff3 feat(web): add skills list + sorting 2026-01-05 04:06:52 +01:00
Peter Steinberger 8ece7713ed fix(web): avoid skill not found flash 2026-01-05 03:56:34 +01:00
Peter Steinberger 3c31b460ff fix(web): keep search request id simple 2026-01-05 03:42:46 +01:00
Peter Steinberger 554d5d4cd0 feat: unify homepage search 2026-01-05 03:37:16 +01:00
Peter Steinberger 236c670015 fix(cli): unbox sync output 2026-01-05 01:59:39 +01:00
221 changed files with 28525 additions and 2547 deletions
+3
View File
@@ -1,6 +1,9 @@
# Frontend
VITE_CONVEX_URL=
VITE_CONVEX_SITE_URL=
VITE_SOULHUB_SITE_URL=
VITE_SOULHUB_HOST=
VITE_SITE_MODE=
SITE_URL=http://localhost:3000
CONVEX_SITE_URL=
+3 -2
View File
@@ -15,10 +15,12 @@ jobs:
- uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.5
bun-version: 1.3.6
- name: Install
run: bun install --frozen-lockfile
- name: Peer deps
run: bun run check:peers
- name: Lint
run: bun run lint
@@ -36,4 +38,3 @@ jobs:
- name: Build
run: bun run build
+6
View File
@@ -1,5 +1,8 @@
node_modules
.DS_Store
.bun-build
*.bun-build
bin/docs-list
dist
dist-ssr
!packages/schema/dist
@@ -18,3 +21,6 @@ todos.json
.vscode
.env*.local
coverage
playwright-report
test-results
.playwright
+5
View File
@@ -38,3 +38,8 @@
- Local env: `.env.local` (never commit secrets).
- Convex env holds JWT keys; Vercel only needs `VITE_CONVEX_URL` + `VITE_CONVEX_SITE_URL`.
- OAuth: GitHub OAuth App credentials required for login.
## Convex Ops (Gotchas)
- New Convex functions must be pushed before `convex run`: use `bunx convex dev --once` (dev) or `bunx convex deploy` (prod).
- For non-interactive prod deploys, use `bunx convex deploy -y` to skip confirmation.
- If `bunx convex run --env-file .env.local ...` returns `401 MissingAccessToken` despite `bunx convex login`, workaround: omit `--env-file` and use `--deployment-name <name>` / `--prod`.
+113 -2
View File
@@ -1,5 +1,115 @@
# Changelog
## Unreleased
### Added
### Changed
### Fixed
## 0.4.0 - 2026-01-30
### Added
- Web: show published skills on user profiles (thanks @njoylab, #20).
- CLI: include OpenClaw + Moltbot fallback skill roots for sync scans.
- CLI: support OpenClaw configuration files (`OPENCLAW_CONFIG_PATH` / `OPENCLAW_STATE_DIR`).
### Changed
- Brand: rebrand to OpenClaw and publish CLI as `clawhub` (legacy `clawdhub` supported).
- Domain: default site/registry now `https://clawhub.ai`; `.well-known/clawhub.json` preferred.
- Theme: persist theme under `clawhub-theme` (legacy key still read).
### Fixed
- Registry: drop missing skills during search hydration (thanks @aaronn, #28).
- CLI: use path-based skill metadata lookup for updates (thanks @daveonkels, #22).
- Search: keep highlighted-only filtering and clamp vector candidates to Convex limits (thanks @aaronn, #30).
## 0.3.0 - 2026-01-19
### Added
- CLI: add `explore` command for latest updates, with limit clamping + tests/docs (thanks @jdrhyne, #14).
- CLI: `explore --json` output + new sorts (`installs`, `installsAllTime`, `trending`) and limit up to 200.
- API: `/api/v1/skills` supports installs + trending sorts (7-day installs).
- API: idempotent `POST/DELETE /api/v1/stars/{slug}` endpoints.
- Registry: trending leaderboard + daily stats backfill for installs-based sorts.
### Fixed
- Web: keep search mode navigation and state in sync (thanks @NACC96, #12).
## 0.2.0 - 2026-01-13
### Added
- Web: dynamic OG image cards for skills (name, description, version).
- CLI: auto-scan Clawdbot skill roots (per-agent workspaces, shared skills, extraDirs).
- Web: import skills from public GitHub URLs (auto-detect `SKILL.md`, smart file selection, provenance).
- Web/API: SoulHub (SOUL.md registry) with v1 endpoints and first-run auto-seed.
### Fixed
- Web: stabilize skill OG image generation on server runtimes.
- Web: prevent skill OG text overflow outside the card.
- Registry: make SoulHub auto-seed idempotent and non-user-owned.
- Registry: keep GitHub backup state + publish backups intact (thanks @joshp123, #1).
- CLI/Registry: restore fork lineage on sync + clamp bulk list queries (thanks @joshp123, #1).
- CLI: default workdir falls back to Clawdbot workspace (override with `--workdir` / `CLAWHUB_WORKDIR`).
## 0.0.6 - 2026-01-07
### Added
- API: v1 public REST endpoints with rate limits, raw file fetch, and OpenAPI spec.
- Docs: `docs/api.md` and `DEPRECATIONS.md` for the v1 cutover plan.
### Changed
- CLI: publish now uses single multipart `POST /api/v1/skills`.
- Registry: legacy `/api/*` + `/api/cli/*` marked for deprecation (kept for now).
## 0.0.5 - 2026-01-06
### Added
- Telemetry: track installs via `clawhub sync` (logged-in only), per root, with 120-day staleness.
- Skills: show current + all-time installs; sort by installs.
- Profile: private "Installed" tab with JSON export + delete telemetry controls.
- Docs: add `docs/telemetry.md` (what we track + how to opt out).
- Web: custom Open Graph image (`/og.png`) + richer OG/Twitter tags.
- Web: dashboard for managing your published skills (thanks @dbhurley!).
### Changed
- CLI: telemetry opt-out via `CLAWHUB_DISABLE_TELEMETRY=1`.
- Web: move theme picker into mobile menu.
### Fixed
- Web: handle shorthand hex colors in diff theme (thanks @dbhurley!).
## 0.0.5 - 2026-01-06
### Added
- Maintenance: admin backfill to re-parse `SKILL.md` and repair stored summaries/parsed metadata.
### Fixed
- CLI sync: ignore plural `skills.md` docs files when scanning for skills.
- Registry: parse YAML frontmatter (incl multiline `description`) and accept YAML `metadata` objects.
## 0.0.4 - 2026-01-05
### Added
- Web: `/skills` list view with sorting (newest/downloads/stars/name) + quick filter.
- Web: admin/moderator highlight toggle on skill detail.
- Web: canonical skill URLs as `/<owner>/<slug>` (legacy `/skills/<slug>` redirects).
- Web: upload auto-generates a changelog via OpenAI when left blank (marked as auto-generated).
### Fixed
- Web: skill detail shows a loading state instead of flashing "Skill not found".
- Web: user profile shows avatar + loading state (no "User not found" flash).
- Web: improved mobile responsiveness (nav menu, skill detail layout, install command overflow).
- Web: upload now unwraps folder picks so `SKILL.md` can be at the bundle root.
- Registry: cap embedding payload size to avoid model context errors.
- CLI: ignore legacy `auth.clawdhub.com` registry and prefer site discovery.
### Changed
- Web: homepage search now expands into full search mode with live results + highlighted toggle.
- CLI: sync no longer prompts for changelog; registry auto-generates when blank.
## 0.0.3 - 2026-01-04
### Added
@@ -8,12 +118,13 @@
### Changed
- CLI sync: default `--concurrency` is now 4 (was 8).
- CLI sync: replace boxed notes with plain output for long lists.
### Fixed
- CLI sync: wrap note output to avoid terminal overflow; cap list lengths.
- CLI sync: label fallback scans as fallback locations.
- CLI package: bundle schema internally (no external `clawdhub-schema` publish).
- Repo: mark `clawdhub-schema` as private to prevent publishing.
- CLI package: bundle schema internally (no external `clawhub-schema` publish).
- Repo: mark `clawhub-schema` as private to prevent publishing.
## 0.0.2 - 2026-01-04
+7
View File
@@ -0,0 +1,7 @@
# Deprecations
## Legacy /api routes (pre-v1)
- Deprecated: 2026-01-07
- TODO: remove legacy `/api/*` and `/api/cli/*` routes after clients migrate to `/api/v1`.
- Legacy handlers live in `convex/http.ts` and `convex/httpApi.ts`.
+115 -10
View File
@@ -1,38 +1,143 @@
# ClawdHub
# OpenClaw
Minimal skill registry powered by TanStack Start + Convex.
<p align="center">
<a href="https://github.com/openclaw/clawhub/actions/workflows/ci.yml?branch=main"><img src="https://img.shields.io/github/actions/workflow/status/openclaw/clawhub/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a>
<a href="https://discord.gg/clawd"><img src="https://img.shields.io/discord/1456350064065904867?label=Discord&logo=discord&logoColor=white&color=5865F2&style=for-the-badge" alt="Discord"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
</p>
## Quick start
OpenClaw is the **public skill registry for Clawdbot**: publish, version, and search text-based agent skills (a `SKILL.md` plus supporting files).
Its designed for fast browsing + a CLI-friendly API, with moderation hooks and vector search.
onlycrabs.ai is the **SOUL.md registry**: publish and share system lore the same way you publish skills.
Live: `https://clawhub.ai`
onlycrabs.ai: `https://onlycrabs.ai`
## What you can do
- Browse skills + render their `SKILL.md`.
- Publish new skill versions with changelogs + tags (including `latest`).
- Browse souls + render their `SOUL.md`.
- Publish new soul versions with changelogs + tags.
- Search via embeddings (vector index) instead of brittle keywords.
- Star + comment; admins/mods can curate and approve skills.
## onlycrabs.ai (SOUL.md registry)
- Entry point is host-based: `onlycrabs.ai`.
- On the onlycrabs.ai host, the home page and nav default to souls.
- On OpenClaw, souls live under `/souls`.
- Soul bundles only accept `SOUL.md` for now (no extra files).
## How it works (high level)
- Web app: TanStack Start (React, Vite/Nitro).
- Backend: Convex (DB + file storage + HTTP actions) + Convex Auth (GitHub OAuth).
- Search: OpenAI embeddings (`text-embedding-3-small`) + Convex vector search.
- API schema + routes: `packages/schema` (`clawhub-schema`).
## Telemetry
OpenClaw tracks minimal **install telemetry** (to compute install counts) when you run `clawhub sync` while logged in.
Disable via:
```bash
export CLAWHUB_DISABLE_TELEMETRY=1
```
Details: `docs/telemetry.md`.
## Repo layout
- `src/` — TanStack Start app (routes, components, styles).
- `convex/` — schema + queries/mutations/actions + HTTP API routes.
- `packages/schema/` — shared API types/routes for the CLI and app.
- `docs/spec.md` — product + implementation spec (good first read).
## Local dev
Prereqs: Bun + Convex CLI.
```bash
bun install
cp .env.local.example .env.local
bun --bun run dev
```
In another terminal:
# terminal A: web app
bun run dev
```bash
# terminal B: Convex dev deployment
bunx convex dev
```
## Convex Auth setup
## Auth (GitHub OAuth) setup
Create a GitHub OAuth App, set `AUTH_GITHUB_ID` / `AUTH_GITHUB_SECRET`, then:
```bash
bunx auth --deployment-name <deployment> --web-server-url http://localhost:3000
```
This writes `JWT_PRIVATE_KEY` + `JWKS` to the deployment and prints the values for local `.env.local`.
This writes `JWT_PRIVATE_KEY` + `JWKS` to the deployment and prints values for your local `.env.local`.
## Environment
- `VITE_CONVEX_URL`: Convex deployment URL (`https://<deployment>.convex.cloud`).
- `VITE_CONVEX_SITE_URL`: Convex site URL (`https://<deployment>.convex.site`).
- `VITE_SOULHUB_SITE_URL`: onlycrabs.ai site URL (`https://onlycrabs.ai`).
- `VITE_SOULHUB_HOST`: onlycrabs.ai host match (`onlycrabs.ai`).
- `VITE_SITE_MODE`: Optional override (`skills` or `souls`) for SSR builds.
- `CONVEX_SITE_URL`: same as `VITE_CONVEX_SITE_URL` (auth + cookies).
- `SITE_URL`: App URL (local: `http://localhost:3000`).
- `AUTH_GITHUB_ID` / `AUTH_GITHUB_SECRET`: GitHub OAuth App.
- `JWT_PRIVATE_KEY` / `JWKS`: Convex Auth keys.
- `OPENAI_API_KEY`: embeddings.
- `OPENAI_API_KEY`: embeddings for search + indexing.
## Nix plugins (nixmode skills)
OpenClaw can store a nix-clawdbot plugin pointer in SKILL frontmatter so the registry knows which
Nix package bundle to install. A nix plugin is different from a regular skill pack: it bundles the
skill pack, the CLI binary, and its config flags/requirements together.
Add this to `SKILL.md`:
```yaml
---
name: peekaboo
description: Capture and automate macOS UI with the Peekaboo CLI.
metadata: {"clawdbot":{"nix":{"plugin":"github:clawdbot/nix-steipete-tools?dir=tools/peekaboo","systems":["aarch64-darwin"]}}}
---
```
Install via nix-clawdbot:
```nix
programs.clawdbot.plugins = [
{ source = "github:clawdbot/nix-steipete-tools?dir=tools/peekaboo"; }
];
```
You can also declare config requirements + an example snippet:
```yaml
---
name: padel
description: Check padel court availability and manage bookings via Playtomic.
metadata: {"clawdbot":{"config":{"requiredEnv":["PADEL_AUTH_FILE"],"stateDirs":[".config/padel"],"example":"config = { env = { PADEL_AUTH_FILE = \\\"/run/agenix/padel-auth\\\"; }; };"}}}
---
```
To show CLI help (recommended for nix plugins), include the `cli --help` output:
```yaml
---
name: padel
description: Check padel court availability and manage bookings via Playtomic.
metadata: {"clawdbot":{"cliHelp":"padel --help\\nUsage: padel [command]\\n"}}
---
```
`metadata.clawdbot` is preferred, but `metadata.clawdis` is accepted as an alias for compatibility.
## Scripts
+5 -2
View File
@@ -1,5 +1,5 @@
{
"$schema": "https://biomejs.dev/schemas/2.3.11/schema.json",
"$schema": "https://biomejs.dev/schemas/2.3.13/schema.json",
"files": {
"includes": [
"**",
@@ -10,9 +10,12 @@
"!**/.output",
"!**/coverage",
"!**/convex/_generated",
"!**/test-results",
"!**/src/routeTree.gen.ts",
"!**/.tanstack",
"!**/public"
"!**/public",
"!**/.devenv",
"!**/.devenv"
]
},
"assist": { "actions": { "source": { "organizeImports": "on" } } },
+309 -239
View File
File diff suppressed because it is too large Load Diff
Executable
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bun
import { existsSync } from 'node:fs'
import { stat } from 'node:fs/promises'
import { fileURLToPath } from 'node:url'
const distCliUrl = new URL('./packages/clawdhub/dist/cli.js', import.meta.url)
const distCliPath = fileURLToPath(distCliUrl)
const srcRootPath = fileURLToPath(new URL('./packages/clawdhub/src/', import.meta.url))
const shouldBuild = await (async () => {
if (!existsSync(distCliPath)) return true
try {
const dist = await stat(distCliPath)
const latestSrcMtime = await getLatestMtime(srcRootPath)
return latestSrcMtime > dist.mtimeMs
} catch {
return true
}
})()
if (shouldBuild) {
const proc = Bun.spawn(['bunx', 'tsc', '-p', 'packages/clawdhub/tsconfig.json'], {
stdin: 'inherit',
stdout: 'inherit',
stderr: 'inherit',
})
const code = await proc.exited
if (code !== 0) process.exit(code)
}
await import(distCliUrl.href)
async function getLatestMtime(root: string) {
let latest = 0
const glob = new Bun.Glob('**/*.ts')
for await (const rel of glob.scan({ cwd: root, onlyFiles: true })) {
const path = `${root}${root.endsWith('/') ? '' : '/'}${rel}`
try {
const entry = await stat(path)
latest = Math.max(latest, entry.mtimeMs)
} catch {
// ignore
}
}
return latest
}
+74
View File
@@ -10,20 +10,57 @@
import type * as auth from "../auth.js";
import type * as comments from "../comments.js";
import type * as crons from "../crons.js";
import type * as devSeed from "../devSeed.js";
import type * as devSeedExtra from "../devSeedExtra.js";
import type * as downloads from "../downloads.js";
import type * as githubBackups from "../githubBackups.js";
import type * as githubBackupsNode from "../githubBackupsNode.js";
import type * as githubImport from "../githubImport.js";
import type * as githubSoulBackups from "../githubSoulBackups.js";
import type * as githubSoulBackupsNode from "../githubSoulBackupsNode.js";
import type * as http from "../http.js";
import type * as httpApi from "../httpApi.js";
import type * as httpApiV1 from "../httpApiV1.js";
import type * as leaderboards from "../leaderboards.js";
import type * as lib_access from "../lib/access.js";
import type * as lib_apiTokenAuth from "../lib/apiTokenAuth.js";
import type * as lib_badges from "../lib/badges.js";
import type * as lib_changelog from "../lib/changelog.js";
import type * as lib_embeddings from "../lib/embeddings.js";
import type * as lib_githubBackup from "../lib/githubBackup.js";
import type * as lib_githubImport from "../lib/githubImport.js";
import type * as lib_githubSoulBackup from "../lib/githubSoulBackup.js";
import type * as lib_leaderboards from "../lib/leaderboards.js";
import type * as lib_moderation from "../lib/moderation.js";
import type * as lib_public from "../lib/public.js";
import type * as lib_searchText from "../lib/searchText.js";
import type * as lib_skillBackfill from "../lib/skillBackfill.js";
import type * as lib_skillPublish from "../lib/skillPublish.js";
import type * as lib_skillStats from "../lib/skillStats.js";
import type * as lib_skills from "../lib/skills.js";
import type * as lib_soulChangelog from "../lib/soulChangelog.js";
import type * as lib_soulPublish from "../lib/soulPublish.js";
import type * as lib_tokens from "../lib/tokens.js";
import type * as lib_webhooks from "../lib/webhooks.js";
import type * as maintenance from "../maintenance.js";
import type * as rateLimits from "../rateLimits.js";
import type * as search from "../search.js";
import type * as seed from "../seed.js";
import type * as seedSouls from "../seedSouls.js";
import type * as skillStatEvents from "../skillStatEvents.js";
import type * as skills from "../skills.js";
import type * as soulComments from "../soulComments.js";
import type * as soulDownloads from "../soulDownloads.js";
import type * as soulStars from "../soulStars.js";
import type * as souls from "../souls.js";
import type * as stars from "../stars.js";
import type * as statsMaintenance from "../statsMaintenance.js";
import type * as telemetry from "../telemetry.js";
import type * as tokens from "../tokens.js";
import type * as uploads from "../uploads.js";
import type * as users from "../users.js";
import type * as webhooks from "../webhooks.js";
import type {
ApiFromModules,
@@ -34,20 +71,57 @@ import type {
declare const fullApi: ApiFromModules<{
auth: typeof auth;
comments: typeof comments;
crons: typeof crons;
devSeed: typeof devSeed;
devSeedExtra: typeof devSeedExtra;
downloads: typeof downloads;
githubBackups: typeof githubBackups;
githubBackupsNode: typeof githubBackupsNode;
githubImport: typeof githubImport;
githubSoulBackups: typeof githubSoulBackups;
githubSoulBackupsNode: typeof githubSoulBackupsNode;
http: typeof http;
httpApi: typeof httpApi;
httpApiV1: typeof httpApiV1;
leaderboards: typeof leaderboards;
"lib/access": typeof lib_access;
"lib/apiTokenAuth": typeof lib_apiTokenAuth;
"lib/badges": typeof lib_badges;
"lib/changelog": typeof lib_changelog;
"lib/embeddings": typeof lib_embeddings;
"lib/githubBackup": typeof lib_githubBackup;
"lib/githubImport": typeof lib_githubImport;
"lib/githubSoulBackup": typeof lib_githubSoulBackup;
"lib/leaderboards": typeof lib_leaderboards;
"lib/moderation": typeof lib_moderation;
"lib/public": typeof lib_public;
"lib/searchText": typeof lib_searchText;
"lib/skillBackfill": typeof lib_skillBackfill;
"lib/skillPublish": typeof lib_skillPublish;
"lib/skillStats": typeof lib_skillStats;
"lib/skills": typeof lib_skills;
"lib/soulChangelog": typeof lib_soulChangelog;
"lib/soulPublish": typeof lib_soulPublish;
"lib/tokens": typeof lib_tokens;
"lib/webhooks": typeof lib_webhooks;
maintenance: typeof maintenance;
rateLimits: typeof rateLimits;
search: typeof search;
seed: typeof seed;
seedSouls: typeof seedSouls;
skillStatEvents: typeof skillStatEvents;
skills: typeof skills;
soulComments: typeof soulComments;
soulDownloads: typeof soulDownloads;
soulStars: typeof soulStars;
souls: typeof souls;
stars: typeof stars;
statsMaintenance: typeof statsMaintenance;
telemetry: typeof telemetry;
tokens: typeof tokens;
uploads: typeof uploads;
users: typeof users;
webhooks: typeof webhooks;
}>;
/**
+5 -4
View File
@@ -1,7 +1,8 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { assertRole, requireUser } from './lib/access'
import { assertModerator, requireUser } from './lib/access'
import { type PublicUser, toPublicUser } from './lib/public'
export const listBySkill = query({
args: { skillId: v.id('skills'), limit: v.optional(v.number()) },
@@ -13,10 +14,10 @@ export const listBySkill = query({
.order('desc')
.take(limit)
const results: Array<{ comment: Doc<'comments'>; user: Doc<'users'> | null }> = []
const results: Array<{ comment: Doc<'comments'>; user: PublicUser | null }> = []
for (const comment of comments) {
if (comment.softDeletedAt) continue
const user = await ctx.db.get(comment.userId)
const user = toPublicUser(await ctx.db.get(comment.userId))
results.push({ comment, user })
}
return results
@@ -59,7 +60,7 @@ export const remove = mutation({
const isOwner = comment.userId === user._id
if (!isOwner) {
assertRole(user, ['admin', 'moderator'])
assertModerator(user)
}
await ctx.db.patch(comment._id, {
+34
View File
@@ -0,0 +1,34 @@
import { cronJobs } from 'convex/server'
import { internal } from './_generated/api'
const crons = cronJobs()
crons.interval(
'github-backup-sync',
{ minutes: 30 },
internal.githubBackupsNode.syncGitHubBackupsInternal,
{ batchSize: 50, maxBatches: 5 },
)
crons.interval(
'trending-leaderboard',
{ minutes: 60 },
internal.leaderboards.rebuildTrendingLeaderboardInternal,
{ limit: 200 },
)
crons.interval(
'skill-stats-backfill',
{ minutes: 10 },
internal.statsMaintenance.runSkillStatBackfillInternal,
{ batchSize: 200, maxBatches: 5 },
)
crons.interval(
'skill-stat-events',
{ minutes: 15 },
internal.skillStatEvents.processSkillStatEventsAction,
{},
)
export default crons
+459
View File
@@ -0,0 +1,459 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { ActionCtx } from './_generated/server'
import { internalAction, internalMutation } from './_generated/server'
import { EMBEDDING_DIMENSIONS } from './lib/embeddings'
import { parseClawdisMetadata, parseFrontmatter } from './lib/skills'
type SeedSkillSpec = {
slug: string
displayName: string
summary: string
version: string
metadata: Record<string, unknown>
rawSkillMd: string
}
type SeedActionArgs = {
reset?: boolean
}
type SeedActionResult = {
ok: true
results: Array<Record<string, unknown> & { slug: string }>
}
type SeedMutationResult = Record<string, unknown>
const SEED_SKILLS: SeedSkillSpec[] = [
{
slug: 'padel',
displayName: 'Padel',
summary: 'Check padel court availability and manage bookings via Playtomic.',
version: '0.1.0',
metadata: {
clawdbot: {
nix: {
plugin: 'github:joshp123/padel-cli',
systems: ['aarch64-darwin', 'x86_64-linux'],
},
config: {
requiredEnv: ['PADEL_AUTH_FILE'],
stateDirs: ['.config/padel'],
example:
'config = { env = { PADEL_AUTH_FILE = "/run/agenix/padel-auth"; }; stateDirs = [ ".config/padel" ]; };',
},
cliHelp: `Padel CLI for availability
Usage:
padel [command]
Available Commands:
auth Manage authentication
availability Show availability for a club on a date
book Book a court
bookings Manage bookings history
search Search for available courts
venues Manage saved venues
Flags:
-h, --help help for padel
--json Output JSON
Use "padel [command] --help" for more information about a command.
`,
},
},
rawSkillMd: `---
name: padel
description: Check padel court availability and manage bookings via the padel CLI.
---
# Padel Booking Skill
## CLI
\`\`\`bash
padel # On PATH (clawdbot plugin bundle)
\`\`\`
## Venues
Use the configured venue list in order of preference. If no venues are configured, ask for a venue name or location.
## Commands
### Check next booking
\`\`\`bash
padel bookings list 2>&1 | head -3
\`\`\`
### Search availability
\`\`\`bash
padel search --venues VENUE1,VENUE2 --date YYYY-MM-DD --time 09:00-12:00
\`\`\`
## Response guidelines
- Keep responses concise.
- Use 🎾 emoji.
- End with a call to action.
## Authorization
Only the authorized booker can confirm bookings. If the requester is not authorized, ask the authorized user to confirm.
`,
},
{
slug: 'gohome',
displayName: 'GoHome',
summary: 'Operate GoHome via gRPC discovery, metrics, and Grafana dashboards.',
version: '0.1.0',
metadata: {
clawdbot: {
nix: {
plugin: 'github:joshp123/gohome',
systems: ['x86_64-linux', 'aarch64-linux'],
},
config: {
requiredEnv: ['GOHOME_GRPC_ADDR', 'GOHOME_HTTP_BASE'],
example:
'config = { env = { GOHOME_GRPC_ADDR = "gohome:9000"; GOHOME_HTTP_BASE = "http://gohome:8080"; }; };',
},
cliHelp: `GoHome CLI
Usage:
gohome-cli [command]
Available Commands:
services List registered services
plugins Inspect loaded plugins
methods List RPC methods
call Call an RPC method
roborock Manage roborock devices
tado Manage tado zones
Flags:
--grpc-addr string gRPC endpoint (host:port)
-h, --help help for gohome-cli
`,
},
},
rawSkillMd: `---
name: gohome
description: Use when Clawdbot needs to test or operate GoHome via gRPC discovery, metrics, and Grafana.
---
# GoHome Skill
## Quick start
\`\`\`bash
export GOHOME_HTTP_BASE="http://gohome:8080"
export GOHOME_GRPC_ADDR="gohome:9000"
\`\`\`
## CLI
\`\`\`bash
gohome-cli services
\`\`\`
## Discovery flow (read-only)
1) List plugins.
2) Describe a plugin.
3) List RPC methods.
4) Call a read-only RPC.
## Metrics validation
\`\`\`bash
curl -s "\${GOHOME_HTTP_BASE}/gohome/metrics" | rg -n "gohome_"
\`\`\`
## Stateful actions
Only call write RPCs after explicit user approval.
`,
},
{
slug: 'xuezh',
displayName: 'Xuezh',
summary: 'Teach Mandarin with the xuezh engine for review, speaking, and audits.',
version: '0.1.0',
metadata: {
clawdbot: {
nix: {
plugin: 'github:joshp123/xuezh',
systems: ['aarch64-darwin', 'x86_64-linux'],
},
config: {
requiredEnv: ['XUEZH_AZURE_SPEECH_KEY_FILE', 'XUEZH_AZURE_SPEECH_REGION'],
stateDirs: ['.config/xuezh'],
example:
'config = { env = { XUEZH_AZURE_SPEECH_KEY_FILE = "/run/agenix/xuezh-azure-speech-key"; XUEZH_AZURE_SPEECH_REGION = "westeurope"; }; stateDirs = [ ".config/xuezh" ]; };',
},
cliHelp: `xuezh - Chinese learning engine
Usage:
xuezh [command]
Available Commands:
snapshot Fetch learner state snapshot
review Review due items
audio Process speech audio
items Manage learning items
events Log learning events
Flags:
-h, --help help for xuezh
--json Output JSON
`,
},
},
rawSkillMd: `---
name: xuezh
description: Teach Mandarin using the xuezh engine for review, speaking, and audits.
---
# Xuezh Skill
## Contract
Use the xuezh CLI exactly as specified. If a command is missing, ask for implementation instead of guessing.
## Default loop
1) Call \`xuezh snapshot\`.
2) Pick a tiny plan (1-2 bullets).
3) Run a short activity.
4) Log outcomes.
## CLI examples
\`\`\`bash
xuezh snapshot --profile default
xuezh review next --limit 10
xuezh audio process-voice --file ./utterance.wav
\`\`\`
`,
},
]
function injectMetadata(rawSkillMd: string, metadata: Record<string, unknown>) {
const frontmatterEnd = rawSkillMd.indexOf('\n---', 3)
if (frontmatterEnd === -1) return rawSkillMd
return `${rawSkillMd.slice(0, frontmatterEnd)}\nmetadata: ${JSON.stringify(
metadata,
)}${rawSkillMd.slice(frontmatterEnd)}`
}
async function seedNixSkillsHandler(
ctx: ActionCtx,
args: SeedActionArgs,
): Promise<SeedActionResult> {
const results: Array<Record<string, unknown> & { slug: string }> = []
for (const spec of SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
const result: SeedMutationResult = await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})
results.push({ slug: spec.slug, ...result })
}
return { ok: true, results }
}
export const seedNixSkills: ReturnType<typeof internalAction> = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: seedNixSkillsHandler,
})
async function seedPadelSkillHandler(
ctx: ActionCtx,
args: SeedActionArgs,
): Promise<SeedMutationResult> {
const spec = SEED_SKILLS.find((entry) => entry.slug === 'padel')
if (!spec) throw new Error('padel seed spec missing')
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
return (await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})) as SeedMutationResult
}
export const seedPadelSkill: ReturnType<typeof internalAction> = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: seedPadelSkillHandler,
})
export const seedSkillMutation = internalMutation({
args: {
reset: v.optional(v.boolean()),
storageId: v.id('_storage'),
metadata: v.any(),
frontmatter: v.any(),
clawdis: v.any(),
skillMd: v.string(),
slug: v.string(),
displayName: v.string(),
summary: v.optional(v.string()),
version: v.string(),
},
handler: async (ctx, args) => {
const existing = await ctx.db
.query('skills')
.withIndex('by_slug', (q) => q.eq('slug', args.slug))
.unique()
if (existing && !args.reset) {
return { ok: true, skipped: true, skillId: existing._id }
}
if (existing && args.reset) {
const versions = await ctx.db
.query('skillVersions')
.withIndex('by_skill', (q) => q.eq('skillId', existing._id))
.collect()
for (const version of versions) {
await ctx.db.delete(version._id)
}
const embeddings = await ctx.db
.query('skillEmbeddings')
.withIndex('by_skill', (q) => q.eq('skillId', existing._id))
.collect()
for (const embedding of embeddings) {
await ctx.db.delete(embedding._id)
}
await ctx.db.delete(existing._id)
}
const now = Date.now()
const existingUsers = await ctx.db
.query('users')
.withIndex('handle', (q) => q.eq('handle', 'local'))
.collect()
const userId =
existingUsers[0]?._id ??
(await ctx.db.insert('users', {
handle: 'local',
displayName: 'Local Dev',
role: 'admin',
createdAt: now,
updatedAt: now,
}))
const skillId = await ctx.db.insert('skills', {
slug: args.slug,
displayName: args.displayName,
summary: args.summary,
ownerUserId: userId,
latestVersionId: undefined,
tags: {},
softDeletedAt: undefined,
badges: { redactionApproved: undefined },
statsDownloads: 0,
statsStars: 0,
statsInstallsCurrent: 0,
statsInstallsAllTime: 0,
stats: {
downloads: 0,
installsCurrent: 0,
installsAllTime: 0,
stars: 0,
versions: 0,
comments: 0,
},
createdAt: now,
updatedAt: now,
})
const versionId = await ctx.db.insert('skillVersions', {
skillId,
version: args.version,
changelog: 'Seeded local version for screenshots.',
files: [
{
path: 'SKILL.md',
size: args.skillMd.length,
storageId: args.storageId,
sha256: 'seeded',
contentType: 'text/markdown',
},
],
parsed: {
frontmatter: args.frontmatter,
metadata: args.metadata,
clawdis: args.clawdis,
},
createdBy: userId,
createdAt: now,
softDeletedAt: undefined,
})
const embeddingId = await ctx.db.insert('skillEmbeddings', {
skillId,
versionId,
ownerId: userId,
embedding: Array.from({ length: EMBEDDING_DIMENSIONS }, () => 0),
isLatest: true,
isApproved: true,
visibility: 'latest-approved',
updatedAt: now,
})
await ctx.db.patch(skillId, {
latestVersionId: versionId,
tags: { latest: versionId },
statsDownloads: 0,
statsStars: 0,
statsInstallsCurrent: 0,
statsInstallsAllTime: 0,
stats: {
downloads: 0,
installsCurrent: 0,
installsAllTime: 0,
stars: 0,
versions: 1,
comments: 0,
},
updatedAt: now,
})
return { ok: true, skillId, versionId, embeddingId }
},
})
+541
View File
@@ -0,0 +1,541 @@
/**
* Extra seed skills for pagination testing.
*
* This file contains 50 placeholder skills to test pagination behavior.
* Run with: bunx convex run internal.devSeedExtra.seedExtraSkillsInternal
* Or with reset: bunx convex run internal.devSeedExtra.seedExtraSkillsInternal '{"reset": true}'
*/
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction, internalMutation } from './_generated/server'
import { parseClawdisMetadata, parseFrontmatter } from './lib/skills'
type SeedSkillSpec = {
slug: string
displayName: string
summary: string
version: string
metadata: Record<string, unknown>
rawSkillMd: string
}
function makeSkill(
slug: string,
displayName: string,
summary: string,
envVars: string[] = [],
commands: string[] = ['help', 'status', 'run'],
): SeedSkillSpec {
const cliHelp = `${slug} - ${summary}
Usage:
${slug} [command]
Commands:
${commands.map((cmd) => ` ${cmd.padEnd(12)} Run ${cmd} operation`).join('\n')}
Flags:
-h, --help Show help
--json Output as JSON
`
const rawSkillMd = `---
name: ${slug}
description: ${summary}
---
# ${displayName}
## CLI
\`\`\`bash
${commands.map((cmd) => `${slug} ${cmd}`).join('\n')}
\`\`\`
## Usage
Use this skill to ${summary.toLowerCase()}.
`
return {
slug,
displayName,
summary,
version: '0.1.0',
metadata: {
clawdbot: {
nix: {
plugin: `github:example/${slug}`,
systems: ['aarch64-darwin', 'x86_64-linux'],
},
config: {
requiredEnv: envVars,
},
cliHelp,
},
},
rawSkillMd,
}
}
// 50 placeholder skills for pagination testing
const EXTRA_SEED_SKILLS: SeedSkillSpec[] = [
// DevOps & Infrastructure (10)
makeSkill(
'kubectl-helper',
'Kubectl Helper',
'Simplified kubectl commands for common Kubernetes operations.',
['KUBECONFIG'],
['pods', 'logs', 'exec', 'describe', 'apply'],
),
makeSkill(
'terraform-runner',
'Terraform Runner',
'Execute Terraform plans and applies with safety checks.',
['TF_VAR_region', 'AWS_PROFILE'],
['plan', 'apply', 'destroy', 'output', 'state'],
),
makeSkill(
'ansible-exec',
'Ansible Exec',
'Run Ansible playbooks and ad-hoc commands.',
['ANSIBLE_INVENTORY'],
['playbook', 'adhoc', 'inventory', 'facts', 'vault'],
),
makeSkill(
'docker-compose-mgr',
'Docker Compose Manager',
'Manage Docker Compose stacks and services.',
['DOCKER_HOST'],
['up', 'down', 'logs', 'ps', 'restart'],
),
makeSkill(
'k9s-wrapper',
'K9s Wrapper',
'Interactive Kubernetes cluster management via K9s.',
['KUBECONFIG'],
['launch', 'contexts', 'namespaces', 'pods', 'logs'],
),
makeSkill(
'helm-charts',
'Helm Charts',
'Manage Helm chart deployments and releases.',
['KUBECONFIG', 'HELM_REPO'],
['install', 'upgrade', 'rollback', 'list', 'search'],
),
makeSkill(
'prometheus-alerts',
'Prometheus Alerts',
'Query Prometheus metrics and manage alerting rules.',
['PROMETHEUS_URL'],
['query', 'alerts', 'rules', 'targets', 'status'],
),
makeSkill(
'grafana-dash',
'Grafana Dashboards',
'Create and manage Grafana dashboards programmatically.',
['GRAFANA_URL', 'GRAFANA_API_KEY'],
['list', 'export', 'import', 'create', 'delete'],
),
makeSkill(
'nginx-config',
'Nginx Config',
'Generate and validate Nginx configuration files.',
['NGINX_CONF_DIR'],
['generate', 'validate', 'reload', 'test', 'sites'],
),
makeSkill(
'jenkins-jobs',
'Jenkins Jobs',
'Manage Jenkins jobs and pipelines.',
['JENKINS_URL', 'JENKINS_TOKEN'],
['list', 'build', 'status', 'logs', 'config'],
),
// Productivity (8)
makeSkill(
'todoist-sync',
'Todoist Sync',
'Sync and manage Todoist tasks from the command line.',
['TODOIST_API_TOKEN'],
['list', 'add', 'complete', 'projects', 'labels'],
),
makeSkill(
'notion-backup',
'Notion Backup',
'Export and backup Notion workspaces.',
['NOTION_TOKEN'],
['export', 'backup', 'restore', 'pages', 'databases'],
),
makeSkill(
'gcal-manager',
'Google Calendar Manager',
'Manage Google Calendar events and schedules.',
['GOOGLE_CREDENTIALS_FILE'],
['events', 'create', 'delete', 'calendars', 'reminders'],
),
makeSkill(
'time-tracker',
'Time Tracker',
'Track time spent on projects and tasks.',
['TIMETRACK_DB'],
['start', 'stop', 'status', 'report', 'projects'],
),
makeSkill(
'email-digest',
'Email Digest',
'Generate email digests and summaries.',
['IMAP_SERVER', 'IMAP_USER'],
['fetch', 'digest', 'search', 'folders', 'unread'],
),
makeSkill(
'habit-tracker',
'Habit Tracker',
'Track daily habits and streaks.',
['HABITS_DB'],
['log', 'streak', 'stats', 'habits', 'remind'],
),
makeSkill(
'bookmark-sync',
'Bookmark Sync',
'Sync bookmarks across browsers and devices.',
['BOOKMARKS_DIR'],
['sync', 'export', 'import', 'search', 'tags'],
),
makeSkill(
'notes-export',
'Notes Export',
'Export notes to various formats.',
['NOTES_DIR'],
['export', 'convert', 'search', 'list', 'tags'],
),
// Media & Entertainment (6)
makeSkill(
'spotify-ctl',
'Spotify Control',
'Control Spotify playback from the terminal.',
['SPOTIFY_CLIENT_ID', 'SPOTIFY_CLIENT_SECRET'],
['play', 'pause', 'next', 'prev', 'search'],
),
makeSkill(
'plex-manager',
'Plex Manager',
'Manage Plex media libraries and playback.',
['PLEX_URL', 'PLEX_TOKEN'],
['libraries', 'scan', 'search', 'play', 'sessions'],
),
makeSkill(
'ytdl-wrapper',
'YouTube Downloader',
'Download videos from YouTube and other platforms.',
['YTDL_OUTPUT_DIR'],
['download', 'info', 'playlist', 'audio', 'formats'],
),
makeSkill(
'podcast-dl',
'Podcast Downloader',
'Download and manage podcast episodes.',
['PODCAST_DIR'],
['subscribe', 'download', 'list', 'play', 'search'],
),
makeSkill(
'audiobook-player',
'Audiobook Player',
'Manage and play audiobook collections.',
['AUDIOBOOK_DIR'],
['play', 'pause', 'bookmark', 'list', 'progress'],
),
makeSkill(
'music-lib',
'Music Library',
'Organize and query local music libraries.',
['MUSIC_DIR'],
['scan', 'search', 'play', 'playlist', 'stats'],
),
// Smart Home (8)
makeSkill(
'hass-control',
'Home Assistant Control',
'Control Home Assistant entities and automations.',
['HASS_URL', 'HASS_TOKEN'],
['entities', 'services', 'automations', 'scenes', 'history'],
),
makeSkill(
'zigbee-mqtt',
'Zigbee2MQTT',
'Manage Zigbee devices via MQTT.',
['MQTT_BROKER', 'ZIGBEE_TOPIC'],
['devices', 'pair', 'remove', 'rename', 'groups'],
),
makeSkill(
'tasmota-ctl',
'Tasmota Control',
'Control Tasmota-flashed devices.',
['TASMOTA_HOSTS'],
['status', 'power', 'config', 'update', 'backup'],
),
makeSkill(
'esphome-mgr',
'ESPHome Manager',
'Manage ESPHome device configurations.',
['ESPHOME_DIR'],
['compile', 'upload', 'logs', 'dashboard', 'config'],
),
makeSkill(
'mqtt-broker',
'MQTT Broker',
'Interact with MQTT brokers for IoT messaging.',
['MQTT_BROKER', 'MQTT_USER'],
['pub', 'sub', 'topics', 'clients', 'stats'],
),
makeSkill(
'hue-lights',
'Philips Hue',
'Control Philips Hue lights and scenes.',
['HUE_BRIDGE_IP', 'HUE_API_KEY'],
['lights', 'scenes', 'groups', 'schedules', 'sensors'],
),
makeSkill(
'smart-thermo',
'Smart Thermostat',
'Control smart thermostats and HVAC systems.',
['THERMOSTAT_API_KEY'],
['status', 'set', 'schedule', 'history', 'zones'],
),
makeSkill(
'cam-viewer',
'Camera Viewer',
'View and manage security camera feeds.',
['CAMERA_URLS'],
['list', 'snapshot', 'stream', 'record', 'events'],
),
// Finance (5)
makeSkill(
'budget-track',
'Budget Tracker',
'Track budgets and spending across categories.',
['BUDGET_DB'],
['summary', 'add', 'categories', 'report', 'goals'],
),
makeSkill(
'crypto-watch',
'Crypto Watcher',
'Monitor cryptocurrency prices and portfolios.',
['CRYPTO_API_KEY'],
['prices', 'portfolio', 'alerts', 'history', 'convert'],
),
makeSkill(
'stock-alerts',
'Stock Alerts',
'Set up stock price alerts and notifications.',
['STOCK_API_KEY'],
['quote', 'watch', 'alerts', 'portfolio', 'news'],
),
makeSkill(
'expense-cat',
'Expense Categorizer',
'Automatically categorize expenses.',
['EXPENSE_DB'],
['import', 'categorize', 'report', 'rules', 'export'],
),
makeSkill(
'invoice-gen',
'Invoice Generator',
'Generate and manage invoices.',
['INVOICE_DIR', 'COMPANY_INFO'],
['create', 'list', 'send', 'paid', 'overdue'],
),
// Communication (5)
makeSkill(
'slack-bot',
'Slack Bot',
'Interact with Slack channels and messages.',
['SLACK_TOKEN'],
['send', 'channels', 'users', 'search', 'files'],
),
makeSkill(
'discord-mgr',
'Discord Manager',
'Manage Discord servers and messages.',
['DISCORD_TOKEN'],
['send', 'servers', 'channels', 'members', 'roles'],
),
makeSkill(
'telegram-bot',
'Telegram Bot',
'Send and receive Telegram messages.',
['TELEGRAM_BOT_TOKEN'],
['send', 'receive', 'chats', 'files', 'inline'],
),
makeSkill(
'matrix-cli',
'Matrix CLI',
'Interact with Matrix chat rooms.',
['MATRIX_HOMESERVER', 'MATRIX_TOKEN'],
['send', 'rooms', 'join', 'leave', 'sync'],
),
makeSkill(
'irc-bridge',
'IRC Bridge',
'Bridge IRC channels to other platforms.',
['IRC_SERVER', 'IRC_NICK'],
['connect', 'join', 'send', 'channels', 'users'],
),
// Data & Analytics (5)
makeSkill(
'pg-queries',
'PostgreSQL Queries',
'Execute PostgreSQL queries and manage databases.',
['DATABASE_URL'],
['query', 'tables', 'schema', 'backup', 'restore'],
),
makeSkill(
'clickhouse-ql',
'ClickHouse Queries',
'Run ClickHouse analytics queries.',
['CLICKHOUSE_URL'],
['query', 'tables', 'insert', 'system', 'optimize'],
),
makeSkill(
'redis-cli',
'Redis CLI',
'Interact with Redis cache and data structures.',
['REDIS_URL'],
['get', 'set', 'keys', 'info', 'flush'],
),
makeSkill(
'elastic-search',
'Elasticsearch',
'Search and manage Elasticsearch indices.',
['ELASTICSEARCH_URL'],
['search', 'index', 'mapping', 'cluster', 'aliases'],
),
makeSkill(
'mongo-shell',
'MongoDB Shell',
'Query and manage MongoDB collections.',
['MONGODB_URI'],
['find', 'insert', 'update', 'delete', 'aggregate'],
),
// Security (3)
makeSkill(
'vault-secrets',
'Vault Secrets',
'Manage secrets in HashiCorp Vault.',
['VAULT_ADDR', 'VAULT_TOKEN'],
['read', 'write', 'list', 'delete', 'seal'],
),
makeSkill(
'gpg-keys',
'GPG Keys',
'Manage GPG keys and encryption.',
['GNUPGHOME'],
['list', 'generate', 'export', 'import', 'encrypt'],
),
makeSkill(
'ssh-rotate',
'SSH Key Rotator',
'Rotate and manage SSH keys.',
['SSH_KEY_DIR'],
['generate', 'rotate', 'deploy', 'list', 'revoke'],
),
]
function injectMetadata(rawSkillMd: string, metadata: Record<string, unknown>) {
const frontmatterEnd = rawSkillMd.indexOf('\n---', 3)
if (frontmatterEnd === -1) return rawSkillMd
return `${rawSkillMd.slice(0, frontmatterEnd)}\nmetadata: ${JSON.stringify(
metadata,
)}${rawSkillMd.slice(frontmatterEnd)}`
}
function randomStats() {
return {
downloads: Math.floor(Math.random() * 5000),
stars: Math.floor(Math.random() * 500),
installsCurrent: Math.floor(Math.random() * 200),
installsAllTime: Math.floor(Math.random() * 1000),
}
}
export const applyRandomStats = internalMutation({
args: {
skillId: v.id('skills'),
stats: v.object({
downloads: v.number(),
stars: v.number(),
installsCurrent: v.number(),
installsAllTime: v.number(),
}),
},
handler: async (ctx, args) => {
await ctx.db.patch(args.skillId, {
statsDownloads: args.stats.downloads,
statsStars: args.stats.stars,
statsInstallsCurrent: args.stats.installsCurrent,
statsInstallsAllTime: args.stats.installsAllTime,
stats: {
downloads: args.stats.downloads,
stars: args.stats.stars,
installsCurrent: args.stats.installsCurrent,
installsAllTime: args.stats.installsAllTime,
versions: 1,
comments: 0,
},
})
},
})
export const seedExtraSkillsInternal = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: async (ctx: ActionCtx, args) => {
const results: Array<{ slug: string; ok: boolean; skipped?: boolean }> = []
for (const spec of EXTRA_SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
const result = (await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})) as { ok: boolean; skipped?: boolean; skillId?: string }
// Apply random stats after creation (only if not skipped)
if (result.skillId && !result.skipped) {
const stats = randomStats()
await ctx.runMutation(internal.devSeedExtra.applyRandomStats, {
skillId: result.skillId as Id<'skills'>,
stats,
})
}
results.push({ slug: spec.slug, ok: result.ok, skipped: result.skipped })
}
const created = results.filter((r) => !r.skipped).length
const skipped = results.filter((r) => r.skipped).length
return { ok: true, total: results.length, created, skipped }
},
})
+4 -3
View File
@@ -2,6 +2,7 @@ import { v } from 'convex/values'
import { zipSync } from 'fflate'
import { api } from './_generated/api'
import { httpAction, mutation } from './_generated/server'
import { insertStatEvent } from './skillStatEvents'
export const downloadZip = httpAction(async (ctx, request) => {
const url = new URL(request.url)
@@ -69,9 +70,9 @@ export const increment = mutation({
handler: async (ctx, args) => {
const skill = await ctx.db.get(args.skillId)
if (!skill) return
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, downloads: skill.stats.downloads + 1 },
updatedAt: Date.now(),
await insertStatEvent(ctx, {
skillId: skill._id,
kind: 'download',
})
},
})
+170
View File
@@ -0,0 +1,170 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalMutation, internalQuery } from './_generated/server'
import { assertRole, requireUserFromAction } from './lib/access'
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const SYNC_STATE_KEY = 'default'
type BackupPageItem =
| {
kind: 'ok'
skillId: Id<'skills'>
versionId: Id<'skillVersions'>
slug: string
displayName: string
version: string
ownerHandle: string
files: Doc<'skillVersions'>['files']
publishedAt: number
}
| { kind: 'missingLatestVersion'; skillId: Id<'skills'> }
| { kind: 'missingVersionDoc'; skillId: Id<'skills'>; versionId: Id<'skillVersions'> }
| { kind: 'missingOwner'; skillId: Id<'skills'>; ownerUserId: Id<'users'> }
type BackupPageResult = {
items: BackupPageItem[]
cursor: string | null
isDone: boolean
}
type BackupSyncState = {
cursor: string | null
}
export type SyncGitHubBackupsResult = {
stats: {
skillsScanned: number
skillsSkipped: number
skillsBackedUp: number
skillsMissingVersion: number
skillsMissingOwner: number
errors: number
}
cursor: string | null
isDone: boolean
}
export const getGitHubBackupPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BackupPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const { page, isDone, continueCursor } = await ctx.db
.query('skills')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
const items: BackupPageItem[] = []
for (const skill of page) {
if (skill.softDeletedAt) continue
if (!skill.latestVersionId) {
items.push({ kind: 'missingLatestVersion', skillId: skill._id })
continue
}
const version = await ctx.db.get(skill.latestVersionId)
if (!version) {
items.push({
kind: 'missingVersionDoc',
skillId: skill._id,
versionId: skill.latestVersionId,
})
continue
}
const owner = await ctx.db.get(skill.ownerUserId)
if (!owner || owner.deletedAt) {
items.push({ kind: 'missingOwner', skillId: skill._id, ownerUserId: skill.ownerUserId })
continue
}
items.push({
kind: 'ok',
skillId: skill._id,
versionId: version._id,
slug: skill.slug,
displayName: skill.displayName,
version: version.version,
ownerHandle: owner.handle ?? owner._id,
files: version.files,
publishedAt: version.createdAt,
})
}
return { items, cursor: continueCursor, isDone }
},
})
export const getGitHubBackupSyncStateInternal = internalQuery({
args: {},
handler: async (ctx): Promise<BackupSyncState> => {
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
return { cursor: state?.cursor ?? null }
},
})
export const setGitHubBackupSyncStateInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
},
handler: async (ctx, args) => {
const now = Date.now()
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
if (!state) {
await ctx.db.insert('githubBackupSyncState', {
key: SYNC_STATE_KEY,
cursor: args.cursor,
updatedAt: now,
})
return { ok: true as const }
}
await ctx.db.patch(state._id, {
cursor: args.cursor,
updatedAt: now,
})
return { ok: true as const }
},
})
export const syncGitHubBackups: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
resetCursor: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SyncGitHubBackupsResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
if (args.resetCursor && !args.dryRun) {
await ctx.runMutation(internal.githubBackups.setGitHubBackupSyncStateInternal, {
cursor: undefined,
})
}
return ctx.runAction(internal.githubBackupsNode.syncGitHubBackupsInternal, {
dryRun: args.dryRun,
batchSize: args.batchSize,
maxBatches: args.maxBatches,
}) as Promise<SyncGitHubBackupsResult>
},
})
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
}
+183
View File
@@ -0,0 +1,183 @@
'use node'
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction } from './_generated/server'
import {
backupSkillToGitHub,
fetchGitHubSkillMeta,
getGitHubBackupContext,
isGitHubBackupConfigured,
} from './lib/githubBackup'
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const DEFAULT_MAX_BATCHES = 5
const MAX_MAX_BATCHES = 200
type BackupPageItem =
| {
kind: 'ok'
slug: string
version: string
displayName: string
ownerHandle: string
files: Doc<'skillVersions'>['files']
publishedAt: number
}
| { kind: 'missingLatestVersion' }
| { kind: 'missingVersionDoc' }
| { kind: 'missingOwner' }
export type GitHubBackupSyncStats = {
skillsScanned: number
skillsSkipped: number
skillsBackedUp: number
skillsMissingVersion: number
skillsMissingOwner: number
errors: number
}
export type SyncGitHubBackupsInternalArgs = {
dryRun?: boolean
batchSize?: number
maxBatches?: number
}
export type SyncGitHubBackupsInternalResult = {
stats: GitHubBackupSyncStats
cursor: string | null
isDone: boolean
}
export const backupSkillForPublishInternal = internalAction({
args: {
slug: v.string(),
version: v.string(),
displayName: v.string(),
ownerHandle: v.string(),
files: v.array(
v.object({
path: v.string(),
size: v.number(),
storageId: v.id('_storage'),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
),
publishedAt: v.number(),
},
handler: async (ctx, args) => {
if (!isGitHubBackupConfigured()) {
return { skipped: true as const }
}
await backupSkillToGitHub(ctx, args)
return { skipped: false as const }
},
})
export async function syncGitHubBackupsInternalHandler(
ctx: ActionCtx,
args: SyncGitHubBackupsInternalArgs,
): Promise<SyncGitHubBackupsInternalResult> {
const dryRun = Boolean(args.dryRun)
const stats: GitHubBackupSyncStats = {
skillsScanned: 0,
skillsSkipped: 0,
skillsBackedUp: 0,
skillsMissingVersion: 0,
skillsMissingOwner: 0,
errors: 0,
}
if (!isGitHubBackupConfigured()) {
return { stats, cursor: null, isDone: true }
}
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const context = await getGitHubBackupContext()
const state = dryRun
? { cursor: null as string | null }
: ((await ctx.runQuery(internal.githubBackups.getGitHubBackupSyncStateInternal, {})) as {
cursor: string | null
})
let cursor: string | null = state.cursor
let isDone = false
for (let batch = 0; batch < maxBatches; batch++) {
const page = (await ctx.runQuery(internal.githubBackups.getGitHubBackupPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as { items: BackupPageItem[]; cursor: string | null; isDone: boolean }
cursor = page.cursor
isDone = page.isDone
for (const item of page.items) {
if (item.kind !== 'ok') {
if (item.kind === 'missingLatestVersion' || item.kind === 'missingVersionDoc') {
stats.skillsMissingVersion += 1
} else if (item.kind === 'missingOwner') {
stats.skillsMissingOwner += 1
}
continue
}
stats.skillsScanned += 1
try {
const meta = await fetchGitHubSkillMeta(context, item.ownerHandle, item.slug)
if (meta?.latest?.version === item.version) {
stats.skillsSkipped += 1
continue
}
if (!dryRun) {
await backupSkillToGitHub(
ctx,
{
slug: item.slug,
version: item.version,
displayName: item.displayName,
ownerHandle: item.ownerHandle,
files: item.files,
publishedAt: item.publishedAt,
},
context,
)
stats.skillsBackedUp += 1
}
} catch (error) {
console.error('GitHub backup sync failed', error)
stats.errors += 1
}
}
if (!dryRun) {
await ctx.runMutation(internal.githubBackups.setGitHubBackupSyncStateInternal, {
cursor: isDone ? undefined : (cursor ?? undefined),
})
}
if (isDone) break
}
return { stats, cursor, isDone }
}
export const syncGitHubBackupsInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: syncGitHubBackupsInternalHandler,
})
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
}
+317
View File
@@ -0,0 +1,317 @@
import { ConvexError, v } from 'convex/values'
import { unzipSync } from 'fflate'
import semver from 'semver'
import { api, internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { action } from './_generated/server'
import { requireUserFromAction } from './lib/access'
import {
buildGitHubImportFileList,
computeDefaultSelectedPaths,
detectGitHubImportCandidates,
fetchGitHubZipBytes,
listTextFilesUnderCandidate,
normalizeRepoPath,
parseGitHubImportUrl,
resolveGitHubCommit,
stripGitHubZipRoot,
suggestDisplayName,
suggestVersion,
} from './lib/githubImport'
import { publishVersionForUser } from './lib/skillPublish'
import { sanitizePath } from './lib/skills'
const MAX_SELECTED_BYTES = 50 * 1024 * 1024
const MAX_UNZIPPED_BYTES = 80 * 1024 * 1024
const MAX_FILE_COUNT = 7_500
const MAX_SINGLE_FILE_BYTES = 10 * 1024 * 1024
export const previewGitHubImport = action({
args: { url: v.string() },
handler: async (ctx, args) => {
await requireUserFromAction(ctx)
const parsed = parseGitHubImportUrl(args.url)
const resolved = await resolveGitHubCommit(parsed, fetch)
const zipBytes = await fetchGitHubZipBytes(resolved, fetch)
const entries = unzipToEntries(zipBytes)
const stripped = stripGitHubZipRoot(entries)
const candidates = detectGitHubImportCandidates(stripped).filter((candidate) =>
isCandidateUnderResolvedPath(candidate.path, resolved.path),
)
if (candidates.length === 0) throw new ConvexError('No SKILL.md found in this repo')
return {
resolved,
candidates: candidates.map((candidate) => ({
path: candidate.path,
readmePath: candidate.readmePath,
name: candidate.name ?? null,
description: candidate.description ?? null,
})),
}
},
})
export const previewGitHubImportCandidate = action({
args: { url: v.string(), candidatePath: v.string() },
handler: async (ctx, args) => {
const { userId } = await requireUserFromAction(ctx)
const parsed = parseGitHubImportUrl(args.url)
const resolved = await resolveGitHubCommit(parsed, fetch)
const zipBytes = await fetchGitHubZipBytes(resolved, fetch)
const entries = unzipToEntries(zipBytes)
const stripped = stripGitHubZipRoot(entries)
const normalizedCandidatePath = normalizeRepoPath(args.candidatePath)
if (!isCandidateUnderResolvedPath(normalizedCandidatePath, resolved.path)) {
throw new ConvexError('Candidate path is outside the requested import scope')
}
const candidates = detectGitHubImportCandidates(stripped).filter((candidate) =>
isCandidateUnderResolvedPath(candidate.path, resolved.path),
)
const candidate = candidates.find((item) => item.path === normalizedCandidatePath)
if (!candidate) throw new ConvexError('Candidate not found')
const files = listTextFilesUnderCandidate(stripped, candidate.path)
const defaultSelectedPaths = computeDefaultSelectedPaths({ candidate, files })
const fileList = buildGitHubImportFileList({
candidate,
files,
defaultSelectedPaths,
})
const baseForNaming = candidate.path ? (candidate.path.split('/').at(-1) ?? '') : resolved.repo
const suggestedDisplayName = suggestDisplayName(candidate, baseForNaming)
const rawSlugBase = sanitizeSlug(candidate.path ? baseForNaming : resolved.repo)
const suggestedSlug = await suggestAvailableSlug(ctx, userId, rawSlugBase)
const existing = await ctx.runQuery(api.skills.getBySlug, { slug: suggestedSlug })
const existingLatest =
existing?.skill && existing.skill.ownerUserId === userId
? (existing.latestVersion?.version ?? null)
: null
const suggestedVersion = suggestVersion(existingLatest)
return {
resolved,
candidate: {
path: candidate.path,
readmePath: candidate.readmePath,
name: candidate.name ?? null,
description: candidate.description ?? null,
},
defaults: {
selectedPaths: defaultSelectedPaths,
slug: suggestedSlug,
displayName: suggestedDisplayName,
version: suggestedVersion,
tags: ['latest'],
},
files: fileList,
}
},
})
export const importGitHubSkill = action({
args: {
url: v.string(),
commit: v.string(),
candidatePath: v.string(),
selectedPaths: v.array(v.string()),
slug: v.optional(v.string()),
displayName: v.optional(v.string()),
version: v.optional(v.string()),
tags: v.optional(v.array(v.string())),
},
handler: async (ctx, args) => {
const { userId } = await requireUserFromAction(ctx)
const parsed = parseGitHubImportUrl(args.url)
const resolved = await resolveGitHubCommit(parsed, fetch)
if (!/^[a-f0-9]{40}$/i.test(args.commit)) throw new ConvexError('Invalid commit')
if (args.commit.toLowerCase() !== resolved.commit.toLowerCase()) {
throw new ConvexError('Import is out of date. Re-run preview.')
}
const normalizedCandidatePath = normalizeRepoPath(args.candidatePath)
if (!isCandidateUnderResolvedPath(normalizedCandidatePath, resolved.path)) {
throw new ConvexError('Candidate path is outside the requested import scope')
}
const zipBytes = await fetchGitHubZipBytes(resolved, fetch)
const entries = stripGitHubZipRoot(unzipToEntries(zipBytes))
const candidates = detectGitHubImportCandidates(entries).filter((candidate) =>
isCandidateUnderResolvedPath(candidate.path, resolved.path),
)
const candidate = candidates.find((item) => item.path === normalizedCandidatePath)
if (!candidate) throw new ConvexError('Candidate not found')
const filesUnderCandidate = listTextFilesUnderCandidate(entries, candidate.path)
const byPath = new Map(filesUnderCandidate.map((file) => [file.path, file.bytes]))
const selected = Array.from(
new Set(args.selectedPaths.map((path) => normalizeRepoPath(path)).filter(Boolean)),
)
if (selected.length === 0) throw new ConvexError('No files selected')
const candidateRoot = candidate.path ? `${candidate.path}/` : ''
const normalizedReadmePath = normalizeRepoPath(candidate.readmePath)
if (!selected.includes(normalizedReadmePath)) {
throw new ConvexError('SKILL.md must be selected')
}
let totalBytes = 0
const storedFiles: Array<{
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}> = []
for (const path of selected.sort()) {
if (candidateRoot && !path.startsWith(candidateRoot)) {
throw new ConvexError('Selected file is outside the chosen skill folder')
}
const bytes = byPath.get(path)
if (!bytes) continue
totalBytes += bytes.byteLength
if (totalBytes > MAX_SELECTED_BYTES) throw new ConvexError('Selected files exceed 50MB limit')
const relPath = candidateRoot ? path.slice(candidateRoot.length) : path
const sanitized = sanitizePath(relPath)
if (!sanitized) throw new ConvexError('Invalid file paths')
const sha256 = await sha256Hex(bytes)
const safeBytes = new Uint8Array(bytes)
const storageId = await ctx.storage.store(new Blob([safeBytes], { type: 'text/plain' }))
storedFiles.push({
path: sanitized,
size: bytes.byteLength,
storageId,
sha256,
contentType: 'text/plain',
})
}
if (storedFiles.length === 0) throw new ConvexError('No files selected')
const slugBase = (args.slug ?? '').trim().toLowerCase()
const displayName = (args.displayName ?? '').trim()
const tags = (args.tags ?? ['latest']).map((tag) => tag.trim()).filter(Boolean)
const version = (args.version ?? '').trim()
if (!slugBase) throw new ConvexError('Slug required')
if (!displayName) throw new ConvexError('Display name required')
if (!version || !semver.valid(version)) throw new ConvexError('Version must be valid semver')
const result = await publishVersionForUser(ctx, userId, {
slug: slugBase,
displayName,
version,
changelog: '',
tags,
files: storedFiles,
source: {
kind: 'github',
url: resolved.originalUrl,
repo: `${resolved.owner}/${resolved.repo}`,
ref: resolved.ref,
commit: resolved.commit,
path: candidate.path,
importedAt: Date.now(),
},
})
return { ok: true, slug: slugBase, version, ...result }
},
})
function unzipToEntries(zipBytes: Uint8Array) {
const entries = unzipSync(zipBytes)
const out: Record<string, Uint8Array> = {}
const rawPaths = Object.keys(entries)
if (rawPaths.length > MAX_FILE_COUNT) throw new ConvexError('Repo archive has too many files')
let totalBytes = 0
for (const [rawPath, bytes] of Object.entries(entries)) {
const normalizedPath = normalizeZipPath(rawPath)
if (!normalizedPath) continue
if (isJunkPath(normalizedPath)) continue
if (!bytes) continue
if (bytes.byteLength > MAX_SINGLE_FILE_BYTES) continue
totalBytes += bytes.byteLength
if (totalBytes > MAX_UNZIPPED_BYTES) throw new ConvexError('Repo archive is too large')
out[normalizedPath] = bytes
}
return out
}
function isCandidateUnderResolvedPath(candidatePath: string, resolvedPath: string) {
const root = normalizeRepoPath(resolvedPath)
if (!root) return true
if (!candidatePath) return false
if (candidatePath === root) return true
return candidatePath.startsWith(`${root}/`)
}
function sanitizeSlug(value: string) {
return value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]+/g, '-')
.replace(/^-+/, '')
.replace(/-+$/, '')
.replace(/--+/g, '-')
}
async function suggestAvailableSlug(ctx: ActionCtx, userId: Id<'users'>, base: string) {
const cleaned = sanitizeSlug(base)
if (!cleaned) throw new ConvexError('Could not derive slug')
for (let i = 0; i < 50; i += 1) {
const candidate = i === 0 ? cleaned : `${cleaned}-${i + 1}`
const existing = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug: candidate })
if (!existing) return candidate
if (existing.ownerUserId === userId) return candidate
}
throw new ConvexError('Could not find an available slug')
}
async function sha256Hex(bytes: Uint8Array) {
const normalized = new Uint8Array(bytes)
const digest = await crypto.subtle.digest('SHA-256', normalized.buffer)
return toHex(new Uint8Array(digest))
}
function toHex(bytes: Uint8Array) {
let out = ''
for (const byte of bytes) out += byte.toString(16).padStart(2, '0')
return out
}
function normalizeZipPath(path: string) {
const normalized = path
.replaceAll('\u0000', '')
.replaceAll('\\', '/')
.trim()
.replace(/^\.\/+/, '')
.replace(/^\/+/, '')
if (!normalized) return ''
if (normalized.includes('..')) return ''
return normalized
}
function isJunkPath(path: string) {
const normalized = path.toLowerCase()
if (normalized.startsWith('__macosx/')) return true
if (normalized.endsWith('/.ds_store')) return true
if (normalized === '.ds_store') return true
return false
}
+170
View File
@@ -0,0 +1,170 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalMutation, internalQuery } from './_generated/server'
import { assertRole, requireUserFromAction } from './lib/access'
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const SYNC_STATE_KEY = 'souls'
type BackupPageItem =
| {
kind: 'ok'
soulId: Id<'souls'>
versionId: Id<'soulVersions'>
slug: string
displayName: string
version: string
ownerHandle: string
files: Doc<'soulVersions'>['files']
publishedAt: number
}
| { kind: 'missingLatestVersion'; soulId: Id<'souls'> }
| { kind: 'missingVersionDoc'; soulId: Id<'souls'>; versionId: Id<'soulVersions'> }
| { kind: 'missingOwner'; soulId: Id<'souls'>; ownerUserId: Id<'users'> }
type BackupPageResult = {
items: BackupPageItem[]
cursor: string | null
isDone: boolean
}
type BackupSyncState = {
cursor: string | null
}
export type SyncGitHubSoulBackupsResult = {
stats: {
soulsScanned: number
soulsSkipped: number
soulsBackedUp: number
soulsMissingVersion: number
soulsMissingOwner: number
errors: number
}
cursor: string | null
isDone: boolean
}
export const getGitHubSoulBackupPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BackupPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const { page, isDone, continueCursor } = await ctx.db
.query('souls')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
const items: BackupPageItem[] = []
for (const soul of page) {
if (soul.softDeletedAt) continue
if (!soul.latestVersionId) {
items.push({ kind: 'missingLatestVersion', soulId: soul._id })
continue
}
const version = await ctx.db.get(soul.latestVersionId)
if (!version) {
items.push({
kind: 'missingVersionDoc',
soulId: soul._id,
versionId: soul.latestVersionId,
})
continue
}
const owner = await ctx.db.get(soul.ownerUserId)
if (!owner || owner.deletedAt) {
items.push({ kind: 'missingOwner', soulId: soul._id, ownerUserId: soul.ownerUserId })
continue
}
items.push({
kind: 'ok',
soulId: soul._id,
versionId: version._id,
slug: soul.slug,
displayName: soul.displayName,
version: version.version,
ownerHandle: owner.handle ?? owner._id,
files: version.files,
publishedAt: version.createdAt,
})
}
return { items, cursor: continueCursor, isDone }
},
})
export const getGitHubSoulBackupSyncStateInternal = internalQuery({
args: {},
handler: async (ctx): Promise<BackupSyncState> => {
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
return { cursor: state?.cursor ?? null }
},
})
export const setGitHubSoulBackupSyncStateInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
},
handler: async (ctx, args) => {
const now = Date.now()
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
if (!state) {
await ctx.db.insert('githubBackupSyncState', {
key: SYNC_STATE_KEY,
cursor: args.cursor,
updatedAt: now,
})
return { ok: true as const }
}
await ctx.db.patch(state._id, {
cursor: args.cursor,
updatedAt: now,
})
return { ok: true as const }
},
})
export const syncGitHubSoulBackups: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
resetCursor: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SyncGitHubSoulBackupsResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
if (args.resetCursor && !args.dryRun) {
await ctx.runMutation(internal.githubSoulBackups.setGitHubSoulBackupSyncStateInternal, {
cursor: undefined,
})
}
return ctx.runAction(internal.githubSoulBackupsNode.syncGitHubSoulBackupsInternal, {
dryRun: args.dryRun,
batchSize: args.batchSize,
maxBatches: args.maxBatches,
}) as Promise<SyncGitHubSoulBackupsResult>
},
})
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
}
+186
View File
@@ -0,0 +1,186 @@
'use node'
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction } from './_generated/server'
import {
backupSoulToGitHub,
fetchGitHubSoulMeta,
getGitHubSoulBackupContext,
isGitHubSoulBackupConfigured,
} from './lib/githubSoulBackup'
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const DEFAULT_MAX_BATCHES = 5
const MAX_MAX_BATCHES = 200
type BackupPageItem =
| {
kind: 'ok'
slug: string
version: string
displayName: string
ownerHandle: string
files: Doc<'soulVersions'>['files']
publishedAt: number
}
| { kind: 'missingLatestVersion' }
| { kind: 'missingVersionDoc' }
| { kind: 'missingOwner' }
export type GitHubSoulBackupSyncStats = {
soulsScanned: number
soulsSkipped: number
soulsBackedUp: number
soulsMissingVersion: number
soulsMissingOwner: number
errors: number
}
export type SyncGitHubSoulBackupsInternalArgs = {
dryRun?: boolean
batchSize?: number
maxBatches?: number
}
export type SyncGitHubSoulBackupsInternalResult = {
stats: GitHubSoulBackupSyncStats
cursor: string | null
isDone: boolean
}
export const backupSoulForPublishInternal = internalAction({
args: {
slug: v.string(),
version: v.string(),
displayName: v.string(),
ownerHandle: v.string(),
files: v.array(
v.object({
path: v.string(),
size: v.number(),
storageId: v.id('_storage'),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
),
publishedAt: v.number(),
},
handler: async (ctx, args) => {
if (!isGitHubSoulBackupConfigured()) {
return { skipped: true as const }
}
await backupSoulToGitHub(ctx, args)
return { skipped: false as const }
},
})
export async function syncGitHubSoulBackupsInternalHandler(
ctx: ActionCtx,
args: SyncGitHubSoulBackupsInternalArgs,
): Promise<SyncGitHubSoulBackupsInternalResult> {
const dryRun = Boolean(args.dryRun)
const stats: GitHubSoulBackupSyncStats = {
soulsScanned: 0,
soulsSkipped: 0,
soulsBackedUp: 0,
soulsMissingVersion: 0,
soulsMissingOwner: 0,
errors: 0,
}
if (!isGitHubSoulBackupConfigured()) {
return { stats, cursor: null, isDone: true }
}
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const context = await getGitHubSoulBackupContext()
const state = dryRun
? { cursor: null as string | null }
: ((await ctx.runQuery(
internal.githubSoulBackups.getGitHubSoulBackupSyncStateInternal,
{},
)) as {
cursor: string | null
})
let cursor: string | null = state.cursor
let isDone = false
for (let batch = 0; batch < maxBatches; batch++) {
const page = (await ctx.runQuery(internal.githubSoulBackups.getGitHubSoulBackupPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as { items: BackupPageItem[]; cursor: string | null; isDone: boolean }
cursor = page.cursor
isDone = page.isDone
for (const item of page.items) {
if (item.kind !== 'ok') {
if (item.kind === 'missingLatestVersion' || item.kind === 'missingVersionDoc') {
stats.soulsMissingVersion += 1
} else if (item.kind === 'missingOwner') {
stats.soulsMissingOwner += 1
}
continue
}
stats.soulsScanned += 1
try {
const meta = await fetchGitHubSoulMeta(context, item.ownerHandle, item.slug)
if (meta?.latest?.version === item.version) {
stats.soulsSkipped += 1
continue
}
if (!dryRun) {
await backupSoulToGitHub(
ctx,
{
slug: item.slug,
version: item.version,
displayName: item.displayName,
ownerHandle: item.ownerHandle,
files: item.files,
publishedAt: item.publishedAt,
},
context,
)
stats.soulsBackedUp += 1
}
} catch (error) {
console.error('GitHub soul backup sync failed', error)
stats.errors += 1
}
}
if (!dryRun) {
await ctx.runMutation(internal.githubSoulBackups.setGitHubSoulBackupSyncStateInternal, {
cursor: isDone ? undefined : (cursor ?? undefined),
})
}
if (isDone) break
}
return { stats, cursor, isDone }
}
export const syncGitHubSoulBackupsInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: syncGitHubSoulBackupsInternalHandler,
})
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
}
+128 -8
View File
@@ -1,4 +1,4 @@
import { ApiRoutes } from 'clawdhub-schema'
import { ApiRoutes, LegacyApiRoutes } from 'clawhub-schema'
import { httpRouter } from 'convex/server'
import { auth } from './auth'
import { downloadZip } from './downloads'
@@ -6,12 +6,30 @@ import {
cliPublishHttp,
cliSkillDeleteHttp,
cliSkillUndeleteHttp,
cliTelemetrySyncHttp,
cliUploadUrlHttp,
cliWhoamiHttp,
getSkillHttp,
resolveSkillVersionHttp,
searchSkillsHttp,
} from './httpApi'
import {
listSkillsV1Http,
listSoulsV1Http,
publishSkillV1Http,
publishSoulV1Http,
resolveSkillVersionV1Http,
searchSkillsV1Http,
skillsDeleteRouterV1Http,
skillsGetRouterV1Http,
skillsPostRouterV1Http,
soulsDeleteRouterV1Http,
soulsGetRouterV1Http,
soulsPostRouterV1Http,
starsDeleteRouterV1Http,
starsPostRouterV1Http,
whoamiV1Http,
} from './httpApiV1'
const http = httpRouter()
@@ -26,47 +44,149 @@ http.route({
http.route({
path: ApiRoutes.search,
method: 'GET',
handler: searchSkillsV1Http,
})
http.route({
path: ApiRoutes.resolve,
method: 'GET',
handler: resolveSkillVersionV1Http,
})
http.route({
path: ApiRoutes.skills,
method: 'GET',
handler: listSkillsV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.skills}/`,
method: 'GET',
handler: skillsGetRouterV1Http,
})
http.route({
path: ApiRoutes.skills,
method: 'POST',
handler: publishSkillV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.skills}/`,
method: 'POST',
handler: skillsPostRouterV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.skills}/`,
method: 'DELETE',
handler: skillsDeleteRouterV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.stars}/`,
method: 'POST',
handler: starsPostRouterV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.stars}/`,
method: 'DELETE',
handler: starsDeleteRouterV1Http,
})
http.route({
path: ApiRoutes.whoami,
method: 'GET',
handler: whoamiV1Http,
})
http.route({
path: ApiRoutes.souls,
method: 'GET',
handler: listSoulsV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: 'GET',
handler: soulsGetRouterV1Http,
})
http.route({
path: ApiRoutes.souls,
method: 'POST',
handler: publishSoulV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: 'POST',
handler: soulsPostRouterV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: 'DELETE',
handler: soulsDeleteRouterV1Http,
})
// TODO: remove legacy /api routes after deprecation window.
http.route({
path: LegacyApiRoutes.download,
method: 'GET',
handler: downloadZip,
})
http.route({
path: LegacyApiRoutes.search,
method: 'GET',
handler: searchSkillsHttp,
})
http.route({
path: ApiRoutes.skill,
path: LegacyApiRoutes.skill,
method: 'GET',
handler: getSkillHttp,
})
http.route({
path: ApiRoutes.skillResolve,
path: LegacyApiRoutes.skillResolve,
method: 'GET',
handler: resolveSkillVersionHttp,
})
http.route({
path: ApiRoutes.cliWhoami,
path: LegacyApiRoutes.cliWhoami,
method: 'GET',
handler: cliWhoamiHttp,
})
http.route({
path: ApiRoutes.cliUploadUrl,
path: LegacyApiRoutes.cliUploadUrl,
method: 'POST',
handler: cliUploadUrlHttp,
})
http.route({
path: ApiRoutes.cliPublish,
path: LegacyApiRoutes.cliPublish,
method: 'POST',
handler: cliPublishHttp,
})
http.route({
path: ApiRoutes.cliSkillDelete,
path: LegacyApiRoutes.cliTelemetrySync,
method: 'POST',
handler: cliTelemetrySyncHttp,
})
http.route({
path: LegacyApiRoutes.cliSkillDelete,
method: 'POST',
handler: cliSkillDeleteHttp,
})
http.route({
path: ApiRoutes.cliSkillUndelete,
path: LegacyApiRoutes.cliSkillUndelete,
method: 'POST',
handler: cliSkillUndeleteHttp,
})
+216 -40
View File
@@ -14,6 +14,10 @@ const { publishVersionForUser } = await import('./skills')
const { __handlers } = await import('./httpApi')
const { hashSkillFiles } = await import('./lib/skills')
function makeCtx(partial: Record<string, unknown>) {
return partial as unknown as import('./_generated/server').ActionCtx
}
describe('httpApi handlers', () => {
afterEach(() => {
vi.mocked(requireApiTokenUser).mockReset()
@@ -22,14 +26,14 @@ describe('httpApi handlers', () => {
it('searchSkillsHttp returns empty results for empty query', async () => {
const response = await __handlers.searchSkillsHandler(
{ runAction: vi.fn() },
makeCtx({ runAction: vi.fn() }),
new Request('https://example.com/api/search?q=%20%20'),
)
expect(response.status).toBe(200)
expect(await response.json()).toEqual({ results: [] })
})
it('searchSkillsHttp forwards args', async () => {
it('searchSkillsHttp forwards args (approvedOnly alias)', async () => {
const runAction = vi.fn().mockResolvedValue([
{
score: 1,
@@ -38,22 +42,48 @@ describe('httpApi handlers', () => {
},
])
const response = await __handlers.searchSkillsHandler(
{ runAction },
makeCtx({ runAction }),
new Request('https://example.com/api/search?q=test&approvedOnly=true&limit=5'),
)
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
limit: 5,
approvedOnly: true,
highlightedOnly: true,
})
expect(response.status).toBe(200)
const json = await response.json()
expect(json.results[0].slug).toBe('a')
})
it('searchSkillsHttp forwards highlightedOnly', async () => {
const runAction = vi.fn().mockResolvedValue([])
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request('https://example.com/api/search?q=test&highlightedOnly=true'),
)
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
limit: undefined,
highlightedOnly: true,
})
})
it('searchSkillsHttp omits highlightedOnly when approvedOnly is false', async () => {
const runAction = vi.fn().mockResolvedValue([])
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request('https://example.com/api/search?q=test&approvedOnly=false'),
)
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
limit: undefined,
highlightedOnly: undefined,
})
})
it('getSkillHttp validates slug', async () => {
const response = await __handlers.getSkillHandler(
{ runQuery: vi.fn() },
makeCtx({ runQuery: vi.fn() }),
new Request('https://example.com/api/skill'),
)
expect(response.status).toBe(400)
@@ -62,7 +92,7 @@ describe('httpApi handlers', () => {
it('getSkillHttp returns 404 when missing', async () => {
const runQuery = vi.fn().mockResolvedValue(null)
const response = await __handlers.getSkillHandler(
{ runQuery },
makeCtx({ runQuery }),
new Request('https://example.com/api/skill?slug=missing'),
)
expect(response.status).toBe(404)
@@ -75,7 +105,14 @@ describe('httpApi handlers', () => {
displayName: 'Demo',
summary: 'x',
tags: {},
stats: { downloads: 0, stars: 0, versions: 1, comments: 0 },
stats: {
downloads: 0,
installsCurrent: 0,
installsAllTime: 0,
stars: 0,
versions: 1,
comments: 0,
},
createdAt: 1,
updatedAt: 2,
},
@@ -83,7 +120,7 @@ describe('httpApi handlers', () => {
owner: { handle: 'p', displayName: 'Peter', image: null },
})
const response = await __handlers.getSkillHandler(
{ runQuery },
makeCtx({ runQuery }),
new Request('https://example.com/api/skill?slug=demo'),
)
expect(response.status).toBe(200)
@@ -93,9 +130,33 @@ describe('httpApi handlers', () => {
expect(json.owner.handle).toBe('p')
})
it('getSkillHttp returns payload with null owner/latestVersion', async () => {
const runQuery = vi.fn().mockResolvedValue({
skill: {
slug: 'demo',
displayName: 'Demo',
summary: null,
tags: {},
stats: {},
createdAt: 1,
updatedAt: 2,
},
latestVersion: null,
owner: null,
})
const response = await __handlers.getSkillHandler(
makeCtx({ runQuery }),
new Request('https://example.com/api/skill?slug=demo'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.latestVersion).toBeNull()
expect(json.owner).toBeNull()
})
it('resolveSkillVersionHttp validates hash', async () => {
const response = await __handlers.resolveSkillVersionHandler(
{ runQuery: vi.fn() },
makeCtx({ runQuery: vi.fn() }),
new Request('https://example.com/api/skill/resolve?slug=demo&hash=bad'),
)
expect(response.status).toBe(400)
@@ -104,7 +165,7 @@ describe('httpApi handlers', () => {
it('resolveSkillVersionHttp returns 404 when missing', async () => {
const runQuery = vi.fn().mockResolvedValue(null)
const response = await __handlers.resolveSkillVersionHandler(
{ runQuery },
makeCtx({ runQuery }),
new Request(
'https://example.com/api/skill/resolve?slug=missing&hash=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
),
@@ -114,25 +175,13 @@ describe('httpApi handlers', () => {
it('resolveSkillVersionHttp returns match and latestVersion', async () => {
const matchHash = await hashSkillFiles([{ path: 'SKILL.md', sha256: 'abc' }])
const runQuery = vi
.fn()
.mockResolvedValueOnce({
skill: {
_id: 's',
slug: 'demo',
displayName: 'Demo',
tags: {},
stats: {},
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '2.0.0', createdAt: 3, changelog: 'c' },
owner: null,
})
.mockResolvedValueOnce([{ version: '1.0.0', files: [{ path: 'SKILL.md', sha256: 'abc' }] }])
const runQuery = vi.fn().mockResolvedValueOnce({
match: { version: '1.0.0' },
latestVersion: { version: '2.0.0' },
})
const response = await __handlers.resolveSkillVersionHandler(
{ runQuery },
makeCtx({ runQuery }),
new Request(`https://example.com/api/skill/resolve?slug=demo&hash=${matchHash}`),
)
expect(response.status).toBe(200)
@@ -144,7 +193,7 @@ describe('httpApi handlers', () => {
it('cliWhoamiHttp returns 401 on auth failure', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const response = await __handlers.cliWhoamiHandler(
{} as unknown,
makeCtx({}),
new Request('https://x/api/cli/whoami'),
)
expect(response.status).toBe(401)
@@ -155,7 +204,7 @@ describe('httpApi handlers', () => {
user: { handle: 'p', displayName: 'Peter', image: 'x' },
} as never)
const response = await __handlers.cliWhoamiHandler(
{} as unknown,
makeCtx({}),
new Request('https://x/api/cli/whoami'),
)
expect(response.status).toBe(200)
@@ -163,11 +212,94 @@ describe('httpApi handlers', () => {
expect(json.user.handle).toBe('p')
})
it('cliTelemetrySyncHttp forwards roots and returns ok', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'users:1' } as never)
const runMutation = vi.fn().mockResolvedValue(null)
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
roots: [
{
rootId: 'abc',
label: '~/skills',
skills: [{ slug: 'weather', version: null }],
},
],
}),
}),
)
expect(response.status).toBe(200)
expect(await response.json()).toEqual({ ok: true })
expect(runMutation).toHaveBeenCalledTimes(1)
})
it('cliTelemetrySyncHttp returns 400 on invalid payload', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'users:1' } as never)
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation: vi.fn() }),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ roots: 'nope' }),
}),
)
expect(response.status).toBe(400)
})
it('cliTelemetrySyncHttp forwards skill versions when provided', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'users:1' } as never)
const runMutation = vi.fn().mockResolvedValue(null)
await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
roots: [
{
rootId: 'abc',
label: '~/skills',
skills: [{ slug: 'weather', version: '1.0.0' }],
},
],
}),
}),
)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'users:1',
roots: [
{ rootId: 'abc', label: '~/skills', skills: [{ slug: 'weather', version: '1.0.0' }] },
],
})
})
it('cliTelemetrySyncHttp returns 400 on invalid json', async () => {
const request = new Request('https://x/api/cli/telemetry/sync', { method: 'POST', body: '{' })
const response = await __handlers.cliTelemetrySyncHandler(makeCtx({}), request)
expect(response.status).toBe(400)
})
it('cliTelemetrySyncHttp returns 401 when unauthorized', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({}),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ roots: [] }),
}),
)
expect(response.status).toBe(401)
})
it('cliUploadUrlHttp returns uploadUrl', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue('https://upload.local')
const response = await __handlers.cliUploadUrlHandler(
{ runMutation } as unknown,
makeCtx({ runMutation }),
new Request('https://x/api/cli/upload-url', { method: 'POST' }),
)
expect(response.status).toBe(200)
@@ -177,7 +309,7 @@ describe('httpApi handlers', () => {
it('cliUploadUrlHttp returns 401 when unauthorized', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const response = await __handlers.cliUploadUrlHandler(
{} as unknown,
makeCtx({}),
new Request('https://x/api/cli/upload-url', { method: 'POST' }),
)
expect(response.status).toBe(401)
@@ -185,7 +317,7 @@ describe('httpApi handlers', () => {
it('cliPublishHttp returns 400 on invalid json', async () => {
const request = new Request('https://x/api/cli/publish', { method: 'POST', body: '{' })
const response = await __handlers.cliPublishHandler({} as unknown, request)
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(400)
})
@@ -196,7 +328,7 @@ describe('httpApi handlers', () => {
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({}),
})
const response = await __handlers.cliPublishHandler({} as unknown, request)
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(401)
})
@@ -214,7 +346,7 @@ describe('httpApi handlers', () => {
files: [{ path: 'SKILL.md', size: 1, storageId: 'id', sha256: 'a' }],
}),
})
const response = await __handlers.cliPublishHandler({} as unknown, request)
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(400)
})
@@ -236,7 +368,7 @@ describe('httpApi handlers', () => {
files: [{ path: 'SKILL.md', size: 1, storageId: 'id', sha256: 'a' }],
}),
})
const response = await __handlers.cliPublishHandler({} as unknown, request)
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.ok).toBe(true)
@@ -250,7 +382,7 @@ describe('httpApi handlers', () => {
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
})
const response = await __handlers.cliSkillDeleteHandler({} as never, request, true)
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true)
expect(response.status).toBe(401)
})
@@ -262,7 +394,7 @@ describe('httpApi handlers', () => {
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
})
const response = await __handlers.cliSkillDeleteHandler({ runMutation } as never, request, true)
const response = await __handlers.cliSkillDeleteHandler(makeCtx({ runMutation }), request, true)
expect(response.status).toBe(200)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'user1',
@@ -281,7 +413,7 @@ describe('httpApi handlers', () => {
body: JSON.stringify({ slug: 'demo' }),
})
const response = await __handlers.cliSkillDeleteHandler(
{ runMutation } as never,
makeCtx({ runMutation }),
request,
false,
)
@@ -293,9 +425,53 @@ describe('httpApi handlers', () => {
})
})
it('cliSkillUndeleteHttp calls delete handler with deleted=false', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/skill/undelete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
}),
false,
)
expect(response.status).toBe(200)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'user1',
slug: 'demo',
deleted: false,
})
warnSpy.mockRestore()
})
it('cliSkillDeleteHttp calls delete handler with deleted=true', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/skill/delete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
}),
true,
)
expect(response.status).toBe(200)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'user1',
slug: 'demo',
deleted: true,
})
warnSpy.mockRestore()
})
it('cliSkillDeleteHandler returns 400 on invalid json', async () => {
const request = new Request('https://x/api/cli/skill/delete', { method: 'POST', body: '{' })
const response = await __handlers.cliSkillDeleteHandler({} as never, request, true)
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true)
expect(response.status).toBe(400)
})
@@ -306,7 +482,7 @@ describe('httpApi handlers', () => {
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({}),
})
const response = await __handlers.cliSkillDeleteHandler({} as never, request, true)
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true)
expect(response.status).toBe(400)
})
})
+23
View File
@@ -25,6 +25,29 @@ describe('httpApi', () => {
expect(parsed.files[0]?.path).toBe('SKILL.md')
})
it('normalizes optional fields in publish payload', () => {
const parsed = __test.parsePublishBody({
slug: 'cool-skill',
displayName: 'Cool Skill',
version: '1.2.3',
changelog: '',
tags: [],
forkOf: { slug: 'base-skill' },
files: [
{
path: 'SKILL.md',
size: 5,
storageId: 'fakeStorageId',
sha256: 'abcd',
contentType: 'text/markdown',
},
],
})
expect(parsed.tags).toBeUndefined()
expect(parsed.source).toBeUndefined()
expect(parsed.forkOf).toEqual({ slug: 'base-skill', version: undefined })
})
it('rejects invalid publish payloads', () => {
expect(() => __test.parsePublishBody(null)).toThrow(/Publish payload/i)
expect(() =>
+57 -36
View File
@@ -1,22 +1,18 @@
import {
ApiCliSkillDeleteResponseSchema,
ApiCliTelemetrySyncResponseSchema,
CliPublishRequestSchema,
CliSkillDeleteRequestSchema,
CliTelemetrySyncRequestSchema,
parseArk,
} from 'clawdhub-schema'
} from 'clawhub-schema'
import { api, internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { httpAction } from './_generated/server'
import { requireApiTokenUser } from './lib/apiTokenAuth'
import { hashSkillFiles } from './lib/skills'
import { publishVersionForUser } from './skills'
type HttpCtx = {
runAction: (fn: unknown, args: unknown) => Promise<unknown>
runQuery: (fn: unknown, args: unknown) => Promise<unknown>
runMutation: (fn: unknown, args: unknown) => Promise<unknown>
}
type SearchSkillEntry = {
score: number
skill: {
@@ -43,18 +39,19 @@ type GetBySlugResult = {
owner: { handle?: string; displayName?: string; image?: string } | null
} | null
async function searchSkillsHandler(ctx: HttpCtx, request: Request) {
async function searchSkillsHandler(ctx: ActionCtx, request: Request) {
const url = new URL(request.url)
const query = url.searchParams.get('q')?.trim() ?? ''
const limit = toOptionalNumber(url.searchParams.get('limit'))
const approvedOnly = url.searchParams.get('approvedOnly') === 'true'
const highlightedOnly = url.searchParams.get('highlightedOnly') === 'true' || approvedOnly
if (!query) return json({ results: [] })
const results = (await ctx.runAction(api.search.searchSkills, {
query,
limit,
approvedOnly: approvedOnly || undefined,
highlightedOnly: highlightedOnly || undefined,
})) as SearchSkillEntry[]
return json({
@@ -71,7 +68,7 @@ async function searchSkillsHandler(ctx: HttpCtx, request: Request) {
export const searchSkillsHttp = httpAction(searchSkillsHandler)
async function getSkillHandler(ctx: HttpCtx, request: Request) {
async function getSkillHandler(ctx: ActionCtx, request: Request) {
const url = new URL(request.url)
const slug = url.searchParams.get('slug')?.trim().toLowerCase()
if (!slug) return text('Missing slug', 400)
@@ -108,39 +105,22 @@ async function getSkillHandler(ctx: HttpCtx, request: Request) {
export const getSkillHttp = httpAction(getSkillHandler)
async function resolveSkillVersionHandler(ctx: HttpCtx, request: Request) {
async function resolveSkillVersionHandler(ctx: ActionCtx, request: Request) {
const url = new URL(request.url)
const slug = url.searchParams.get('slug')?.trim().toLowerCase()
const hash = url.searchParams.get('hash')?.trim().toLowerCase()
if (!slug || !hash) return text('Missing slug or hash', 400)
if (!/^[a-f0-9]{64}$/.test(hash)) return text('Invalid hash', 400)
const result = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult
if (!result?.skill) return text('Skill not found', 404)
const resolved = await ctx.runQuery(api.skills.resolveVersionByHash, { slug, hash })
if (!resolved) return text('Skill not found', 404)
const versions = (await ctx.runQuery(api.skills.listVersions, {
skillId: result.skill._id,
limit: 200,
})) as Array<{ version: string; files: Array<{ path: string; sha256: string }> }>
let match: { version: string } | null = null
for (const version of versions) {
const fingerprint = await hashSkillFiles(version.files)
if (fingerprint === hash) {
match = { version: version.version }
break
}
}
return json({
slug,
match,
latestVersion: result.latestVersion ? { version: result.latestVersion.version } : null,
})
return json({ slug, match: resolved.match, latestVersion: resolved.latestVersion })
}
export const resolveSkillVersionHttp = httpAction(resolveSkillVersionHandler)
async function cliWhoamiHandler(ctx: HttpCtx, request: Request) {
async function cliWhoamiHandler(ctx: ActionCtx, request: Request) {
try {
const { user } = await requireApiTokenUser(ctx, request)
return json({
@@ -157,7 +137,7 @@ async function cliWhoamiHandler(ctx: HttpCtx, request: Request) {
export const cliWhoamiHttp = httpAction(cliWhoamiHandler)
async function cliUploadUrlHandler(ctx: HttpCtx, request: Request) {
async function cliUploadUrlHandler(ctx: ActionCtx, request: Request) {
try {
const { userId } = await requireApiTokenUser(ctx, request)
const uploadUrl = await ctx.runMutation(internal.uploads.generateUploadUrlForUserInternal, {
@@ -171,7 +151,7 @@ async function cliUploadUrlHandler(ctx: HttpCtx, request: Request) {
export const cliUploadUrlHttp = httpAction(cliUploadUrlHandler)
async function cliPublishHandler(ctx: HttpCtx, request: Request) {
async function cliPublishHandler(ctx: ActionCtx, request: Request) {
let body: unknown
try {
body = await request.json()
@@ -193,7 +173,7 @@ async function cliPublishHandler(ctx: HttpCtx, request: Request) {
export const cliPublishHttp = httpAction(cliPublishHandler)
async function cliSkillDeleteHandler(ctx: HttpCtx, request: Request, deleted: boolean) {
async function cliSkillDeleteHandler(ctx: ActionCtx, request: Request, deleted: boolean) {
let body: unknown
try {
body = await request.json()
@@ -225,6 +205,39 @@ export const cliSkillUndeleteHttp = httpAction((ctx, request) =>
cliSkillDeleteHandler(ctx, request, false),
)
async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
let body: unknown
try {
body = await request.json()
} catch {
return text('Invalid JSON', 400)
}
try {
const { userId } = await requireApiTokenUser(ctx, request)
const args = parseArk(CliTelemetrySyncRequestSchema, body, 'Telemetry payload')
await ctx.runMutation(internal.telemetry.reportCliSyncInternal, {
userId,
roots: args.roots.map((root) => ({
rootId: root.rootId,
label: root.label,
skills: root.skills.map((skill) => ({
slug: skill.slug,
version: skill.version ?? undefined,
})),
})),
})
const ok = parseArk(ApiCliTelemetrySyncResponseSchema, { ok: true }, 'Telemetry response')
return json(ok)
} catch (error) {
const message = error instanceof Error ? error.message : 'Telemetry failed'
if (message.toLowerCase().includes('unauthorized')) return text('Unauthorized', 401)
return text(message, 400)
}
}
export const cliTelemetrySyncHttp = httpAction(cliTelemetrySyncHandler)
function json(value: unknown, status = 200) {
return new Response(JSON.stringify(value), {
status,
@@ -261,6 +274,13 @@ function parsePublishBody(body: unknown) {
version: parsed.version,
changelog: parsed.changelog,
tags,
source: parsed.source ?? undefined,
forkOf: parsed.forkOf
? {
slug: parsed.forkOf.slug,
version: parsed.forkOf.version ?? undefined,
}
: undefined,
files: parsed.files.map((file) => ({
...file,
storageId: file.storageId as Id<'_storage'>,
@@ -281,4 +301,5 @@ export const __handlers = {
cliUploadUrlHandler,
cliPublishHandler,
cliSkillDeleteHandler,
cliTelemetrySyncHandler,
}
+584
View File
@@ -0,0 +1,584 @@
/* @vitest-environment node */
import { beforeEach, describe, expect, it, vi } from 'vitest'
vi.mock('./lib/apiTokenAuth', () => ({
requireApiTokenUser: vi.fn(),
}))
vi.mock('./skills', () => ({
publishVersionForUser: vi.fn(),
}))
const { requireApiTokenUser } = await import('./lib/apiTokenAuth')
const { publishVersionForUser } = await import('./skills')
const { __handlers } = await import('./httpApiV1')
type ActionCtx = import('./_generated/server').ActionCtx
function makeCtx(partial: Record<string, unknown>) {
return partial as unknown as ActionCtx
}
const okRate = () => ({
allowed: true,
remaining: 10,
limit: 100,
resetAt: Date.now() + 60_000,
})
const blockedRate = () => ({
allowed: false,
remaining: 0,
limit: 100,
resetAt: Date.now() + 60_000,
})
beforeEach(() => {
vi.mocked(requireApiTokenUser).mockReset()
vi.mocked(publishVersionForUser).mockReset()
})
describe('httpApiV1 handlers', () => {
it('search returns empty results for blank query', async () => {
const runAction = vi.fn()
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.searchSkillsV1Handler(
makeCtx({ runAction, runMutation }),
new Request('https://example.com/api/v1/search?q=%20%20'),
)
if (response.status !== 200) {
throw new Error(await response.text())
}
expect(await response.json()).toEqual({ results: [] })
expect(runAction).not.toHaveBeenCalled()
})
it('search forwards limit and highlightedOnly', async () => {
const runAction = vi.fn().mockResolvedValue([
{
score: 1,
skill: { slug: 'a', displayName: 'A', summary: null, updatedAt: 1 },
version: { version: '1.0.0' },
},
])
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.searchSkillsV1Handler(
makeCtx({ runAction, runMutation }),
new Request('https://example.com/api/v1/search?q=test&limit=5&highlightedOnly=true'),
)
if (response.status !== 200) {
throw new Error(await response.text())
}
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
limit: 5,
highlightedOnly: true,
})
})
it('search rate limits', async () => {
const runMutation = vi.fn().mockResolvedValue(blockedRate())
const response = await __handlers.searchSkillsV1Handler(
makeCtx({ runAction: vi.fn(), runMutation }),
new Request('https://example.com/api/v1/search?q=test'),
)
expect(response.status).toBe(429)
})
it('resolve validates hash', async () => {
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.resolveSkillVersionV1Handler(
makeCtx({ runQuery: vi.fn(), runMutation }),
new Request('https://example.com/api/v1/resolve?slug=demo&hash=bad'),
)
expect(response.status).toBe(400)
})
it('resolve returns 404 when missing', async () => {
const runQuery = vi.fn().mockResolvedValue(null)
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.resolveSkillVersionV1Handler(
makeCtx({ runQuery, runMutation }),
new Request(
'https://example.com/api/v1/resolve?slug=demo&hash=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
),
)
expect(response.status).toBe(404)
})
it('resolve returns match and latestVersion', async () => {
const runQuery = vi.fn().mockResolvedValue({
match: { version: '1.0.0' },
latestVersion: { version: '2.0.0' },
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.resolveSkillVersionV1Handler(
makeCtx({ runQuery, runMutation }),
new Request(
'https://example.com/api/v1/resolve?slug=demo&hash=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.match.version).toBe('1.0.0')
})
it('lists skills with resolved tags', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('cursor' in args || 'limit' in args) {
return {
items: [
{
skill: {
_id: 'skills:1',
slug: 'demo',
displayName: 'Demo',
summary: 's',
tags: { latest: 'versions:1' },
stats: { downloads: 0, stars: 0, versions: 1, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '1.0.0', createdAt: 3, changelog: 'c' },
},
],
nextCursor: null,
}
}
if ('versionId' in args) return { version: '1.0.0' }
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.listSkillsV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills?limit=1'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.items[0].tags.latest).toBe('1.0.0')
})
it('lists skills supports sort aliases', async () => {
const checks: Array<[string, string]> = [
['rating', 'stars'],
['installs', 'installsCurrent'],
['installs-all-time', 'installsAllTime'],
['trending', 'trending'],
]
for (const [input, expected] of checks) {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('sort' in args || 'cursor' in args || 'limit' in args) {
expect(args.sort).toBe(expected)
return { items: [], nextCursor: null }
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.listSkillsV1Handler(
makeCtx({ runQuery, runMutation }),
new Request(`https://example.com/api/v1/skills?sort=${input}`),
)
expect(response.status).toBe(200)
}
})
it('get skill returns 404 when missing', async () => {
const runQuery = vi.fn().mockResolvedValue(null)
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills/missing'),
)
expect(response.status).toBe(404)
})
it('get skill returns payload', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('slug' in args) {
return {
skill: {
_id: 'skills:1',
slug: 'demo',
displayName: 'Demo',
summary: 's',
tags: { latest: 'versions:1' },
stats: { downloads: 0, stars: 0, versions: 1, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: {
version: '1.0.0',
createdAt: 3,
changelog: 'c',
files: [],
},
owner: { handle: 'p', displayName: 'Peter', image: null },
}
}
if ('versionId' in args) return { version: '1.0.0' }
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills/demo'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.skill.slug).toBe('demo')
expect(json.latestVersion.version).toBe('1.0.0')
})
it('lists versions', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('slug' in args) {
return { _id: 'skills:1', slug: 'demo', displayName: 'Demo' }
}
if ('skillId' in args && 'cursor' in args) {
return {
items: [
{
version: '1.0.0',
createdAt: 1,
changelog: 'c',
changelogSource: 'user',
files: [],
},
],
nextCursor: null,
}
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills/demo/versions?limit=1'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.items[0].version).toBe('1.0.0')
})
it('returns version detail', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('slug' in args) {
return { _id: 'skills:1', slug: 'demo', displayName: 'Demo' }
}
if ('skillId' in args && 'version' in args) {
return {
version: '1.0.0',
createdAt: 1,
changelog: 'c',
changelogSource: 'auto',
files: [
{
path: 'SKILL.md',
size: 1,
storageId: 'storage:1',
sha256: 'abc',
contentType: 'text/plain',
},
],
}
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills/demo/versions/1.0.0'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.version.files[0].path).toBe('SKILL.md')
})
it('returns raw file content', async () => {
const version = {
version: '1.0.0',
createdAt: 1,
changelog: 'c',
files: [
{
path: 'SKILL.md',
size: 5,
storageId: 'storage:1',
sha256: 'abcd',
contentType: 'text/plain',
},
],
softDeletedAt: undefined,
}
const runQuery = vi.fn().mockResolvedValue({
skill: {
_id: 'skills:1',
slug: 'demo',
displayName: 'Demo',
summary: 's',
tags: {},
stats: {},
createdAt: 1,
updatedAt: 2,
},
latestVersion: version,
owner: null,
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const storage = {
get: vi.fn().mockResolvedValue(new Blob(['hello'], { type: 'text/plain' })),
}
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation, storage }),
new Request('https://example.com/api/v1/skills/demo/file?path=SKILL.md'),
)
expect(response.status).toBe(200)
expect(await response.text()).toBe('hello')
expect(response.headers.get('X-Content-SHA256')).toBe('abcd')
})
it('returns 413 when raw file too large', async () => {
const version = {
version: '1.0.0',
createdAt: 1,
changelog: 'c',
files: [
{
path: 'SKILL.md',
size: 210 * 1024,
storageId: 'storage:1',
sha256: 'abcd',
contentType: 'text/plain',
},
],
softDeletedAt: undefined,
}
const runQuery = vi.fn().mockResolvedValue({
skill: {
_id: 'skills:1',
slug: 'demo',
displayName: 'Demo',
summary: 's',
tags: {},
stats: {},
createdAt: 1,
updatedAt: 2,
},
latestVersion: version,
owner: null,
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation, storage: { get: vi.fn() } }),
new Request('https://example.com/api/v1/skills/demo/file?path=SKILL.md'),
)
expect(response.status).toBe(413)
})
it('publish json succeeds', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({
userId: 'users:1',
user: { handle: 'p' },
} as never)
vi.mocked(publishVersionForUser).mockResolvedValueOnce({
skillId: 's',
versionId: 'v',
embeddingId: 'e',
} as never)
const runMutation = vi.fn().mockResolvedValue(okRate())
const body = JSON.stringify({
slug: 'demo',
displayName: 'Demo',
version: '1.0.0',
changelog: 'c',
files: [
{
path: 'SKILL.md',
size: 1,
storageId: 'storage:1',
sha256: 'abc',
contentType: 'text/plain',
},
],
})
const response = await __handlers.publishSkillV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/skills', {
method: 'POST',
headers: { 'Content-Type': 'application/json', Authorization: 'Bearer clh_test' },
body,
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.ok).toBe(true)
expect(publishVersionForUser).toHaveBeenCalled()
})
it('publish multipart succeeds', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({
userId: 'users:1',
user: { handle: 'p' },
} as never)
vi.mocked(publishVersionForUser).mockResolvedValueOnce({
skillId: 's',
versionId: 'v',
embeddingId: 'e',
} as never)
const runMutation = vi.fn().mockResolvedValue(okRate())
const form = new FormData()
form.set(
'payload',
JSON.stringify({
slug: 'demo',
displayName: 'Demo',
version: '1.0.0',
changelog: '',
tags: ['latest'],
}),
)
form.append('files', new Blob(['hello'], { type: 'text/plain' }), 'SKILL.md')
const response = await __handlers.publishSkillV1Handler(
makeCtx({ runMutation, storage: { store: vi.fn().mockResolvedValue('storage:1') } }),
new Request('https://example.com/api/v1/skills', {
method: 'POST',
headers: { Authorization: 'Bearer clh_test' },
body: form,
}),
)
if (response.status !== 200) {
throw new Error(await response.text())
}
})
it('publish rejects missing token', async () => {
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.publishSkillV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/skills', { method: 'POST' }),
)
expect(response.status).toBe(401)
})
it('whoami returns user payload', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({
userId: 'users:1',
user: { handle: 'p', displayName: 'Peter', image: null },
} as never)
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.whoamiV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/whoami', {
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.user.handle).toBe('p')
})
it('delete and undelete require auth', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsDeleteRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/skills/demo', { method: 'DELETE' }),
)
expect(response.status).toBe(401)
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const response2 = await __handlers.skillsPostRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/skills/demo/undelete', { method: 'POST' }),
)
expect(response2.status).toBe(401)
})
it('delete and undelete succeed', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runMutation = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('key' in args) return okRate()
return { ok: true }
})
const response = await __handlers.skillsDeleteRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/skills/demo', {
method: 'DELETE',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(response.status).toBe(200)
const response2 = await __handlers.skillsPostRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/skills/demo/undelete', {
method: 'POST',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(response2.status).toBe(200)
})
it('stars require auth', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.starsPostRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/stars/demo', { method: 'POST' }),
)
expect(response.status).toBe(401)
})
it('stars add succeeds', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runQuery = vi.fn().mockResolvedValue({ _id: 'skills:1' })
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, starred: true, alreadyStarred: false })
const response = await __handlers.starsPostRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/stars/demo', {
method: 'POST',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.ok).toBe(true)
expect(json.starred).toBe(true)
})
it('stars delete succeeds', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runQuery = vi.fn().mockResolvedValue({ _id: 'skills:1' })
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, unstarred: true, alreadyUnstarred: false })
const response = await __handlers.starsDeleteRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/stars/demo', {
method: 'DELETE',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.ok).toBe(true)
expect(json.unstarred).toBe(true)
})
})
+1172
View File
File diff suppressed because it is too large Load Diff
+39
View File
@@ -0,0 +1,39 @@
import { v } from 'convex/values'
import { internalMutation } from './_generated/server'
import { buildTrendingLeaderboard } from './lib/leaderboards'
const MAX_TRENDING_LIMIT = 200
const KEEP_LEADERBOARD_ENTRIES = 3
export const rebuildTrendingLeaderboardInternal = internalMutation({
args: { limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = clampInt(args.limit ?? MAX_TRENDING_LIMIT, 1, MAX_TRENDING_LIMIT)
const now = Date.now()
const { startDay, endDay, items } = await buildTrendingLeaderboard(ctx, { limit, now })
await ctx.db.insert('skillLeaderboards', {
kind: 'trending',
generatedAt: now,
rangeStartDay: startDay,
rangeEndDay: endDay,
items,
})
const recent = await ctx.db
.query('skillLeaderboards')
.withIndex('by_kind', (q) => q.eq('kind', 'trending'))
.order('desc')
.take(KEEP_LEADERBOARD_ENTRIES + 5)
for (const entry of recent.slice(KEEP_LEADERBOARD_ENTRIES)) {
await ctx.db.delete(entry._id)
}
return { ok: true as const, count: items.length }
},
})
function clampInt(value: number, min: number, max: number) {
return Math.min(Math.max(value, min), max)
}
+10 -2
View File
@@ -1,5 +1,5 @@
import { getAuthUserId } from '@convex-dev/auth/server'
import { api } from '../_generated/api'
import { internal } from '../_generated/api'
import type { Doc } from '../_generated/dataModel'
import type { ActionCtx, MutationCtx, QueryCtx } from '../_generated/server'
@@ -16,7 +16,7 @@ export async function requireUser(ctx: MutationCtx | QueryCtx) {
export async function requireUserFromAction(ctx: ActionCtx) {
const userId = await getAuthUserId(ctx)
if (!userId) throw new Error('Unauthorized')
const user = await ctx.runQuery(api.users.getById, { userId })
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId })
if (!user || user.deletedAt) throw new Error('User not found')
return { userId, user: user as Doc<'users'> }
}
@@ -26,3 +26,11 @@ export function assertRole(user: Doc<'users'>, allowed: Role[]) {
throw new Error('Forbidden')
}
}
export function assertAdmin(user: Doc<'users'>) {
assertRole(user, ['admin'])
}
export function assertModerator(user: Doc<'users'>) {
assertRole(user, ['admin', 'moderator'])
}
+50
View File
@@ -0,0 +1,50 @@
import type { Doc, Id } from '../_generated/dataModel'
import type { QueryCtx } from '../_generated/server'
type BadgeKind = Doc<'skillBadges'>['kind']
export type SkillBadgeMap = Partial<Record<BadgeKind, { byUserId: Id<'users'>; at: number }>>
export type SkillBadgeSource = { badges?: SkillBadgeMap | null }
type BadgeCtx = Pick<QueryCtx, 'db'>
export function isSkillHighlighted(skill: SkillBadgeSource) {
return Boolean(skill.badges?.highlighted)
}
export function isSkillOfficial(skill: SkillBadgeSource) {
return Boolean(skill.badges?.official)
}
export function isSkillDeprecated(skill: SkillBadgeSource) {
return Boolean(skill.badges?.deprecated)
}
export function buildBadgeMap(records: Doc<'skillBadges'>[]): SkillBadgeMap {
return records.reduce<SkillBadgeMap>((acc, record) => {
acc[record.kind] = { byUserId: record.byUserId, at: record.at }
return acc
}, {})
}
export async function getSkillBadgeMap(
ctx: BadgeCtx,
skillId: Id<'skills'>,
): Promise<SkillBadgeMap> {
const records = await ctx.db
.query('skillBadges')
.withIndex('by_skill', (q) => q.eq('skillId', skillId))
.collect()
return buildBadgeMap(records)
}
export async function getSkillBadgeMaps(
ctx: BadgeCtx,
skillIds: Array<Id<'skills'>>,
): Promise<Map<Id<'skills'>, SkillBadgeMap>> {
const entries = await Promise.all(
skillIds.map(async (skillId) => [skillId, await getSkillBadgeMap(ctx, skillId)] as const),
)
return new Map(entries)
}
+34
View File
@@ -0,0 +1,34 @@
import { describe, expect, it } from 'vitest'
import { __test } from './changelog'
describe('changelog utils', () => {
it('summarizes file diffs', () => {
const diff = __test.summarizeFileDiff(
[
{ path: 'a.txt', sha256: 'aaa' },
{ path: 'b.txt', sha256: 'bbb' },
],
[
{ path: 'a.txt', sha256: 'aaa' },
{ path: 'b.txt', sha256: 'ccc' },
{ path: 'c.txt', sha256: 'ddd' },
],
)
expect(diff.added).toEqual(['c.txt'])
expect(diff.removed).toEqual([])
expect(diff.changed).toEqual(['b.txt'])
expect(__test.formatDiffSummary(diff)).toBe('1 added, 1 changed')
})
it('generates a fallback initial release note', () => {
const text = __test.generateFallback({
slug: 'demo',
version: '1.0.0',
oldReadme: null,
nextReadme: 'hi',
fileDiff: null,
})
expect(text).toMatch(/Initial release/i)
})
})
+278
View File
@@ -0,0 +1,278 @@
import { internal } from '../_generated/api'
import type { Doc, Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? 'gpt-4.1'
const MAX_README_CHARS = 8_000
const MAX_PATHS_IN_PROMPT = 30
type FileMeta = { path: string; sha256?: string }
type FileDiffSummary = {
added: string[]
removed: string[]
changed: string[]
}
function clampText(value: string, maxChars: number) {
const trimmed = value.trim()
if (trimmed.length <= maxChars) return trimmed
return `${trimmed.slice(0, maxChars).trimEnd()}\n…`
}
function summarizeFileDiff(oldFiles: FileMeta[], nextFiles: FileMeta[]): FileDiffSummary {
const oldByPath = new Map(oldFiles.map((f) => [f.path, f] as const))
const nextByPath = new Map(nextFiles.map((f) => [f.path, f] as const))
const added: string[] = []
const removed: string[] = []
const changed: string[] = []
for (const [path, file] of nextByPath.entries()) {
const prev = oldByPath.get(path)
if (!prev) {
added.push(path)
continue
}
if (file.sha256 && prev.sha256 && file.sha256 !== prev.sha256) changed.push(path)
}
for (const path of oldByPath.keys()) {
if (!nextByPath.has(path)) removed.push(path)
}
added.sort()
removed.sort()
changed.sort()
return { added, removed, changed }
}
function formatDiffSummary(diff: FileDiffSummary) {
const parts: string[] = []
if (diff.added.length) parts.push(`${diff.added.length} added`)
if (diff.changed.length) parts.push(`${diff.changed.length} changed`)
if (diff.removed.length) parts.push(`${diff.removed.length} removed`)
return parts.join(', ') || 'no file changes detected'
}
function pickPaths(values: string[]) {
if (values.length <= MAX_PATHS_IN_PROMPT) return values
return values.slice(0, MAX_PATHS_IN_PROMPT)
}
function extractResponseText(payload: unknown) {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
async function generateWithOpenAI(args: {
slug: string
version: string
oldReadme: string | null
nextReadme: string
fileDiff: FileDiffSummary | null
}) {
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) return null
const oldReadme = args.oldReadme ? clampText(args.oldReadme, MAX_README_CHARS) : ''
const nextReadme = clampText(args.nextReadme, MAX_README_CHARS)
const fileDiff = args.fileDiff
const diffSummary = fileDiff ? formatDiffSummary(fileDiff) : 'unknown'
const changedPaths = fileDiff ? pickPaths(fileDiff.changed) : []
const addedPaths = fileDiff ? pickPaths(fileDiff.added) : []
const removedPaths = fileDiff ? pickPaths(fileDiff.removed) : []
const input = [
`Skill: ${args.slug}`,
`Version: ${args.version}`,
`File changes: ${diffSummary}`,
changedPaths.length ? `Changed files (sample): ${changedPaths.join(', ')}` : null,
addedPaths.length ? `Added files (sample): ${addedPaths.join(', ')}` : null,
removedPaths.length ? `Removed files (sample): ${removedPaths.join(', ')}` : null,
oldReadme ? `Previous SKILL.md:\n${oldReadme}` : null,
`New SKILL.md:\n${nextReadme}`,
]
.filter(Boolean)
.join('\n\n')
const response = await fetch('https://api.openai.com/v1/responses', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: CHANGELOG_MODEL,
instructions:
'Write a concise changelog for this skill version. Audience: everyone. Output plain text. Prefer 26 bullet points. If it is a big change, include a short 1-line summary first, then bullets. Dont mention that you are AI. Dont invent details; only use the inputs.',
input,
max_output_tokens: 220,
}),
})
if (!response.ok) return null
const payload = (await response.json()) as unknown
return extractResponseText(payload)
}
function generateFallback(args: {
slug: string
version: string
oldReadme: string | null
nextReadme: string
fileDiff: FileDiffSummary | null
}) {
const lines: string[] = []
if (!args.oldReadme) {
lines.push(`- Initial release.`)
return lines.join('\n')
}
const diff = args.fileDiff
if (diff) {
const parts: string[] = []
if (diff.added.length) parts.push(`added ${diff.added.length}`)
if (diff.changed.length) parts.push(`updated ${diff.changed.length}`)
if (diff.removed.length) parts.push(`removed ${diff.removed.length}`)
if (parts.length) lines.push(`- ${parts.join(', ')} file(s).`)
}
lines.push(`- Updated SKILL.md and bundle contents.`)
return lines.join('\n')
}
export async function generateChangelogForPublish(
ctx: ActionCtx,
args: { slug: string; version: string; readmeText: string; files: FileMeta[] },
): Promise<string> {
try {
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: args.slug,
})) as Doc<'skills'> | null
const previous: Doc<'skillVersions'> | null =
skill?.latestVersionId && !skill.softDeletedAt
? ((await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: skill.latestVersionId,
})) as Doc<'skillVersions'> | null)
: null
const oldReadmeText: string | null = previous
? await readReadmeFromVersion(ctx, previous)
: null
const oldFiles = previous
? previous.files.map((file) => ({ path: file.path, sha256: file.sha256 }))
: []
const fileDiff = previous ? summarizeFileDiff(oldFiles, args.files) : null
const ai = await generateWithOpenAI({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
}).catch(() => null)
return (
ai ??
generateFallback({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
})
)
} catch {
return '- Updated skill.'
}
}
export async function generateChangelogPreview(
ctx: ActionCtx,
args: {
slug: string
version: string
readmeText: string
filePaths?: string[]
},
): Promise<string> {
try {
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: args.slug,
})) as Doc<'skills'> | null
const previous: Doc<'skillVersions'> | null =
skill?.latestVersionId && !skill.softDeletedAt
? ((await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: skill.latestVersionId,
})) as Doc<'skillVersions'> | null)
: null
const oldReadmeText: string | null = previous
? await readReadmeFromVersion(ctx, previous)
: null
const fileDiff =
previous && args.filePaths
? summarizeFileDiff(
previous.files.map((file) => ({ path: file.path, sha256: file.sha256 })),
args.filePaths.map((path) => ({ path })),
)
: null
const ai = await generateWithOpenAI({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
}).catch(() => null)
return (
ai ??
generateFallback({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
})
)
} catch {
return '- Updated skill.'
}
}
async function readReadmeFromVersion(ctx: ActionCtx, version: Doc<'skillVersions'>) {
const readmeFile = version.files.find((file) => {
const lower = file.path.toLowerCase()
return lower === 'skill.md' || lower === 'skills.md'
})
if (!readmeFile) return null
const blob = await ctx.storage.get(readmeFile.storageId as Id<'_storage'>)
if (!blob) return null
return blob.text()
}
export const __test = {
clampText,
extractResponseText,
formatDiffSummary,
summarizeFileDiff,
generateFallback,
}
+8 -1
View File
@@ -1,9 +1,16 @@
export const EMBEDDING_MODEL = 'text-embedding-3-small'
export const EMBEDDING_DIMENSIONS = 1536
function emptyEmbedding() {
return Array.from({ length: EMBEDDING_DIMENSIONS }, () => 0)
}
export async function generateEmbedding(text: string) {
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) throw new Error('OPENAI_API_KEY is not configured')
if (!apiKey) {
console.warn('OPENAI_API_KEY is not configured; using zero embeddings')
return emptyEmbedding()
}
const response = await fetch('https://api.openai.com/v1/embeddings', {
method: 'POST',
+443
View File
@@ -0,0 +1,443 @@
'use node'
import { createPrivateKey, createSign } from 'node:crypto'
import type { Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
const GITHUB_API = 'https://api.github.com'
const DEFAULT_REPO = 'clawdbot/skills'
const DEFAULT_ROOT = 'skills'
const META_FILENAME = '_meta.json'
const USER_AGENT = 'clawhub/skills-backup'
type BackupFile = {
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}
type BackupParams = {
slug: string
version: string
displayName: string
ownerHandle: string
files: BackupFile[]
publishedAt: number
}
type RepoInfo = {
default_branch?: string
}
type GitRef = {
object: { sha: string }
}
type GitCommit = {
sha: string
tree: { sha: string }
}
type GitTreeEntry = {
path?: string
type?: string
}
type GitTree = {
tree?: GitTreeEntry[]
}
type MetaFile = {
owner: string
slug: string
displayName: string
latest: {
version: string
publishedAt: number
commit: string | null
}
history: Array<{
version: string
publishedAt: number
commit: string
}>
}
export type GitHubBackupContext = {
token: string
repo: string
repoOwner: string
repoName: string
branch: string
root: string
}
export function isGitHubBackupConfigured() {
return Boolean(
process.env.GITHUB_APP_ID &&
process.env.GITHUB_APP_PRIVATE_KEY &&
process.env.GITHUB_APP_INSTALLATION_ID,
)
}
export async function getGitHubBackupContext(): Promise<GitHubBackupContext> {
const repo = process.env.GITHUB_SKILLS_REPO ?? DEFAULT_REPO
const root = process.env.GITHUB_SKILLS_ROOT ?? DEFAULT_ROOT
const [repoOwner, repoName] = parseRepo(repo)
const token = await createInstallationToken()
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`)
const branch = repoInfo.default_branch ?? 'main'
return { token, repo, repoOwner, repoName, branch, root }
}
export async function fetchGitHubSkillMeta(
context: GitHubBackupContext,
ownerHandle: string,
slug: string,
): Promise<MetaFile | null> {
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug)
return fetchMetaFile(
context.token,
context.repoOwner,
context.repoName,
`${skillRoot}/${META_FILENAME}`,
context.branch,
)
}
export async function backupSkillToGitHub(
ctx: ActionCtx,
params: BackupParams,
context?: GitHubBackupContext,
) {
if (!isGitHubBackupConfigured()) return
const resolved = context ?? (await getGitHubBackupContext())
const skillRoot = buildSkillRoot(resolved.root, params.ownerHandle, params.slug)
const ref = await githubGet<GitRef>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/ref/heads/${resolved.branch}`,
)
const baseCommitSha = ref.object.sha
const baseCommit = await githubGet<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits/${baseCommitSha}`,
)
const baseTreeSha = baseCommit.tree.sha
const existingTree = await githubGet<GitTree>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees/${baseTreeSha}?recursive=1`,
)
const prefix = `${skillRoot}/`
const existingPaths = new Set(
(existingTree.tree ?? [])
.filter((entry) => entry.type === 'blob' && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? ''),
)
const newPaths = new Set<string>()
const treeEntries: Array<{
path: string
mode: '100644'
type: 'blob'
sha: string | null
}> = []
for (const file of params.files) {
const content = await fetchStorageBase64(ctx, file.storageId)
const blobSha = await createBlob(resolved.token, resolved.repoOwner, resolved.repoName, content)
const path = `${skillRoot}/${file.path}`
newPaths.add(path)
treeEntries.push({ path, mode: '100644', type: 'blob', sha: blobSha })
}
const existingMeta = await fetchMetaFile(
resolved.token,
resolved.repoOwner,
resolved.repoName,
`${skillRoot}/${META_FILENAME}`,
resolved.branch,
)
const metaPath = `${skillRoot}/${META_FILENAME}`
const metaDraft = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, null)
const metaDraftContent = `${JSON.stringify(metaDraft, null, 2)}\n`
const metaDraftSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaDraftContent),
)
newPaths.add(metaPath)
treeEntries.push({ path: metaPath, mode: '100644', type: 'blob', sha: metaDraftSha })
for (const path of existingPaths) {
if (newPaths.has(path)) continue
treeEntries.push({ path, mode: '100644', type: 'blob', sha: null })
}
const newTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: baseTreeSha,
tree: treeEntries,
},
)
const commit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `skill: ${params.slug} v${params.version}`,
tree: newTree.sha,
parents: [baseCommitSha],
},
)
const metaFinal = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, commit.sha)
const metaFinalContent = `${JSON.stringify(metaFinal, null, 2)}\n`
const metaFinalSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaFinalContent),
)
const metaTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: commit.tree.sha,
tree: [{ path: metaPath, mode: '100644', type: 'blob', sha: metaFinalSha }],
},
)
const metaCommit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `meta: ${params.slug} v${params.version}`,
tree: metaTree.sha,
parents: [commit.sha],
},
)
await githubPatch(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/refs/heads/${resolved.branch}`,
{
sha: metaCommit.sha,
},
)
}
function buildMetaFile(
params: BackupParams,
existing: MetaFile | null,
repo: string,
baseCommitSha: string,
latestCommitSha: string | null,
): MetaFile {
let history = [...(existing?.history ?? [])]
if (existing?.latest?.version) {
const previousCommit = existing.latest.commit ?? commitUrl(repo, baseCommitSha)
const previous = {
version: existing.latest.version,
publishedAt: existing.latest.publishedAt,
commit: previousCommit,
}
history = [previous, ...history.filter((entry) => entry.version !== previous.version)]
}
return {
owner: normalizeOwner(params.ownerHandle),
slug: params.slug,
displayName: params.displayName,
latest: {
version: params.version,
publishedAt: params.publishedAt,
commit: latestCommitSha ? commitUrl(repo, latestCommitSha) : null,
},
history: history.slice(0, 200),
}
}
async function fetchMetaFile(
token: string,
repoOwner: string,
repoName: string,
path: string,
branch: string,
): Promise<MetaFile | null> {
try {
const response = await githubGet<{ content?: string }>(
token,
`/repos/${repoOwner}/${repoName}/contents/${encodePath(path)}?ref=${branch}`,
)
if (!response.content) return null
const raw = fromBase64(response.content)
return JSON.parse(raw) as MetaFile
} catch (error) {
if (isNotFoundError(error)) return null
throw error
}
}
async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<'_storage'>) {
const blob = await ctx.storage.get(storageId)
if (!blob) throw new Error('File missing in storage')
const buffer = Buffer.from(await blob.arrayBuffer())
return buffer.toString('base64')
}
async function createInstallationToken() {
const appId = process.env.GITHUB_APP_ID
const installationId = process.env.GITHUB_APP_INSTALLATION_ID
if (!appId || !installationId) {
throw new Error('GitHub App credentials missing')
}
const jwt = createAppJwt(appId)
const response = await fetch(`${GITHUB_API}/app/installations/${installationId}/access_tokens`, {
method: 'POST',
headers: buildHeaders(jwt, true),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub App token failed: ${message}`)
}
const payload = (await response.json()) as { token?: string }
if (!payload.token) throw new Error('GitHub App token missing')
return payload.token
}
function createAppJwt(appId: string) {
const privateKey = loadPrivateKey()
const now = Math.floor(Date.now() / 1000)
const header = { alg: 'RS256', typ: 'JWT' }
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId }
const encodedHeader = base64Url(JSON.stringify(header))
const encodedPayload = base64Url(JSON.stringify(payload))
const signingInput = `${encodedHeader}.${encodedPayload}`
const sign = createSign('RSA-SHA256')
sign.update(signingInput)
sign.end()
const signature = sign.sign(privateKey)
return `${signingInput}.${base64Url(signature)}`
}
function loadPrivateKey() {
const raw = process.env.GITHUB_APP_PRIVATE_KEY
if (!raw) throw new Error('GITHUB_APP_PRIVATE_KEY is not configured')
const normalized = raw.replace(/\\n/g, '\n')
return createPrivateKey(normalized)
}
async function createBlob(token: string, repoOwner: string, repoName: string, content: string) {
const result = await githubPost<{ sha: string }>(
token,
`/repos/${repoOwner}/${repoName}/git/blobs`,
{
content,
encoding: 'base64',
},
)
if (!result.sha) throw new Error('GitHub blob missing sha')
return result.sha
}
async function githubGet<T>(token: string, path: string): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
headers: buildHeaders(token),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub GET ${path} failed: ${message}`)
}
return (await response.json()) as T
}
async function githubPost<T>(token: string, path: string, body: unknown): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'POST',
headers: buildHeaders(token),
body: JSON.stringify(body),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub POST ${path} failed: ${message}`)
}
return (await response.json()) as T
}
async function githubPatch(token: string, path: string, body: unknown) {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'PATCH',
headers: buildHeaders(token),
body: JSON.stringify(body),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub PATCH ${path} failed: ${message}`)
}
}
function buildHeaders(token: string, isAppJwt = false) {
return {
Authorization: `${isAppJwt ? 'Bearer' : 'token'} ${token}`,
Accept: 'application/vnd.github+json',
'User-Agent': USER_AGENT,
}
}
function parseRepo(repo: string) {
const [owner, name] = repo.split('/')
if (!owner || !name) throw new Error('GITHUB_SKILLS_REPO must be owner/repo')
return [owner, name] as const
}
function normalizeOwner(value: string) {
const normalized = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]/g, '-')
.replace(/-+/g, '-')
.replace(/^-+|-+$/g, '')
return normalized || 'unknown'
}
function commitUrl(repo: string, sha: string) {
return `https://github.com/${repo}/commit/${sha}`
}
function buildSkillRoot(root: string, ownerHandle: string, slug: string) {
const ownerSegment = normalizeOwner(ownerHandle)
return `${root}/${ownerSegment}/${slug}`
}
function encodePath(path: string) {
return path
.split('/')
.map((segment) => encodeURIComponent(segment))
.join('/')
}
function base64Url(value: string | Buffer) {
const buffer = typeof value === 'string' ? Buffer.from(value) : value
return buffer.toString('base64').replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/g, '')
}
function toBase64(value: string) {
return Buffer.from(value).toString('base64')
}
function fromBase64(value: string) {
return Buffer.from(value, 'base64').toString('utf8')
}
function isNotFoundError(error: unknown) {
return (
error instanceof Error && (error.message.includes('404') || error.message.includes('Not Found'))
)
}
+247
View File
@@ -0,0 +1,247 @@
/* @vitest-environment node */
import { unzipSync } from 'fflate'
import { describe, expect, it } from 'vitest'
import {
buildGitHubZipForTests,
computeDefaultSelectedPaths,
detectGitHubImportCandidates,
extractMarkdownRelativeTargets,
fetchGitHubZipBytes,
parseGitHubImportUrl,
resolveGitHubCommit,
resolveMarkdownTarget,
stripGitHubZipRoot,
} from './githubImport'
function requestInfoToUrlString(input: RequestInfo | URL): string {
if (typeof input === 'string') return input
if (input instanceof URL) return input.toString()
if (input instanceof Request) return input.url
throw new Error('Unexpected fetch input type')
}
describe('github import', () => {
it('parses repo root urls', () => {
expect(parseGitHubImportUrl('https://github.com/visionik/ouracli')).toEqual({
owner: 'visionik',
repo: 'ouracli',
originalUrl: 'https://github.com/visionik/ouracli',
})
})
it('rejects non-https and non-github urls', () => {
expect(() => parseGitHubImportUrl('http://github.com/a/b')).toThrow(/https/i)
expect(() => parseGitHubImportUrl('https://example.com/a/b')).toThrow(/github\.com/i)
expect(() => parseGitHubImportUrl('not-a-url')).toThrow(/Invalid URL/i)
})
it('rejects malformed tree/blob urls', () => {
expect(() => parseGitHubImportUrl('https://github.com/a/b/tree/')).toThrow(/Missing ref/i)
expect(() => parseGitHubImportUrl('https://github.com/a/b/blob/main')).toThrow(/Missing path/i)
expect(() => parseGitHubImportUrl('https://github.com/a/b/tree/main/bad%5cpath')).toThrow()
})
it('parses tree urls with ref and path', () => {
expect(parseGitHubImportUrl('https://github.com/a/b/tree/main/skills/foo')).toEqual({
owner: 'a',
repo: 'b',
ref: 'main',
path: 'skills/foo',
originalUrl: 'https://github.com/a/b/tree/main/skills/foo',
})
})
it('parses blob urls and derives folder path', () => {
expect(parseGitHubImportUrl('https://github.com/a/b/blob/main/skills/foo/SKILL.md')).toEqual({
owner: 'a',
repo: 'b',
ref: 'main',
path: 'skills/foo',
originalUrl: 'https://github.com/a/b/blob/main/skills/foo/SKILL.md',
})
})
it('strips single top-level folder from GitHub zip entries', () => {
const zip = buildGitHubZipForTests({
'repo-1/skill/SKILL.md': 'Body',
'repo-1/skill/a.txt': 'a',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
expect(Object.keys(stripped).sort()).toEqual(['skill/SKILL.md', 'skill/a.txt'])
})
it('keeps paths when zip has multiple top-level roots', () => {
const zip = buildGitHubZipForTests({
'a/SKILL.md': 'Body',
'b/SKILL.md': 'Body',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
expect(Object.keys(stripped).sort()).toEqual(['a/SKILL.md', 'b/SKILL.md'])
})
it('detects candidates in a GitHub zip and strips the root folder', () => {
const zip = buildGitHubZipForTests({
'ouracli-123/SKILL.md': `---\nname: demo\ndescription: Hello\n---\nBody`,
'ouracli-123/src/index.ts': 'export {}',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
const candidates = detectGitHubImportCandidates(stripped)
expect(candidates.map((c) => c.path)).toEqual([''])
expect(candidates[0]?.name).toBe('demo')
})
it('detects multiple candidates and supports skills.md', () => {
const zip = buildGitHubZipForTests({
'repo-1/alpha/SKILL.md': `---\nname: Alpha\n---\nBody`,
'repo-1/beta/skills.md': `---\nname: Beta\n---\nBody`,
'repo-1/readme.md': 'x',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
const candidates = detectGitHubImportCandidates(stripped)
expect(candidates.map((c) => c.path)).toEqual(['alpha', 'beta'])
expect(candidates.map((c) => c.name)).toEqual(['Alpha', 'Beta'])
})
it('computes default selection via markdown references', () => {
const entries = {
'skill/SKILL.md': `---\nname: demo\n---\nSee [usage](docs/usage.md) and ![logo](img/logo.svg).\nIgnore [web](https://example.com).`,
'skill/docs/usage.md': `See [more](more.md)`,
'skill/docs/more.md': `Ok`,
'skill/img/logo.svg': `<svg/>`,
'skill/extra.txt': 'not referenced',
}
const zip = buildGitHubZipForTests(
Object.fromEntries(Object.entries(entries).map(([k, v]) => [`repo-1/${k}`, v])),
)
const raw = unzipSync(zip)
const stripped = stripGitHubZipRoot(raw)
const candidates = detectGitHubImportCandidates(stripped)
const candidate = candidates.find((c) => c.path === 'skill')
expect(candidate).toBeTruthy()
if (!candidate) throw new Error('candidate not found')
const files = Object.entries(stripped)
.filter(([path]) => path.startsWith('skill/'))
.map(([path, bytes]) => ({ path, bytes }))
const selected = computeDefaultSelectedPaths({ candidate, files })
expect(selected).toContain('skill/SKILL.md')
expect(selected).toContain('skill/docs/usage.md')
expect(selected).toContain('skill/docs/more.md')
expect(selected).toContain('skill/img/logo.svg')
expect(selected).not.toContain('skill/extra.txt')
})
it('does not select files outside skill folder (even when referenced)', () => {
const entries = {
'skill/SKILL.md': `See [outside](../outside.md) and [abs](/abs.md) and [mail](mailto:test@example.com).`,
'outside.md': `secret`,
'skill/docs/usage.md': `Ok`,
}
const zip = buildGitHubZipForTests(
Object.fromEntries(Object.entries(entries).map(([k, v]) => [`repo-1/${k}`, v])),
)
const stripped = stripGitHubZipRoot(unzipSync(zip))
const candidate = detectGitHubImportCandidates(stripped).find((c) => c.path === 'skill')
expect(candidate).toBeTruthy()
if (!candidate) throw new Error('candidate not found')
const files = Object.entries(stripped).map(([path, bytes]) => ({ path, bytes }))
const selected = computeDefaultSelectedPaths({ candidate, files })
expect(selected).toContain('skill/SKILL.md')
expect(selected).not.toContain('outside.md')
})
it('extracts markdown targets with titles and angle brackets', () => {
const targets = extractMarkdownRelativeTargets(
`See [a](docs/usage.md "Title") and [b](<docs/my file.md>) and ![c](img/logo.svg)`,
)
expect(targets).toEqual(['docs/usage.md', 'docs/my file.md', 'img/logo.svg'])
})
it('resolves markdown targets safely', () => {
expect(resolveMarkdownTarget('a/SKILL.md', 'docs/usage.md')).toBe('a/docs/usage.md')
expect(resolveMarkdownTarget('a/SKILL.md', '../oops.md')).toBeNull()
expect(resolveMarkdownTarget('a/SKILL.md', '/abs.md')).toBeNull()
expect(resolveMarkdownTarget('a/SKILL.md', 'docs/usage.md#section')).toBe('a/docs/usage.md')
expect(resolveMarkdownTarget('a/SKILL.md', 'docs/usage.md?x=1')).toBe('a/docs/usage.md')
})
it('resolves HEAD commit via redirect chain and refuses unexpected redirect hosts', async () => {
const fetcher: typeof fetch = async (input) => {
const url = requestInfoToUrlString(input)
if (url.includes('/archive/HEAD.zip')) {
return new Response(null, {
status: 302,
headers: {
location:
'https://codeload.github.com/a/b/zip/0123456789012345678901234567890123456789',
},
})
}
if (url.startsWith('https://codeload.github.com/a/b/zip/')) {
return new Response(null, { status: 200 })
}
throw new Error(`Unexpected fetch: ${url}`)
}
const resolved = await resolveGitHubCommit(
{ owner: 'a', repo: 'b', originalUrl: 'https://github.com/a/b' },
fetcher,
)
expect(resolved.commit).toBe('0123456789012345678901234567890123456789')
const badFetcher: typeof fetch = async (input) => {
const url = requestInfoToUrlString(input)
if (url.includes('/archive/HEAD.zip')) {
return new Response(null, {
status: 302,
headers: { location: 'https://evil.example/zip/abc' },
})
}
throw new Error(`Unexpected fetch: ${url}`)
}
await expect(
resolveGitHubCommit(
{ owner: 'a', repo: 'b', originalUrl: 'https://github.com/a/b' },
badFetcher,
),
).rejects.toThrow(/redirect/i)
})
it('resolves explicit ref commit via GitHub API', async () => {
const fetcher: typeof fetch = async (input) => {
const url = requestInfoToUrlString(input)
if (url.startsWith('https://api.github.com/repos/a/b/commits/')) {
return new Response(JSON.stringify({ sha: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }), {
status: 200,
})
}
throw new Error(`Unexpected fetch: ${url}`)
}
const resolved = await resolveGitHubCommit(
{ owner: 'a', repo: 'b', ref: 'main', originalUrl: 'https://github.com/a/b' },
fetcher,
)
expect(resolved.commit).toBe('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')
})
it('enforces zip byte cap when content-length is too large', async () => {
const resolved = {
owner: 'a',
repo: 'b',
ref: 'main',
commit: '0123456789012345678901234567890123456789',
path: '',
repoUrl: 'https://github.com/a/b',
originalUrl: 'https://github.com/a/b',
} as const
const fetcher: typeof fetch = async () =>
new Response(new Blob([new Uint8Array([1, 2, 3])]), {
status: 200,
headers: { 'content-length': String(999_999_999) },
})
await expect(fetchGitHubZipBytes(resolved, fetcher, { maxZipBytes: 10 })).rejects.toThrow(
/too large/i,
)
})
})
+425
View File
@@ -0,0 +1,425 @@
import { TEXT_FILE_EXTENSION_SET } from 'clawhub-schema'
import { zipSync } from 'fflate'
import semver from 'semver'
import { parseFrontmatter } from './skills'
export type GitHubImportUrl = {
owner: string
repo: string
ref?: string
path?: string
originalUrl: string
}
export type GitHubImportResolved = {
owner: string
repo: string
ref: string
commit: string
path: string
repoUrl: string
originalUrl: string
}
export type GitHubImportCandidate = {
path: string
readmePath: string
name?: string
description?: string
}
export type GitHubImportFileEntry = {
path: string
size: number
defaultSelected: boolean
}
const MAX_REDIRECTS = 6
const GITHUB_HOST = 'github.com'
const CODELOAD_HOST = 'codeload.github.com'
const SKILL_FILENAMES = ['skill.md', 'skills.md']
export function parseGitHubImportUrl(input: string): GitHubImportUrl {
const originalUrl = input.trim()
let url: URL
try {
url = new URL(originalUrl)
} catch {
throw new Error('Invalid URL')
}
if (url.protocol !== 'https:') throw new Error('Only https:// URLs are supported')
if (url.hostname !== GITHUB_HOST) throw new Error('Only github.com URLs are supported')
const segments = url.pathname
.split('/')
.map((segment) => segment.trim())
.filter(Boolean)
.map((segment) => {
try {
return decodeURIComponent(segment)
} catch {
throw new Error('Invalid URL')
}
})
const owner = segments[0] ?? ''
const repo = (segments[1] ?? '').replace(/\.git$/, '')
if (!owner || !repo) throw new Error('GitHub URL must be /<owner>/<repo>')
const kind = segments[2] ?? ''
if (!kind) return { owner, repo, originalUrl }
if (kind !== 'tree' && kind !== 'blob') {
return { owner, repo, originalUrl }
}
const ref = segments[3] ?? ''
if (!ref) throw new Error('Missing ref in GitHub URL')
const rest = segments.slice(4).join('/')
const normalizedRest = normalizeRepoPath(rest)
if (kind === 'blob') {
if (!rest) throw new Error('Missing path in GitHub URL')
if (!normalizedRest) throw new Error('Invalid path in GitHub URL')
const dir = normalizedRest.split('/').slice(0, -1).join('/')
return { owner, repo, ref, path: dir || undefined, originalUrl }
}
if (rest && !normalizedRest) throw new Error('Invalid path in GitHub URL')
return { owner, repo, ref, path: normalizedRest || undefined, originalUrl }
}
export async function resolveGitHubCommit(
parsed: GitHubImportUrl,
fetcher: typeof fetch,
): Promise<GitHubImportResolved> {
const repoUrl = `https://${GITHUB_HOST}/${parsed.owner}/${parsed.repo}`
const ref = parsed.ref?.trim() || 'HEAD'
const path = normalizeRepoPath(parsed.path ?? '')
const commit =
ref === 'HEAD'
? await resolveHeadCommit(parsed, fetcher)
: await resolveRefCommit(parsed, ref, fetcher)
return {
owner: parsed.owner,
repo: parsed.repo,
ref,
commit,
path,
repoUrl,
originalUrl: parsed.originalUrl,
}
}
async function resolveRefCommit(parsed: GitHubImportUrl, ref: string, fetcher: typeof fetch) {
const apiUrl = `https://api.github.com/repos/${parsed.owner}/${parsed.repo}/commits/${encodeURIComponent(ref)}`
const response = await fetcher(apiUrl, {
headers: {
Accept: 'application/vnd.github+json',
'User-Agent': 'clawhub/github-import',
},
})
if (!response.ok) throw new Error('GitHub ref not found')
const body = (await response.json()) as { sha?: unknown }
const sha = typeof body.sha === 'string' ? body.sha : ''
if (!/^[a-f0-9]{40}$/i.test(sha)) throw new Error('GitHub commit sha missing')
return sha.toLowerCase()
}
async function resolveHeadCommit(parsed: GitHubImportUrl, fetcher: typeof fetch) {
let url = `https://${GITHUB_HOST}/${parsed.owner}/${parsed.repo}/archive/HEAD.zip`
for (let i = 0; i < MAX_REDIRECTS; i += 1) {
const response = await fetcher(url, { redirect: 'manual' })
const location = response.headers.get('location')
if (!location) break
const next = new URL(location, url)
if (next.hostname !== GITHUB_HOST && next.hostname !== CODELOAD_HOST) {
throw new Error('Unexpected redirect host')
}
url = next.toString()
}
const maybe = url.split('/').at(-1) ?? ''
if (!/^[a-f0-9]{40}$/i.test(maybe)) {
throw new Error('Could not resolve commit for HEAD')
}
return maybe.toLowerCase()
}
export async function fetchGitHubZipBytes(
resolved: GitHubImportResolved,
fetcher: typeof fetch,
limits?: { maxZipBytes?: number },
): Promise<Uint8Array> {
const maxZipBytes = limits?.maxZipBytes ?? 25 * 1024 * 1024
const url = `https://${CODELOAD_HOST}/${resolved.owner}/${resolved.repo}/zip/${resolved.commit}`
const response = await fetcher(url, {
headers: { 'User-Agent': 'clawhub/github-import' },
})
if (!response.ok) throw new Error('GitHub archive download failed')
const lengthHeader = response.headers.get('content-length')
if (lengthHeader) {
const contentLength = Number.parseInt(lengthHeader, 10)
if (Number.isFinite(contentLength) && contentLength > maxZipBytes) {
throw new Error('GitHub archive too large')
}
}
const reader = response.body?.getReader()
if (!reader) {
const buffer = new Uint8Array(await response.arrayBuffer())
if (buffer.byteLength > maxZipBytes) throw new Error('GitHub archive too large')
return buffer
}
const chunks: Uint8Array[] = []
let total = 0
while (true) {
const { done, value } = await reader.read()
if (done) break
if (!value) continue
total += value.byteLength
if (total > maxZipBytes) throw new Error('GitHub archive too large')
chunks.push(value)
}
const out = new Uint8Array(total)
let offset = 0
for (const chunk of chunks) {
out.set(chunk, offset)
offset += chunk.byteLength
}
return out
}
export type ZipEntryMap = Record<string, Uint8Array>
export function buildGitHubZipForTests(entries: Record<string, string>) {
const asBytes = Object.fromEntries(
Object.entries(entries).map(([path, text]) => [path, new TextEncoder().encode(text)]),
)
return Uint8Array.from(zipSync(asBytes, { level: 1 }))
}
export function stripGitHubZipRoot(entries: ZipEntryMap): ZipEntryMap {
const paths = Object.keys(entries)
if (paths.length === 0) return {}
const first = paths[0] ?? ''
const firstRoot = first.split('/')[0] ?? ''
if (!firstRoot) return entries
const prefix = `${firstRoot}/`
if (!paths.every((path) => path.startsWith(prefix))) return entries
const out: ZipEntryMap = {}
for (const [path, data] of Object.entries(entries)) {
const stripped = path.slice(prefix.length)
if (!stripped) continue
out[stripped] = data
}
return out
}
export function detectGitHubImportCandidates(entries: ZipEntryMap): GitHubImportCandidate[] {
const candidates: GitHubImportCandidate[] = []
for (const path of Object.keys(entries)) {
const normalized = normalizeRepoPath(path)
const lower = normalized.toLowerCase()
const isSkill = SKILL_FILENAMES.some((name) => lower === name || lower.endsWith(`/${name}`))
if (!isSkill) continue
const dir = normalized.split('/').slice(0, -1).join('/')
const readmePath = normalized
const raw = new TextDecoder().decode(entries[path] ?? new Uint8Array())
const frontmatter = parseFrontmatter(raw)
const name = typeof frontmatter.name === 'string' ? frontmatter.name : undefined
const description =
typeof frontmatter.description === 'string' ? frontmatter.description : undefined
candidates.push({
path: normalizeRepoPath(dir),
readmePath,
name: name?.trim() || undefined,
description: description?.trim() || undefined,
})
}
return uniqCandidates(candidates)
}
function uniqCandidates(candidates: GitHubImportCandidate[]) {
const seen = new Set<string>()
const out: GitHubImportCandidate[] = []
for (const candidate of candidates) {
const key = `${candidate.path}::${candidate.readmePath}`
if (seen.has(key)) continue
seen.add(key)
out.push(candidate)
}
return out.sort((a, b) => a.path.localeCompare(b.path))
}
export function listTextFilesUnderCandidate(
entries: ZipEntryMap,
candidatePath: string,
): Array<{ path: string; bytes: Uint8Array }> {
const root = normalizeCandidateRoot(candidatePath)
const out: Array<{ path: string; bytes: Uint8Array }> = []
for (const [path, bytes] of Object.entries(entries)) {
const normalized = normalizeRepoPath(path)
if (!isUnderRoot(normalized, root)) continue
if (!isTextPath(normalized)) continue
out.push({ path: normalized, bytes })
}
return out.sort((a, b) => a.path.localeCompare(b.path))
}
export function computeDefaultSelectedPaths(params: {
candidate: GitHubImportCandidate
files: Array<{ path: string; bytes: Uint8Array }>
maxDepth?: number
maxAdds?: number
}) {
const maxDepth = params.maxDepth ?? 4
const maxAdds = params.maxAdds ?? 200
const byPath = new Map(params.files.map((file) => [file.path, file.bytes]))
const candidateRoot = normalizeCandidateRoot(params.candidate.path)
const selected = new Set<string>()
let added = 0
const add = (path: string) => {
const normalized = normalizeRepoPath(path)
if (!isUnderRoot(normalized, candidateRoot)) return
if (!byPath.has(normalized)) return
if (!selected.has(normalized)) {
selected.add(normalized)
added += 1
}
}
add(params.candidate.readmePath)
const visited = new Set<string>()
const queue: Array<{ path: string; depth: number }> = [
{ path: params.candidate.readmePath, depth: 0 },
]
while (queue.length > 0) {
const item = queue.shift()
if (!item) break
if (item.depth >= maxDepth) continue
if (visited.has(item.path)) continue
visited.add(item.path)
const bytes = byPath.get(item.path)
if (!bytes) continue
if (!item.path.toLowerCase().endsWith('.md')) continue
const text = new TextDecoder().decode(bytes)
const refs = extractMarkdownRelativeTargets(text)
for (const ref of refs) {
if (added >= maxAdds) break
const resolved = resolveMarkdownTarget(item.path, ref)
if (!resolved) continue
add(resolved)
if (resolved.toLowerCase().endsWith('.md') && byPath.has(resolved)) {
queue.push({ path: resolved, depth: item.depth + 1 })
}
}
if (added >= maxAdds) break
}
return Array.from(selected).sort()
}
export function buildGitHubImportFileList(params: {
candidate: GitHubImportCandidate
files: Array<{ path: string; bytes: Uint8Array }>
defaultSelectedPaths: string[]
}): GitHubImportFileEntry[] {
const selected = new Set(params.defaultSelectedPaths)
return params.files.map((file) => ({
path: file.path,
size: file.bytes.byteLength,
defaultSelected: selected.has(file.path),
}))
}
export function normalizeRepoPath(path: string) {
const stripped = path.replace(/^\/+/, '').trim()
if (!stripped) return ''
const cleaned = stripped.split('/').filter(Boolean).join('/')
if (!cleaned || cleaned.includes('\\') || cleaned.includes('..')) return ''
return cleaned
}
export function normalizeCandidateRoot(candidatePath: string) {
const normalized = normalizeRepoPath(candidatePath)
return normalized ? `${normalized}/` : ''
}
function isUnderRoot(path: string, rootWithSlash: string) {
if (!rootWithSlash) return true
return path === rootWithSlash.slice(0, -1) || path.startsWith(rootWithSlash)
}
function isTextPath(path: string) {
const lower = path.toLowerCase()
const ext = lower.split('.').at(-1) ?? ''
if (!ext) return false
return TEXT_FILE_EXTENSION_SET.has(ext)
}
export function suggestDisplayName(candidate: GitHubImportCandidate, fallbackBase: string) {
const base = candidate.name?.trim() || fallbackBase.trim()
if (!base) return ''
return base
.replace(/[-_]+/g, ' ')
.replace(/\s+/g, ' ')
.replace(/\b\w/g, (char) => char.toUpperCase())
}
export function suggestVersion(latestVersion?: string | null) {
const latest = latestVersion?.trim() || ''
if (latest && semver.valid(latest)) {
return semver.inc(latest, 'patch') ?? '0.1.0'
}
return '0.1.0'
}
export function extractMarkdownRelativeTargets(markdown: string): string[] {
const out: string[] = []
const pattern = /!?\[[^\]]*]\(([^)]+)\)/g
for (const match of markdown.matchAll(pattern)) {
const raw = (match[1] ?? '').trim()
if (!raw) continue
const isAngleWrapped = raw.startsWith('<') && raw.endsWith('>')
const cleaned = raw.replace(/^<|>$/g, '').trim()
if (!cleaned) continue
const target = isAngleWrapped ? cleaned : (cleaned.split(/\s+/)[0] ?? '')
if (!target) continue
if (target.startsWith('#')) continue
const lower = target.toLowerCase()
if (lower.startsWith('http:') || lower.startsWith('https:')) continue
if (lower.startsWith('mailto:')) continue
out.push(target)
}
return out
}
export function resolveMarkdownTarget(fromPath: string, target: string) {
const withoutHash = target.split('#')[0] ?? ''
const withoutQuery = (withoutHash.split('?')[0] ?? '').trim()
if (!withoutQuery) return null
if (withoutQuery.startsWith('/')) return null
if (withoutQuery.includes('\\') || withoutQuery.includes('..')) return null
const fromDirParts = normalizeRepoPath(fromPath).split('/').slice(0, -1)
const targetParts = withoutQuery.split('/').filter(Boolean)
const combined = [...fromDirParts, ...targetParts]
const normalized: string[] = []
for (const part of combined) {
if (part === '.') continue
if (part === '..') return null
normalized.push(part)
}
return normalizeRepoPath(normalized.join('/')) || null
}
+443
View File
@@ -0,0 +1,443 @@
'use node'
import { createPrivateKey, createSign } from 'node:crypto'
import type { Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
const GITHUB_API = 'https://api.github.com'
const DEFAULT_REPO = 'clawdbot/souls'
const DEFAULT_ROOT = 'souls'
const META_FILENAME = '_meta.json'
const USER_AGENT = 'clawhub/souls-backup'
type BackupFile = {
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}
type BackupParams = {
slug: string
version: string
displayName: string
ownerHandle: string
files: BackupFile[]
publishedAt: number
}
type RepoInfo = {
default_branch?: string
}
type GitRef = {
object: { sha: string }
}
type GitCommit = {
sha: string
tree: { sha: string }
}
type GitTreeEntry = {
path?: string
type?: string
}
type GitTree = {
tree?: GitTreeEntry[]
}
type MetaFile = {
owner: string
slug: string
displayName: string
latest: {
version: string
publishedAt: number
commit: string | null
}
history: Array<{
version: string
publishedAt: number
commit: string
}>
}
export type GitHubBackupContext = {
token: string
repo: string
repoOwner: string
repoName: string
branch: string
root: string
}
export function isGitHubSoulBackupConfigured() {
return Boolean(
process.env.GITHUB_APP_ID &&
process.env.GITHUB_APP_PRIVATE_KEY &&
process.env.GITHUB_APP_INSTALLATION_ID,
)
}
export async function getGitHubSoulBackupContext(): Promise<GitHubBackupContext> {
const repo = process.env.GITHUB_SOULS_REPO ?? DEFAULT_REPO
const root = process.env.GITHUB_SOULS_ROOT ?? DEFAULT_ROOT
const [repoOwner, repoName] = parseRepo(repo)
const token = await createInstallationToken()
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`)
const branch = repoInfo.default_branch ?? 'main'
return { token, repo, repoOwner, repoName, branch, root }
}
export async function fetchGitHubSoulMeta(
context: GitHubBackupContext,
ownerHandle: string,
slug: string,
): Promise<MetaFile | null> {
const soulRoot = buildSoulRoot(context.root, ownerHandle, slug)
return fetchMetaFile(
context.token,
context.repoOwner,
context.repoName,
`${soulRoot}/${META_FILENAME}`,
context.branch,
)
}
export async function backupSoulToGitHub(
ctx: ActionCtx,
params: BackupParams,
context?: GitHubBackupContext,
) {
if (!isGitHubSoulBackupConfigured()) return
const resolved = context ?? (await getGitHubSoulBackupContext())
const soulRoot = buildSoulRoot(resolved.root, params.ownerHandle, params.slug)
const ref = await githubGet<GitRef>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/ref/heads/${resolved.branch}`,
)
const baseCommitSha = ref.object.sha
const baseCommit = await githubGet<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits/${baseCommitSha}`,
)
const baseTreeSha = baseCommit.tree.sha
const existingTree = await githubGet<GitTree>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees/${baseTreeSha}?recursive=1`,
)
const prefix = `${soulRoot}/`
const existingPaths = new Set(
(existingTree.tree ?? [])
.filter((entry) => entry.type === 'blob' && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? ''),
)
const newPaths = new Set<string>()
const treeEntries: Array<{
path: string
mode: '100644'
type: 'blob'
sha: string | null
}> = []
for (const file of params.files) {
const content = await fetchStorageBase64(ctx, file.storageId)
const blobSha = await createBlob(resolved.token, resolved.repoOwner, resolved.repoName, content)
const path = `${soulRoot}/${file.path}`
newPaths.add(path)
treeEntries.push({ path, mode: '100644', type: 'blob', sha: blobSha })
}
const existingMeta = await fetchMetaFile(
resolved.token,
resolved.repoOwner,
resolved.repoName,
`${soulRoot}/${META_FILENAME}`,
resolved.branch,
)
const metaPath = `${soulRoot}/${META_FILENAME}`
const metaDraft = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, null)
const metaDraftContent = `${JSON.stringify(metaDraft, null, 2)}\n`
const metaDraftSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaDraftContent),
)
newPaths.add(metaPath)
treeEntries.push({ path: metaPath, mode: '100644', type: 'blob', sha: metaDraftSha })
for (const path of existingPaths) {
if (newPaths.has(path)) continue
treeEntries.push({ path, mode: '100644', type: 'blob', sha: null })
}
const newTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: baseTreeSha,
tree: treeEntries,
},
)
const commit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `soul: ${params.slug} v${params.version}`,
tree: newTree.sha,
parents: [baseCommitSha],
},
)
const metaFinal = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, commit.sha)
const metaFinalContent = `${JSON.stringify(metaFinal, null, 2)}\n`
const metaFinalSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaFinalContent),
)
const metaTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: commit.tree.sha,
tree: [{ path: metaPath, mode: '100644', type: 'blob', sha: metaFinalSha }],
},
)
const metaCommit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `meta: ${params.slug} v${params.version}`,
tree: metaTree.sha,
parents: [commit.sha],
},
)
await githubPatch(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/refs/heads/${resolved.branch}`,
{
sha: metaCommit.sha,
},
)
}
function buildMetaFile(
params: BackupParams,
existing: MetaFile | null,
repo: string,
baseCommitSha: string,
latestCommitSha: string | null,
): MetaFile {
let history = [...(existing?.history ?? [])]
if (existing?.latest?.version) {
const previousCommit = existing.latest.commit ?? commitUrl(repo, baseCommitSha)
const previous = {
version: existing.latest.version,
publishedAt: existing.latest.publishedAt,
commit: previousCommit,
}
history = [previous, ...history.filter((entry) => entry.version !== previous.version)]
}
return {
owner: normalizeOwner(params.ownerHandle),
slug: params.slug,
displayName: params.displayName,
latest: {
version: params.version,
publishedAt: params.publishedAt,
commit: latestCommitSha ? commitUrl(repo, latestCommitSha) : null,
},
history: history.slice(0, 200),
}
}
async function fetchMetaFile(
token: string,
repoOwner: string,
repoName: string,
path: string,
branch: string,
): Promise<MetaFile | null> {
try {
const response = await githubGet<{ content?: string }>(
token,
`/repos/${repoOwner}/${repoName}/contents/${encodePath(path)}?ref=${branch}`,
)
if (!response.content) return null
const raw = fromBase64(response.content)
return JSON.parse(raw) as MetaFile
} catch (error) {
if (isNotFoundError(error)) return null
throw error
}
}
async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<'_storage'>) {
const blob = await ctx.storage.get(storageId)
if (!blob) throw new Error('File missing in storage')
const buffer = Buffer.from(await blob.arrayBuffer())
return buffer.toString('base64')
}
async function createInstallationToken() {
const appId = process.env.GITHUB_APP_ID
const installationId = process.env.GITHUB_APP_INSTALLATION_ID
if (!appId || !installationId) {
throw new Error('GitHub App credentials missing')
}
const jwt = createAppJwt(appId)
const response = await fetch(`${GITHUB_API}/app/installations/${installationId}/access_tokens`, {
method: 'POST',
headers: buildHeaders(jwt, true),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub App token failed: ${message}`)
}
const payload = (await response.json()) as { token?: string }
if (!payload.token) throw new Error('GitHub App token missing')
return payload.token
}
function createAppJwt(appId: string) {
const privateKey = loadPrivateKey()
const now = Math.floor(Date.now() / 1000)
const header = { alg: 'RS256', typ: 'JWT' }
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId }
const encodedHeader = base64Url(JSON.stringify(header))
const encodedPayload = base64Url(JSON.stringify(payload))
const signingInput = `${encodedHeader}.${encodedPayload}`
const sign = createSign('RSA-SHA256')
sign.update(signingInput)
sign.end()
const signature = sign.sign(privateKey)
return `${signingInput}.${base64Url(signature)}`
}
function loadPrivateKey() {
const raw = process.env.GITHUB_APP_PRIVATE_KEY
if (!raw) throw new Error('GITHUB_APP_PRIVATE_KEY is not configured')
const normalized = raw.replace(/\\n/g, '\n')
return createPrivateKey(normalized)
}
async function createBlob(token: string, repoOwner: string, repoName: string, content: string) {
const result = await githubPost<{ sha: string }>(
token,
`/repos/${repoOwner}/${repoName}/git/blobs`,
{
content,
encoding: 'base64',
},
)
if (!result.sha) throw new Error('GitHub blob missing sha')
return result.sha
}
async function githubGet<T>(token: string, path: string): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
headers: buildHeaders(token),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub GET ${path} failed: ${message}`)
}
return (await response.json()) as T
}
async function githubPost<T>(token: string, path: string, body: unknown): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'POST',
headers: buildHeaders(token),
body: JSON.stringify(body),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub POST ${path} failed: ${message}`)
}
return (await response.json()) as T
}
async function githubPatch(token: string, path: string, body: unknown) {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'PATCH',
headers: buildHeaders(token),
body: JSON.stringify(body),
})
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub PATCH ${path} failed: ${message}`)
}
}
function buildHeaders(token: string, isAppJwt = false) {
return {
Authorization: `${isAppJwt ? 'Bearer' : 'token'} ${token}`,
Accept: 'application/vnd.github+json',
'User-Agent': USER_AGENT,
}
}
function parseRepo(repo: string) {
const [owner, name] = repo.split('/')
if (!owner || !name) throw new Error('GITHUB_SOULS_REPO must be owner/repo')
return [owner, name] as const
}
function normalizeOwner(value: string) {
const normalized = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]/g, '-')
.replace(/-+/g, '-')
.replace(/^-+|-+$/g, '')
return normalized || 'unknown'
}
function commitUrl(repo: string, sha: string) {
return `https://github.com/${repo}/commit/${sha}`
}
function buildSoulRoot(root: string, ownerHandle: string, slug: string) {
const ownerSegment = normalizeOwner(ownerHandle)
return `${root}/${ownerSegment}/${slug}`
}
function encodePath(path: string) {
return path
.split('/')
.map((segment) => encodeURIComponent(segment))
.join('/')
}
function base64Url(value: string | Buffer) {
const buffer = typeof value === 'string' ? Buffer.from(value) : value
return buffer.toString('base64').replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/g, '')
}
function toBase64(value: string) {
return Buffer.from(value).toString('base64')
}
function fromBase64(value: string) {
return Buffer.from(value, 'base64').toString('utf8')
}
function isNotFoundError(error: unknown) {
return (
error instanceof Error && (error.message.includes('404') || error.message.includes('Not Found'))
)
}
+103
View File
@@ -0,0 +1,103 @@
import type { Id } from '../_generated/dataModel'
import type { MutationCtx, QueryCtx } from '../_generated/server'
const DAY_MS = 24 * 60 * 60 * 1000
export const TRENDING_DAYS = 7
type LeaderboardEntry = {
skillId: Id<'skills'>
score: number
installs: number
downloads: number
}
export function toDayKey(timestamp: number) {
return Math.floor(timestamp / DAY_MS)
}
export function getTrendingRange(now: number) {
const endDay = toDayKey(now)
const startDay = endDay - (TRENDING_DAYS - 1)
return { startDay, endDay }
}
export async function buildTrendingLeaderboard(
ctx: QueryCtx | MutationCtx,
params: { limit: number; now?: number },
) {
const now = params.now ?? Date.now()
const { startDay, endDay } = getTrendingRange(now)
const rows = await ctx.db
.query('skillDailyStats')
.withIndex('by_day', (q) => q.gte('day', startDay).lte('day', endDay))
.collect()
const totals = new Map<Id<'skills'>, { installs: number; downloads: number }>()
for (const row of rows) {
const current = totals.get(row.skillId) ?? { installs: 0, downloads: 0 }
current.installs += row.installs
current.downloads += row.downloads
totals.set(row.skillId, current)
}
const entries = Array.from(totals, ([skillId, totalsEntry]) => ({
skillId,
installs: totalsEntry.installs,
downloads: totalsEntry.downloads,
score: totalsEntry.installs,
}))
const items = topN(entries, params.limit, compareTrendingEntries).sort((a, b) =>
compareTrendingEntries(b, a),
)
return { startDay, endDay, items }
}
function compareTrendingEntries(a: LeaderboardEntry, b: LeaderboardEntry) {
if (a.score !== b.score) return a.score - b.score
if (a.downloads !== b.downloads) return a.downloads - b.downloads
return 0
}
function topN<T>(entries: T[], limit: number, compare: (a: T, b: T) => number) {
if (entries.length <= limit) return entries.slice()
const heap: T[] = []
for (const entry of entries) {
if (heap.length < limit) {
heap.push(entry)
siftUp(heap, heap.length - 1, compare)
continue
}
if (compare(entry, heap[0]) <= 0) continue
heap[0] = entry
siftDown(heap, 0, compare)
}
return heap
}
function siftUp<T>(heap: T[], index: number, compare: (a: T, b: T) => number) {
let current = index
while (current > 0) {
const parent = Math.floor((current - 1) / 2)
if (compare(heap[current], heap[parent]) >= 0) break
;[heap[current], heap[parent]] = [heap[parent], heap[current]]
current = parent
}
}
function siftDown<T>(heap: T[], index: number, compare: (a: T, b: T) => number) {
let current = index
const length = heap.length
while (true) {
const left = current * 2 + 1
const right = current * 2 + 2
let smallest = current
if (left < length && compare(heap[left], heap[smallest]) < 0) smallest = left
if (right < length && compare(heap[right], heap[smallest]) < 0) smallest = right
if (smallest === current) break
;[heap[current], heap[smallest]] = [heap[smallest], heap[current]]
current = smallest
}
}
+49
View File
@@ -0,0 +1,49 @@
import type { Doc } from '../_generated/dataModel'
const FLAG_RULES: Array<{ flag: string; pattern: RegExp }> = [
// Known-bad / known-suspicious identifiers.
// NOTE: keep these narrowly scoped; use staff review to confirm removals.
{
flag: 'blocked.malware',
pattern: /(keepcold131\/ClawdAuthenticatorTool|ClawdAuthenticatorTool)/i,
},
{ flag: 'suspicious.keyword', pattern: /(malware|stealer|phish|phishing|keylogger)/i },
{ flag: 'suspicious.secrets', pattern: /(api[-_ ]?key|token|password|private key|secret)/i },
{ flag: 'suspicious.crypto', pattern: /(wallet|seed phrase|mnemonic|crypto)/i },
{ flag: 'suspicious.webhook', pattern: /(discord\.gg|webhook|hooks\.slack)/i },
{ flag: 'suspicious.script', pattern: /(curl[^\n]+\|\s*(sh|bash))/i },
{ flag: 'suspicious.url_shortener', pattern: /(bit\.ly|tinyurl\.com|t\.co|goo\.gl|is\.gd)/i },
]
export function deriveModerationFlags({
skill,
parsed,
files,
}: {
skill: Pick<Doc<'skills'>, 'slug' | 'displayName' | 'summary'>
parsed: Doc<'skillVersions'>['parsed']
files: Doc<'skillVersions'>['files']
}) {
const text = [
skill.slug,
skill.displayName,
skill.summary ?? '',
JSON.stringify(parsed?.frontmatter ?? {}),
JSON.stringify(parsed?.metadata ?? {}),
JSON.stringify((parsed as { moltbot?: unknown } | undefined)?.moltbot ?? {}),
...files.map((file) => file.path),
]
.filter(Boolean)
.join('\n')
const flags = new Set<string>()
for (const rule of FLAG_RULES) {
if (rule.pattern.test(text)) {
flags.add(rule.flag)
}
}
return Array.from(flags)
}
+91
View File
@@ -0,0 +1,91 @@
import type { Doc } from '../_generated/dataModel'
export type PublicUser = Pick<
Doc<'users'>,
'_id' | '_creationTime' | 'handle' | 'name' | 'displayName' | 'image' | 'bio'
>
export type PublicSkill = Pick<
Doc<'skills'>,
| '_id'
| '_creationTime'
| 'slug'
| 'displayName'
| 'summary'
| 'ownerUserId'
| 'canonicalSkillId'
| 'forkOf'
| 'latestVersionId'
| 'tags'
| 'badges'
| 'stats'
| 'createdAt'
| 'updatedAt'
>
export type PublicSoul = Pick<
Doc<'souls'>,
| '_id'
| '_creationTime'
| 'slug'
| 'displayName'
| 'summary'
| 'ownerUserId'
| 'latestVersionId'
| 'tags'
| 'stats'
| 'createdAt'
| 'updatedAt'
>
export function toPublicUser(user: Doc<'users'> | null | undefined): PublicUser | null {
if (!user || user.deletedAt) return null
return {
_id: user._id,
_creationTime: user._creationTime,
handle: user.handle,
name: user.name,
displayName: user.displayName,
image: user.image,
bio: user.bio,
}
}
export function toPublicSkill(skill: Doc<'skills'> | null | undefined): PublicSkill | null {
if (!skill || skill.softDeletedAt) return null
if (skill.moderationStatus && skill.moderationStatus !== 'active') return null
if (skill.moderationFlags?.includes('blocked.malware')) return null
return {
_id: skill._id,
_creationTime: skill._creationTime,
slug: skill.slug,
displayName: skill.displayName,
summary: skill.summary,
ownerUserId: skill.ownerUserId,
canonicalSkillId: skill.canonicalSkillId,
forkOf: skill.forkOf,
latestVersionId: skill.latestVersionId,
tags: skill.tags,
badges: skill.badges,
stats: skill.stats,
createdAt: skill.createdAt,
updatedAt: skill.updatedAt,
}
}
export function toPublicSoul(soul: Doc<'souls'> | null | undefined): PublicSoul | null {
if (!soul || soul.softDeletedAt) return null
return {
_id: soul._id,
_creationTime: soul._creationTime,
slug: soul.slug,
displayName: soul.displayName,
summary: soul.summary,
ownerUserId: soul.ownerUserId,
latestVersionId: soul.latestVersionId,
tags: soul.tags,
stats: soul.stats,
createdAt: soul.createdAt,
updatedAt: soul.updatedAt,
}
}
+46
View File
@@ -0,0 +1,46 @@
/* @vitest-environment node */
import { describe, expect, it } from 'vitest'
import { __test, matchesExactTokens, tokenize } from './searchText'
describe('searchText', () => {
it('tokenize lowercases and splits on punctuation', () => {
expect(tokenize('Minimax Usage /minimax-usage')).toEqual([
'minimax',
'usage',
'minimax',
'usage',
])
})
it('matchesExactTokens requires at least one query token to prefix-match', () => {
const queryTokens = tokenize('Remind Me')
expect(matchesExactTokens(queryTokens, ['Remind Me', '/remind-me', 'Short summary'])).toBe(true)
// "Reminder" starts with "remind", so it matches with prefix matching
expect(matchesExactTokens(queryTokens, ['Reminder tool', '/reminder', 'Short summary'])).toBe(
true,
)
// Matches because "remind" token is present
expect(matchesExactTokens(queryTokens, ['Remind tool', '/remind', 'Short summary'])).toBe(true)
// No matching tokens at all
expect(matchesExactTokens(queryTokens, ['Other tool', '/other', 'Short summary'])).toBe(false)
})
it('matchesExactTokens supports prefix matching for partial queries', () => {
// "go" should match "gohome" because "gohome" starts with "go"
expect(matchesExactTokens(['go'], ['GoHome', '/gohome', 'Navigate home'])).toBe(true)
// "pad" should match "padel"
expect(matchesExactTokens(['pad'], ['Padel', '/padel', 'Tennis-like sport'])).toBe(true)
// "xyz" should not match anything
expect(matchesExactTokens(['xyz'], ['GoHome', '/gohome', 'Navigate home'])).toBe(false)
})
it('matchesExactTokens ignores empty inputs', () => {
expect(matchesExactTokens([], ['text'])).toBe(false)
expect(matchesExactTokens(['token'], [' ', null, undefined])).toBe(false)
})
it('normalize uses lowercase', () => {
expect(__test.normalize('AbC')).toBe('abc')
})
})
+27
View File
@@ -0,0 +1,27 @@
const WORD_RE = /[a-z0-9]+/g
function normalize(value: string) {
return value.toLowerCase()
}
export function tokenize(value: string): string[] {
if (!value) return []
return normalize(value).match(WORD_RE) ?? []
}
export function matchesExactTokens(
queryTokens: string[],
parts: Array<string | null | undefined>,
): boolean {
if (queryTokens.length === 0) return false
const text = parts.filter((part) => Boolean(part?.trim())).join(' ')
if (!text) return false
const textTokens = tokenize(text)
if (textTokens.length === 0) return false
// Require at least one token to prefix-match, allowing vector similarity to determine relevance
return queryTokens.some((queryToken) =>
textTokens.some((textToken) => textToken.includes(queryToken)),
)
}
export const __test = { normalize, tokenize, matchesExactTokens }
+34
View File
@@ -0,0 +1,34 @@
import { describe, expect, it } from 'vitest'
import { buildSkillSummaryBackfillPatch } from './skillBackfill'
describe('skill backfill', () => {
it('produces summary + parsed patch from block scalar', () => {
const patch = buildSkillSummaryBackfillPatch({
readmeText: `---\ndescription: >\n Hello\n world.\n---\nBody`,
currentSummary: '>',
currentParsed: { frontmatter: { description: '>' } },
})
expect(patch.summary).toBe('Hello world.')
expect(patch.parsed?.frontmatter.description).toBe('Hello world.')
})
it('does not set summary when description is not a string', () => {
const patch = buildSkillSummaryBackfillPatch({
readmeText: `---\ndescription:\n - a\n---\nBody`,
currentSummary: 'Old',
currentParsed: { frontmatter: {} },
})
expect(patch.summary).toBeUndefined()
expect(patch.parsed?.frontmatter.description).toEqual(['a'])
})
it('keeps legacy summary when unchanged and still updates parsed', () => {
const patch = buildSkillSummaryBackfillPatch({
readmeText: `---\ndescription: Hello\n---\nBody`,
currentSummary: 'Hello',
currentParsed: { frontmatter: { description: 'nope' } },
})
expect(patch.summary).toBeUndefined()
expect(patch.parsed?.frontmatter.description).toBe('Hello')
})
})
+67
View File
@@ -0,0 +1,67 @@
import {
getFrontmatterMetadata,
getFrontmatterValue,
type ParsedSkillFrontmatter,
parseClawdisMetadata,
parseFrontmatter,
} from './skills'
export type ParsedSkillData = {
frontmatter: ParsedSkillFrontmatter
metadata?: unknown
clawdis?: unknown
}
export type SkillSummaryBackfillPatch = {
summary?: string
parsed?: ParsedSkillData
}
export function buildSkillSummaryBackfillPatch(args: {
readmeText: string
currentSummary?: string
currentParsed?: ParsedSkillData
}): SkillSummaryBackfillPatch {
const frontmatter = parseFrontmatter(args.readmeText)
const summary = getFrontmatterValue(frontmatter, 'description') ?? undefined
const metadata = getFrontmatterMetadata(frontmatter)
const clawdis = parseClawdisMetadata(frontmatter)
const parsed: ParsedSkillData = { frontmatter, metadata, clawdis }
const patch: SkillSummaryBackfillPatch = {}
if (summary && summary !== args.currentSummary) {
patch.summary = summary
}
if (!deepEqual(parsed, args.currentParsed)) {
patch.parsed = parsed
}
return patch
}
function deepEqual(a: unknown, b: unknown): boolean {
if (a === b) return true
if (!a || !b) return a === b
if (typeof a !== typeof b) return false
if (Array.isArray(a) || Array.isArray(b)) {
if (!Array.isArray(a) || !Array.isArray(b)) return false
if (a.length !== b.length) return false
for (let i = 0; i < a.length; i++) {
if (!deepEqual(a[i], b[i])) return false
}
return true
}
if (typeof a === 'object' && typeof b === 'object') {
const aObj = a as Record<string, unknown>
const bObj = b as Record<string, unknown>
const aKeys = Object.keys(aObj).sort()
const bKeys = Object.keys(bObj).sort()
if (aKeys.length !== bKeys.length) return false
for (let i = 0; i < aKeys.length; i++) {
if (aKeys[i] !== bKeys[i]) return false
const key = aKeys[i] as string
if (!deepEqual(aObj[key], bObj[key])) return false
}
return true
}
return false
}
+28
View File
@@ -0,0 +1,28 @@
import { describe, expect, it } from 'vitest'
import { __test } from './skillPublish'
describe('skillPublish', () => {
it('merges github source into metadata', () => {
const merged = __test.mergeSourceIntoMetadata(
{ clawdis: { emoji: 'x' } },
{
kind: 'github',
url: 'https://github.com/a/b',
repo: 'a/b',
ref: 'main',
commit: '0123456789012345678901234567890123456789',
path: 'skills/demo',
importedAt: 123,
},
)
expect((merged as Record<string, unknown>).clawdis).toEqual({ emoji: 'x' })
const source = (merged as Record<string, unknown>).source
expect(source).toEqual(
expect.objectContaining({
kind: 'github',
repo: 'a/b',
path: 'skills/demo',
}),
)
})
})
+284
View File
@@ -0,0 +1,284 @@
import { ConvexError } from 'convex/values'
import semver from 'semver'
import { api, internal } from '../_generated/api'
import type { Doc, Id } from '../_generated/dataModel'
import type { ActionCtx, MutationCtx } from '../_generated/server'
import { getSkillBadgeMap, isSkillHighlighted } from './badges'
import { generateChangelogForPublish } from './changelog'
import { generateEmbedding } from './embeddings'
import type { PublicUser } from './public'
import {
buildEmbeddingText,
getFrontmatterMetadata,
hashSkillFiles,
isTextFile,
parseClawdisMetadata,
parseFrontmatter,
sanitizePath,
} from './skills'
import type { WebhookSkillPayload } from './webhooks'
const MAX_TOTAL_BYTES = 50 * 1024 * 1024
const MAX_FILES_FOR_EMBEDDING = 40
export type PublishResult = {
skillId: Id<'skills'>
versionId: Id<'skillVersions'>
embeddingId: Id<'skillEmbeddings'>
}
export type PublishVersionArgs = {
slug: string
displayName: string
version: string
changelog: string
tags?: string[]
forkOf?: { slug: string; version?: string }
source?: {
kind: 'github'
url: string
repo: string
ref: string
commit: string
path: string
importedAt: number
}
files: Array<{
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}>
}
export async function publishVersionForUser(
ctx: ActionCtx,
userId: Id<'users'>,
args: PublishVersionArgs,
): Promise<PublishResult> {
const version = args.version.trim()
const slug = args.slug.trim().toLowerCase()
const displayName = args.displayName.trim()
if (!slug || !displayName) throw new ConvexError('Slug and display name required')
if (!/^[a-z0-9][a-z0-9-]*$/.test(slug)) {
throw new ConvexError('Slug must be lowercase and url-safe')
}
if (!semver.valid(version)) {
throw new ConvexError('Version must be valid semver')
}
const suppliedChangelog = args.changelog.trim()
const changelogSource = suppliedChangelog ? ('user' as const) : ('auto' as const)
const sanitizedFiles = args.files.map((file) => ({
...file,
path: sanitizePath(file.path),
}))
if (sanitizedFiles.some((file) => !file.path)) {
throw new ConvexError('Invalid file paths')
}
const safeFiles = sanitizedFiles.map((file) => ({
...file,
path: file.path as string,
}))
if (safeFiles.some((file) => !isTextFile(file.path, file.contentType ?? undefined))) {
throw new ConvexError('Only text-based files are allowed')
}
const totalBytes = safeFiles.reduce((sum, file) => sum + file.size, 0)
if (totalBytes > MAX_TOTAL_BYTES) {
throw new ConvexError('Skill bundle exceeds 50MB limit')
}
const readmeFile = safeFiles.find(
(file) => file.path?.toLowerCase() === 'skill.md' || file.path?.toLowerCase() === 'skills.md',
)
if (!readmeFile) throw new ConvexError('SKILL.md is required')
const readmeText = await fetchText(ctx, readmeFile.storageId)
const frontmatter = parseFrontmatter(readmeText)
const clawdis = parseClawdisMetadata(frontmatter)
const metadata = mergeSourceIntoMetadata(getFrontmatterMetadata(frontmatter), args.source)
const otherFiles = [] as Array<{ path: string; content: string }>
for (const file of safeFiles) {
if (!file.path || file.path.toLowerCase().endsWith('.md')) continue
if (!isTextFile(file.path, file.contentType ?? undefined)) continue
const content = await fetchText(ctx, file.storageId)
otherFiles.push({ path: file.path, content })
if (otherFiles.length >= MAX_FILES_FOR_EMBEDDING) break
}
const embeddingText = buildEmbeddingText({
frontmatter,
readme: readmeText,
otherFiles,
})
const fingerprintPromise = hashSkillFiles(
safeFiles.map((file) => ({ path: file.path, sha256: file.sha256 })),
)
const changelogPromise =
changelogSource === 'user'
? Promise.resolve(suppliedChangelog)
: generateChangelogForPublish(ctx, {
slug,
version,
readmeText,
files: safeFiles.map((file) => ({ path: file.path, sha256: file.sha256 })),
})
const embeddingPromise = generateEmbedding(embeddingText)
const [fingerprint, changelogText, embedding] = await Promise.all([
fingerprintPromise,
changelogPromise,
embeddingPromise.catch((error) => {
throw new ConvexError(formatEmbeddingError(error))
}),
])
const publishResult = (await ctx.runMutation(internal.skills.insertVersion, {
userId,
slug,
displayName,
version,
changelog: changelogText,
changelogSource,
tags: args.tags?.map((tag) => tag.trim()).filter(Boolean),
fingerprint,
forkOf: args.forkOf
? {
slug: args.forkOf.slug.trim().toLowerCase(),
version: args.forkOf.version?.trim() || undefined,
}
: undefined,
files: safeFiles.map((file) => ({
...file,
path: file.path,
})),
parsed: {
frontmatter,
metadata,
clawdis,
},
embedding,
})) as PublishResult
const owner = (await ctx.runQuery(internal.users.getByIdInternal, {
userId,
})) as Doc<'users'> | null
const ownerHandle = owner?.handle ?? owner?.displayName ?? owner?.name ?? 'unknown'
void ctx.scheduler
.runAfter(0, internal.githubBackupsNode.backupSkillForPublishInternal, {
slug,
version,
displayName,
ownerHandle,
files: safeFiles,
publishedAt: Date.now(),
})
.catch((error) => {
console.error('GitHub backup scheduling failed', error)
})
void schedulePublishWebhook(ctx, {
slug,
version,
displayName,
})
return publishResult
}
function mergeSourceIntoMetadata(metadata: unknown, source: PublishVersionArgs['source']) {
if (!source) return metadata === undefined ? undefined : metadata
const sourceValue = {
kind: source.kind,
url: source.url,
repo: source.repo,
ref: source.ref,
commit: source.commit,
path: source.path,
importedAt: source.importedAt,
}
if (!metadata) return { source: sourceValue }
if (typeof metadata !== 'object' || Array.isArray(metadata)) return { source: sourceValue }
return { ...(metadata as Record<string, unknown>), source: sourceValue }
}
export const __test = {
mergeSourceIntoMetadata,
}
export async function queueHighlightedWebhook(ctx: MutationCtx, skillId: Id<'skills'>) {
const skill = await ctx.db.get(skillId)
if (!skill) return
const owner = await ctx.db.get(skill.ownerUserId)
const latestVersion = skill.latestVersionId ? await ctx.db.get(skill.latestVersionId) : null
const badges = await getSkillBadgeMap(ctx, skillId)
const payload: WebhookSkillPayload = {
slug: skill.slug,
displayName: skill.displayName,
summary: skill.summary ?? undefined,
version: latestVersion?.version ?? undefined,
ownerHandle: owner?.handle ?? owner?.name ?? undefined,
highlighted: isSkillHighlighted({ badges }),
tags: Object.keys(skill.tags ?? {}),
}
await ctx.scheduler.runAfter(0, internal.webhooks.sendDiscordWebhook, {
event: 'skill.highlighted',
skill: payload,
})
}
export async function fetchText(
ctx: { storage: { get: (id: Id<'_storage'>) => Promise<Blob | null> } },
storageId: Id<'_storage'>,
) {
const blob = await ctx.storage.get(storageId)
if (!blob) throw new Error('File missing in storage')
return blob.text()
}
function formatEmbeddingError(error: unknown) {
if (error instanceof Error) {
if (error.message.includes('OPENAI_API_KEY')) {
return 'OPENAI_API_KEY is not configured.'
}
if (error.message.startsWith('Embedding failed')) {
return error.message
}
}
return 'Embedding failed. Please try again.'
}
async function schedulePublishWebhook(
ctx: ActionCtx,
params: { slug: string; version: string; displayName: string },
) {
const result = (await ctx.runQuery(api.skills.getBySlug, {
slug: params.slug,
})) as { skill: Doc<'skills'>; owner: PublicUser | null } | null
if (!result?.skill) return
const payload: WebhookSkillPayload = {
slug: result.skill.slug,
displayName: result.skill.displayName || params.displayName,
summary: result.skill.summary ?? undefined,
version: params.version,
ownerHandle: result.owner?.handle ?? result.owner?.name ?? undefined,
highlighted: isSkillHighlighted(result.skill),
tags: Object.keys(result.skill.tags ?? {}),
}
await ctx.scheduler.runAfter(0, internal.webhooks.sendDiscordWebhook, {
event: 'skill.publish',
skill: payload,
})
}
+80
View File
@@ -0,0 +1,80 @@
import type { Doc, Id } from '../_generated/dataModel'
import type { MutationCtx } from '../_generated/server'
import { toDayKey } from './leaderboards'
type SkillStatDeltas = {
downloads?: number
stars?: number
installsCurrent?: number
installsAllTime?: number
}
export function applySkillStatDeltas(skill: Doc<'skills'>, deltas: SkillStatDeltas) {
const currentDownloads =
typeof skill.statsDownloads === 'number' ? skill.statsDownloads : skill.stats.downloads
const currentStars = typeof skill.statsStars === 'number' ? skill.statsStars : skill.stats.stars
const currentInstallsCurrent =
typeof skill.statsInstallsCurrent === 'number'
? skill.statsInstallsCurrent
: (skill.stats.installsCurrent ?? 0)
const currentInstallsAllTime =
typeof skill.statsInstallsAllTime === 'number'
? skill.statsInstallsAllTime
: (skill.stats.installsAllTime ?? 0)
const nextDownloads = Math.max(0, currentDownloads + (deltas.downloads ?? 0))
const nextStars = Math.max(0, currentStars + (deltas.stars ?? 0))
const nextInstallsCurrent = Math.max(0, currentInstallsCurrent + (deltas.installsCurrent ?? 0))
const nextInstallsAllTime = Math.max(0, currentInstallsAllTime + (deltas.installsAllTime ?? 0))
return {
statsDownloads: nextDownloads,
statsStars: nextStars,
statsInstallsCurrent: nextInstallsCurrent,
statsInstallsAllTime: nextInstallsAllTime,
stats: {
...skill.stats,
downloads: nextDownloads,
stars: nextStars,
installsCurrent: nextInstallsCurrent,
installsAllTime: nextInstallsAllTime,
},
}
}
export async function bumpDailySkillStats(
ctx: MutationCtx,
params: {
skillId: Id<'skills'>
now: number
downloads?: number
installs?: number
},
) {
const downloads = params.downloads ?? 0
const installs = params.installs ?? 0
if (downloads === 0 && installs === 0) return
const day = toDayKey(params.now)
const existing = await ctx.db
.query('skillDailyStats')
.withIndex('by_skill_day', (q) => q.eq('skillId', params.skillId).eq('day', day))
.unique()
if (existing) {
await ctx.db.patch(existing._id, {
downloads: Math.max(0, existing.downloads + downloads),
installs: Math.max(0, existing.installs + installs),
updatedAt: params.now,
})
return
}
await ctx.db.insert('skillDailyStats', {
skillId: params.skillId,
day,
downloads: Math.max(0, downloads),
installs: Math.max(0, installs),
updatedAt: params.now,
})
}
+95
View File
@@ -1,6 +1,8 @@
import { describe, expect, it } from 'vitest'
import {
buildEmbeddingText,
getFrontmatterMetadata,
getFrontmatterValue,
hashSkillFiles,
isTextFile,
parseClawdisMetadata,
@@ -26,6 +28,29 @@ describe('skills utils', () => {
expect(frontmatter.description).toBe('Hello')
})
it('parses block scalars in frontmatter', () => {
const folded = parseFrontmatter(
`---\nname: demo\ndescription: >\n Hello\n world.\n\n Next paragraph.\n---\nBody`,
)
expect(folded.description).toBe('Hello world.\nNext paragraph.')
const literal = parseFrontmatter(
`---\nname: demo\ndescription: |\n Hello\n world.\n---\nBody`,
)
expect(literal.description).toBe('Hello\nworld.')
})
it('keeps structured YAML values in frontmatter', () => {
const frontmatter = parseFrontmatter(
`---\nname: demo\ncount: 3\nnums: [1, 2]\nobj:\n a: b\n---\nBody`,
)
expect(frontmatter.nums).toEqual([1, 2])
expect(frontmatter.obj).toEqual({ a: 'b' })
expect(frontmatter.name).toBe('demo')
expect(frontmatter.count).toBe(3)
expect(getFrontmatterValue(frontmatter, 'count')).toBeUndefined()
})
it('parses clawdis metadata', () => {
const frontmatter = parseFrontmatter(
`---\nmetadata: {"clawdis":{"requires":{"bins":["rg"]},"emoji":"🦞"}}\n---\nBody`,
@@ -40,6 +65,38 @@ describe('skills utils', () => {
expect(parseClawdisMetadata(frontmatter)).toBeUndefined()
})
it('accepts metadata as YAML object (no JSON string)', () => {
const frontmatter = parseFrontmatter(
`---\nmetadata:\n clawdis:\n emoji: "🦞"\n requires:\n bins:\n - rg\n---\nBody`,
)
expect(getFrontmatterMetadata(frontmatter)).toEqual({
clawdis: { emoji: '🦞', requires: { bins: ['rg'] } },
})
const clawdis = parseClawdisMetadata(frontmatter)
expect(clawdis?.emoji).toBe('🦞')
expect(clawdis?.requires?.bins).toEqual(['rg'])
})
it('accepts clawdis as top-level YAML key', () => {
const frontmatter = parseFrontmatter(
`---\nclawdis:\n emoji: "🦞"\n requires:\n anyBins: [rg, fd]\n---\nBody`,
)
const clawdis = parseClawdisMetadata(frontmatter)
expect(clawdis?.emoji).toBe('🦞')
expect(clawdis?.requires?.anyBins).toEqual(['rg', 'fd'])
})
it('accepts legacy metadata JSON string (quoted)', () => {
const frontmatter = parseFrontmatter(
`---\nmetadata: '{"clawdis":{"emoji":"🦞","requires":{"bins":["rg"]}}}'\n---\nBody`,
)
const metadata = getFrontmatterMetadata(frontmatter)
expect(metadata).toEqual({ clawdis: { emoji: '🦞', requires: { bins: ['rg'] } } })
const clawdis = parseClawdisMetadata(frontmatter)
expect(clawdis?.emoji).toBe('🦞')
expect(clawdis?.requires?.bins).toEqual(['rg'])
})
it('parses clawdis install specs and os', () => {
const frontmatter = parseFrontmatter(
`---\nmetadata: {"clawdis":{"install":[{"kind":"brew","formula":"rg"},{"kind":"nope"},{"kind":"node","package":"x"}],"os":"macos,linux","requires":{"anyBins":["rg","fd"]}}}\n---\nBody`,
@@ -50,6 +107,35 @@ describe('skills utils', () => {
expect(clawdis?.requires?.anyBins).toEqual(['rg', 'fd'])
})
it('parses clawdbot metadata with nix plugin pointer', () => {
const frontmatter = parseFrontmatter(
`---\nmetadata: {"clawdbot":{"nix":{"plugin":"github:clawdbot/nix-steipete-tools?dir=tools/peekaboo","systems":["aarch64-darwin"]}}}\n---\nBody`,
)
const clawdis = parseClawdisMetadata(frontmatter)
expect(clawdis?.nix?.plugin).toBe('github:clawdbot/nix-steipete-tools?dir=tools/peekaboo')
expect(clawdis?.nix?.systems).toEqual(['aarch64-darwin'])
})
it('parses clawdbot config requirements with example', () => {
const frontmatter = parseFrontmatter(
`---\nmetadata: {"clawdbot":{"config":{"requiredEnv":["PADEL_AUTH_FILE"],"stateDirs":[".config/padel"],"example":"config = { env = { PADEL_AUTH_FILE = \\"/run/agenix/padel-auth\\"; }; };"}}}\n---\nBody`,
)
const clawdis = parseClawdisMetadata(frontmatter)
expect(clawdis?.config?.requiredEnv).toEqual(['PADEL_AUTH_FILE'])
expect(clawdis?.config?.stateDirs).toEqual(['.config/padel'])
expect(clawdis?.config?.example).toBe(
'config = { env = { PADEL_AUTH_FILE = "/run/agenix/padel-auth"; }; };',
)
})
it('parses cli help output', () => {
const frontmatter = parseFrontmatter(
`---\nmetadata: {"clawdbot":{"cliHelp":"padel --help\\nUsage: padel [command]\\n"}}\n---\nBody`,
)
const clawdis = parseClawdisMetadata(frontmatter)
expect(clawdis?.cliHelp).toBe('padel --help\nUsage: padel [command]')
})
it('sanitizes file paths', () => {
expect(sanitizePath('good/file.md')).toBe('good/file.md')
expect(sanitizePath('../bad/file.md')).toBeNull()
@@ -88,6 +174,15 @@ describe('skills utils', () => {
expect(text.length).toBe(10)
})
it('truncates embedding text by default max chars', () => {
const text = buildEmbeddingText({
frontmatter: {},
readme: 'x'.repeat(40_000),
otherFiles: [],
})
expect(text.length).toBeLessThanOrEqual(12_000)
})
it('hashes skill files deterministically', async () => {
const a = await hashSkillFiles([
{ path: 'b.txt', sha256: 'b' },
+118 -30
View File
@@ -1,16 +1,20 @@
import {
type ClawdbotConfigSpec,
type ClawdisSkillMetadata,
ClawdisSkillMetadataSchema,
isTextContentType,
type NixPluginSpec,
parseArk,
type SkillInstallSpec,
TEXT_FILE_EXTENSION_SET,
} from 'clawdhub-schema'
} from 'clawhub-schema'
import { parse as parseYaml } from 'yaml'
export type ParsedSkillFrontmatter = Record<string, string>
export type ParsedSkillFrontmatter = Record<string, unknown>
export type { ClawdisSkillMetadata, SkillInstallSpec }
const FRONTMATTER_START = '---'
const DEFAULT_EMBEDDING_MAX_CHARS = 12_000
export function parseFrontmatter(content: string): ParsedSkillFrontmatter {
const frontmatter: ParsedSkillFrontmatter = {}
@@ -19,14 +23,19 @@ export function parseFrontmatter(content: string): ParsedSkillFrontmatter {
const endIndex = normalized.indexOf(`\n${FRONTMATTER_START}`, 3)
if (endIndex === -1) return frontmatter
const block = normalized.slice(4, endIndex)
for (const line of block.split('\n')) {
const match = line.match(/^([\w-]+):\s*(.*)$/)
if (!match) continue
const key = match[1]
const rawValue = match[2].trim()
if (!key || !rawValue) continue
frontmatter[key] = stripQuotes(rawValue)
try {
const parsed = parseYaml(block) as unknown
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return frontmatter
for (const [key, value] of Object.entries(parsed as Record<string, unknown>)) {
if (!/^[\w-]+$/.test(key)) continue
const jsonValue = toJsonValue(value)
if (jsonValue !== undefined) frontmatter[key] = jsonValue
}
} catch {
return frontmatter
}
return frontmatter
}
@@ -35,15 +44,40 @@ export function getFrontmatterValue(frontmatter: ParsedSkillFrontmatter, key: st
return typeof raw === 'string' ? raw : undefined
}
export function parseClawdisMetadata(frontmatter: ParsedSkillFrontmatter) {
const raw = getFrontmatterValue(frontmatter, 'metadata')
export function getFrontmatterMetadata(frontmatter: ParsedSkillFrontmatter) {
const raw = frontmatter.metadata
if (!raw) return undefined
if (typeof raw === 'string') {
try {
const parsed = JSON.parse(raw) as unknown
return parsed ?? undefined
} catch {
return undefined
}
}
if (typeof raw === 'object') return raw
return undefined
}
export function parseClawdisMetadata(frontmatter: ParsedSkillFrontmatter) {
const metadata = getFrontmatterMetadata(frontmatter)
const metadataRecord =
metadata && typeof metadata === 'object' && !Array.isArray(metadata)
? (metadata as Record<string, unknown>)
: undefined
const clawdbotMeta = metadataRecord?.clawdbot
const clawdisMeta = metadataRecord?.clawdis
const metadataSource =
clawdbotMeta && typeof clawdbotMeta === 'object' && !Array.isArray(clawdbotMeta)
? (clawdbotMeta as Record<string, unknown>)
: clawdisMeta && typeof clawdisMeta === 'object' && !Array.isArray(clawdisMeta)
? (clawdisMeta as Record<string, unknown>)
: undefined
const clawdisRaw = metadataSource ?? frontmatter.clawdis
if (!clawdisRaw || typeof clawdisRaw !== 'object' || Array.isArray(clawdisRaw)) return undefined
try {
const parsed = JSON.parse(raw) as { clawdis?: unknown }
if (!parsed || typeof parsed !== 'object') return undefined
const clawdis = (parsed as { clawdis?: unknown }).clawdis
if (!clawdis || typeof clawdis !== 'object') return undefined
const clawdisObj = clawdis as Record<string, unknown>
const clawdisObj = clawdisRaw as Record<string, unknown>
const requiresRaw =
typeof clawdisObj.requires === 'object' && clawdisObj.requires !== null
? (clawdisObj.requires as Record<string, unknown>)
@@ -60,6 +94,7 @@ export function parseClawdisMetadata(frontmatter: ParsedSkillFrontmatter) {
if (typeof clawdisObj.homepage === 'string') metadata.homepage = clawdisObj.homepage
if (typeof clawdisObj.skillKey === 'string') metadata.skillKey = clawdisObj.skillKey
if (typeof clawdisObj.primaryEnv === 'string') metadata.primaryEnv = clawdisObj.primaryEnv
if (typeof clawdisObj.cliHelp === 'string') metadata.cliHelp = clawdisObj.cliHelp
if (osRaw.length > 0) metadata.os = osRaw
if (requiresRaw) {
@@ -77,6 +112,10 @@ export function parseClawdisMetadata(frontmatter: ParsedSkillFrontmatter) {
}
if (install.length > 0) metadata.install = install
const nix = parseNixPluginSpec(clawdisObj.nix)
if (nix) metadata.nix = nix
const config = parseClawdbotConfigSpec(clawdisObj.config)
if (config) metadata.config = config
return parseArk(ClawdisSkillMetadataSchema, metadata, 'Clawdis metadata')
} catch {
@@ -110,14 +149,14 @@ export function buildEmbeddingText(params: {
otherFiles: Array<{ path: string; content: string }>
maxChars?: number
}) {
const { frontmatter, readme, otherFiles, maxChars = 200_000 } = params
const { frontmatter, readme, otherFiles, maxChars = DEFAULT_EMBEDDING_MAX_CHARS } = params
const headerParts = [
frontmatter.name,
frontmatter.description,
frontmatter.homepage,
frontmatter.website,
frontmatter.url,
frontmatter.emoji,
getFrontmatterValue(frontmatter, 'name'),
getFrontmatterValue(frontmatter, 'description'),
getFrontmatterValue(frontmatter, 'homepage'),
getFrontmatterValue(frontmatter, 'website'),
getFrontmatterValue(frontmatter, 'url'),
getFrontmatterValue(frontmatter, 'emoji'),
].filter(Boolean)
const fileParts = otherFiles.map((file) => `# ${file.path}\n${file.content}`)
const raw = [headerParts.join('\n'), readme, ...fileParts].filter(Boolean).join('\n\n')
@@ -137,14 +176,32 @@ export async function hashSkillFiles(files: Array<{ path: string; sha256: string
return toHex(new Uint8Array(digest))
}
function stripQuotes(value: string) {
if (
(value.startsWith('"') && value.endsWith('"')) ||
(value.startsWith("'") && value.endsWith("'"))
) {
return value.slice(1, -1)
function toJsonValue(value: unknown): unknown {
if (value === null) return null
if (value === undefined) return undefined
if (typeof value === 'string') {
const trimmedEnd = value.trimEnd()
return trimmedEnd.trim() ? trimmedEnd : undefined
}
return value
if (typeof value === 'number') return Number.isFinite(value) ? value : undefined
if (typeof value === 'boolean') return value
if (typeof value === 'bigint') return value.toString()
if (value instanceof Date) return value.toISOString()
if (Array.isArray(value)) {
return value.map((entry) => {
const next = toJsonValue(entry)
return next === undefined ? null : next
})
}
if (isPlainObject(value)) {
const out: Record<string, unknown> = {}
for (const [key, entry] of Object.entries(value)) {
const next = toJsonValue(entry)
if (next !== undefined) out[key] = next
}
return out
}
return undefined
}
function normalizeStringList(input: unknown): string[] {
@@ -181,8 +238,39 @@ function parseInstallSpec(input: unknown): SkillInstallSpec | undefined {
return spec
}
function parseNixPluginSpec(input: unknown): NixPluginSpec | undefined {
if (!input || typeof input !== 'object') return undefined
const raw = input as Record<string, unknown>
if (typeof raw.plugin !== 'string') return undefined
const plugin = raw.plugin.trim()
if (!plugin) return undefined
const systems = normalizeStringList(raw.systems)
const spec: NixPluginSpec = { plugin }
if (systems.length > 0) spec.systems = systems
return spec
}
function parseClawdbotConfigSpec(input: unknown): ClawdbotConfigSpec | undefined {
if (!input || typeof input !== 'object') return undefined
const raw = input as Record<string, unknown>
const requiredEnv = normalizeStringList(raw.requiredEnv)
const stateDirs = normalizeStringList(raw.stateDirs)
const example = typeof raw.example === 'string' ? raw.example.trim() : ''
const spec: ClawdbotConfigSpec = {}
if (requiredEnv.length > 0) spec.requiredEnv = requiredEnv
if (stateDirs.length > 0) spec.stateDirs = stateDirs
if (example) spec.example = example
return Object.keys(spec).length > 0 ? spec : undefined
}
function toHex(bytes: Uint8Array) {
let out = ''
for (const byte of bytes) out += byte.toString(16).padStart(2, '0')
return out
}
function isPlainObject(value: unknown): value is Record<string, unknown> {
if (!value || typeof value !== 'object') return false
const proto = Object.getPrototypeOf(value)
return proto === Object.prototype || proto === null
}
+273
View File
@@ -0,0 +1,273 @@
import { internal } from '../_generated/api'
import type { Doc } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? 'gpt-4.1'
const MAX_README_CHARS = 8_000
const MAX_PATHS_IN_PROMPT = 30
type FileMeta = { path: string; sha256?: string }
type FileDiffSummary = {
added: string[]
removed: string[]
changed: string[]
}
function clampText(value: string, maxChars: number) {
const trimmed = value.trim()
if (trimmed.length <= maxChars) return trimmed
return `${trimmed.slice(0, maxChars).trimEnd()}\n…`
}
function summarizeFileDiff(oldFiles: FileMeta[], nextFiles: FileMeta[]): FileDiffSummary {
const oldByPath = new Map(oldFiles.map((f) => [f.path, f] as const))
const nextByPath = new Map(nextFiles.map((f) => [f.path, f] as const))
const added: string[] = []
const removed: string[] = []
const changed: string[] = []
for (const [path, file] of nextByPath.entries()) {
const prev = oldByPath.get(path)
if (!prev) {
added.push(path)
continue
}
if (file.sha256 && prev.sha256 && file.sha256 !== prev.sha256) changed.push(path)
}
for (const path of oldByPath.keys()) {
if (!nextByPath.has(path)) removed.push(path)
}
added.sort()
removed.sort()
changed.sort()
return { added, removed, changed }
}
function formatDiffSummary(diff: FileDiffSummary) {
const parts: string[] = []
if (diff.added.length) parts.push(`${diff.added.length} added`)
if (diff.changed.length) parts.push(`${diff.changed.length} changed`)
if (diff.removed.length) parts.push(`${diff.removed.length} removed`)
return parts.join(', ') || 'no file changes detected'
}
function pickPaths(values: string[]) {
if (values.length <= MAX_PATHS_IN_PROMPT) return values
return values.slice(0, MAX_PATHS_IN_PROMPT)
}
function extractResponseText(payload: unknown) {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
async function generateWithOpenAI(args: {
slug: string
version: string
oldReadme: string | null
nextReadme: string
fileDiff: FileDiffSummary | null
}) {
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) return null
const oldReadme = args.oldReadme ? clampText(args.oldReadme, MAX_README_CHARS) : ''
const nextReadme = clampText(args.nextReadme, MAX_README_CHARS)
const fileDiff = args.fileDiff
const diffSummary = fileDiff ? formatDiffSummary(fileDiff) : 'unknown'
const changedPaths = fileDiff ? pickPaths(fileDiff.changed) : []
const addedPaths = fileDiff ? pickPaths(fileDiff.added) : []
const removedPaths = fileDiff ? pickPaths(fileDiff.removed) : []
const input = [
`Soul: ${args.slug}`,
`Version: ${args.version}`,
`File changes: ${diffSummary}`,
changedPaths.length ? `Changed files (sample): ${changedPaths.join(', ')}` : null,
addedPaths.length ? `Added files (sample): ${addedPaths.join(', ')}` : null,
removedPaths.length ? `Removed files (sample): ${removedPaths.join(', ')}` : null,
oldReadme ? `Previous SOUL.md:\n${oldReadme}` : null,
`New SOUL.md:\n${nextReadme}`,
]
.filter(Boolean)
.join('\n\n')
const response = await fetch('https://api.openai.com/v1/responses', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: CHANGELOG_MODEL,
instructions:
'Write a concise changelog for this soul version. Audience: everyone. Output plain text. Prefer 26 bullet points. If it is a big change, include a short 1-line summary first, then bullets. Dont mention that you are AI. Dont invent details; only use the inputs.',
input,
max_output_tokens: 220,
}),
})
if (!response.ok) return null
const payload = (await response.json()) as unknown
return extractResponseText(payload)
}
function generateFallback(args: {
slug: string
version: string
oldReadme: string | null
nextReadme: string
fileDiff: FileDiffSummary | null
}) {
const lines: string[] = []
if (!args.oldReadme) {
lines.push(`- Initial release.`)
return lines.join('\n')
}
const diff = args.fileDiff
if (diff) {
const parts: string[] = []
if (diff.added.length) parts.push(`added ${diff.added.length}`)
if (diff.changed.length) parts.push(`updated ${diff.changed.length}`)
if (diff.removed.length) parts.push(`removed ${diff.removed.length}`)
if (parts.length) lines.push(`- ${parts.join(', ')} file(s).`)
}
lines.push(`- Updated SOUL.md.`)
return lines.join('\n')
}
export async function generateSoulChangelogForPublish(
ctx: ActionCtx,
args: { slug: string; version: string; readmeText: string; files: FileMeta[] },
): Promise<string> {
try {
const soul = (await ctx.runQuery(internal.souls.getSoulBySlugInternal, {
slug: args.slug,
})) as Doc<'souls'> | null
const previous: Doc<'soulVersions'> | null =
soul?.latestVersionId && !soul.softDeletedAt
? ((await ctx.runQuery(internal.souls.getVersionByIdInternal, {
versionId: soul.latestVersionId,
})) as Doc<'soulVersions'> | null)
: null
const oldReadmeText: string | null = previous
? await readReadmeFromVersion(ctx, previous)
: null
const oldFiles = previous
? previous.files.map((file) => ({ path: file.path, sha256: file.sha256 }))
: []
const fileDiff = previous ? summarizeFileDiff(oldFiles, args.files) : null
const ai = await generateWithOpenAI({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
}).catch(() => null)
return (
ai ??
generateFallback({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
})
)
} catch {
return '- Updated soul.'
}
}
export async function generateSoulChangelogPreview(
ctx: ActionCtx,
args: {
slug: string
version: string
readmeText: string
filePaths?: string[]
},
): Promise<string> {
try {
const soul = (await ctx.runQuery(internal.souls.getSoulBySlugInternal, {
slug: args.slug,
})) as Doc<'souls'> | null
const previous: Doc<'soulVersions'> | null =
soul?.latestVersionId && !soul.softDeletedAt
? ((await ctx.runQuery(internal.souls.getVersionByIdInternal, {
versionId: soul.latestVersionId,
})) as Doc<'soulVersions'> | null)
: null
const oldReadmeText: string | null = previous
? await readReadmeFromVersion(ctx, previous)
: null
const oldPaths = previous ? previous.files.map((file) => file.path) : []
const nextPaths = args.filePaths ?? []
const diff = previous ? summarizeFileDiffFromPaths(oldPaths, nextPaths) : null
const ai = await generateWithOpenAI({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff: diff,
}).catch(() => null)
return (
ai ??
generateFallback({
slug: args.slug,
version: args.version,
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff: diff,
})
)
} catch {
return '- Updated soul.'
}
}
async function readReadmeFromVersion(ctx: ActionCtx, version: Doc<'soulVersions'>) {
const file = version.files.find((entry) => entry.path.toLowerCase() === 'soul.md')
if (!file) return null
const blob = await ctx.storage.get(file.storageId)
if (!blob) return null
return blob.text()
}
function summarizeFileDiffFromPaths(oldPaths: string[], nextPaths: string[]) {
const oldFiles = oldPaths.map((path) => ({ path }))
const nextFiles = nextPaths.map((path) => ({ path }))
return summarizeFileDiff(oldFiles, nextFiles)
}
export const __test = {
summarizeFileDiff,
}
+236
View File
@@ -0,0 +1,236 @@
import { ConvexError } from 'convex/values'
import semver from 'semver'
import { internal } from '../_generated/api'
import type { Doc, Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { generateEmbedding } from './embeddings'
import {
buildEmbeddingText,
getFrontmatterMetadata,
getFrontmatterValue,
hashSkillFiles,
isTextFile,
parseFrontmatter,
sanitizePath,
} from './skills'
import { generateSoulChangelogForPublish } from './soulChangelog'
const MAX_TOTAL_BYTES = 50 * 1024 * 1024
const MAX_SUMMARY_LENGTH = 160
function deriveSoulSummary(readmeText: string) {
const lines = readmeText.split(/\r?\n/)
let inFrontmatter = false
for (const raw of lines) {
const trimmed = raw.trim()
if (!trimmed) continue
if (!inFrontmatter && trimmed === '---') {
inFrontmatter = true
continue
}
if (inFrontmatter) {
if (trimmed === '---') {
inFrontmatter = false
}
continue
}
const cleaned = trimmed.replace(/^#+\s*/, '')
if (!cleaned) continue
if (cleaned.length > MAX_SUMMARY_LENGTH) {
return `${cleaned.slice(0, MAX_SUMMARY_LENGTH - 3).trimEnd()}...`
}
return cleaned
}
return undefined
}
export type PublishResult = {
soulId: Id<'souls'>
versionId: Id<'soulVersions'>
embeddingId: Id<'soulEmbeddings'>
}
export type PublishVersionArgs = {
slug: string
displayName: string
version: string
changelog: string
tags?: string[]
source?: {
kind: 'github'
url: string
repo: string
ref: string
commit: string
path: string
importedAt: number
}
files: Array<{
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}>
}
export async function publishSoulVersionForUser(
ctx: ActionCtx,
userId: Id<'users'>,
args: PublishVersionArgs,
): Promise<PublishResult> {
const version = args.version.trim()
const slug = args.slug.trim().toLowerCase()
const displayName = args.displayName.trim()
if (!slug || !displayName) throw new ConvexError('Slug and display name required')
if (!/^[a-z0-9][a-z0-9-]*$/.test(slug)) {
throw new ConvexError('Slug must be lowercase and url-safe')
}
if (!semver.valid(version)) {
throw new ConvexError('Version must be valid semver')
}
const suppliedChangelog = args.changelog.trim()
const changelogSource = suppliedChangelog ? ('user' as const) : ('auto' as const)
const sanitizedFiles = args.files.map((file) => {
const path = sanitizePath(file.path)
if (!path) throw new ConvexError('Invalid file paths')
if (!isTextFile(path, file.contentType ?? undefined)) {
throw new ConvexError('Only text-based files are allowed')
}
return { ...file, path }
})
const totalBytes = sanitizedFiles.reduce((sum, file) => sum + file.size, 0)
if (totalBytes > MAX_TOTAL_BYTES) {
throw new ConvexError('Soul bundle exceeds 50MB limit')
}
const isSoulFile = (path: string) => path.toLowerCase() === 'soul.md'
const readmeFile = sanitizedFiles.find((file) => isSoulFile(file.path))
if (!readmeFile) throw new ConvexError('SOUL.md is required')
const nonSoulFiles = sanitizedFiles.filter((file) => !isSoulFile(file.path))
if (nonSoulFiles.length > 0) {
throw new ConvexError('Only SOUL.md is allowed for soul bundles')
}
const readmeText = await fetchText(ctx, readmeFile.storageId)
const frontmatter = parseFrontmatter(readmeText)
const summary = getFrontmatterValue(frontmatter, 'description') ?? deriveSoulSummary(readmeText)
const metadata = mergeSourceIntoMetadata(getFrontmatterMetadata(frontmatter), args.source)
const embeddingText = buildEmbeddingText({
frontmatter,
readme: readmeText,
otherFiles: [],
})
const fingerprint = await hashSkillFiles(
sanitizedFiles.map((file) => ({
path: file.path ?? '',
sha256: file.sha256,
})),
)
const changelogPromise =
changelogSource === 'user'
? Promise.resolve(suppliedChangelog)
: generateSoulChangelogForPublish(ctx, {
slug,
version,
readmeText,
files: sanitizedFiles.map((file) => ({ path: file.path ?? '', sha256: file.sha256 })),
})
const embeddingPromise = generateEmbedding(embeddingText)
const [changelogText, embedding] = await Promise.all([
changelogPromise,
embeddingPromise.catch((error) => {
throw new ConvexError(formatEmbeddingError(error))
}),
])
const publishResult = (await ctx.runMutation(internal.souls.insertVersion, {
userId,
slug,
displayName,
version,
changelog: changelogText,
changelogSource,
tags: args.tags?.map((tag) => tag.trim()).filter(Boolean),
fingerprint,
files: sanitizedFiles,
parsed: {
frontmatter,
metadata,
},
summary,
embedding,
})) as PublishResult
const owner = (await ctx.runQuery(internal.users.getByIdInternal, {
userId,
})) as Doc<'users'> | null
const ownerHandle = owner?.handle ?? owner?.name ?? userId
void ctx.scheduler
.runAfter(0, internal.githubSoulBackupsNode.backupSoulForPublishInternal, {
slug,
version,
displayName,
ownerHandle,
files: sanitizedFiles,
publishedAt: Date.now(),
})
.catch((error) => {
console.error('GitHub soul backup scheduling failed', error)
})
return publishResult
}
function mergeSourceIntoMetadata(metadata: unknown, source: PublishVersionArgs['source']) {
if (!source) return metadata === undefined ? undefined : metadata
const sourceValue = {
kind: source.kind,
url: source.url,
repo: source.repo,
ref: source.ref,
commit: source.commit,
path: source.path,
importedAt: source.importedAt,
}
if (!metadata) return { source: sourceValue }
if (typeof metadata !== 'object' || Array.isArray(metadata)) return { source: sourceValue }
return { ...(metadata as Record<string, unknown>), source: sourceValue }
}
export async function fetchText(
ctx: { storage: { get: (id: Id<'_storage'>) => Promise<Blob | null> } },
storageId: Id<'_storage'>,
) {
const blob = await ctx.storage.get(storageId)
if (!blob) throw new Error('File missing in storage')
return blob.text()
}
function formatEmbeddingError(error: unknown) {
if (error instanceof Error) {
if (error.message.includes('OPENAI_API_KEY')) {
return 'OPENAI_API_KEY is not configured.'
}
if (error.message.startsWith('Embedding failed')) {
return error.message
}
}
return 'Embedding failed. Please try again.'
}
export const __test = {
getSummary: (frontmatter: Record<string, unknown>) =>
getFrontmatterValue(frontmatter, 'description'),
}
+26 -13
View File
@@ -1,20 +1,33 @@
/* @vitest-environment node */
import { describe, expect, it } from 'vitest'
import { API_TOKEN_PREFIX, generateToken, hashToken } from './tokens'
import { __test, generateToken, hashToken } from './tokens'
describe('tokens', () => {
it('generates token with prefix and url-safe chars', () => {
const { token, prefix } = generateToken()
expect(token.startsWith(API_TOKEN_PREFIX)).toBe(true)
expect(prefix).toBe(token.slice(0, 12))
expect(token).toMatch(/^[a-z0-9_-]+$/i)
it('hashToken returns sha256 hex', async () => {
await expect(hashToken('test')).resolves.toBe(
'9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08',
)
})
it('hashes tokens deterministically', async () => {
const a = await hashToken('clh_test')
const b = await hashToken('clh_test')
const c = await hashToken('clh_other')
expect(a).toBe(b)
expect(a).not.toBe(c)
expect(a).toMatch(/^[a-f0-9]{64}$/)
it('generateToken returns token + prefix', () => {
const { token, prefix } = generateToken()
expect(token).toMatch(/^clh_[A-Za-z0-9_-]+$/)
expect(prefix).toBe(token.slice(0, 12))
})
it('toHex encodes bytes', () => {
expect(__test.toHex(new Uint8Array([0, 15, 255]))).toBe('000fff')
})
it('toBase64 encodes 1/2/3-byte tails', () => {
expect(__test.toBase64(new Uint8Array([0xff]))).toBe('/w==')
expect(__test.toBase64(new Uint8Array([0xff, 0xee]))).toBe('/+4=')
expect(__test.toBase64(new Uint8Array([0xff, 0xee, 0xdd]))).toBe('/+7d')
})
it('toBase64Url replaces alphabet and strips padding', () => {
expect(__test.toBase64Url(new Uint8Array([0xff]))).toBe('_w')
expect(__test.toBase64Url(new Uint8Array([0xfa, 0x00, 0x00]))).toBe('-gAA')
})
})
+6
View File
@@ -43,3 +43,9 @@ function toBase64(bytes: Uint8Array) {
}
return output
}
export const __test = {
toHex,
toBase64,
toBase64Url,
}
+91
View File
@@ -0,0 +1,91 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it } from 'vitest'
import { buildDiscordPayload, buildSkillUrl, getWebhookConfig, shouldSendWebhook } from './webhooks'
const originalEnv = { ...process.env }
afterEach(() => {
process.env = { ...originalEnv }
})
describe('webhook config', () => {
it('parses highlighted-only flag', () => {
process.env.DISCORD_WEBHOOK_URL = 'https://example.com'
process.env.DISCORD_WEBHOOK_HIGHLIGHTED_ONLY = 'true'
const config = getWebhookConfig()
expect(config.highlightedOnly).toBe(true)
})
it('defaults site url when missing', () => {
delete process.env.SITE_URL
process.env.DISCORD_WEBHOOK_URL = 'https://example.com'
const config = getWebhookConfig()
expect(config.siteUrl).toBe('https://clawhub.ai')
})
})
describe('webhook filtering', () => {
it('skips when url missing', () => {
const config = getWebhookConfig({} as NodeJS.ProcessEnv)
expect(shouldSendWebhook('skill.publish', { slug: 'demo', displayName: 'Demo' }, config)).toBe(
false,
)
})
it('filters non-highlighted when highlighted-only', () => {
const config = {
url: 'https://example.com',
highlightedOnly: true,
siteUrl: 'https://clawhub.ai',
}
const allowed = shouldSendWebhook(
'skill.publish',
{ slug: 'demo', displayName: 'Demo', highlighted: false },
config,
)
expect(allowed).toBe(false)
})
it('allows highlighted event when highlighted-only', () => {
const config = {
url: 'https://example.com',
highlightedOnly: true,
siteUrl: 'https://clawhub.ai',
}
const allowed = shouldSendWebhook(
'skill.highlighted',
{ slug: 'demo', displayName: 'Demo', highlighted: true },
config,
)
expect(allowed).toBe(true)
})
})
describe('payload building', () => {
it('builds canonical url with owner', () => {
const url = buildSkillUrl(
{ slug: 'beeper', displayName: 'Beeper', ownerHandle: 'KrauseFx' },
'https://clawhub.ai',
)
expect(url).toBe('https://clawhub.ai/KrauseFx/beeper')
})
it('builds a publish embed', () => {
const payload = buildDiscordPayload(
'skill.publish',
{
slug: 'demo',
displayName: 'Demo Skill',
summary: 'Nice skill',
version: '1.2.3',
ownerHandle: 'steipete',
tags: ['latest', 'discord'],
},
{ url: 'https://example.com', highlightedOnly: false, siteUrl: 'https://clawhub.ai' },
)
const embed = payload.embeds[0]
expect(embed.title).toBe('Demo Skill')
expect(embed.description).toBe('Nice skill')
expect(embed.fields[0].value).toBe('v1.2.3')
})
})
+112
View File
@@ -0,0 +1,112 @@
export type WebhookEvent = 'skill.publish' | 'skill.highlighted'
export type WebhookSkillPayload = {
slug: string
displayName: string
summary?: string
version?: string
ownerHandle?: string
highlighted?: boolean
tags?: string[]
}
export type WebhookConfig = {
url: string | null
highlightedOnly: boolean
siteUrl: string
}
const DEFAULT_SITE_URL = 'https://clawhub.ai'
export function getWebhookConfig(env: NodeJS.ProcessEnv = process.env): WebhookConfig {
const url = env.DISCORD_WEBHOOK_URL?.trim() || null
const highlightedOnly = parseBoolean(env.DISCORD_WEBHOOK_HIGHLIGHTED_ONLY)
const siteUrl = env.SITE_URL?.trim() || DEFAULT_SITE_URL
return { url, highlightedOnly, siteUrl }
}
export function shouldSendWebhook(
event: WebhookEvent,
skill: WebhookSkillPayload,
config: WebhookConfig,
) {
if (!config.url) return false
if (!config.highlightedOnly) return true
if (event === 'skill.highlighted') return true
return Boolean(skill.highlighted)
}
export function buildDiscordPayload(
event: WebhookEvent,
skill: WebhookSkillPayload,
config: WebhookConfig,
) {
const titleBase = skill.displayName || skill.slug
const title = event === 'skill.highlighted' ? `Highlighted: ${titleBase}` : titleBase
const description = buildDescription(event, skill)
const url = buildSkillUrl(skill, config.siteUrl)
const tags = formatTags(skill.tags)
return {
embeds: [
{
title,
description,
url,
color: event === 'skill.highlighted' ? 0xff6b4a : 0x2f76ff,
fields: [
{
name: 'Version',
value: skill.version ? `v${skill.version}` : '—',
inline: true,
},
{
name: 'Owner',
value: skill.ownerHandle ? `@${skill.ownerHandle}` : '—',
inline: true,
},
{
name: 'Tags',
value: tags,
inline: false,
},
],
footer: {
text: 'OpenClaw',
},
timestamp: new Date().toISOString(),
},
],
}
}
export function buildSkillUrl(skill: WebhookSkillPayload, siteUrl: string) {
const owner = skill.ownerHandle?.trim()
if (owner) return `${siteUrl}/${owner}/${skill.slug}`
return `${siteUrl}/skills/${skill.slug}`
}
function buildDescription(event: WebhookEvent, skill: WebhookSkillPayload) {
const summary = (skill.summary ?? '').trim()
if (summary) return truncate(summary, 200)
if (event === 'skill.highlighted') return 'Newly highlighted skill on OpenClaw.'
if (skill.version) return `New version v${skill.version} published on OpenClaw.`
return 'New skill published on OpenClaw.'
}
function parseBoolean(value?: string) {
if (!value) return false
const normalized = value.trim().toLowerCase()
return normalized === '1' || normalized === 'true' || normalized === 'yes' || normalized === 'on'
}
function formatTags(tags?: string[] | null) {
const cleaned = (tags ?? []).map((tag) => tag.trim()).filter(Boolean)
if (cleaned.length === 0) return '—'
return cleaned.slice(0, 8).join(', ')
}
function truncate(value: string, max: number) {
if (value.length <= max) return value
return `${value.slice(0, max - 1).trim()}`
}
+270
View File
@@ -0,0 +1,270 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from 'vitest'
vi.mock('./_generated/api', () => ({
internal: {
maintenance: {
getSkillBackfillPageInternal: Symbol('getSkillBackfillPageInternal'),
applySkillBackfillPatchInternal: Symbol('applySkillBackfillPatchInternal'),
backfillSkillSummariesInternal: Symbol('backfillSkillSummariesInternal'),
getSkillFingerprintBackfillPageInternal: Symbol('getSkillFingerprintBackfillPageInternal'),
applySkillFingerprintBackfillPatchInternal: Symbol(
'applySkillFingerprintBackfillPatchInternal',
),
backfillSkillFingerprintsInternal: Symbol('backfillSkillFingerprintsInternal'),
},
},
}))
const { backfillSkillFingerprintsInternalHandler, backfillSkillSummariesInternalHandler } =
await import('./maintenance')
function makeBlob(text: string) {
return { text: () => Promise.resolve(text) } as unknown as Blob
}
describe('maintenance backfill', () => {
it('repairs summary + parsed by reparsing SKILL.md', async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [
{
kind: 'ok',
skillId: 'skills:1',
versionId: 'skillVersions:1',
skillSummary: '>',
versionParsed: { frontmatter: { description: '>' } },
readmeStorageId: 'storage:1',
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const storageGet = vi
.fn()
.mockResolvedValue(makeBlob(`---\ndescription: >\n Hello\n world.\n---\nBody`))
const result = await backfillSkillSummariesInternalHandler(
{ runQuery, runMutation, storage: { get: storageGet } } as never,
{ dryRun: false, batchSize: 10, maxBatches: 1 },
)
expect(result.ok).toBe(true)
expect(result.stats.skillsScanned).toBe(1)
expect(result.stats.skillsPatched).toBe(1)
expect(result.stats.versionsPatched).toBe(1)
expect(runMutation).toHaveBeenCalledTimes(1)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
skillId: 'skills:1',
versionId: 'skillVersions:1',
summary: 'Hello world.',
parsed: {
frontmatter: { description: 'Hello world.' },
metadata: undefined,
clawdis: undefined,
},
})
})
it('dryRun does not patch', async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [
{
kind: 'ok',
skillId: 'skills:1',
versionId: 'skillVersions:1',
skillSummary: '>',
versionParsed: { frontmatter: { description: '>' } },
readmeStorageId: 'storage:1',
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn()
const storageGet = vi.fn().mockResolvedValue(makeBlob(`---\ndescription: Hello\n---\nBody`))
const result = await backfillSkillSummariesInternalHandler(
{ runQuery, runMutation, storage: { get: storageGet } } as never,
{ dryRun: true, batchSize: 10, maxBatches: 1 },
)
expect(result.ok).toBe(true)
expect(result.stats.skillsPatched).toBe(1)
expect(runMutation).not.toHaveBeenCalled()
})
it('counts missing storage blob', async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [
{
kind: 'ok',
skillId: 'skills:1',
versionId: 'skillVersions:1',
skillSummary: null,
versionParsed: { frontmatter: {} },
readmeStorageId: 'storage:missing',
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn()
const storageGet = vi.fn().mockResolvedValue(null)
const result = await backfillSkillSummariesInternalHandler(
{ runQuery, runMutation, storage: { get: storageGet } } as never,
{ dryRun: false, batchSize: 10, maxBatches: 1 },
)
expect(result.stats.missingStorageBlob).toBe(1)
expect(runMutation).not.toHaveBeenCalled()
})
})
describe('maintenance fingerprint backfill', () => {
it('backfills fingerprint field and inserts index entry', async () => {
const { hashSkillFiles } = await import('./lib/skills')
const expected = await hashSkillFiles([{ path: 'SKILL.md', sha256: 'abc' }])
const runQuery = vi.fn().mockResolvedValue({
items: [
{
skillId: 'skills:1',
versionId: 'skillVersions:1',
versionFingerprint: undefined,
files: [{ path: 'SKILL.md', sha256: 'abc' }],
existingEntries: [],
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const result = await backfillSkillFingerprintsInternalHandler(
{ runQuery, runMutation } as never,
{ dryRun: false, batchSize: 10, maxBatches: 1 },
)
expect(result.ok).toBe(true)
expect(result.stats.versionsScanned).toBe(1)
expect(result.stats.versionsPatched).toBe(1)
expect(result.stats.fingerprintsInserted).toBe(1)
expect(result.stats.fingerprintMismatches).toBe(0)
expect(runMutation).toHaveBeenCalledTimes(1)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
versionId: 'skillVersions:1',
fingerprint: expected,
patchVersion: true,
replaceEntries: true,
existingEntryIds: [],
})
})
it('dryRun does not patch', async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [
{
skillId: 'skills:1',
versionId: 'skillVersions:1',
versionFingerprint: undefined,
files: [{ path: 'SKILL.md', sha256: 'abc' }],
existingEntries: [],
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn()
const result = await backfillSkillFingerprintsInternalHandler(
{ runQuery, runMutation } as never,
{ dryRun: true, batchSize: 10, maxBatches: 1 },
)
expect(result.ok).toBe(true)
expect(result.stats.versionsPatched).toBe(1)
expect(result.stats.fingerprintsInserted).toBe(1)
expect(runMutation).not.toHaveBeenCalled()
})
it('patches missing version fingerprint without touching correct entries', async () => {
const { hashSkillFiles } = await import('./lib/skills')
const expected = await hashSkillFiles([{ path: 'SKILL.md', sha256: 'abc' }])
const runQuery = vi.fn().mockResolvedValue({
items: [
{
skillId: 'skills:1',
versionId: 'skillVersions:1',
versionFingerprint: undefined,
files: [{ path: 'SKILL.md', sha256: 'abc' }],
existingEntries: [{ id: 'skillVersionFingerprints:1', fingerprint: expected }],
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const result = await backfillSkillFingerprintsInternalHandler(
{ runQuery, runMutation } as never,
{ dryRun: false, batchSize: 10, maxBatches: 1 },
)
expect(result.ok).toBe(true)
expect(result.stats.versionsPatched).toBe(1)
expect(result.stats.fingerprintsInserted).toBe(0)
expect(result.stats.fingerprintMismatches).toBe(0)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
versionId: 'skillVersions:1',
fingerprint: expected,
patchVersion: true,
replaceEntries: false,
existingEntryIds: [],
})
})
it('replaces mismatched fingerprint entries', async () => {
const { hashSkillFiles } = await import('./lib/skills')
const expected = await hashSkillFiles([{ path: 'SKILL.md', sha256: 'abc' }])
const runQuery = vi.fn().mockResolvedValue({
items: [
{
skillId: 'skills:1',
versionId: 'skillVersions:1',
versionFingerprint: 'wrong',
files: [{ path: 'SKILL.md', sha256: 'abc' }],
existingEntries: [{ id: 'skillVersionFingerprints:1', fingerprint: 'wrong' }],
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const result = await backfillSkillFingerprintsInternalHandler(
{ runQuery, runMutation } as never,
{ dryRun: false, batchSize: 10, maxBatches: 1 },
)
expect(result.ok).toBe(true)
expect(result.stats.fingerprintMismatches).toBe(1)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
versionId: 'skillVersions:1',
fingerprint: expected,
patchVersion: true,
replaceEntries: true,
existingEntryIds: ['skillVersionFingerprints:1'],
})
})
})
+840
View File
@@ -0,0 +1,840 @@
import { ConvexError, v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { action, internalAction, internalMutation, internalQuery } from './_generated/server'
import { assertRole, requireUserFromAction } from './lib/access'
import { buildSkillSummaryBackfillPatch, type ParsedSkillData } from './lib/skillBackfill'
import { hashSkillFiles } from './lib/skills'
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const DEFAULT_MAX_BATCHES = 20
const MAX_MAX_BATCHES = 200
type BackfillStats = {
skillsScanned: number
skillsPatched: number
versionsPatched: number
missingLatestVersion: number
missingReadme: number
missingStorageBlob: number
}
type BackfillPageItem =
| {
kind: 'ok'
skillId: Id<'skills'>
versionId: Id<'skillVersions'>
skillSummary: Doc<'skills'>['summary']
versionParsed: Doc<'skillVersions'>['parsed']
readmeStorageId: Id<'_storage'>
}
| { kind: 'missingLatestVersion'; skillId: Id<'skills'> }
| { kind: 'missingVersionDoc'; skillId: Id<'skills'>; versionId: Id<'skillVersions'> }
| { kind: 'missingReadme'; skillId: Id<'skills'>; versionId: Id<'skillVersions'> }
type BackfillPageResult = {
items: BackfillPageItem[]
cursor: string | null
isDone: boolean
}
export const getSkillBackfillPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BackfillPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const { page, isDone, continueCursor } = await ctx.db
.query('skills')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
const items: BackfillPageItem[] = []
for (const skill of page) {
if (!skill.latestVersionId) {
items.push({ kind: 'missingLatestVersion', skillId: skill._id })
continue
}
const version = await ctx.db.get(skill.latestVersionId)
if (!version) {
items.push({
kind: 'missingVersionDoc',
skillId: skill._id,
versionId: skill.latestVersionId,
})
continue
}
const readmeFile = version.files.find(
(file) => file.path.toLowerCase() === 'skill.md' || file.path.toLowerCase() === 'skills.md',
)
if (!readmeFile) {
items.push({ kind: 'missingReadme', skillId: skill._id, versionId: version._id })
continue
}
items.push({
kind: 'ok',
skillId: skill._id,
versionId: version._id,
skillSummary: skill.summary,
versionParsed: version.parsed,
readmeStorageId: readmeFile.storageId,
})
}
return { items, cursor: continueCursor, isDone }
},
})
export const applySkillBackfillPatchInternal = internalMutation({
args: {
skillId: v.id('skills'),
versionId: v.id('skillVersions'),
summary: v.optional(v.string()),
parsed: v.optional(
v.object({
frontmatter: v.record(v.string(), v.any()),
metadata: v.optional(v.any()),
clawdis: v.optional(v.any()),
}),
),
},
handler: async (ctx, args) => {
const now = Date.now()
if (typeof args.summary === 'string') {
await ctx.db.patch(args.skillId, { summary: args.summary, updatedAt: now })
}
if (args.parsed) {
await ctx.db.patch(args.versionId, { parsed: args.parsed })
}
return { ok: true as const }
},
})
export type BackfillActionArgs = {
dryRun?: boolean
batchSize?: number
maxBatches?: number
}
export type BackfillActionResult = { ok: true; stats: BackfillStats }
export async function backfillSkillSummariesInternalHandler(
ctx: ActionCtx,
args: BackfillActionArgs,
): Promise<BackfillActionResult> {
const dryRun = Boolean(args.dryRun)
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const totals: BackfillStats = {
skillsScanned: 0,
skillsPatched: 0,
versionsPatched: 0,
missingLatestVersion: 0,
missingReadme: 0,
missingStorageBlob: 0,
}
let cursor: string | null = null
let isDone = false
for (let i = 0; i < maxBatches; i++) {
const page = (await ctx.runQuery(internal.maintenance.getSkillBackfillPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as BackfillPageResult
cursor = page.cursor
isDone = page.isDone
for (const item of page.items) {
totals.skillsScanned++
if (item.kind === 'missingLatestVersion') {
totals.missingLatestVersion++
continue
}
if (item.kind === 'missingVersionDoc') {
totals.missingLatestVersion++
continue
}
if (item.kind === 'missingReadme') {
totals.missingReadme++
continue
}
const blob = await ctx.storage.get(item.readmeStorageId)
if (!blob) {
totals.missingStorageBlob++
continue
}
const readmeText = await blob.text()
const patch = buildSkillSummaryBackfillPatch({
readmeText,
currentSummary: item.skillSummary ?? undefined,
currentParsed: item.versionParsed as ParsedSkillData,
})
if (!patch.summary && !patch.parsed) continue
if (patch.summary) totals.skillsPatched++
if (patch.parsed) totals.versionsPatched++
if (dryRun) continue
await ctx.runMutation(internal.maintenance.applySkillBackfillPatchInternal, {
skillId: item.skillId,
versionId: item.versionId,
summary: patch.summary,
parsed: patch.parsed,
})
}
if (isDone) break
}
if (!isDone) {
throw new ConvexError('Backfill incomplete (maxBatches reached)')
}
return { ok: true as const, stats: totals }
}
export const backfillSkillSummariesInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: backfillSkillSummariesInternalHandler,
})
export const backfillSkillSummaries: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BackfillActionResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
return ctx.runAction(
internal.maintenance.backfillSkillSummariesInternal,
args,
) as Promise<BackfillActionResult>
},
})
export const scheduleBackfillSkillSummaries: ReturnType<typeof action> = action({
args: { dryRun: v.optional(v.boolean()) },
handler: async (ctx, args) => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
await ctx.scheduler.runAfter(0, internal.maintenance.backfillSkillSummariesInternal, {
dryRun: Boolean(args.dryRun),
batchSize: DEFAULT_BATCH_SIZE,
maxBatches: DEFAULT_MAX_BATCHES,
})
return { ok: true as const }
},
})
type FingerprintBackfillStats = {
versionsScanned: number
versionsPatched: number
fingerprintsInserted: number
fingerprintMismatches: number
}
type FingerprintBackfillPageItem = {
skillId: Id<'skills'>
versionId: Id<'skillVersions'>
versionFingerprint?: string
files: Array<{ path: string; sha256: string }>
existingEntries: Array<{ id: Id<'skillVersionFingerprints'>; fingerprint: string }>
}
type FingerprintBackfillPageResult = {
items: FingerprintBackfillPageItem[]
cursor: string | null
isDone: boolean
}
type BadgeBackfillStats = {
skillsScanned: number
skillsPatched: number
highlightsPatched: number
}
type SkillBadgeTableBackfillStats = {
skillsScanned: number
recordsInserted: number
}
type BadgeBackfillPageItem = {
skillId: Id<'skills'>
ownerUserId: Id<'users'>
createdAt?: number
updatedAt?: number
batch?: string
badges?: Doc<'skills'>['badges']
}
type BadgeBackfillPageResult = {
items: BadgeBackfillPageItem[]
cursor: string | null
isDone: boolean
}
type BadgeKind = Doc<'skillBadges'>['kind']
export const getSkillFingerprintBackfillPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<FingerprintBackfillPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const { page, isDone, continueCursor } = await ctx.db
.query('skillVersions')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
const items: FingerprintBackfillPageItem[] = []
for (const version of page) {
const existingEntries = await ctx.db
.query('skillVersionFingerprints')
.withIndex('by_version', (q) => q.eq('versionId', version._id))
.take(20)
const normalizedFiles = version.files.map((file) => ({
path: file.path,
sha256: file.sha256,
}))
const hasAnyEntry = existingEntries.length > 0
const entryFingerprints = new Set(existingEntries.map((entry) => entry.fingerprint))
const hasFingerprintMismatch =
typeof version.fingerprint === 'string' &&
hasAnyEntry &&
(entryFingerprints.size !== 1 || !entryFingerprints.has(version.fingerprint))
const needsFingerprintField = !version.fingerprint
const needsFingerprintEntry = !hasAnyEntry
if (!needsFingerprintField && !needsFingerprintEntry && !hasFingerprintMismatch) continue
items.push({
skillId: version.skillId,
versionId: version._id,
versionFingerprint: version.fingerprint ?? undefined,
files: normalizedFiles,
existingEntries: existingEntries.map((entry) => ({
id: entry._id,
fingerprint: entry.fingerprint,
})),
})
}
return { items, cursor: continueCursor, isDone }
},
})
export const applySkillFingerprintBackfillPatchInternal = internalMutation({
args: {
versionId: v.id('skillVersions'),
fingerprint: v.string(),
patchVersion: v.boolean(),
replaceEntries: v.boolean(),
existingEntryIds: v.optional(v.array(v.id('skillVersionFingerprints'))),
},
handler: async (ctx, args) => {
const version = await ctx.db.get(args.versionId)
if (!version) return { ok: false as const, reason: 'missingVersion' as const }
const now = Date.now()
if (args.patchVersion) {
await ctx.db.patch(version._id, { fingerprint: args.fingerprint })
}
if (args.replaceEntries) {
const existing = args.existingEntryIds ?? []
for (const id of existing) {
await ctx.db.delete(id)
}
await ctx.db.insert('skillVersionFingerprints', {
skillId: version.skillId,
versionId: version._id,
fingerprint: args.fingerprint,
createdAt: now,
})
}
return { ok: true as const }
},
})
export type FingerprintBackfillActionArgs = {
dryRun?: boolean
batchSize?: number
maxBatches?: number
}
export type FingerprintBackfillActionResult = { ok: true; stats: FingerprintBackfillStats }
export async function backfillSkillFingerprintsInternalHandler(
ctx: ActionCtx,
args: FingerprintBackfillActionArgs,
): Promise<FingerprintBackfillActionResult> {
const dryRun = Boolean(args.dryRun)
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const totals: FingerprintBackfillStats = {
versionsScanned: 0,
versionsPatched: 0,
fingerprintsInserted: 0,
fingerprintMismatches: 0,
}
let cursor: string | null = null
let isDone = false
for (let i = 0; i < maxBatches; i++) {
const page = (await ctx.runQuery(internal.maintenance.getSkillFingerprintBackfillPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as FingerprintBackfillPageResult
cursor = page.cursor
isDone = page.isDone
for (const item of page.items) {
totals.versionsScanned++
const fingerprint = await hashSkillFiles(item.files)
const existingFingerprints = new Set(item.existingEntries.map((entry) => entry.fingerprint))
const hasAnyEntry = item.existingEntries.length > 0
const entryIsCorrect =
hasAnyEntry && existingFingerprints.size === 1 && existingFingerprints.has(fingerprint)
const versionFingerprintIsCorrect = item.versionFingerprint === fingerprint
if (hasAnyEntry && !entryIsCorrect) totals.fingerprintMismatches++
const shouldPatchVersion = !versionFingerprintIsCorrect
const shouldReplaceEntries = !entryIsCorrect
if (!shouldPatchVersion && !shouldReplaceEntries) continue
if (shouldPatchVersion) totals.versionsPatched++
if (shouldReplaceEntries) totals.fingerprintsInserted++
if (dryRun) continue
await ctx.runMutation(internal.maintenance.applySkillFingerprintBackfillPatchInternal, {
versionId: item.versionId,
fingerprint,
patchVersion: shouldPatchVersion,
replaceEntries: shouldReplaceEntries,
existingEntryIds: shouldReplaceEntries ? item.existingEntries.map((entry) => entry.id) : [],
})
}
if (isDone) break
}
if (!isDone) {
throw new ConvexError('Backfill incomplete (maxBatches reached)')
}
return { ok: true as const, stats: totals }
}
export const backfillSkillFingerprintsInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: backfillSkillFingerprintsInternalHandler,
})
export const backfillSkillFingerprints: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: async (ctx, args): Promise<FingerprintBackfillActionResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
return ctx.runAction(
internal.maintenance.backfillSkillFingerprintsInternal,
args,
) as Promise<FingerprintBackfillActionResult>
},
})
export const scheduleBackfillSkillFingerprints: ReturnType<typeof action> = action({
args: { dryRun: v.optional(v.boolean()) },
handler: async (ctx, args) => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
await ctx.scheduler.runAfter(0, internal.maintenance.backfillSkillFingerprintsInternal, {
dryRun: Boolean(args.dryRun),
batchSize: DEFAULT_BATCH_SIZE,
maxBatches: DEFAULT_MAX_BATCHES,
})
return { ok: true as const }
},
})
export const getSkillBadgeBackfillPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BadgeBackfillPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const { page, isDone, continueCursor } = await ctx.db
.query('skills')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
const items: BadgeBackfillPageItem[] = page.map((skill) => ({
skillId: skill._id,
ownerUserId: skill.ownerUserId,
createdAt: skill.createdAt ?? undefined,
updatedAt: skill.updatedAt ?? undefined,
batch: skill.batch ?? undefined,
badges: skill.badges ?? undefined,
}))
return { items, cursor: continueCursor, isDone }
},
})
export const applySkillBadgeBackfillPatchInternal = internalMutation({
args: {
skillId: v.id('skills'),
badges: v.optional(
v.object({
redactionApproved: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
highlighted: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
official: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
deprecated: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
}),
),
},
handler: async (ctx, args) => {
await ctx.db.patch(args.skillId, { badges: args.badges ?? undefined, updatedAt: Date.now() })
return { ok: true as const }
},
})
export const upsertSkillBadgeRecordInternal = internalMutation({
args: {
skillId: v.id('skills'),
kind: v.union(
v.literal('highlighted'),
v.literal('official'),
v.literal('deprecated'),
v.literal('redactionApproved'),
),
byUserId: v.id('users'),
at: v.number(),
},
handler: async (ctx, args) => {
const existing = await ctx.db
.query('skillBadges')
.withIndex('by_skill_kind', (q) => q.eq('skillId', args.skillId).eq('kind', args.kind))
.unique()
if (existing) return { inserted: false as const }
await ctx.db.insert('skillBadges', {
skillId: args.skillId,
kind: args.kind,
byUserId: args.byUserId,
at: args.at,
})
return { inserted: true as const }
},
})
export type BadgeBackfillActionArgs = {
dryRun?: boolean
batchSize?: number
maxBatches?: number
}
export type BadgeBackfillActionResult = { ok: true; stats: BadgeBackfillStats }
export async function backfillSkillBadgesInternalHandler(
ctx: ActionCtx,
args: BadgeBackfillActionArgs,
): Promise<BadgeBackfillActionResult> {
const dryRun = Boolean(args.dryRun)
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const totals: BadgeBackfillStats = {
skillsScanned: 0,
skillsPatched: 0,
highlightsPatched: 0,
}
let cursor: string | null = null
let isDone = false
for (let i = 0; i < maxBatches; i++) {
const page = (await ctx.runQuery(internal.maintenance.getSkillBadgeBackfillPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as BadgeBackfillPageResult
cursor = page.cursor
isDone = page.isDone
for (const item of page.items) {
totals.skillsScanned++
const shouldHighlight = item.batch === 'highlighted' && !item.badges?.highlighted
if (!shouldHighlight) continue
totals.skillsPatched++
totals.highlightsPatched++
if (dryRun) continue
const at = item.updatedAt ?? item.createdAt ?? Date.now()
await ctx.runMutation(internal.maintenance.applySkillBadgeBackfillPatchInternal, {
skillId: item.skillId,
badges: {
...item.badges,
highlighted: {
byUserId: item.ownerUserId,
at,
},
},
})
}
if (isDone) break
}
if (!isDone) {
throw new ConvexError('Backfill incomplete (maxBatches reached)')
}
return { ok: true as const, stats: totals }
}
export const backfillSkillBadgesInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: backfillSkillBadgesInternalHandler,
})
export const backfillSkillBadges: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BadgeBackfillActionResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
return ctx.runAction(
internal.maintenance.backfillSkillBadgesInternal,
args,
) as Promise<BadgeBackfillActionResult>
},
})
export const scheduleBackfillSkillBadges: ReturnType<typeof action> = action({
args: { dryRun: v.optional(v.boolean()) },
handler: async (ctx, args) => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
await ctx.scheduler.runAfter(0, internal.maintenance.backfillSkillBadgesInternal, {
dryRun: Boolean(args.dryRun),
batchSize: DEFAULT_BATCH_SIZE,
maxBatches: DEFAULT_MAX_BATCHES,
})
return { ok: true as const }
},
})
export type SkillBadgeTableBackfillActionResult = {
ok: true
stats: SkillBadgeTableBackfillStats
}
export async function backfillSkillBadgeTableInternalHandler(
ctx: ActionCtx,
args: BadgeBackfillActionArgs,
): Promise<SkillBadgeTableBackfillActionResult> {
const dryRun = Boolean(args.dryRun)
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const totals: SkillBadgeTableBackfillStats = {
skillsScanned: 0,
recordsInserted: 0,
}
let cursor: string | null = null
let isDone = false
for (let i = 0; i < maxBatches; i++) {
const page = (await ctx.runQuery(internal.maintenance.getSkillBadgeBackfillPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as BadgeBackfillPageResult
cursor = page.cursor
isDone = page.isDone
for (const item of page.items) {
totals.skillsScanned++
const badges = item.badges ?? {}
const entries: Array<{ kind: BadgeKind; byUserId: Id<'users'>; at: number }> = []
if (badges.redactionApproved) {
entries.push({
kind: 'redactionApproved',
byUserId: badges.redactionApproved.byUserId,
at: badges.redactionApproved.at,
})
}
if (badges.official) {
entries.push({
kind: 'official',
byUserId: badges.official.byUserId,
at: badges.official.at,
})
}
if (badges.deprecated) {
entries.push({
kind: 'deprecated',
byUserId: badges.deprecated.byUserId,
at: badges.deprecated.at,
})
}
const highlighted =
badges.highlighted ??
(item.batch === 'highlighted'
? {
byUserId: item.ownerUserId,
at: item.updatedAt ?? item.createdAt ?? Date.now(),
}
: undefined)
if (highlighted) {
entries.push({
kind: 'highlighted',
byUserId: highlighted.byUserId,
at: highlighted.at,
})
}
if (dryRun) continue
for (const entry of entries) {
const result = await ctx.runMutation(internal.maintenance.upsertSkillBadgeRecordInternal, {
skillId: item.skillId,
kind: entry.kind,
byUserId: entry.byUserId,
at: entry.at,
})
if (result.inserted) {
totals.recordsInserted++
}
}
}
if (isDone) break
}
if (!isDone) {
throw new ConvexError('Backfill incomplete (maxBatches reached)')
}
return { ok: true as const, stats: totals }
}
export const backfillSkillBadgeTableInternal = internalAction({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: backfillSkillBadgeTableInternalHandler,
})
export const backfillSkillBadgeTable: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: async (ctx, args): Promise<SkillBadgeTableBackfillActionResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
return ctx.runAction(
internal.maintenance.backfillSkillBadgeTableInternal,
args,
) as Promise<SkillBadgeTableBackfillActionResult>
},
})
export const scheduleBackfillSkillBadgeTable: ReturnType<typeof action> = action({
args: { dryRun: v.optional(v.boolean()) },
handler: async (ctx, args) => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
await ctx.scheduler.runAfter(0, internal.maintenance.backfillSkillBadgeTableInternal, {
dryRun: Boolean(args.dryRun),
batchSize: DEFAULT_BATCH_SIZE,
maxBatches: DEFAULT_MAX_BATCHES,
})
return { ok: true as const }
},
})
function clampInt(value: number, min: number, max: number) {
const rounded = Math.trunc(value)
if (!Number.isFinite(rounded)) return min
return Math.min(max, Math.max(min, rounded))
}
+50
View File
@@ -0,0 +1,50 @@
import { v } from 'convex/values'
import { internalMutation } from './_generated/server'
export const checkRateLimitInternal = internalMutation({
args: {
key: v.string(),
limit: v.number(),
windowMs: v.number(),
},
handler: async (ctx, args) => {
const now = Date.now()
const windowStart = Math.floor(now / args.windowMs) * args.windowMs
const resetAt = windowStart + args.windowMs
if (args.limit <= 0) {
return { allowed: false, remaining: 0, limit: args.limit, resetAt }
}
const existing = await ctx.db
.query('rateLimits')
.withIndex('by_key_window', (q) => q.eq('key', args.key).eq('windowStart', windowStart))
.unique()
if (!existing) {
await ctx.db.insert('rateLimits', {
key: args.key,
windowStart,
count: 1,
limit: args.limit,
updatedAt: now,
})
return { allowed: true, remaining: Math.max(0, args.limit - 1), limit: args.limit, resetAt }
}
if (existing.count >= args.limit) {
return { allowed: false, remaining: 0, limit: args.limit, resetAt }
}
await ctx.db.patch(existing._id, {
count: existing.count + 1,
limit: args.limit,
updatedAt: now,
})
return {
allowed: true,
remaining: Math.max(0, args.limit - existing.count - 1),
limit: args.limit,
resetAt,
}
},
})
+323 -3
View File
@@ -3,6 +3,8 @@ import { defineSchema, defineTable } from 'convex/server'
import { v } from 'convex/values'
import { EMBEDDING_DIMENSIONS } from './lib/embeddings'
const authSchema = authTables as unknown as Record<string, ReturnType<typeof defineTable>>
const users = defineTable({
name: v.optional(v.string()),
image: v.optional(v.string()),
@@ -28,6 +30,15 @@ const skills = defineTable({
displayName: v.string(),
summary: v.optional(v.string()),
ownerUserId: v.id('users'),
canonicalSkillId: v.optional(v.id('skills')),
forkOf: v.optional(
v.object({
skillId: v.id('skills'),
kind: v.union(v.literal('fork'), v.literal('duplicate')),
version: v.optional(v.string()),
at: v.number(),
}),
),
latestVersionId: v.optional(v.id('skillVersions')),
tags: v.record(v.string(), v.id('skillVersions')),
softDeletedAt: v.optional(v.number()),
@@ -38,8 +49,70 @@ const skills = defineTable({
at: v.number(),
}),
),
highlighted: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
official: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
deprecated: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
}),
moderationStatus: v.optional(
v.union(v.literal('active'), v.literal('hidden'), v.literal('removed')),
),
moderationNotes: v.optional(v.string()),
moderationReason: v.optional(v.string()),
moderationFlags: v.optional(v.array(v.string())),
lastReviewedAt: v.optional(v.number()),
hiddenAt: v.optional(v.number()),
hiddenBy: v.optional(v.id('users')),
reportCount: v.optional(v.number()),
lastReportedAt: v.optional(v.number()),
batch: v.optional(v.string()),
statsDownloads: v.optional(v.number()),
statsStars: v.optional(v.number()),
statsInstallsCurrent: v.optional(v.number()),
statsInstallsAllTime: v.optional(v.number()),
stats: v.object({
downloads: v.number(),
installsCurrent: v.optional(v.number()),
installsAllTime: v.optional(v.number()),
stars: v.number(),
versions: v.number(),
comments: v.number(),
}),
createdAt: v.number(),
updatedAt: v.number(),
})
.index('by_slug', ['slug'])
.index('by_owner', ['ownerUserId'])
.index('by_updated', ['updatedAt'])
.index('by_stats_downloads', ['statsDownloads', 'updatedAt'])
.index('by_stats_stars', ['statsStars', 'updatedAt'])
.index('by_stats_installs_current', ['statsInstallsCurrent', 'updatedAt'])
.index('by_stats_installs_all_time', ['statsInstallsAllTime', 'updatedAt'])
.index('by_batch', ['batch'])
.index('by_active_updated', ['softDeletedAt', 'updatedAt'])
const souls = defineTable({
slug: v.string(),
displayName: v.string(),
summary: v.optional(v.string()),
ownerUserId: v.id('users'),
latestVersionId: v.optional(v.id('soulVersions')),
tags: v.record(v.string(), v.id('soulVersions')),
softDeletedAt: v.optional(v.number()),
stats: v.object({
downloads: v.number(),
stars: v.number(),
@@ -52,12 +125,13 @@ const skills = defineTable({
.index('by_slug', ['slug'])
.index('by_owner', ['ownerUserId'])
.index('by_updated', ['updatedAt'])
.index('by_batch', ['batch'])
const skillVersions = defineTable({
skillId: v.id('skills'),
version: v.string(),
fingerprint: v.optional(v.string()),
changelog: v.string(),
changelogSource: v.optional(v.union(v.literal('auto'), v.literal('user'))),
files: v.array(
v.object({
path: v.string(),
@@ -68,9 +142,10 @@ const skillVersions = defineTable({
}),
),
parsed: v.object({
frontmatter: v.record(v.string(), v.string()),
frontmatter: v.record(v.string(), v.any()),
metadata: v.optional(v.any()),
clawdis: v.optional(v.any()),
moltbot: v.optional(v.any()),
}),
createdBy: v.id('users'),
createdAt: v.number(),
@@ -79,6 +154,69 @@ const skillVersions = defineTable({
.index('by_skill', ['skillId'])
.index('by_skill_version', ['skillId', 'version'])
const soulVersions = defineTable({
soulId: v.id('souls'),
version: v.string(),
fingerprint: v.optional(v.string()),
changelog: v.string(),
changelogSource: v.optional(v.union(v.literal('auto'), v.literal('user'))),
files: v.array(
v.object({
path: v.string(),
size: v.number(),
storageId: v.id('_storage'),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
),
parsed: v.object({
frontmatter: v.record(v.string(), v.any()),
metadata: v.optional(v.any()),
clawdis: v.optional(v.any()),
moltbot: v.optional(v.any()),
}),
createdBy: v.id('users'),
createdAt: v.number(),
softDeletedAt: v.optional(v.number()),
})
.index('by_soul', ['soulId'])
.index('by_soul_version', ['soulId', 'version'])
const skillVersionFingerprints = defineTable({
skillId: v.id('skills'),
versionId: v.id('skillVersions'),
fingerprint: v.string(),
createdAt: v.number(),
})
.index('by_version', ['versionId'])
.index('by_fingerprint', ['fingerprint'])
.index('by_skill_fingerprint', ['skillId', 'fingerprint'])
const skillBadges = defineTable({
skillId: v.id('skills'),
kind: v.union(
v.literal('highlighted'),
v.literal('official'),
v.literal('deprecated'),
v.literal('redactionApproved'),
),
byUserId: v.id('users'),
at: v.number(),
})
.index('by_skill', ['skillId'])
.index('by_skill_kind', ['skillId', 'kind'])
.index('by_kind_at', ['kind', 'at'])
const soulVersionFingerprints = defineTable({
soulId: v.id('souls'),
versionId: v.id('soulVersions'),
fingerprint: v.string(),
createdAt: v.number(),
})
.index('by_version', ['versionId'])
.index('by_fingerprint', ['fingerprint'])
.index('by_soul_fingerprint', ['soulId', 'fingerprint'])
const skillEmbeddings = defineTable({
skillId: v.id('skills'),
versionId: v.id('skillVersions'),
@@ -97,6 +235,85 @@ const skillEmbeddings = defineTable({
filterFields: ['visibility'],
})
const skillDailyStats = defineTable({
skillId: v.id('skills'),
day: v.number(),
downloads: v.number(),
installs: v.number(),
updatedAt: v.number(),
})
.index('by_skill_day', ['skillId', 'day'])
.index('by_day', ['day'])
const skillLeaderboards = defineTable({
kind: v.string(),
generatedAt: v.number(),
rangeStartDay: v.number(),
rangeEndDay: v.number(),
items: v.array(
v.object({
skillId: v.id('skills'),
score: v.number(),
installs: v.number(),
downloads: v.number(),
}),
),
}).index('by_kind', ['kind', 'generatedAt'])
const skillStatBackfillState = defineTable({
key: v.string(),
cursor: v.optional(v.string()),
doneAt: v.optional(v.number()),
updatedAt: v.number(),
}).index('by_key', ['key'])
const skillStatEvents = defineTable({
skillId: v.id('skills'),
kind: v.union(
v.literal('download'),
v.literal('star'),
v.literal('unstar'),
v.literal('install_new'),
v.literal('install_reactivate'),
v.literal('install_deactivate'),
v.literal('install_clear'),
),
delta: v.optional(
v.object({
allTime: v.number(),
current: v.number(),
}),
),
occurredAt: v.number(),
processedAt: v.optional(v.number()),
})
.index('by_unprocessed', ['processedAt'])
.index('by_skill', ['skillId'])
const skillStatUpdateCursors = defineTable({
key: v.string(),
cursorCreationTime: v.optional(v.number()),
updatedAt: v.number(),
}).index('by_key', ['key'])
const soulEmbeddings = defineTable({
soulId: v.id('souls'),
versionId: v.id('soulVersions'),
ownerId: v.id('users'),
embedding: v.array(v.number()),
isLatest: v.boolean(),
isApproved: v.boolean(),
visibility: v.string(),
updatedAt: v.number(),
})
.index('by_soul', ['soulId'])
.index('by_version', ['versionId'])
.vectorIndex('by_embedding', {
vectorField: 'embedding',
dimensions: EMBEDDING_DIMENSIONS,
filterFields: ['visibility'],
})
const comments = defineTable({
skillId: v.id('skills'),
userId: v.id('users'),
@@ -108,6 +325,27 @@ const comments = defineTable({
.index('by_skill', ['skillId'])
.index('by_user', ['userId'])
const skillReports = defineTable({
skillId: v.id('skills'),
userId: v.id('users'),
reason: v.optional(v.string()),
createdAt: v.number(),
})
.index('by_skill', ['skillId'])
.index('by_user', ['userId'])
.index('by_skill_user', ['skillId', 'userId'])
const soulComments = defineTable({
soulId: v.id('souls'),
userId: v.id('users'),
body: v.string(),
createdAt: v.number(),
softDeletedAt: v.optional(v.number()),
deletedBy: v.optional(v.id('users')),
})
.index('by_soul', ['soulId'])
.index('by_user', ['userId'])
const stars = defineTable({
skillId: v.id('skills'),
userId: v.id('users'),
@@ -117,6 +355,15 @@ const stars = defineTable({
.index('by_user', ['userId'])
.index('by_skill_user', ['skillId', 'userId'])
const soulStars = defineTable({
soulId: v.id('souls'),
userId: v.id('users'),
createdAt: v.number(),
})
.index('by_soul', ['soulId'])
.index('by_user', ['userId'])
.index('by_soul_user', ['soulId', 'userId'])
const auditLogs = defineTable({
actorUserId: v.id('users'),
action: v.string(),
@@ -140,14 +387,87 @@ const apiTokens = defineTable({
.index('by_user', ['userId'])
.index('by_hash', ['tokenHash'])
const rateLimits = defineTable({
key: v.string(),
windowStart: v.number(),
count: v.number(),
limit: v.number(),
updatedAt: v.number(),
})
.index('by_key_window', ['key', 'windowStart'])
.index('by_key', ['key'])
const githubBackupSyncState = defineTable({
key: v.string(),
cursor: v.optional(v.string()),
updatedAt: v.number(),
}).index('by_key', ['key'])
const userSyncRoots = defineTable({
userId: v.id('users'),
rootId: v.string(),
label: v.string(),
firstSeenAt: v.number(),
lastSeenAt: v.number(),
expiredAt: v.optional(v.number()),
})
.index('by_user', ['userId'])
.index('by_user_root', ['userId', 'rootId'])
const userSkillInstalls = defineTable({
userId: v.id('users'),
skillId: v.id('skills'),
firstSeenAt: v.number(),
lastSeenAt: v.number(),
activeRoots: v.number(),
lastVersion: v.optional(v.string()),
})
.index('by_user', ['userId'])
.index('by_user_skill', ['userId', 'skillId'])
.index('by_skill', ['skillId'])
const userSkillRootInstalls = defineTable({
userId: v.id('users'),
rootId: v.string(),
skillId: v.id('skills'),
firstSeenAt: v.number(),
lastSeenAt: v.number(),
lastVersion: v.optional(v.string()),
removedAt: v.optional(v.number()),
})
.index('by_user', ['userId'])
.index('by_user_root', ['userId', 'rootId'])
.index('by_user_root_skill', ['userId', 'rootId', 'skillId'])
.index('by_user_skill', ['userId', 'skillId'])
.index('by_skill', ['skillId'])
export default defineSchema({
...authTables,
...authSchema,
users,
skills,
souls,
skillVersions,
soulVersions,
skillVersionFingerprints,
skillBadges,
soulVersionFingerprints,
skillEmbeddings,
soulEmbeddings,
skillDailyStats,
skillLeaderboards,
skillStatBackfillState,
skillStatEvents,
skillStatUpdateCursors,
comments,
skillReports,
soulComments,
stars,
soulStars,
auditLogs,
apiTokens,
rateLimits,
githubBackupSyncState,
userSyncRoots,
userSkillInstalls,
userSkillRootInstalls,
})
+12
View File
@@ -0,0 +1,12 @@
/* @vitest-environment node */
import { describe, expect, it } from 'vitest'
import { __test } from './search'
describe('search helpers', () => {
it('advances candidate limit until max', () => {
expect(__test.getNextCandidateLimit(50, 1000)).toBe(100)
expect(__test.getNextCandidateLimit(800, 1000)).toBe(1000)
expect(__test.getNextCandidateLimit(1000, 1000)).toBeNull()
})
})
+207 -21
View File
@@ -2,16 +2,25 @@ import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalQuery } from './_generated/server'
import { getSkillBadgeMaps, isSkillHighlighted, type SkillBadgeMap } from './lib/badges'
import { generateEmbedding } from './lib/embeddings'
import { toPublicSkill, toPublicSoul } from './lib/public'
import { matchesExactTokens, tokenize } from './lib/searchText'
type HydratedEntry = {
embeddingId: Id<'skillEmbeddings'>
skill: Doc<'skills'> | null
skill: NonNullable<ReturnType<typeof toPublicSkill>>
version: Doc<'skillVersions'> | null
ownerHandle: string | null
}
type SearchResult = HydratedEntry & { score: number }
function getNextCandidateLimit(current: number, max: number) {
const next = Math.min(current * 2, max)
return next > current ? next : null
}
export const searchSkills: ReturnType<typeof action> = action({
args: {
query: v.string(),
@@ -21,48 +30,225 @@ export const searchSkills: ReturnType<typeof action> = action({
handler: async (ctx, args): Promise<SearchResult[]> => {
const query = args.query.trim()
if (!query) return []
const vector = await generateEmbedding(query)
const results = await ctx.vectorSearch('skillEmbeddings', 'by_embedding', {
vector,
limit: args.limit ?? 10,
filter: (q) => q.or(q.eq('visibility', 'latest'), q.eq('visibility', 'latest-approved')),
})
const queryTokens = tokenize(query)
if (queryTokens.length === 0) return []
let vector: number[]
try {
vector = await generateEmbedding(query)
} catch (error) {
console.warn('Search embedding generation failed', error)
return []
}
const limit = args.limit ?? 10
// Convex vectorSearch max limit is 256; clamp candidate sizes accordingly.
const maxCandidate = Math.min(Math.max(limit * 10, 200), 256)
let candidateLimit = Math.min(Math.max(limit * 3, 50), 256)
let hydrated: HydratedEntry[] = []
let scoreById = new Map<Id<'skillEmbeddings'>, number>()
let exactMatches: HydratedEntry[] = []
const hydrated = (await ctx.runQuery(internal.search.hydrateResults, {
embeddingIds: results.map((result) => result._id),
})) as HydratedEntry[]
while (candidateLimit <= maxCandidate) {
const results = await ctx.vectorSearch('skillEmbeddings', 'by_embedding', {
vector,
limit: candidateLimit,
filter: (q) => q.or(q.eq('visibility', 'latest'), q.eq('visibility', 'latest-approved')),
})
const scoreById = new Map<Id<'skillEmbeddings'>, number>(
results.map((result) => [result._id, result._score]),
)
hydrated = (await ctx.runQuery(internal.search.hydrateResults, {
embeddingIds: results.map((result) => result._id),
})) as HydratedEntry[]
const filtered = args.highlightedOnly
? hydrated.filter((entry) => entry.skill?.batch === 'highlighted')
: hydrated
scoreById = new Map<Id<'skillEmbeddings'>, number>(
results.map((result) => [result._id, result._score]),
)
return filtered
const badgeMapEntries = (await ctx.runQuery(internal.search.getSkillBadgeMapsInternal, {
skillIds: hydrated.map((entry) => entry.skill._id),
})) as Array<[Id<'skills'>, SkillBadgeMap]>
const badgeMapBySkillId = new Map(badgeMapEntries)
const hydratedWithBadges = hydrated.map((entry) => ({
...entry,
skill: {
...entry.skill,
badges: badgeMapBySkillId.get(entry.skill._id) ?? {},
},
}))
const filtered = args.highlightedOnly
? hydratedWithBadges.filter((entry) => isSkillHighlighted(entry.skill))
: hydratedWithBadges
exactMatches = filtered.filter((entry) =>
matchesExactTokens(queryTokens, [
entry.skill.displayName,
entry.skill.slug,
entry.skill.summary,
]),
)
if (exactMatches.length >= limit || results.length < candidateLimit) {
break
}
const nextLimit = getNextCandidateLimit(candidateLimit, maxCandidate)
if (!nextLimit) break
candidateLimit = nextLimit
}
return exactMatches
.map((entry) => ({
...entry,
score: scoreById.get(entry.embeddingId) ?? 0,
}))
.filter((entry) => entry.skill)
.slice(0, limit)
},
})
export const getBadgeMapsForSkills = internalQuery({
args: { skillIds: v.array(v.id('skills')) },
handler: async (ctx, args): Promise<Array<[Id<'skills'>, SkillBadgeMap]>> => {
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
return Array.from(badgeMap.entries())
},
})
export const hydrateResults = internalQuery({
args: { embeddingIds: v.array(v.id('skillEmbeddings')) },
handler: async (ctx, args): Promise<HydratedEntry[]> => {
const entries: HydratedEntry[] = []
const ownerHandleCache = new Map<Id<'users'>, Promise<string | null>>()
const getOwnerHandle = (ownerUserId: Id<'users'>) => {
const cached = ownerHandleCache.get(ownerUserId)
if (cached) return cached
const handlePromise = ctx.db
.get(ownerUserId)
.then((owner) => owner?.handle ?? owner?._id ?? null)
ownerHandleCache.set(ownerUserId, handlePromise)
return handlePromise
}
const entries = await Promise.all(
args.embeddingIds.map(async (embeddingId) => {
const embedding = await ctx.db.get(embeddingId)
if (!embedding) return null
const skill = await ctx.db.get(embedding.skillId)
if (!skill || skill.softDeletedAt) return null
const [version, ownerHandle] = await Promise.all([
ctx.db.get(embedding.versionId),
getOwnerHandle(skill.ownerUserId),
])
const publicSkill = toPublicSkill(skill)
if (!publicSkill) return null
return { embeddingId, skill: publicSkill, version, ownerHandle }
}),
)
return entries.filter((entry): entry is HydratedEntry => entry !== null)
},
})
type HydratedSoulEntry = {
embeddingId: Id<'soulEmbeddings'>
soul: NonNullable<ReturnType<typeof toPublicSoul>>
version: Doc<'soulVersions'> | null
}
type SoulSearchResult = HydratedSoulEntry & { score: number }
export const searchSouls: ReturnType<typeof action> = action({
args: {
query: v.string(),
limit: v.optional(v.number()),
},
handler: async (ctx, args): Promise<SoulSearchResult[]> => {
const query = args.query.trim()
if (!query) return []
const queryTokens = tokenize(query)
if (queryTokens.length === 0) return []
let vector: number[]
try {
vector = await generateEmbedding(query)
} catch (error) {
console.warn('Search embedding generation failed', error)
return []
}
const limit = args.limit ?? 10
// Convex vectorSearch max limit is 256; clamp candidate sizes accordingly.
const maxCandidate = Math.min(Math.max(limit * 10, 200), 256)
let candidateLimit = Math.min(Math.max(limit * 3, 50), 256)
let hydrated: HydratedSoulEntry[] = []
let scoreById = new Map<Id<'soulEmbeddings'>, number>()
let exactMatches: HydratedSoulEntry[] = []
while (candidateLimit <= maxCandidate) {
const results = await ctx.vectorSearch('soulEmbeddings', 'by_embedding', {
vector,
limit: candidateLimit,
filter: (q) => q.or(q.eq('visibility', 'latest'), q.eq('visibility', 'latest-approved')),
})
hydrated = (await ctx.runQuery(internal.search.hydrateSoulResults, {
embeddingIds: results.map((result) => result._id),
})) as HydratedSoulEntry[]
scoreById = new Map<Id<'soulEmbeddings'>, number>(
results.map((result) => [result._id, result._score]),
)
exactMatches = hydrated.filter((entry) =>
matchesExactTokens(queryTokens, [
entry.soul.displayName,
entry.soul.slug,
entry.soul.summary,
]),
)
if (exactMatches.length >= limit || results.length < candidateLimit) {
break
}
const nextLimit = getNextCandidateLimit(candidateLimit, maxCandidate)
if (!nextLimit) break
candidateLimit = nextLimit
}
return exactMatches
.map((entry) => ({
...entry,
score: scoreById.get(entry.embeddingId) ?? 0,
}))
.filter((entry) => entry.soul)
.slice(0, limit)
},
})
export const hydrateSoulResults = internalQuery({
args: { embeddingIds: v.array(v.id('soulEmbeddings')) },
handler: async (ctx, args): Promise<HydratedSoulEntry[]> => {
const entries: HydratedSoulEntry[] = []
for (const embeddingId of args.embeddingIds) {
const embedding = await ctx.db.get(embeddingId)
if (!embedding) continue
const skill = await ctx.db.get(embedding.skillId)
if (skill?.softDeletedAt) continue
const soul = await ctx.db.get(embedding.soulId)
if (soul?.softDeletedAt) continue
const version = await ctx.db.get(embedding.versionId)
entries.push({ embeddingId, skill, version })
const publicSoul = toPublicSoul(soul)
if (!publicSoul) continue
entries.push({ embeddingId, soul: publicSoul, version })
}
return entries
},
})
export const getSkillBadgeMapsInternal = internalQuery({
args: { skillIds: v.array(v.id('skills')) },
handler: async (ctx, args) => {
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
return Array.from(badgeMap.entries())
},
})
export const __test = { getNextCandidateLimit }
+37
View File
@@ -0,0 +1,37 @@
import { describe, expect, it } from 'vitest'
import type { Doc } from './_generated/dataModel'
import { decideSeedStart } from './seed'
function seedState(cursor: string, updatedAt: number) {
return { cursor, updatedAt } as unknown as Doc<'githubBackupSyncState'>
}
describe('decideSeedStart', () => {
it('returns done when done', () => {
expect(decideSeedStart(seedState('done', Date.now()), Date.now())).toEqual({
started: false,
reason: 'done',
})
})
it('returns running when lock fresh', () => {
const now = Date.now()
expect(decideSeedStart(seedState('running', now), now + 1000)).toEqual({
started: false,
reason: 'running',
})
})
it('starts when lock stale', () => {
const now = Date.now()
const stale = now - 10 * 60 * 1000 - 1
expect(decideSeedStart(seedState('running', stale), now)).toEqual({
started: true,
reason: 'patched',
})
})
it('starts when missing', () => {
expect(decideSeedStart(null, Date.now())).toEqual({ started: true, reason: 'inserted' })
})
})
+254
View File
@@ -0,0 +1,254 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import type { ActionCtx, DatabaseReader, DatabaseWriter } from './_generated/server'
import { action, internalMutation, internalQuery } from './_generated/server'
import { publishSoulVersionForUser } from './lib/soulPublish'
import { SOUL_SEED_DISPLAY_NAME, SOUL_SEED_HANDLE, SOUL_SEED_KEY, SOUL_SEEDS } from './seedSouls'
const SEED_LOCK_STALE_MS = 10 * 60 * 1000
type SeedStateDoc = Doc<'githubBackupSyncState'>
type SeedStartDecision = {
started: boolean
reason: 'done' | 'running' | 'patched' | 'inserted'
}
async function getSeedState(ctx: { db: DatabaseReader }): Promise<SeedStateDoc | null> {
const entries = (await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SOUL_SEED_KEY))
.order('desc')
.take(2)) as SeedStateDoc[]
return entries[0] ?? null
}
async function cleanupSeedState(ctx: { db: DatabaseWriter }, keepId: Id<'githubBackupSyncState'>) {
const entries = (await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SOUL_SEED_KEY))
.order('desc')
.take(50)) as SeedStateDoc[]
for (const entry of entries) {
if (entry._id === keepId) continue
await ctx.db.delete(entry._id)
}
}
export function decideSeedStart(existing: SeedStateDoc | null, now: number): SeedStartDecision {
const cursor = existing?.cursor ?? null
if (cursor === 'done') return { started: false, reason: 'done' }
if (cursor === 'running' && existing && now - existing.updatedAt < SEED_LOCK_STALE_MS) {
return { started: false, reason: 'running' }
}
return existing ? { started: true, reason: 'patched' } : { started: true, reason: 'inserted' }
}
export const getSoulSeedStateInternal = internalQuery({
args: {},
handler: async (ctx) => getSeedState(ctx),
})
export const setSoulSeedStateInternal = internalMutation({
args: { status: v.string() },
handler: async (ctx, args) => {
const existing = await getSeedState(ctx)
const now = Date.now()
if (existing) {
await ctx.db.patch(existing._id, { cursor: args.status, updatedAt: now })
await cleanupSeedState(ctx, existing._id)
return existing._id
}
const id = await ctx.db.insert('githubBackupSyncState', {
key: SOUL_SEED_KEY,
cursor: args.status,
updatedAt: now,
})
await cleanupSeedState(ctx, id)
return id
},
})
export const tryStartSoulSeedInternal = internalMutation({
args: {},
handler: async (ctx) => {
const now = Date.now()
const existing = await getSeedState(ctx)
const decision = decideSeedStart(existing, now)
if (!decision.started) return decision
if (existing) {
await ctx.db.patch(existing._id, { cursor: 'running', updatedAt: now })
await cleanupSeedState(ctx, existing._id)
return { started: true, reason: 'patched' as const }
}
const id = await ctx.db.insert('githubBackupSyncState', {
key: SOUL_SEED_KEY,
cursor: 'running',
updatedAt: now,
})
await cleanupSeedState(ctx, id)
return { started: true, reason: 'inserted' as const }
},
})
export const hasAnySoulsInternal = internalQuery({
args: {},
handler: async (ctx) => {
const entry = await ctx.db.query('souls').take(1)
return entry.length > 0
},
})
export const ensureSoulSeeds = action({
args: {},
handler: async (ctx) => {
const started = (await ctx.runMutation(internal.seed.tryStartSoulSeedInternal, {})) as {
started: boolean
reason: 'done' | 'running' | 'patched' | 'inserted'
}
if (!started.started) {
if (started.reason === 'done') return { seeded: false, reason: 'already-seeded' as const }
return { seeded: false, reason: 'in-progress' as const }
}
const hasSouls = (await ctx.runQuery(internal.seed.hasAnySoulsInternal, {})) as boolean
if (hasSouls) {
await ctx.runMutation(internal.seed.setSoulSeedStateInternal, { status: 'done' })
return { seeded: false, reason: 'souls-exist' as const }
}
try {
const result = await runSeed(ctx)
await ctx.runMutation(internal.seed.setSoulSeedStateInternal, { status: 'done' })
return { seeded: true, reason: 'seeded' as const, ...result }
} catch (error) {
await ctx.runMutation(internal.seed.setSoulSeedStateInternal, { status: 'error' })
throw error
}
},
})
export const seed = action({
args: {},
handler: async (ctx) => runSeed(ctx),
})
async function runSeed(ctx: ActionCtx) {
const userId = (await ctx.runMutation(internal.seed.ensureSeedUserInternal, {
handle: SOUL_SEED_HANDLE,
displayName: SOUL_SEED_DISPLAY_NAME,
})) as Id<'users'>
const created: string[] = []
const skipped: string[] = []
for (const seedEntry of SOUL_SEEDS) {
const existing = (await ctx.runQuery(internal.souls.getSoulBySlugInternal, {
slug: seedEntry.slug,
})) as Doc<'souls'> | null
if (existing) {
if (existing.softDeletedAt && existing.ownerUserId === userId) {
await ctx.runMutation(internal.souls.setSoulSoftDeletedInternal, {
userId,
slug: seedEntry.slug,
deleted: false,
})
}
skipped.push(seedEntry.slug)
continue
}
const body = seedEntry.readme
if (!body) {
skipped.push(seedEntry.slug)
continue
}
const bytes = new TextEncoder().encode(body)
const sha256 = await sha256Hex(bytes)
const storageId = await ctx.storage.store(new Blob([bytes], { type: 'text/markdown' }))
try {
await publishSoulVersionForUser(ctx, userId, {
slug: seedEntry.slug,
displayName: seedEntry.displayName,
version: seedEntry.version,
changelog: '',
tags: seedEntry.tags,
files: [
{
path: 'SOUL.md',
size: bytes.byteLength,
storageId,
sha256,
contentType: 'text/markdown',
},
],
})
created.push(seedEntry.slug)
} catch (error) {
if (!isExpectedSeedSkipError(error)) throw error
skipped.push(seedEntry.slug)
}
}
return { created, skipped }
}
function isExpectedSeedSkipError(error: unknown) {
const message = error instanceof Error ? error.message : String(error)
return (
message.includes('Version already exists') || message.includes('Only the owner can publish')
)
}
export const ensureSeedUserInternal = internalMutation({
args: {
handle: v.string(),
displayName: v.string(),
},
handler: async (ctx, args) => {
const baseHandle = args.handle.trim()
const displayName = args.displayName.trim()
const candidates = [baseHandle, `${baseHandle}-bot`]
for (let i = 2; i <= 6; i += 1) candidates.push(`${baseHandle}-bot-${i}`)
for (const candidate of candidates) {
const existing = await ctx.db
.query('users')
.withIndex('handle', (q) => q.eq('handle', candidate))
.take(2)
const user = (existing[0] ?? null) as Doc<'users'> | null
if (user) {
if ((user.displayName ?? user.name) === displayName) return user._id
continue
}
return ctx.db.insert('users', {
handle: candidate,
displayName,
createdAt: Date.now(),
updatedAt: Date.now(),
})
}
throw new Error('Unable to allocate seed user handle')
},
})
async function sha256Hex(bytes: Uint8Array) {
const data = new Uint8Array(bytes)
const digest = await crypto.subtle.digest('SHA-256', data)
return toHex(new Uint8Array(digest))
}
function toHex(bytes: Uint8Array) {
let out = ''
for (const byte of bytes) out += byte.toString(16).padStart(2, '0')
return out
}
+111
View File
File diff suppressed because one or more lines are too long
+568
View File
@@ -0,0 +1,568 @@
/**
* Skill Stat Events - Event-sourced stats processing for skills
*
* Instead of updating skill stats synchronously in the hot path (which can cause
* contention when multiple users download/star/install the same skill), we insert
* lightweight event records and process them in batches via a cron job.
*
* Flow:
* 1. User action (download, star, install) → insertStatEvent() writes to skillStatEvents table
* 2. Cron job runs every 5 minutes → processSkillStatEventsInternal() processes batches
* 3. Events are aggregated per-skill to minimize database operations
* 4. Stats are applied to skill documents and daily stats tables
* 5. Events are marked as processed (kept forever for auditing)
*/
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import type { MutationCtx } from './_generated/server'
import { internalAction, internalMutation, internalQuery } from './_generated/server'
import { applySkillStatDeltas, bumpDailySkillStats } from './lib/skillStats'
/**
* Event types that affect skill stats:
*
* - download: User downloaded skill as zip (+1 downloads)
* - star: User starred the skill (+1 stars)
* - unstar: User removed their star (-1 stars)
* - install_new: First time this user installed this skill (+1 installsAllTime, +1 installsCurrent)
* - install_reactivate: User re-added skill after removing it (+1 installsCurrent only)
* - install_deactivate: User removed skill from all projects (-1 installsCurrent)
* - install_clear: User cleared all telemetry data (custom delta for both allTime and current)
*/
export type StatEventKind =
| 'download'
| 'star'
| 'unstar'
| 'install_new'
| 'install_reactivate'
| 'install_deactivate'
| 'install_clear'
/**
* Insert a stat event to be processed later by the cron job.
*
* This is called from the hot path (downloads, stars, telemetry) instead of
* directly updating skill stats. It's a single insert with no read-modify-write
* cycle, so it's fast and doesn't contend with other operations on the same skill.
*
* @param ctx - Mutation context
* @param params.skillId - The skill being affected
* @param params.kind - Type of event (download, star, install_new, etc.)
* @param params.occurredAt - When the event happened (defaults to now). Important for
* daily stats bucketing - we want downloads at 11:55 PM Monday
* to count toward Monday's stats even if processed on Tuesday.
* @param params.delta - Only used for install_clear events, specifies exact delta amounts
*/
export async function insertStatEvent(
ctx: MutationCtx,
params: {
skillId: Id<'skills'>
kind: StatEventKind
occurredAt?: number
delta?: { allTime: number; current: number }
},
) {
await ctx.db.insert('skillStatEvents', {
skillId: params.skillId,
kind: params.kind,
delta: params.delta,
occurredAt: params.occurredAt ?? Date.now(),
processedAt: undefined,
})
}
/**
* Aggregated deltas for a single skill after processing multiple events.
*
* When we process a batch of 100 events, many might be for the same skill.
* Instead of updating the skill document once per event, we aggregate all
* events for each skill and apply a single update.
*
* The downloadEvents and installNewEvents arrays store the original timestamps
* so we can update daily stats with the correct day bucket for each event.
*/
type AggregatedDeltas = {
downloads: number
stars: number
installsAllTime: number
installsCurrent: number
/** Original timestamps for each download event (for daily stats bucketing) */
downloadEvents: number[]
/** Original timestamps for each new install event (for daily stats bucketing) */
installNewEvents: number[]
}
/**
* Aggregate multiple events for a single skill into net deltas.
*
* Example: If a skill has these events in the batch:
* - download (Mon 11pm)
* - download (Tue 1am)
* - star
* - unstar
* - star
*
* The result would be:
* - downloads: 2
* - stars: 1 (net: +1 -1 +1 = +1)
* - downloadEvents: [<Mon 11pm timestamp>, <Tue 1am timestamp>]
*
* This aggregation reduces the number of database operations from N events
* to 1 skill update + N daily stat updates (which themselves may coalesce
* if multiple events fall on the same day).
*/
function aggregateEvents(events: Doc<'skillStatEvents'>[]): AggregatedDeltas {
const result: AggregatedDeltas = {
downloads: 0,
stars: 0,
installsAllTime: 0,
installsCurrent: 0,
downloadEvents: [],
installNewEvents: [],
}
for (const event of events) {
switch (event.kind) {
case 'download':
result.downloads += 1
result.downloadEvents.push(event.occurredAt)
break
case 'star':
result.stars += 1
break
case 'unstar':
result.stars -= 1
break
case 'install_new':
// New user installing for the first time: count toward both lifetime and current
result.installsAllTime += 1
result.installsCurrent += 1
result.installNewEvents.push(event.occurredAt)
break
case 'install_reactivate':
// User re-added skill after removing: only affects current count
result.installsCurrent += 1
break
case 'install_deactivate':
// User removed skill from all projects: only affects current count
result.installsCurrent -= 1
break
case 'install_clear':
// User cleared telemetry: uses custom delta values (typically negative)
if (event.delta) {
result.installsAllTime += event.delta.allTime
result.installsCurrent += event.delta.current
}
break
}
}
return result
}
/**
* Process a batch of unprocessed stat events.
*
* Called by cron every 5 minutes. Processes up to batchSize events (default 100).
* If the batch is full, schedules an immediate follow-up run to drain the queue.
*
* Processing steps:
* 1. Query unprocessed events (processedAt is undefined)
* 2. Group events by skillId to minimize skill document fetches
* 3. For each skill:
* a. Fetch the skill document once
* b. Aggregate all events for this skill into net deltas
* c. Apply deltas to skill stats (downloads, stars, installs)
* d. Update daily stats for trending (using original event timestamps)
* e. Mark all events as processed
* 4. If batch was full, schedule another run immediately
*
* Aggregation levels:
* - Level 1: Batch of 100 events from the queue
* - Level 2: Group by skillId (e.g., 100 events → 30 unique skills)
* - Level 3: Aggregate events per skill (e.g., 5 events → 1 skill update)
* - Level 4: Daily stats may coalesce (e.g., 3 downloads same day → 1 upsert)
*/
export const processSkillStatEventsInternal = internalMutation({
args: { batchSize: v.optional(v.number()) },
handler: async (ctx, args) => {
const batchSize = args.batchSize ?? 100
const now = Date.now()
// Level 1: Fetch a batch of unprocessed events
const events = await ctx.db
.query('skillStatEvents')
.withIndex('by_unprocessed', (q) => q.eq('processedAt', undefined))
.take(batchSize)
if (events.length === 0) {
return { processed: 0 }
}
// Level 2: Group events by skillId to minimize database reads
// Instead of fetching the same skill document multiple times,
// we fetch it once and process all its events together
const eventsBySkill = new Map<Id<'skills'>, Doc<'skillStatEvents'>[]>()
for (const event of events) {
const existing = eventsBySkill.get(event.skillId) ?? []
existing.push(event)
eventsBySkill.set(event.skillId, existing)
}
// Process each skill's events
for (const [skillId, skillEvents] of eventsBySkill) {
const skill = await ctx.db.get(skillId)
// Skill was deleted - just mark events as processed
if (!skill) {
for (const event of skillEvents) {
await ctx.db.patch(event._id, { processedAt: now })
}
continue
}
// Level 3: Aggregate all events for this skill into net deltas
// e.g., 3 downloads + 2 stars - 1 unstar → { downloads: 3, stars: 1 }
const deltas = aggregateEvents(skillEvents)
// Apply aggregated deltas to skill stats (single update per skill)
if (
deltas.downloads !== 0 ||
deltas.stars !== 0 ||
deltas.installsAllTime !== 0 ||
deltas.installsCurrent !== 0
) {
const patch = applySkillStatDeltas(skill, {
downloads: deltas.downloads,
stars: deltas.stars,
installsAllTime: deltas.installsAllTime,
installsCurrent: deltas.installsCurrent,
})
await ctx.db.patch(skill._id, {
...patch,
updatedAt: now,
})
}
// Update daily stats for trending/leaderboards
// We use the ORIGINAL event timestamp (occurredAt) so that:
// - A download at Mon 11:55 PM counts toward Monday's stats
// - Even if the cron processes it on Tuesday
//
// Level 4: bumpDailySkillStats does its own coalescing - multiple
// events on the same day will update the same daily record
for (const occurredAt of deltas.downloadEvents) {
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, downloads: 1 })
}
for (const occurredAt of deltas.installNewEvents) {
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, installs: 1 })
}
// Mark all events for this skill as processed
for (const event of skillEvents) {
await ctx.db.patch(event._id, { processedAt: now })
}
}
// If we hit the batch limit, there may be more events waiting.
// Schedule an immediate follow-up run to drain the queue.
// This ensures high-volume periods don't create a backlog.
if (events.length === batchSize) {
await ctx.scheduler.runAfter(0, internal.skillStatEvents.processSkillStatEventsInternal, {
batchSize,
})
}
return { processed: events.length }
},
})
// ============================================================================
// Action-based processing (cursor-based, runs outside transaction window)
// ============================================================================
const CURSOR_KEY = 'skill_stat_events'
const EVENT_BATCH_SIZE = 500
const MAX_SKILLS_PER_RUN = 500
/**
* Fetch a batch of events after the given cursor (by _creationTime).
* Returns events sorted by _creationTime ascending.
*/
export const getUnprocessedEventBatch = internalQuery({
args: {
cursorCreationTime: v.optional(v.number()),
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const limit = args.limit ?? EVENT_BATCH_SIZE
const cursor = args.cursorCreationTime
// Query events after the cursor using the built-in creation time index
const events = await ctx.db
.query('skillStatEvents')
.withIndex('by_creation_time', (q) =>
cursor !== undefined ? q.gt('_creationTime', cursor) : q,
)
.take(limit)
return events
},
})
/**
* Get the current cursor position from the cursors table.
*/
export const getStatEventCursor = internalQuery({
args: {},
handler: async (ctx) => {
const cursor = await ctx.db
.query('skillStatUpdateCursors')
.withIndex('by_key', (q) => q.eq('key', CURSOR_KEY))
.unique()
return cursor?.cursorCreationTime
},
})
/**
* Validator for skill deltas passed to the mutation.
*/
const skillDeltaValidator = v.object({
skillId: v.id('skills'),
downloads: v.number(),
stars: v.number(),
installsAllTime: v.number(),
installsCurrent: v.number(),
downloadEvents: v.array(v.number()),
installNewEvents: v.array(v.number()),
})
/**
* Apply aggregated stats to skills and update the cursor.
* This is a single atomic mutation that:
* 1. Updates all affected skills with their aggregated deltas
* 2. Updates daily stats for trending
* 3. Advances the cursor to the new position
*/
export const applyAggregatedStatsAndUpdateCursor = internalMutation({
args: {
skillDeltas: v.array(skillDeltaValidator),
newCursor: v.number(),
},
handler: async (ctx, args) => {
const now = Date.now()
// Process each skill's aggregated deltas
for (const delta of args.skillDeltas) {
const skill = await ctx.db.get(delta.skillId)
// Skill was deleted - skip
if (!skill) {
continue
}
// Apply aggregated deltas to skill stats
if (
delta.downloads !== 0 ||
delta.stars !== 0 ||
delta.installsAllTime !== 0 ||
delta.installsCurrent !== 0
) {
const patch = applySkillStatDeltas(skill, {
downloads: delta.downloads,
stars: delta.stars,
installsAllTime: delta.installsAllTime,
installsCurrent: delta.installsCurrent,
})
await ctx.db.patch(skill._id, {
...patch,
updatedAt: now,
})
}
// Update daily stats for trending/leaderboards
for (const occurredAt of delta.downloadEvents) {
await bumpDailySkillStats(ctx, { skillId: delta.skillId, now: occurredAt, downloads: 1 })
}
for (const occurredAt of delta.installNewEvents) {
await bumpDailySkillStats(ctx, { skillId: delta.skillId, now: occurredAt, installs: 1 })
}
}
// Update cursor position (upsert)
const existingCursor = await ctx.db
.query('skillStatUpdateCursors')
.withIndex('by_key', (q) => q.eq('key', CURSOR_KEY))
.unique()
if (existingCursor) {
await ctx.db.patch(existingCursor._id, {
cursorCreationTime: args.newCursor,
updatedAt: now,
})
} else {
await ctx.db.insert('skillStatUpdateCursors', {
key: CURSOR_KEY,
cursorCreationTime: args.newCursor,
updatedAt: now,
})
}
return { skillsUpdated: args.skillDeltas.length }
},
})
/**
* Action that processes skill stat events in batches outside the transaction window.
*
* Algorithm:
* 1. Get current cursor position
* 2. Fetch events in batches of 500, aggregating as we go
* 3. Stop when we have >= 500 unique skills OR run out of events
* 4. Call mutation to apply all deltas and update cursor atomically
* 5. Self-schedule if we stopped due to skill limit (not exhaustion)
*/
export const processSkillStatEventsAction = internalAction({
args: {},
handler: async (ctx) => {
// Get current cursor position (convert null to undefined for consistency)
const cursorResult = await ctx.runQuery(internal.skillStatEvents.getStatEventCursor)
let cursor: number | undefined = cursorResult ?? undefined
console.log(`[STAT-AGG] Starting aggregation, cursor=${cursor ?? 'none'}`)
// Aggregated deltas per skill
const aggregatedBySkill = new Map<
Id<'skills'>,
{
downloads: number
stars: number
installsAllTime: number
installsCurrent: number
downloadEvents: number[]
installNewEvents: number[]
}
>()
let maxCreationTime: number | undefined = cursor
let exhausted = false
let totalEventsFetched = 0
// Fetch and aggregate until we have enough skills or run out of events
while (aggregatedBySkill.size < MAX_SKILLS_PER_RUN) {
const events = await ctx.runQuery(internal.skillStatEvents.getUnprocessedEventBatch, {
cursorCreationTime: cursor,
limit: EVENT_BATCH_SIZE,
})
if (events.length === 0) {
exhausted = true
break
}
totalEventsFetched += events.length
const skillsBefore = aggregatedBySkill.size
// Aggregate events into per-skill deltas
for (const event of events) {
let skillDelta = aggregatedBySkill.get(event.skillId)
if (!skillDelta) {
skillDelta = {
downloads: 0,
stars: 0,
installsAllTime: 0,
installsCurrent: 0,
downloadEvents: [],
installNewEvents: [],
}
aggregatedBySkill.set(event.skillId, skillDelta)
}
// Apply event to aggregated deltas
switch (event.kind) {
case 'download':
skillDelta.downloads += 1
skillDelta.downloadEvents.push(event.occurredAt)
break
case 'star':
skillDelta.stars += 1
break
case 'unstar':
skillDelta.stars -= 1
break
case 'install_new':
skillDelta.installsAllTime += 1
skillDelta.installsCurrent += 1
skillDelta.installNewEvents.push(event.occurredAt)
break
case 'install_reactivate':
skillDelta.installsCurrent += 1
break
case 'install_deactivate':
skillDelta.installsCurrent -= 1
break
case 'install_clear':
if (event.delta) {
skillDelta.installsAllTime += event.delta.allTime
skillDelta.installsCurrent += event.delta.current
}
break
}
// Track highest _creationTime seen
if (maxCreationTime === undefined || event._creationTime > maxCreationTime) {
maxCreationTime = event._creationTime
}
}
// Update cursor for next batch fetch
cursor = events[events.length - 1]._creationTime
console.log(
`[STAT-AGG] Fetched ${events.length} events, ${aggregatedBySkill.size - skillsBefore} new skills (${aggregatedBySkill.size} total)`,
)
// If we got fewer than requested, we've exhausted the events
if (events.length < EVENT_BATCH_SIZE) {
exhausted = true
break
}
}
// If we have nothing to process, we're done
if (aggregatedBySkill.size === 0 || maxCreationTime === undefined) {
console.log('[STAT-AGG] No events to process, done')
return { processed: 0, skillsUpdated: 0, exhausted: true }
}
// Convert map to array for mutation
const skillDeltas = Array.from(aggregatedBySkill.entries()).map(([skillId, delta]) => ({
skillId,
...delta,
}))
console.log(
`[STAT-AGG] Running mutation for ${skillDeltas.length} skills (${totalEventsFetched} total events)`,
)
// Apply all deltas and update cursor atomically
await ctx.runMutation(internal.skillStatEvents.applyAggregatedStatsAndUpdateCursor, {
skillDeltas,
newCursor: maxCreationTime,
})
// Self-schedule if we stopped because of skill limit, not exhaustion
if (!exhausted) {
console.log('[STAT-AGG] More events remaining, self-scheduling')
await ctx.scheduler.runAfter(0, internal.skillStatEvents.processSkillStatEventsAction, {})
} else {
console.log('[STAT-AGG] All events processed, done')
}
return {
skillsUpdated: skillDeltas.length,
exhausted,
}
},
})
+1208 -179
View File
File diff suppressed because it is too large Load Diff
+88
View File
@@ -0,0 +1,88 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { assertModerator, requireUser } from './lib/access'
import { type PublicUser, toPublicUser } from './lib/public'
export const listBySoul = query({
args: { soulId: v.id('souls'), limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = args.limit ?? 50
const comments = await ctx.db
.query('soulComments')
.withIndex('by_soul', (q) => q.eq('soulId', args.soulId))
.order('desc')
.take(limit)
const results: Array<{ comment: Doc<'soulComments'>; user: PublicUser | null }> = []
for (const comment of comments) {
if (comment.softDeletedAt) continue
const user = toPublicUser(await ctx.db.get(comment.userId))
results.push({ comment, user })
}
return results
},
})
export const add = mutation({
args: { soulId: v.id('souls'), body: v.string() },
handler: async (ctx, args) => {
const { userId } = await requireUser(ctx)
const body = args.body.trim()
if (!body) throw new Error('Comment body required')
const soul = await ctx.db.get(args.soulId)
if (!soul) throw new Error('Soul not found')
await ctx.db.insert('soulComments', {
soulId: args.soulId,
userId,
body,
createdAt: Date.now(),
softDeletedAt: undefined,
deletedBy: undefined,
})
await ctx.db.patch(soul._id, {
stats: { ...soul.stats, comments: soul.stats.comments + 1 },
updatedAt: Date.now(),
})
},
})
export const remove = mutation({
args: { commentId: v.id('soulComments') },
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
const comment = await ctx.db.get(args.commentId)
if (!comment) throw new Error('Comment not found')
if (comment.softDeletedAt) return
const isOwner = comment.userId === user._id
if (!isOwner) {
assertModerator(user)
}
await ctx.db.patch(comment._id, {
softDeletedAt: Date.now(),
deletedBy: user._id,
})
const soul = await ctx.db.get(comment.soulId)
if (soul) {
await ctx.db.patch(soul._id, {
stats: { ...soul.stats, comments: Math.max(0, soul.stats.comments - 1) },
updatedAt: Date.now(),
})
}
await ctx.db.insert('auditLogs', {
actorUserId: user._id,
action: 'soul.comment.delete',
targetType: 'soulComment',
targetId: comment._id,
metadata: { soulId: comment.soulId },
createdAt: Date.now(),
})
},
})
+14
View File
@@ -0,0 +1,14 @@
import { v } from 'convex/values'
import { mutation } from './_generated/server'
export const increment = mutation({
args: { soulId: v.id('souls') },
handler: async (ctx, args) => {
const soul = await ctx.db.get(args.soulId)
if (!soul) return
await ctx.db.patch(soul._id, {
stats: { ...soul.stats, downloads: soul.stats.downloads + 1 },
updatedAt: Date.now(),
})
},
})
+71
View File
@@ -0,0 +1,71 @@
import { v } from 'convex/values'
import { mutation, query } from './_generated/server'
import { requireUser } from './lib/access'
import { toPublicSoul } from './lib/public'
export const isStarred = query({
args: { soulId: v.id('souls') },
handler: async (ctx, args) => {
const { userId } = await requireUser(ctx)
const existing = await ctx.db
.query('soulStars')
.withIndex('by_soul_user', (q) => q.eq('soulId', args.soulId).eq('userId', userId))
.unique()
return Boolean(existing)
},
})
export const toggle = mutation({
args: { soulId: v.id('souls') },
handler: async (ctx, args) => {
const { userId } = await requireUser(ctx)
const soul = await ctx.db.get(args.soulId)
if (!soul) throw new Error('Soul not found')
const existing = await ctx.db
.query('soulStars')
.withIndex('by_soul_user', (q) => q.eq('soulId', args.soulId).eq('userId', userId))
.unique()
if (existing) {
await ctx.db.delete(existing._id)
await ctx.db.patch(soul._id, {
stats: { ...soul.stats, stars: Math.max(0, soul.stats.stars - 1) },
updatedAt: Date.now(),
})
return { starred: false }
}
await ctx.db.insert('soulStars', {
soulId: args.soulId,
userId,
createdAt: Date.now(),
})
await ctx.db.patch(soul._id, {
stats: { ...soul.stats, stars: soul.stats.stars + 1 },
updatedAt: Date.now(),
})
return { starred: true }
},
})
export const listByUser = query({
args: { userId: v.id('users'), limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = args.limit ?? 50
const stars = await ctx.db
.query('soulStars')
.withIndex('by_user', (q) => q.eq('userId', args.userId))
.order('desc')
.take(limit)
const souls: NonNullable<ReturnType<typeof toPublicSoul>>[] = []
for (const star of stars) {
const soul = await ctx.db.get(star.soulId)
const publicSoul = toPublicSoul(soul)
if (!publicSoul) continue
souls.push(publicSoul)
}
return souls
},
})
+570
View File
@@ -0,0 +1,570 @@
import { ConvexError, v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalMutation, internalQuery, mutation, query } from './_generated/server'
import { assertModerator, requireUser, requireUserFromAction } from './lib/access'
import { toPublicSoul, toPublicUser } from './lib/public'
import { getFrontmatterValue, hashSkillFiles } from './lib/skills'
import { generateSoulChangelogPreview } from './lib/soulChangelog'
import { fetchText, type PublishResult, publishSoulVersionForUser } from './lib/soulPublish'
export { publishSoulVersionForUser } from './lib/soulPublish'
type ReadmeResult = { path: string; text: string }
type FileTextResult = { path: string; text: string; size: number; sha256: string }
const MAX_DIFF_FILE_BYTES = 200 * 1024
const MAX_LIST_LIMIT = 50
export const getBySlug = query({
args: { slug: v.string() },
handler: async (ctx, args) => {
const matches = await ctx.db
.query('souls')
.withIndex('by_slug', (q) => q.eq('slug', args.slug))
.order('desc')
.take(2)
const soul = matches[0] ?? null
if (!soul || soul.softDeletedAt) return null
const latestVersion = soul.latestVersionId ? await ctx.db.get(soul.latestVersionId) : null
const owner = toPublicUser(await ctx.db.get(soul.ownerUserId))
const publicSoul = toPublicSoul(soul)
if (!publicSoul) return null
return { soul: publicSoul, latestVersion, owner }
},
})
export const getSoulBySlugInternal = internalQuery({
args: { slug: v.string() },
handler: async (ctx, args) => {
const matches = await ctx.db
.query('souls')
.withIndex('by_slug', (q) => q.eq('slug', args.slug))
.order('desc')
.take(2)
return matches[0] ?? null
},
})
export const list = query({
args: {
ownerUserId: v.optional(v.id('users')),
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const limit = args.limit ?? 24
const ownerUserId = args.ownerUserId
if (ownerUserId) {
const entries = await ctx.db
.query('souls')
.withIndex('by_owner', (q) => q.eq('ownerUserId', ownerUserId))
.order('desc')
.take(limit * 5)
return entries
.filter((soul) => !soul.softDeletedAt)
.slice(0, limit)
.map((soul) => toPublicSoul(soul))
.filter((soul): soul is NonNullable<typeof soul> => Boolean(soul))
}
const entries = await ctx.db
.query('souls')
.order('desc')
.take(limit * 5)
return entries
.filter((soul) => !soul.softDeletedAt)
.slice(0, limit)
.map((soul) => toPublicSoul(soul))
.filter((soul): soul is NonNullable<typeof soul> => Boolean(soul))
},
})
export const listPublicPage = query({
args: {
cursor: v.optional(v.string()),
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const limit = clampInt(args.limit ?? 24, 1, MAX_LIST_LIMIT)
const { page, isDone, continueCursor } = await ctx.db
.query('souls')
.withIndex('by_updated', (q) => q)
.order('desc')
.paginate({ cursor: args.cursor ?? null, numItems: limit })
const items: Array<{
soul: NonNullable<ReturnType<typeof toPublicSoul>>
latestVersion: Doc<'soulVersions'> | null
}> = []
for (const soul of page) {
if (soul.softDeletedAt) continue
const latestVersion = soul.latestVersionId ? await ctx.db.get(soul.latestVersionId) : null
const publicSoul = toPublicSoul(soul)
if (!publicSoul) continue
items.push({ soul: publicSoul, latestVersion })
}
return { items, nextCursor: isDone ? null : continueCursor }
},
})
export const listVersions = query({
args: { soulId: v.id('souls'), limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = args.limit ?? 20
return ctx.db
.query('soulVersions')
.withIndex('by_soul', (q) => q.eq('soulId', args.soulId))
.order('desc')
.take(limit)
},
})
export const listVersionsPage = query({
args: {
soulId: v.id('souls'),
cursor: v.optional(v.string()),
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const limit = clampInt(args.limit ?? 20, 1, MAX_LIST_LIMIT)
const { page, isDone, continueCursor } = await ctx.db
.query('soulVersions')
.withIndex('by_soul', (q) => q.eq('soulId', args.soulId))
.order('desc')
.paginate({ cursor: args.cursor ?? null, numItems: limit })
const items = page.filter((version) => !version.softDeletedAt)
return { items, nextCursor: isDone ? null : continueCursor }
},
})
export const getVersionById = query({
args: { versionId: v.id('soulVersions') },
handler: async (ctx, args) => ctx.db.get(args.versionId),
})
export const getVersionByIdInternal = internalQuery({
args: { versionId: v.id('soulVersions') },
handler: async (ctx, args) => ctx.db.get(args.versionId),
})
export const getVersionBySoulAndVersion = query({
args: { soulId: v.id('souls'), version: v.string() },
handler: async (ctx, args) => {
return ctx.db
.query('soulVersions')
.withIndex('by_soul_version', (q) => q.eq('soulId', args.soulId).eq('version', args.version))
.unique()
},
})
export const publishVersion: ReturnType<typeof action> = action({
args: {
slug: v.string(),
displayName: v.string(),
version: v.string(),
changelog: v.string(),
tags: v.optional(v.array(v.string())),
source: v.optional(
v.object({
kind: v.literal('github'),
url: v.string(),
repo: v.string(),
ref: v.string(),
commit: v.string(),
path: v.string(),
importedAt: v.number(),
}),
),
files: v.array(
v.object({
path: v.string(),
size: v.number(),
storageId: v.id('_storage'),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
),
},
handler: async (ctx, args): Promise<PublishResult> => {
const { userId } = await requireUserFromAction(ctx)
return publishSoulVersionForUser(ctx, userId, args)
},
})
export const generateChangelogPreview = action({
args: {
slug: v.string(),
version: v.string(),
readmeText: v.string(),
filePaths: v.optional(v.array(v.string())),
},
handler: async (ctx, args) => {
await requireUserFromAction(ctx)
const changelog = await generateSoulChangelogPreview(ctx, {
slug: args.slug.trim().toLowerCase(),
version: args.version.trim(),
readmeText: args.readmeText,
filePaths: args.filePaths?.map((value) => value.trim()).filter(Boolean),
})
return { changelog, source: 'auto' as const }
},
})
export const getReadme: ReturnType<typeof action> = action({
args: { versionId: v.id('soulVersions') },
handler: async (ctx, args): Promise<ReadmeResult> => {
const version = (await ctx.runQuery(internal.souls.getVersionByIdInternal, {
versionId: args.versionId,
})) as Doc<'soulVersions'> | null
if (!version) throw new ConvexError('Version not found')
const readmeFile = version.files.find((file) => file.path.toLowerCase() === 'soul.md')
if (!readmeFile) throw new ConvexError('SOUL.md not found')
const text = await fetchText(ctx, readmeFile.storageId)
return { path: readmeFile.path, text }
},
})
export const getFileText: ReturnType<typeof action> = action({
args: { versionId: v.id('soulVersions'), path: v.string() },
handler: async (ctx, args): Promise<FileTextResult> => {
const version = (await ctx.runQuery(internal.souls.getVersionByIdInternal, {
versionId: args.versionId,
})) as Doc<'soulVersions'> | null
if (!version) throw new ConvexError('Version not found')
const normalizedPath = args.path.trim()
const normalizedLower = normalizedPath.toLowerCase()
const file =
version.files.find((entry) => entry.path === normalizedPath) ??
version.files.find((entry) => entry.path.toLowerCase() === normalizedLower)
if (!file) throw new ConvexError('File not found')
if (file.size > MAX_DIFF_FILE_BYTES) {
throw new ConvexError('File exceeds 200KB limit')
}
const text = await fetchText(ctx, file.storageId)
return { path: file.path, text, size: file.size, sha256: file.sha256 }
},
})
export const resolveVersionByHash = query({
args: { slug: v.string(), hash: v.string() },
handler: async (ctx, args) => {
const slug = args.slug.trim().toLowerCase()
const hash = args.hash.trim().toLowerCase()
if (!slug || !/^[a-f0-9]{64}$/.test(hash)) return null
const soulMatches = await ctx.db
.query('souls')
.withIndex('by_slug', (q) => q.eq('slug', slug))
.order('desc')
.take(2)
const soul = soulMatches[0] ?? null
if (!soul || soul.softDeletedAt) return null
const latestVersion = soul.latestVersionId ? await ctx.db.get(soul.latestVersionId) : null
const fingerprintMatches = await ctx.db
.query('soulVersionFingerprints')
.withIndex('by_soul_fingerprint', (q) => q.eq('soulId', soul._id).eq('fingerprint', hash))
.take(25)
let match: { version: string } | null = null
if (fingerprintMatches.length > 0) {
const newest = fingerprintMatches.reduce(
(best, entry) => (entry.createdAt > best.createdAt ? entry : best),
fingerprintMatches[0] as (typeof fingerprintMatches)[number],
)
const version = await ctx.db.get(newest.versionId)
if (version && !version.softDeletedAt) {
match = { version: version.version }
}
}
if (!match) {
const versions = await ctx.db
.query('soulVersions')
.withIndex('by_soul', (q) => q.eq('soulId', soul._id))
.order('desc')
.take(200)
for (const version of versions) {
if (version.softDeletedAt) continue
if (typeof version.fingerprint === 'string' && version.fingerprint === hash) {
match = { version: version.version }
break
}
const fingerprint = await hashSkillFiles(
version.files.map((file) => ({ path: file.path, sha256: file.sha256 })),
)
if (fingerprint === hash) {
match = { version: version.version }
break
}
}
}
return {
match,
latestVersion: latestVersion ? { version: latestVersion.version } : null,
}
},
})
export const updateTags = mutation({
args: {
soulId: v.id('souls'),
tags: v.array(v.object({ tag: v.string(), versionId: v.id('soulVersions') })),
},
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
const soul = await ctx.db.get(args.soulId)
if (!soul) throw new Error('Soul not found')
if (soul.ownerUserId !== user._id) {
assertModerator(user)
}
const nextTags = { ...soul.tags }
for (const entry of args.tags) {
nextTags[entry.tag] = entry.versionId
}
const latestEntry = args.tags.find((entry) => entry.tag === 'latest')
await ctx.db.patch(soul._id, {
tags: nextTags,
latestVersionId: latestEntry ? latestEntry.versionId : soul.latestVersionId,
updatedAt: Date.now(),
})
if (latestEntry) {
const embeddings = await ctx.db
.query('soulEmbeddings')
.withIndex('by_soul', (q) => q.eq('soulId', soul._id))
.collect()
for (const embedding of embeddings) {
const isLatest = embedding.versionId === latestEntry.versionId
await ctx.db.patch(embedding._id, {
isLatest,
visibility: visibilityFor(isLatest, embedding.isApproved),
updatedAt: Date.now(),
})
}
}
},
})
export const insertVersion = internalMutation({
args: {
userId: v.id('users'),
slug: v.string(),
displayName: v.string(),
version: v.string(),
changelog: v.string(),
changelogSource: v.optional(v.union(v.literal('auto'), v.literal('user'))),
tags: v.optional(v.array(v.string())),
fingerprint: v.string(),
summary: v.optional(v.string()),
files: v.array(
v.object({
path: v.string(),
size: v.number(),
storageId: v.id('_storage'),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
),
parsed: v.object({
frontmatter: v.record(v.string(), v.any()),
metadata: v.optional(v.any()),
}),
embedding: v.array(v.number()),
},
handler: async (ctx, args) => {
const userId = args.userId
const user = await ctx.db.get(userId)
if (!user || user.deletedAt) throw new Error('User not found')
const soulMatches = await ctx.db
.query('souls')
.withIndex('by_slug', (q) => q.eq('slug', args.slug))
.order('desc')
.take(2)
let soul: Doc<'souls'> | null = soulMatches[0] ?? null
if (soul && soul.ownerUserId !== userId) {
throw new Error('Only the owner can publish updates')
}
const now = Date.now()
if (!soul) {
const summary = args.summary ?? getFrontmatterValue(args.parsed.frontmatter, 'description')
const soulId = await ctx.db.insert('souls', {
slug: args.slug,
displayName: args.displayName,
summary: summary ?? undefined,
ownerUserId: userId,
latestVersionId: undefined,
tags: {},
softDeletedAt: undefined,
stats: {
downloads: 0,
stars: 0,
versions: 0,
comments: 0,
},
createdAt: now,
updatedAt: now,
})
soul = await ctx.db.get(soulId)
}
if (!soul) throw new Error('Soul creation failed')
const existingVersion = await ctx.db
.query('soulVersions')
.withIndex('by_soul_version', (q) => q.eq('soulId', soul._id).eq('version', args.version))
.unique()
if (existingVersion) {
throw new Error('Version already exists')
}
const versionId = await ctx.db.insert('soulVersions', {
soulId: soul._id,
version: args.version,
fingerprint: args.fingerprint,
changelog: args.changelog,
changelogSource: args.changelogSource,
files: args.files,
parsed: args.parsed,
createdBy: userId,
createdAt: now,
softDeletedAt: undefined,
})
const nextTags: Record<string, Id<'soulVersions'>> = { ...soul.tags }
nextTags.latest = versionId
for (const tag of args.tags ?? []) {
nextTags[tag] = versionId
}
const latestBefore = soul.latestVersionId
await ctx.db.patch(soul._id, {
displayName: args.displayName,
summary:
args.summary ?? getFrontmatterValue(args.parsed.frontmatter, 'description') ?? soul.summary,
latestVersionId: versionId,
tags: nextTags,
stats: { ...soul.stats, versions: soul.stats.versions + 1 },
softDeletedAt: undefined,
updatedAt: now,
})
const embeddingId = await ctx.db.insert('soulEmbeddings', {
soulId: soul._id,
versionId,
ownerId: userId,
embedding: args.embedding,
isLatest: true,
isApproved: true,
visibility: visibilityFor(true, true),
updatedAt: now,
})
if (latestBefore) {
const previousEmbedding = await ctx.db
.query('soulEmbeddings')
.withIndex('by_version', (q) => q.eq('versionId', latestBefore))
.unique()
if (previousEmbedding) {
await ctx.db.patch(previousEmbedding._id, {
isLatest: false,
visibility: visibilityFor(false, previousEmbedding.isApproved),
updatedAt: now,
})
}
}
await ctx.db.insert('soulVersionFingerprints', {
soulId: soul._id,
versionId,
fingerprint: args.fingerprint,
createdAt: now,
})
return { soulId: soul._id, versionId, embeddingId }
},
})
export const setSoulSoftDeletedInternal = internalMutation({
args: {
userId: v.id('users'),
slug: v.string(),
deleted: v.boolean(),
},
handler: async (ctx, args) => {
const user = await ctx.db.get(args.userId)
if (!user || user.deletedAt) throw new Error('User not found')
const slug = args.slug.trim().toLowerCase()
if (!slug) throw new Error('Slug required')
const soulMatches = await ctx.db
.query('souls')
.withIndex('by_slug', (q) => q.eq('slug', slug))
.order('desc')
.take(2)
const soul = soulMatches[0] ?? null
if (!soul) throw new Error('Soul not found')
if (soul.ownerUserId !== args.userId) {
assertModerator(user)
}
const now = Date.now()
await ctx.db.patch(soul._id, {
softDeletedAt: args.deleted ? now : undefined,
updatedAt: now,
})
const embeddings = await ctx.db
.query('soulEmbeddings')
.withIndex('by_soul', (q) => q.eq('soulId', soul._id))
.collect()
for (const embedding of embeddings) {
await ctx.db.patch(embedding._id, {
visibility: args.deleted
? 'deleted'
: visibilityFor(embedding.isLatest, embedding.isApproved),
updatedAt: now,
})
}
await ctx.db.insert('auditLogs', {
actorUserId: args.userId,
action: args.deleted ? 'soul.delete' : 'soul.undelete',
targetType: 'soul',
targetId: soul._id,
metadata: { slug, softDeletedAt: args.deleted ? now : null },
createdAt: now,
})
return { ok: true as const }
},
})
function visibilityFor(isLatest: boolean, isApproved: boolean) {
if (isLatest && isApproved) return 'latest-approved'
if (isLatest) return 'latest'
if (isApproved) return 'archived-approved'
return 'archived'
}
function clampInt(value: number, min: number, max: number) {
const rounded = Number.isFinite(value) ? Math.round(value) : min
return Math.min(max, Math.max(min, rounded))
}
+50 -12
View File
@@ -1,7 +1,8 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { internalMutation, mutation, query } from './_generated/server'
import { requireUser } from './lib/access'
import { toPublicSkill } from './lib/public'
import { insertStatEvent } from './skillStatEvents'
export const isStarred = query({
args: { skillId: v.id('skills') },
@@ -29,10 +30,7 @@ export const toggle = mutation({
if (existing) {
await ctx.db.delete(existing._id)
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, stars: Math.max(0, skill.stats.stars - 1) },
updatedAt: Date.now(),
})
await insertStatEvent(ctx, { skillId: skill._id, kind: 'unstar' })
return { starred: false }
}
@@ -42,10 +40,7 @@ export const toggle = mutation({
createdAt: Date.now(),
})
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, stars: skill.stats.stars + 1 },
updatedAt: Date.now(),
})
await insertStatEvent(ctx, { skillId: skill._id, kind: 'star' })
return { starred: true }
},
@@ -60,11 +55,54 @@ export const listByUser = query({
.withIndex('by_user', (q) => q.eq('userId', args.userId))
.order('desc')
.take(limit)
const skills: Doc<'skills'>[] = []
const skills: NonNullable<ReturnType<typeof toPublicSkill>>[] = []
for (const star of stars) {
const skill = await ctx.db.get(star.skillId)
if (skill) skills.push(skill)
const publicSkill = toPublicSkill(skill)
if (!publicSkill) continue
skills.push(publicSkill)
}
return skills
},
})
export const addStarInternal = internalMutation({
args: { userId: v.id('users'), skillId: v.id('skills') },
handler: async (ctx, args) => {
const skill = await ctx.db.get(args.skillId)
if (!skill) throw new Error('Skill not found')
const existing = await ctx.db
.query('stars')
.withIndex('by_skill_user', (q) => q.eq('skillId', args.skillId).eq('userId', args.userId))
.unique()
if (existing) return { ok: true as const, starred: true, alreadyStarred: true }
await ctx.db.insert('stars', {
skillId: args.skillId,
userId: args.userId,
createdAt: Date.now(),
})
await insertStatEvent(ctx, { skillId: skill._id, kind: 'star' })
return { ok: true as const, starred: true, alreadyStarred: false }
},
})
export const removeStarInternal = internalMutation({
args: { userId: v.id('users'), skillId: v.id('skills') },
handler: async (ctx, args) => {
const skill = await ctx.db.get(args.skillId)
if (!skill) throw new Error('Skill not found')
const existing = await ctx.db
.query('stars')
.withIndex('by_skill_user', (q) => q.eq('skillId', args.skillId).eq('userId', args.userId))
.unique()
if (!existing) return { ok: true as const, unstarred: false, alreadyUnstarred: true }
await ctx.db.delete(existing._id)
await insertStatEvent(ctx, { skillId: skill._id, kind: 'unstar' })
return { ok: true as const, unstarred: true, alreadyUnstarred: false }
},
})
+205
View File
@@ -0,0 +1,205 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction, internalMutation, internalQuery } from './_generated/server'
const DEFAULT_BATCH_SIZE = 200
const MAX_BATCH_SIZE = 1000
const DEFAULT_MAX_BATCHES = 5
const MAX_MAX_BATCHES = 50
const BACKFILL_STATE_KEY = 'default'
export const backfillSkillStatFieldsInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const { page, isDone, continueCursor } = await ctx.db
.query('skills')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
let patched = 0
for (const skill of page) {
const next = buildSkillStatPatch(skill)
if (!next) continue
await ctx.db.patch(skill._id, next)
patched += 1
}
return {
ok: true as const,
scanned: page.length,
patched,
cursor: isDone ? null : continueCursor,
isDone,
}
},
})
type BackfillState = {
cursor: string | null
doneAt?: number
}
type BackfillActionArgs = {
batchSize?: number
maxBatches?: number
resetCursor?: boolean
}
type BackfillStats = {
scanned: number
patched: number
batches: number
}
type BackfillActionResult = {
ok: true
isDone: boolean
cursor: string | null
stats: BackfillStats
}
export const getSkillStatBackfillStateInternal = internalQuery({
args: {},
handler: async (ctx): Promise<BackfillState> => {
const state = await ctx.db
.query('skillStatBackfillState')
.withIndex('by_key', (q) => q.eq('key', BACKFILL_STATE_KEY))
.unique()
return { cursor: state?.cursor ?? null, doneAt: state?.doneAt }
},
})
export const setSkillStatBackfillStateInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
doneAt: v.optional(v.number()),
},
handler: async (ctx, args) => {
const now = Date.now()
const state = await ctx.db
.query('skillStatBackfillState')
.withIndex('by_key', (q) => q.eq('key', BACKFILL_STATE_KEY))
.unique()
if (!state) {
await ctx.db.insert('skillStatBackfillState', {
key: BACKFILL_STATE_KEY,
cursor: args.cursor,
doneAt: args.doneAt,
updatedAt: now,
})
return { ok: true as const }
}
await ctx.db.patch(state._id, {
cursor: args.cursor,
doneAt: args.doneAt,
updatedAt: now,
})
return { ok: true as const }
},
})
async function runSkillStatBackfillInternalHandler(
ctx: ActionCtx,
args: BackfillActionArgs,
): Promise<BackfillActionResult> {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
if (args.resetCursor) {
await ctx.runMutation(internal.statsMaintenance.setSkillStatBackfillStateInternal, {
cursor: undefined,
doneAt: undefined,
})
}
const state = (await ctx.runQuery(
internal.statsMaintenance.getSkillStatBackfillStateInternal,
{},
)) as BackfillState
if (state.doneAt && !args.resetCursor) {
return {
ok: true,
isDone: true,
cursor: null,
stats: { scanned: 0, patched: 0, batches: 0 },
}
}
let cursor: string | null = state.cursor ?? null
const stats: BackfillStats = { scanned: 0, patched: 0, batches: 0 }
for (let i = 0; i < maxBatches; i += 1) {
const result = (await ctx.runMutation(
internal.statsMaintenance.backfillSkillStatFieldsInternal,
{
cursor: cursor ?? undefined,
batchSize,
},
)) as { scanned: number; patched: number; cursor: string | null; isDone: boolean }
stats.scanned += result.scanned
stats.patched += result.patched
stats.batches += 1
cursor = result.cursor
if (result.isDone) {
await ctx.runMutation(internal.statsMaintenance.setSkillStatBackfillStateInternal, {
cursor: undefined,
doneAt: Date.now(),
})
return { ok: true, isDone: true, cursor: null, stats }
}
await ctx.runMutation(internal.statsMaintenance.setSkillStatBackfillStateInternal, {
cursor: cursor ?? undefined,
doneAt: undefined,
})
}
return { ok: true, isDone: false, cursor, stats }
}
export const runSkillStatBackfillInternal: ReturnType<typeof internalAction> = internalAction({
args: {
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
resetCursor: v.optional(v.boolean()),
},
handler: runSkillStatBackfillInternalHandler,
})
function buildSkillStatPatch(skill: Doc<'skills'>) {
const stats = skill.stats
const nextDownloads = stats.downloads
const nextStars = stats.stars
const nextInstallsCurrent = stats.installsCurrent ?? 0
const nextInstallsAllTime = stats.installsAllTime ?? 0
if (
skill.statsDownloads === nextDownloads &&
skill.statsStars === nextStars &&
skill.statsInstallsCurrent === nextInstallsCurrent &&
skill.statsInstallsAllTime === nextInstallsAllTime
) {
return null
}
return {
statsDownloads: nextDownloads,
statsStars: nextStars,
statsInstallsCurrent: nextInstallsCurrent,
statsInstallsAllTime: nextInstallsAllTime,
}
}
function clampInt(value: number, min: number, max: number) {
return Math.min(Math.max(value, min), max)
}
+434
View File
@@ -0,0 +1,434 @@
import { getAuthUserId } from '@convex-dev/auth/server'
import { v } from 'convex/values'
import type { Id } from './_generated/dataModel'
import type { MutationCtx, QueryCtx } from './_generated/server'
import { internalMutation, mutation, query } from './_generated/server'
import { requireUser } from './lib/access'
import { insertStatEvent } from './skillStatEvents'
const TELEMETRY_STALE_MS = 120 * 24 * 60 * 60 * 1000
type RootPayload = {
rootId: string
label: string
skills: Array<{ slug: string; version?: string | null }>
}
export const reportCliSyncInternal = internalMutation({
args: {
userId: v.id('users'),
roots: v.array(
v.object({
rootId: v.string(),
label: v.string(),
skills: v.array(
v.object({
slug: v.string(),
version: v.optional(v.string()),
}),
),
}),
),
},
handler: async (ctx, args) => {
const now = Date.now()
const stalenessCutoff = now - TELEMETRY_STALE_MS
await expireStaleRoots(ctx, { userId: args.userId, stalenessCutoff, now })
const roots = normalizeRoots(args.roots)
const skillsBySlug = await resolveSkillsBySlug(ctx, roots)
for (const root of roots) {
await upsertRoot(ctx, { userId: args.userId, rootId: root.rootId, now, label: root.label })
await applyRootReport(ctx, {
userId: args.userId,
root,
skillsBySlug,
now,
})
}
},
})
export const clearMyTelemetry = mutation({
args: {},
handler: async (ctx) => {
const { userId } = await requireUser(ctx)
await clearTelemetryForUser(ctx, { userId })
},
})
export const clearUserTelemetryInternal = internalMutation({
args: { userId: v.id('users') },
handler: async (ctx, args) => {
await clearTelemetryForUser(ctx, { userId: args.userId })
},
})
export const getMyInstalled = query({
args: {
includeRemoved: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const userId = await getAuthUserId(ctx)
if (!userId) return null
const roots = await ctx.db
.query('userSyncRoots')
.withIndex('by_user', (q) => q.eq('userId', userId))
.order('desc')
.take(200)
const includeRemoved = Boolean(args.includeRemoved)
const resultRoots: Array<{
rootId: string
label: string
firstSeenAt: number
lastSeenAt: number
expiredAt?: number
skills: Array<{
skill: {
slug: string
displayName: string
summary?: string
stats: unknown
ownerUserId: Id<'users'>
}
firstSeenAt: number
lastSeenAt: number
lastVersion?: string
removedAt?: number
}>
}> = []
for (const root of roots) {
const installs = await ctx.db
.query('userSkillRootInstalls')
.withIndex('by_user_root', (q) => q.eq('userId', userId).eq('rootId', root.rootId))
.order('desc')
.take(2000)
const filtered = includeRemoved ? installs : installs.filter((entry) => !entry.removedAt)
const skills: Array<{
skill: {
slug: string
displayName: string
summary?: string
stats: unknown
ownerUserId: Id<'users'>
}
firstSeenAt: number
lastSeenAt: number
lastVersion?: string
removedAt?: number
}> = []
for (const entry of filtered) {
const skill = await ctx.db.get(entry.skillId)
if (!skill) continue
skills.push({
skill: {
slug: skill.slug,
displayName: skill.displayName,
summary: skill.summary,
stats: skill.stats,
ownerUserId: skill.ownerUserId,
},
firstSeenAt: entry.firstSeenAt,
lastSeenAt: entry.lastSeenAt,
lastVersion: entry.lastVersion,
removedAt: entry.removedAt,
})
}
resultRoots.push({
rootId: root.rootId,
label: root.label,
firstSeenAt: root.firstSeenAt,
lastSeenAt: root.lastSeenAt,
expiredAt: root.expiredAt,
skills,
})
}
return {
roots: resultRoots,
cutoffDays: 120,
}
},
})
async function clearTelemetryForUser(ctx: MutationCtx, params: { userId: Id<'users'> }) {
const installs = await ctx.db
.query('userSkillInstalls')
.withIndex('by_user', (q) => q.eq('userId', params.userId))
.take(5000)
for (const entry of installs) {
const skill = await ctx.db.get(entry.skillId)
if (!skill) {
await ctx.db.delete(entry._id)
continue
}
await insertStatEvent(ctx, {
skillId: skill._id,
kind: 'install_clear',
delta: {
allTime: -1,
current: entry.activeRoots > 0 ? -1 : 0,
},
})
await ctx.db.delete(entry._id)
}
const roots = await ctx.db
.query('userSyncRoots')
.withIndex('by_user', (q) => q.eq('userId', params.userId))
.take(5000)
for (const root of roots) {
await ctx.db.delete(root._id)
}
const rootInstalls = await ctx.db
.query('userSkillRootInstalls')
.withIndex('by_user', (q) => q.eq('userId', params.userId))
.take(10000)
for (const entry of rootInstalls) {
await ctx.db.delete(entry._id)
}
}
function normalizeRoots(roots: RootPayload[]): RootPayload[] {
const seen = new Set<string>()
const unique: RootPayload[] = []
for (const root of roots) {
const id = root.rootId.trim()
if (!id) continue
if (seen.has(id)) continue
seen.add(id)
unique.push({
rootId: id,
label: root.label.trim() || 'Unknown',
skills: root.skills
.map((skill) => ({
slug: skill.slug.trim().toLowerCase(),
version: skill.version ?? null,
}))
.filter((skill) => Boolean(skill.slug)),
})
}
return unique
}
async function upsertRoot(
ctx: MutationCtx,
params: { userId: Id<'users'>; rootId: string; now: number; label: string },
) {
const existing = await ctx.db
.query('userSyncRoots')
.withIndex('by_user_root', (q) => q.eq('userId', params.userId).eq('rootId', params.rootId))
.unique()
if (existing) {
await ctx.db.patch(existing._id, {
label: params.label,
lastSeenAt: params.now,
expiredAt: undefined,
})
return
}
await ctx.db.insert('userSyncRoots', {
userId: params.userId,
rootId: params.rootId,
label: params.label,
firstSeenAt: params.now,
lastSeenAt: params.now,
expiredAt: undefined,
})
}
async function applyRootReport(
ctx: MutationCtx,
params: {
userId: Id<'users'>
root: RootPayload
skillsBySlug: Map<string, { skillId: Id<'skills'> }>
now: number
},
) {
const expected = new Set<Id<'skills'>>()
const versionsBySkill = new Map<Id<'skills'>, string | undefined>()
for (const entry of params.root.skills) {
const resolved = params.skillsBySlug.get(entry.slug)
if (!resolved) continue
expected.add(resolved.skillId)
const version = entry.version?.trim() || undefined
if (version) versionsBySkill.set(resolved.skillId, version)
}
const previous = await ctx.db
.query('userSkillRootInstalls')
.withIndex('by_user_root', (q) =>
q.eq('userId', params.userId).eq('rootId', params.root.rootId),
)
.take(5000)
const active = previous.filter((entry) => !entry.removedAt)
for (const skillId of expected) {
const existing = await ctx.db
.query('userSkillRootInstalls')
.withIndex('by_user_root_skill', (q) =>
q.eq('userId', params.userId).eq('rootId', params.root.rootId).eq('skillId', skillId),
)
.unique()
const reportedVersion = versionsBySkill.get(skillId)
if (existing) {
const wasRemoved = Boolean(existing.removedAt)
await ctx.db.patch(existing._id, {
lastSeenAt: params.now,
lastVersion: reportedVersion ?? existing.lastVersion,
removedAt: undefined,
})
if (wasRemoved) {
await incrementActiveRoots(ctx, {
userId: params.userId,
skillId,
now: params.now,
version: reportedVersion,
})
}
continue
}
await ctx.db.insert('userSkillRootInstalls', {
userId: params.userId,
rootId: params.root.rootId,
skillId,
firstSeenAt: params.now,
lastSeenAt: params.now,
lastVersion: reportedVersion,
})
await incrementActiveRoots(ctx, {
userId: params.userId,
skillId,
now: params.now,
version: reportedVersion,
})
}
for (const entry of active) {
if (expected.has(entry.skillId)) continue
await ctx.db.patch(entry._id, { removedAt: params.now })
await decrementActiveRoots(ctx, { userId: params.userId, skillId: entry.skillId })
}
}
async function incrementActiveRoots(
ctx: MutationCtx,
params: { userId: Id<'users'>; skillId: Id<'skills'>; now: number; version?: string },
) {
const existing = await ctx.db
.query('userSkillInstalls')
.withIndex('by_user_skill', (q) => q.eq('userId', params.userId).eq('skillId', params.skillId))
.unique()
if (!existing) {
await ctx.db.insert('userSkillInstalls', {
userId: params.userId,
skillId: params.skillId,
firstSeenAt: params.now,
lastSeenAt: params.now,
activeRoots: 1,
lastVersion: params.version,
})
await bumpSkillInstallCounts(ctx, { skillId: params.skillId, deltaAllTime: 1, deltaCurrent: 1 })
return
}
const nextActive = Math.max(0, (existing.activeRoots ?? 0) + 1)
await ctx.db.patch(existing._id, {
activeRoots: nextActive,
lastSeenAt: params.now,
lastVersion: params.version ?? existing.lastVersion,
})
if ((existing.activeRoots ?? 0) === 0 && nextActive > 0) {
await bumpSkillInstallCounts(ctx, { skillId: params.skillId, deltaAllTime: 0, deltaCurrent: 1 })
}
}
async function decrementActiveRoots(
ctx: MutationCtx,
params: { userId: Id<'users'>; skillId: Id<'skills'> },
) {
const existing = await ctx.db
.query('userSkillInstalls')
.withIndex('by_user_skill', (q) => q.eq('userId', params.userId).eq('skillId', params.skillId))
.unique()
if (!existing) return
const nextActive = Math.max(0, (existing.activeRoots ?? 0) - 1)
await ctx.db.patch(existing._id, { activeRoots: nextActive })
if ((existing.activeRoots ?? 0) > 0 && nextActive === 0) {
await bumpSkillInstallCounts(ctx, {
skillId: params.skillId,
deltaAllTime: 0,
deltaCurrent: -1,
})
}
}
async function bumpSkillInstallCounts(
ctx: MutationCtx,
params: { skillId: Id<'skills'>; deltaAllTime: number; deltaCurrent: number },
) {
if (params.deltaAllTime === 1 && params.deltaCurrent === 1) {
await insertStatEvent(ctx, { skillId: params.skillId, kind: 'install_new' })
} else if (params.deltaAllTime === 0 && params.deltaCurrent === 1) {
await insertStatEvent(ctx, { skillId: params.skillId, kind: 'install_reactivate' })
} else if (params.deltaAllTime === 0 && params.deltaCurrent === -1) {
await insertStatEvent(ctx, { skillId: params.skillId, kind: 'install_deactivate' })
}
}
async function expireStaleRoots(
ctx: MutationCtx,
params: { userId: Id<'users'>; stalenessCutoff: number; now: number },
) {
const roots = await ctx.db
.query('userSyncRoots')
.withIndex('by_user', (q) => q.eq('userId', params.userId))
.take(5000)
const stale = roots.filter((root) => !root.expiredAt && root.lastSeenAt < params.stalenessCutoff)
for (const root of stale) {
await ctx.db.patch(root._id, { expiredAt: params.now })
const installs = await ctx.db
.query('userSkillRootInstalls')
.withIndex('by_user_root', (q) => q.eq('userId', params.userId).eq('rootId', root.rootId))
.take(5000)
for (const entry of installs) {
if (entry.removedAt) continue
await ctx.db.patch(entry._id, { removedAt: params.now })
await decrementActiveRoots(ctx, { userId: params.userId, skillId: entry.skillId })
}
}
}
async function resolveSkillsBySlug(ctx: QueryCtx | MutationCtx, roots: RootPayload[]) {
const slugs = new Set<string>()
for (const root of roots) {
for (const entry of root.skills) slugs.add(entry.slug)
}
const map = new Map<string, { skillId: Id<'skills'> }>()
for (const slug of slugs) {
const skill = await ctx.db
.query('skills')
.withIndex('by_slug', (q) => q.eq('slug', slug))
.unique()
if (skill && !skill.softDeletedAt) map.set(slug, { skillId: skill._id })
}
return map
}
+14 -5
View File
@@ -1,12 +1,19 @@
import { getAuthUserId } from '@convex-dev/auth/server'
import { v } from 'convex/values'
import { mutation, query } from './_generated/server'
import { assertRole, requireUser } from './lib/access'
import { internal } from './_generated/api'
import { internalQuery, mutation, query } from './_generated/server'
import { assertAdmin, requireUser } from './lib/access'
import { toPublicUser } from './lib/public'
const DEFAULT_ROLE = 'user'
const ADMIN_HANDLE = 'steipete'
export const getById = query({
args: { userId: v.id('users') },
handler: async (ctx, args) => toPublicUser(await ctx.db.get(args.userId)),
})
export const getByIdInternal = internalQuery({
args: { userId: v.id('users') },
handler: async (ctx, args) => ctx.db.get(args.userId),
})
@@ -69,6 +76,7 @@ export const deleteAccount = mutation({
deletedAt: Date.now(),
updatedAt: Date.now(),
})
await ctx.runMutation(internal.telemetry.clearUserTelemetryInternal, { userId })
},
})
@@ -76,7 +84,7 @@ export const list = query({
args: { limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
assertRole(user, ['admin'])
assertAdmin(user)
const limit = args.limit ?? 50
return ctx.db.query('users').order('desc').take(limit)
},
@@ -85,10 +93,11 @@ export const list = query({
export const getByHandle = query({
args: { handle: v.string() },
handler: async (ctx, args) => {
return ctx.db
const user = await ctx.db
.query('users')
.withIndex('handle', (q) => q.eq('handle', args.handle))
.unique()
return toPublicUser(user)
},
})
@@ -99,7 +108,7 @@ export const setRole = mutation({
},
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
assertRole(user, ['admin'])
assertAdmin(user)
await ctx.db.patch(args.userId, { role: args.role, updatedAt: Date.now() })
await ctx.db.insert('auditLogs', {
actorUserId: user._id,
+50
View File
@@ -0,0 +1,50 @@
import { v } from 'convex/values'
import { internalAction } from './_generated/server'
import { buildDiscordPayload, getWebhookConfig, shouldSendWebhook } from './lib/webhooks'
export const sendDiscordWebhook = internalAction({
args: {
event: v.union(v.literal('skill.publish'), v.literal('skill.highlighted')),
skill: v.object({
slug: v.string(),
displayName: v.string(),
summary: v.optional(v.string()),
version: v.optional(v.string()),
ownerHandle: v.optional(v.string()),
highlighted: v.optional(v.boolean()),
tags: v.optional(v.array(v.string())),
}),
},
handler: async (_ctx, args) => {
const config = getWebhookConfig()
const logMeta = {
event: args.event,
slug: args.skill.slug,
version: args.skill.version ?? null,
highlighted: args.skill.highlighted ?? false,
highlightedOnly: config.highlightedOnly,
}
if (!shouldSendWebhook(args.event, args.skill, config)) {
console.info('[webhook] skipped', logMeta)
return { ok: false, skipped: true }
}
const payload = buildDiscordPayload(args.event, args.skill, config)
const response = await fetch(config.url as string, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload),
})
if (!response.ok) {
const message = await response.text()
console.error('[webhook] failed', {
...logMeta,
status: response.status,
body: message.slice(0, 300),
})
throw new Error(`Discord webhook failed: ${response.status} ${message}`)
}
console.info('[webhook] sent', { ...logMeta, status: response.status })
return { ok: true }
},
})
+32
View File
@@ -0,0 +1,32 @@
---
summary: 'Documentation index + reading order.'
read_when:
- New contributor onboarding
- Looking for the right doc
---
# Docs
Reading order (new contributor):
1. `README.md` (repo root): run locally.
2. `docs/quickstart.md`: end-to-end: search → install → publish → sync.
3. `docs/architecture.md`: how the pieces fit (TanStack Start + Convex + CLI).
4. `docs/skill-format.md`: what a “skill” is on disk + on the registry.
5. `docs/cli.md`: CLI reference (flags, config, lockfiles, sync rules).
6. `docs/http-api.md`: HTTP endpoints used by the CLI + public API.
7. `docs/auth.md`: GitHub OAuth + API tokens + CLI loopback login.
8. `docs/deploy.md`: Convex + Vercel deployment + rewrites.
9. `docs/troubleshooting.md`: common failure modes.
Feature/ops docs (already present):
- `docs/spec.md`: product + implementation spec (data model + flows).
- `docs/telemetry.md`: what `clawhub sync` reports; opt-out.
- `docs/webhook.md`: Discord webhook events/payload.
- `docs/diffing.md`: version-to-version diff UI spec.
- `docs/manual-testing.md`: CLI smoke scripts.
Docs tooling:
- `docs/mintlify.md`: publish these docs with Mintlify.
+51
View File
@@ -0,0 +1,51 @@
---
summary: 'Public REST API (v1) overview and conventions.'
read_when:
- Building API clients
- Adding endpoints or schemas
---
# API v1
Base: `https://clawhub.ai`
OpenAPI: `/api/v1/openapi.json`
## Auth
- Public read: no token required.
- Write + account: `Authorization: Bearer clh_...`.
## Rate limits
Per IP + per API key:
- Read: 120/min per IP, 600/min per key
- Write: 30/min per IP, 120/min per key
Headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` (on 429).
## Endpoints
Public read:
- `GET /api/v1/search?q=...`
- `GET /api/v1/skills?limit=&cursor=&sort=`
- `sort`: `updated` (default), `downloads`, `stars` (`rating`), `installsCurrent` (`installs`), `installsAllTime`, `trending`
- `GET /api/v1/skills/{slug}`
- `GET /api/v1/skills/{slug}/versions?limit=&cursor=`
- `GET /api/v1/skills/{slug}/versions/{version}`
- `GET /api/v1/skills/{slug}/file?path=&version=&tag=`
- `GET /api/v1/resolve?slug=&hash=`
- `GET /api/v1/download?slug=&version=&tag=`
Auth required:
- `POST /api/v1/skills` (publish, multipart preferred)
- `DELETE /api/v1/skills/{slug}`
- `POST /api/v1/skills/{slug}/undelete`
- `GET /api/v1/whoami`
## Legacy
Legacy `/api/*` and `/api/cli/*` still available. See `DEPRECATIONS.md`.
+61
View File
@@ -0,0 +1,61 @@
---
summary: 'System overview: web app + Convex backend + CLI + shared schema.'
read_when:
- Orienting in codebase
- Tracing a user flow across layers
---
# Architecture
## Pieces
- Web app: TanStack Start (React) under `src/`.
- Backend: Convex under `convex/` (DB, storage, actions, HTTP routes).
- CLI: `packages/clawdhub/` (published as `clawhub`, legacy `clawdhub`).
- Shared schemas/routes: `packages/schema/` (`clawhub-schema`).
## Data + storage
- Skill “bundle” = versioned set of text files stored in Convex `_storage`.
- Metadata extracted from `SKILL.md` frontmatter.
- Stats stored on `skills` (downloads, installs, stars, comments, …).
## Main flows
### Browse (web)
- UI reads skill metadata + latest version from Convex queries/actions.
- `SKILL.md` rendered as Markdown.
### Search (HTTP)
- `/api/v1/search?q=...` routes to Convex action for vector search.
- Embeddings currently generated during publish.
### Install (CLI)
- Resolve latest version via `/api/v1/skills/<slug>`.
- Download zip via `/api/v1/download?slug=...&version=...`.
- Extract into `./skills/<slug>` (default).
- Persist install state:
- `./.clawhub/lock.json` (per workdir, legacy `.clawdhub`)
- `./skills/<slug>/.clawhub/origin.json` (per skill folder, legacy `.clawdhub`)
### Update (CLI)
- Hash local files, call `/api/v1/resolve?slug=...&hash=<sha256>`.
- If local matches a known version → use that for “current”.
- If local doesnt match:
- refuse by default
- or overwrite with `--force`
### Publish (CLI)
- Publish via `POST /api/v1/skills` (multipart; requires Bearer token).
### Sync (CLI)
- Scan roots for skill folders (contain `SKILL.md`).
- Compute fingerprint; compare to registry state.
- Optionally reports telemetry (see `docs/telemetry.md`).
- Publishes new/changed skills (skips modified installed skills inside install root).
+54
View File
@@ -0,0 +1,54 @@
---
summary: 'Auth overview: GitHub OAuth (web) + API tokens (CLI).'
read_when:
- Working on login/token flows
- Debugging 401s
---
# Auth
## Web auth (GitHub OAuth)
- Convex Auth + GitHub OAuth App.
- Env vars:
- `AUTH_GITHUB_ID`
- `AUTH_GITHUB_SECRET`
- `CONVEX_SITE_URL` (used by auth config)
Local setup steps are in the repo root `README.md`.
## API tokens (CLI)
The CLI uses a long-lived API token (Bearer token) for publish/sync/delete.
### Browser flow (default)
`clawhub login` does:
1. Starts a loopback HTTP server on `127.0.0.1` (random port).
2. Opens `<site>/cli/auth?redirect_uri=http://127.0.0.1:<port>/callback&state=...`.
3. Web UI requires GitHub login, then creates a token and redirects back to the loopback server.
4. CLI stores the token in the global config file.
### Headless flow
Create a token in the web UI (Settings → API tokens) and paste it:
```bash
clawhub login --token clh_...
```
### Token storage
Default global config path:
- macOS: `~/Library/Application Support/clawhub/config.json`
Override:
- `CLAWHUB_CONFIG_PATH=/path/to/config.json` (legacy `CLAWDHUB_CONFIG_PATH`)
### Revocation
- Tokens can be revoked in the web UI.
- Revoked tokens return `401 Unauthorized` on CLI endpoints.
+117
View File
@@ -0,0 +1,117 @@
---
summary: 'CLI reference: commands, flags, config, lockfile, sync behavior.'
read_when:
- Working on CLI behavior
- Debugging install/update/sync
---
# CLI
CLI package: `packages/clawdhub/` (published as `clawhub`, bin: `clawhub`).
From this repo you can run it via the wrapper script:
```bash
bun clawhub --help
```
## Global flags
- `--workdir <dir>`: working directory (default: cwd; falls back to Clawdbot workspace if configured)
- `--dir <dir>`: install dir under workdir (default: `skills`)
- `--site <url>`: base URL for browser login (default: `https://clawhub.ai`)
- `--registry <url>`: API base URL (default: discovered, else `https://clawhub.ai`)
- `--no-input`: disable prompts
Env equivalents:
- `CLAWHUB_SITE` (legacy `CLAWDHUB_SITE`)
- `CLAWHUB_REGISTRY` (legacy `CLAWDHUB_REGISTRY`)
- `CLAWHUB_WORKDIR` (legacy `CLAWDHUB_WORKDIR`)
## Config file
Stores your API token + cached registry URL.
- macOS: `~/Library/Application Support/clawhub/config.json`
- override: `CLAWHUB_CONFIG_PATH` (legacy `CLAWDHUB_CONFIG_PATH`)
## Commands
### `login` / `auth login`
- Default: opens browser to `<site>/cli/auth` and completes via loopback callback.
- Headless: `clawhub login --token clh_...`
### `whoami`
- Verifies the stored token via `/api/v1/whoami`.
### `star <slug>` / `unstar <slug>`
- Adds/removes a skill from your highlights.
- Calls `POST /api/v1/stars/<slug>` and `DELETE /api/v1/stars/<slug>`.
- `--yes` skips confirmation.
### `search <query...>`
- Calls `/api/v1/search?q=...`.
### `explore`
- Lists latest updated skills via `/api/v1/skills?limit=...` (sorted by `updatedAt` desc).
- Flags:
- `--limit <n>` (1200, default: 25)
- `--sort newest|downloads|rating|installs|installsAllTime|trending` (default: newest)
- `--json` (machine-readable output)
- Output: `<slug> v<version> <age> <summary>` (summary truncated to 50 chars).
### `install <slug>`
- Resolves latest version via `/api/v1/skills/<slug>`.
- Downloads zip via `/api/v1/download`.
- Extracts into `<workdir>/<dir>/<slug>`.
- Writes:
- `<workdir>/.clawhub/lock.json` (legacy `.clawdhub`)
- `<skill>/.clawhub/origin.json` (legacy `.clawdhub`)
### `list`
- Reads `<workdir>/.clawhub/lock.json` (legacy `.clawdhub`).
### `update [slug]` / `update --all`
- Computes fingerprint from local files.
- If fingerprint matches a known version: no prompt.
- If fingerprint does not match:
- refuses by default
- overwrites with `--force` (or prompt, if interactive)
### `publish <path>`
- Publishes via `POST /api/v1/skills` (multipart).
- Requires semver: `--version 1.2.3`.
### `sync`
- Scans for local skill folders and publishes new/changed ones.
- Roots can be any folder: a skills directory or a single skill folder with `SKILL.md`.
- Auto-adds Clawdbot skill roots when `~/.clawdbot/clawdbot.json` is present:
- `agent.workspace/skills` (main agent)
- `routing.agents.*.workspace/skills` (per-agent)
- `~/.clawdbot/skills` (shared)
- `skills.load.extraDirs` (shared packs)
- Respects `CLAWDBOT_CONFIG_PATH` / `CLAWDBOT_STATE_DIR` and `OPENCLAW_CONFIG_PATH` / `OPENCLAW_STATE_DIR`.
- Flags:
- `--root <dir...>` extra scan roots
- `--all` upload without prompting
- `--dry-run` show plan only
- `--bump patch|minor|major` (default: patch)
- `--changelog <text>` (non-interactive)
- `--tags a,b,c` (default: latest)
- `--concurrency <n>` (default: 4)
Telemetry:
- Sent during `sync` when logged in, unless `CLAWHUB_DISABLE_TELEMETRY=1` (legacy `CLAWDHUB_DISABLE_TELEMETRY=1`).
- Details: `docs/telemetry.md`.
+79
View File
@@ -0,0 +1,79 @@
---
summary: 'Deploy checklist: Convex backend + Vercel web app + /api rewrites.'
read_when:
- Shipping to production
- Debugging /api routing
---
# Deploy
OpenClaw is two deployables:
- Web app (TanStack Start) → typically Vercel.
- Convex backend → Convex deployment (serves `/api/...` routes).
## 1) Deploy Convex
From your local machine:
```bash
bunx convex deploy
```
Ensure Convex env is set (auth + embeddings):
- `AUTH_GITHUB_ID`
- `AUTH_GITHUB_SECRET`
- `CONVEX_SITE_URL`
- `JWT_PRIVATE_KEY`
- `JWKS`
- `OPENAI_API_KEY`
- `SITE_URL` (your web app URL)
- Optional webhook env (see `docs/webhook.md`)
## 2) Deploy web app (Vercel)
Set env vars:
- `VITE_CONVEX_URL`
- `VITE_CONVEX_SITE_URL` (Convex “site” URL)
- `CONVEX_SITE_URL` (same value; used by auth provider config)
- `SITE_URL` (web app URL)
## 3) Route `/api/*` to Convex
This repo currently uses `vercel.json` rewrites:
- `source: /api/:path*`
- `destination: https://<deployment>.convex.site/api/:path*`
For self-host:
- update `vercel.json` to your deployments Convex site URL.
## 4) Registry discovery
The CLI can discover the API base from:
- `/.well-known/clawhub.json` (preferred)
- `/.well-known/clawdhub.json` (legacy)
If you dont serve that file, users must set:
```bash
export CLAWHUB_REGISTRY=https://your-site.example
```
## 5) Post-deploy checks
```bash
curl -i "https://<site>/api/v1/search?q=test"
curl -i "https://<site>/api/v1/skills/gifgrep"
```
Then:
```bash
clawhub login --site https://<site>
clawhub whoami
```
+84
View File
@@ -0,0 +1,84 @@
---
summary: "Skill version diffing mode (Monaco-backed)"
read_when:
- Implementing skill diff UI
- Adding version comparisons
---
# Diffing mode
## Goals
- Compare any file between two versions.
- Default compare: `latest` vs `previous` (SemVer precedence).
- UX feels native to OpenClaw (theme + typography + motion).
- Inline or side-by-side toggle.
- Public access.
## UX
- Diff card on skill detail page.
- Two selectors: Left/Right.
- Items: version strings, plus tags (e.g. `latest`), plus `previous`.
- Default: Left = `previous`, Right = `latest`.
- File list with status: added / removed / changed / same.
- Default file: `SKILL.md` if present; else first changed file.
- Toggle: Inline vs Side-by-side.
- Show size guard message when file > 200KB.
## SemVer ordering
- Use SemVer precedence to sort versions.
- `previous` = immediate predecessor of `latest` by SemVer.
- If `latest` missing or only one version:
- Disable `previous` and show empty-state copy.
## Data sources
- Versions: `api.skills.listVersions` (all, not just latest 10).
- Tags: `skill.tags` map.
- File list: `version.files` with `path`, `sha256`, `size`.
## API
Add action:
- `skills.getFileText({ versionId, path }) -> { text, size, sha256 }`
- Validate version exists + file path exists in version.
- Enforce size <= 200KB (both in action and client).
- Use `fetchText` from `convex/lib/skillPublish.ts`.
Optional helper action:
- `skills.getVersionFiles({ versionId }) -> files[]`
- If we want lightweight fetch without full version object.
## Client flow
1. Fetch versions + tags.
2. Resolve default compare pair:
- Right = tag `latest` if present else highest SemVer.
- Left = `previous` (SemVer predecessor).
3. Build file union by path.
4. For selected file:
- Fetch left/right text (guard by size).
- Feed into Monaco diff editor.
## Monaco theming
- Define `clawhub-light` / `clawhub-dark` via `monaco.editor.defineTheme`.
- Derive colors from CSS variables on `document.documentElement`:
- `--surface`, `--surface-muted`, `--ink`, `--ink-soft`, `--line`, `--accent`.
- Apply theme on load + when theme changes (`data-theme`).
- Match font: `var(--font-mono)`.
- Set diff options:
- `renderSideBySide` toggle
- `diffAlgorithm: 'advanced'`
- `renderSideBySideInlineBreakpoint` for mobile
- `wordWrap: 'on'`
## Edge cases
- File removed/added: show empty buffer on missing side + label.
- Non-text file should not exist (upload rejects), but still guard.
- Large file: show size warning + disable fetch.
- Missing version: show error state.
## Perf
- Cache file text per version+path in client state.
- Debounce selector changes (100-200ms).
- Limit concurrent fetches to 2.
## Tests
- Unit: SemVer ordering + `previous` selection.
- Component: default selectors, tag inclusion, size guard.
+171
View File
@@ -0,0 +1,171 @@
---
summary: 'Feature spec: import a skill from a public GitHub URL (auto-detect SKILL.md, selective file upload, provenance).'
read_when:
- Adding GitHub import (web + API)
- Reviewing safety limits (SSRF/zip-bombs)
- Implementing provenance + canonical-claim flows
---
# GitHub import (public repos)
Goal: paste a GitHub URL → auto-detect skill → preview files → publish (selective) → persist provenance.
Non-goal (v1): private repos (no OAuth/PAT support).
Related:
- `docs/skill-format.md` (what counts as a skill; text-only limits)
- `docs/api.md` / `docs/http-api.md` (REST patterns + auth)
## UX
Upload page: “Import from GitHub” mode.
Flow:
1) URL input
2) Detect skill candidates (SKILL.md)
3) If multiple candidates: choose one
4) File picker: check/uncheck; smart-select referenced files
5) Confirm slug/name/version/tags
6) Import → publish
## Accepted URLs
Allowlist: `https://github.com/...` only.
Supported shapes:
- Repo root: `https://github.com/<owner>/<repo>`
- Tree path: `https://github.com/<owner>/<repo>/tree/<ref>/<path>`
- Blob path (file): `https://github.com/<owner>/<repo>/blob/<ref>/<path>`
Normalization:
- Strip query/hash for fetch.
- From `blob/.../SKILL.md` derive `path` as parent folder.
- If `ref` missing: use `HEAD`.
Reject:
- Non-GitHub hosts.
- Unknown URL patterns.
- Paths containing `..` after normalization.
## Fetch strategy (public)
Download archive:
- `https://github.com/<owner>/<repo>/archive/<ref>.zip`
- Follow redirects. Final redirect usually pins a commit via `codeload.github.com/.../zip/<sha-or-branch>`.
Unzip server-side (Node or Convex node action). Scan for skill candidates.
Skill candidate definition:
- Any folder containing `SKILL.md` or `skill.md` (also accept `skills.md` for compatibility).
- Treat repo root as a folder too.
Multiple skills:
- Return candidate list: `{ path, frontmatter.name, frontmatter.description }`.
- User chooses one.
## Smart file selection
Defaults:
- Always select `SKILL.md` (or chosen readme file).
- Prefer selecting only within chosen skill folder; allow “include out-of-folder refs” if explicitly toggled.
Referenced file expansion:
- Parse Markdown links/images from selected `.md` files:
- `[](<rel>)`, `![](<rel>)`, `<rel>` only when relative.
- Ignore `http(s):`, `mailto:`, `#anchors`.
- Strip query/hash from relative targets.
- Resolve against the current files directory.
- Normalize, reject escapes (`..`).
- Add referenced file if present in archive and is text-allowed.
- Recurse for newly added `.md` files.
Hard caps:
- Max recursion depth (e.g. 4).
- Max referenced additions (e.g. 200).
UI affordances:
- “Select referenced”
- “Select all text”
- “Clear”
- Search/filter by path
## Publish behavior
Server publishes using existing pipeline:
- Text-only enforced (see `docs/skill-format.md`).
- Total ≤ 50MB (selected set).
- Must include `SKILL.md` (or accepted variant).
Suggested defaults (UI):
- `displayName`: frontmatter `name` else folder basename → title case.
- `slug`: sanitize folder basename; if collision, suffix (`-2`, `-3`, …).
- `version`: if new skill → `0.1.0`; if updating own existing skill → bump patch.
- `tags`: default `latest`.
## Provenance (persist source)
Persist on each published version (server-side injection; no mutation of imported files):
- Store in `skillVersions.parsed.metadata.source`:
Example:
```json
{
"kind": "github",
"url": "https://github.com/visionik/ouracli",
"repo": "visionik/ouracli",
"ref": "HEAD",
"commit": "66ac8fb266b7c5ff6519431862be6a375bbfb883",
"path": "",
"importedAt": 1767930000000
}
```
Why `parsed.metadata`:
- Already optional and stored with each version.
- No schema churn for v1.
Future: canonical-claim
- “claim canonical” can key off `{ kind:'github', repo, path }`.
- Prefer commit-pinned provenance for auditability; allow UI to show “Imported from …”.
## API sketch (internal actions)
Two-step (recommended):
- `previewGitHubImport(url)``{ commit, candidates:[...], files:[...], defaults:{...} }`
- `importGitHubSkill({ url, commit, candidatePath, selectedPaths, slug, displayName, version, tags })`
Notes:
- `importGitHubSkill` should re-fetch by pinned `commit` (not floating branch), to avoid TOCTOU.
- Validate `selectedPaths` subset of fetched archive manifest.
## Security / abuse controls
SSRF:
- Only `github.com` (+ `codeload.github.com` during redirect follow).
- No arbitrary redirects to other hosts.
Zip safety:
- Max compressed bytes (from `Content-Length` if present; else streaming cap).
- Max uncompressed total bytes.
- Max file count.
- Max single file size.
- Reject symlinks; reject absolute paths; reject `..` segments.
Rate limits:
- Tie to existing write limits (import == publish).
- Cache preview results briefly (e.g. 60s) keyed by `{repo, commit}`.
Error UX:
- “No SKILL.md found.”
- “Multiple skills found; pick one.”
- “Repo too large / too many files.”
- “Selected files exceed 50MB.”
## Manual test checklist
- Repo root skill (`SKILL.md` at root).
- Nested skill (`skills/foo/SKILL.md`).
- Multi-skill repo (two SKILL.md).
- SKILL.md references `docs/usage.md` + images; smart-select picks `.md` and referenced text files; ignores external links.
- Huge repo → clean “too large” error.
- Redirect pinning → import stores commit sha in provenance.
+188
View File
@@ -0,0 +1,188 @@
---
summary: 'HTTP API reference (public + CLI endpoints + auth).'
read_when:
- Adding/changing endpoints
- Debugging CLI ↔ registry requests
---
# HTTP API
Base URL: `https://clawhub.ai` (default).
All v1 paths are under `/api/v1/...` and implemented by Convex HTTP routes (`convex/http.ts`).
Legacy `/api/...` and `/api/cli/...` remain for compatibility (see `DEPRECATIONS.md`).
OpenAPI: `/api/v1/openapi.json`.
## Rate limits
Enforced per IP + per API key:
- Read: 120/min per IP, 600/min per key
- Write: 30/min per IP, 120/min per key
Headers:
- `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` (when limited)
## Public endpoints (no auth)
### `GET /api/v1/search`
Query params:
- `q` (required): query string
- `limit` (optional): integer
- `highlightedOnly` (optional): `true` to filter to highlighted skills
Response:
```json
{ "results": [{ "score": 0.123, "slug": "gifgrep", "displayName": "GifGrep", "summary": "…", "version": "1.2.3", "updatedAt": 1730000000000 }] }
```
### `GET /api/v1/skills`
Query params:
- `limit` (optional): integer (1200)
- `cursor` (optional): pagination cursor (only for `sort=updated`)
- `sort` (optional): `updated` (default), `downloads`, `stars` (alias: `rating`), `installsCurrent` (alias: `installs`), `installsAllTime`, `trending`
Notes:
- `trending` ranks by installs in the last 7 days (telemetry-based).
Response:
```json
{ "items": [{ "slug": "gifgrep", "displayName": "GifGrep", "summary": "…", "tags": { "latest": "1.2.3" }, "stats": {}, "createdAt": 0, "updatedAt": 0, "latestVersion": { "version": "1.2.3", "createdAt": 0, "changelog": "…" } }], "nextCursor": null }
```
### `GET /api/v1/skills/{slug}`
Response:
```json
{ "skill": { "slug": "gifgrep", "displayName": "GifGrep", "summary": "…", "tags": { "latest": "1.2.3" }, "stats": {}, "createdAt": 0, "updatedAt": 0 }, "latestVersion": { "version": "1.2.3", "createdAt": 0, "changelog": "…" }, "owner": { "handle": "steipete", "displayName": "Peter", "image": null } }
```
### `GET /api/v1/skills/{slug}/versions`
Query params:
- `limit` (optional): integer
- `cursor` (optional): pagination cursor
### `GET /api/v1/skills/{slug}/versions/{version}`
Returns version metadata + files list.
### `GET /api/v1/skills/{slug}/file`
Returns raw text content.
Query params:
- `path` (required)
- `version` (optional)
- `tag` (optional)
Notes:
- Defaults to latest version.
- File size limit: 200KB.
### `GET /api/v1/resolve`
Used by the CLI to map a local fingerprint to a known version.
Query params:
- `slug` (required)
- `hash` (required): 64-char hex sha256 of the bundle fingerprint
Response:
```json
{ "slug": "gifgrep", "match": { "version": "1.2.2" }, "latestVersion": { "version": "1.2.3" } }
```
### `GET /api/v1/download`
Downloads a zip of a skill version.
Query params:
- `slug` (required)
- `version` (optional): semver string
- `tag` (optional): tag name (e.g. `latest`)
Notes:
- If neither `version` nor `tag` is provided, the latest version is used.
- Soft-deleted versions return `410`.
## Auth endpoints (Bearer token)
All endpoints require:
```
Authorization: Bearer clh_...
```
### `GET /api/v1/whoami`
Validates token and returns the user handle.
### `POST /api/v1/skills`
Publishes a new version.
- Preferred: `multipart/form-data` with `payload` JSON + `files[]` blobs.
- JSON body with `files` (storageId-based) is also accepted.
### `DELETE /api/v1/skills/{slug}` / `POST /api/v1/skills/{slug}/undelete`
Soft-delete / restore a skill (owner/admin only).
### `POST /api/v1/stars/{slug}` / `DELETE /api/v1/stars/{slug}`
Add/remove a star (highlights). Both endpoints are idempotent.
Responses:
```json
{ "ok": true, "starred": true, "alreadyStarred": false }
```
```json
{ "ok": true, "unstarred": true, "alreadyUnstarred": false }
```
## Legacy CLI endpoints (deprecated)
Still supported for older CLI versions:
- `GET /api/cli/whoami`
- `POST /api/cli/upload-url`
- `POST /api/cli/publish`
- `POST /api/cli/telemetry/sync`
- `POST /api/cli/skill/delete`
- `POST /api/cli/skill/undelete`
See `DEPRECATIONS.md` for removal plan.
## Registry discovery (`/.well-known/clawhub.json`)
The CLI can discover registry/auth settings from the site:
- `/.well-known/clawhub.json` (JSON, preferred)
- `/.well-known/clawdhub.json` (legacy)
Schema:
```json
{ "apiBase": "https://clawhub.ai", "authBase": "https://clawhub.ai", "minCliVersion": "0.0.5" }
```
If you self-host, serve this file (or set `CLAWHUB_REGISTRY` explicitly; legacy `CLAWDHUB_REGISTRY`).
+40 -19
View File
@@ -1,43 +1,64 @@
---
summary: 'Copy/paste CLI smoke checklist for local verification.'
read_when:
- Pre-merge validation
- Reproducing a reported CLI bug
---
# Manual testing (CLI)
## Setup
- Ensure logged in: `bun clawdhub whoami` (or `bun clawdhub login`).
- Ensure logged in: `bun clawhub whoami` (or `bun clawhub login`).
- Optional: set env
- `CLAWDHUB_SITE=https://clawdhub.com`
- `CLAWDHUB_REGISTRY=https://clawdhub.com`
- `CLAWHUB_SITE=https://clawhub.ai`
- `CLAWHUB_REGISTRY=https://clawhub.ai`
## Smoke
- `bun clawdhub --help`
- `bun clawdhub --cli-version`
- `bun clawdhub whoami`
- `bun clawhub --help`
- `bun clawhub --cli-version`
- `bun clawhub whoami`
## Search
- `bun clawdhub search gif --limit 5`
- `bun clawhub search gif --limit 5`
## Install / list / update
- `mkdir -p /tmp/clawdhub-manual && cd /tmp/clawdhub-manual`
- `bunx clawdhub@beta install gifgrep --force`
- `bunx clawdhub@beta list`
- `bunx clawdhub@beta update gifgrep --force`
- `mkdir -p /tmp/clawhub-manual && cd /tmp/clawhub-manual`
- `bunx clawhub@beta install gifgrep --force`
- `bunx clawhub@beta list`
- `bunx clawhub@beta update gifgrep --force`
## Publish (changelog optional)
- `mkdir -p /tmp/clawdhub-skill-demo/SKILL && cd /tmp/clawdhub-skill-demo`
- `mkdir -p /tmp/clawhub-skill-demo/SKILL && cd /tmp/clawhub-skill-demo`
- Create files:
- `SKILL.md`
- `notes.md`
- Publish:
- `bun clawdhub publish . --slug clawdhub-manual-<ts> --name "Manual <ts>" --version 1.0.0 --tags latest`
- `bun clawhub publish . --slug clawhub-manual-<ts> --name "Manual <ts>" --version 1.0.0 --tags latest`
- Publish update with empty changelog:
- `bun clawdhub publish . --slug clawdhub-manual-<ts> --name "Manual <ts>" --version 1.0.1 --tags latest`
- `bun clawhub publish . --slug clawhub-manual-<ts> --name "Manual <ts>" --version 1.0.1 --tags latest`
## Delete / undelete (owner/admin)
- `bun clawdhub delete clawdhub-manual-<ts> --yes`
- `bun clawhub delete clawhub-manual-<ts> --yes`
- Verify hidden:
- `curl -i "https://clawdhub.com/api/skill?slug=clawdhub-manual-<ts>"`
- `curl -i "https://clawhub.ai/api/v1/skills/clawhub-manual-<ts>"`
- Restore:
- `bun clawdhub undelete clawdhub-manual-<ts> --yes`
- `bun clawhub undelete clawhub-manual-<ts> --yes`
- Cleanup:
- `bun clawdhub delete clawdhub-manual-<ts> --yes`
- `bun clawhub delete clawhub-manual-<ts> --yes`
## Sync
- `bun clawdhub sync --dry-run --all`
- `bun clawhub sync --dry-run --all`
## Playwright (menu smoke)
Run against prod:
```
PLAYWRIGHT_BASE_URL=https://clawhub.ai bun run test:pw
```
Run against a local preview server:
```
bun run test:e2e:local
```
+43
View File
@@ -0,0 +1,43 @@
---
summary: 'Mintlify setup notes for publishing docs/.'
read_when:
- Setting up docs site
---
# Mintlify
Goal: publish `docs/` as a browsable docs site (nice UX for OSS users).
This repo does **not** include Mintlify config yet (`mint.json` missing).
## Minimal setup
1) Install Mintlify CLI (per Mintlify docs).
2) Add a `mint.json` at repo root that points to `docs/` pages.
Example (starter):
```json
{
"name": "OpenClaw",
"logo": "public/logo.svg",
"navigation": [
{ "group": "Start", "pages": ["docs/README", "docs/quickstart"] },
{ "group": "Concepts", "pages": ["docs/architecture", "docs/skill-format", "docs/telemetry"] },
{ "group": "Reference", "pages": ["docs/cli", "docs/http-api", "docs/auth", "docs/deploy"] }
]
}
```
Notes:
- Mintlify usually wants page paths without extension; keep files as `.md`.
- If you prefer Mintlify conventions, rename to `.mdx` later (optional).
## Recommended “docs UX” additions
- Add an “Overview” page (use `docs/README.md`).
- Keep “Quickstart” copy/paste friendly.
- Provide CLI + HTTP API reference pages (done here).
- Add a Troubleshooting page for common setup failures.
+120
View File
@@ -0,0 +1,120 @@
---
summary: 'Local setup + CLI smoke: login, search, install, publish, sync.'
read_when:
- First run / local dev setup
- Verifying end-to-end flows
---
# Quickstart
## 0) Prereqs
- Bun
- Convex CLI (`bunx convex ...`)
- GitHub OAuth App (for login)
- OpenAI key (for embeddings/search)
## 1) Local dev (web + Convex)
```bash
bun install
cp .env.local.example .env.local
# terminal A
bun run dev
# terminal B
bunx convex dev
```
## 2) Auth setup (GitHub OAuth + Convex Auth keys)
Fill in `.env.local`:
- `AUTH_GITHUB_ID`
- `AUTH_GITHUB_SECRET`
- `VITE_CONVEX_URL`
- `VITE_CONVEX_SITE_URL`
- `CONVEX_SITE_URL` (same as `VITE_CONVEX_SITE_URL`)
- `OPENAI_API_KEY`
Generate Convex Auth keys for your deployment:
```bash
bunx auth --deployment-name <deployment> --web-server-url http://localhost:3000
```
Then paste the printed `JWT_PRIVATE_KEY` + `JWKS` into `.env.local` (and ensure the deployment got them too).
## 3) CLI: login + basic commands
From this repo:
```bash
bun clawhub --help
bun clawhub login
bun clawhub whoami
bun clawhub search gif --limit 5
```
Install a skill into `./skills/<slug>` (if Clawdbot is configured, installs into that workspace instead):
```bash
bun clawhub install <slug>
bun clawhub list
```
You can also install into any folder:
```bash
bun clawhub install <slug> --workdir /tmp/clawhub-demo --dir skills
```
Update:
```bash
bun clawhub update --all
```
## 4) Publish a skill
Create a folder containing `SKILL.md` (required) plus any supporting text files:
```bash
mkdir -p /tmp/clawhub-skill-demo && cd /tmp/clawhub-skill-demo
cat > SKILL.md <<'EOF'
---
name: Demo Skill
description: Demo skill for local testing
---
# Demo Skill
Hello.
EOF
```
Publish:
```bash
bun clawhub publish . \
--slug clawhub-demo-$(date +%s) \
--name "Demo $(date +%s)" \
--version 1.0.0 \
--tags latest \
--changelog "Initial release"
```
## 5) Sync local skills (auto-publish new/changed)
`sync` scans for local skill folders and publishes the ones that arent “synced” yet.
```bash
bun clawhub sync
```
Dry run + non-interactive:
```bash
bun clawhub sync --all --dry-run --no-input
```
+58
View File
@@ -0,0 +1,58 @@
---
summary: 'Skill folder format, required files, allowed file types, limits.'
read_when:
- Publishing skills
- Debugging publish/sync failures
---
# Skill format
## On disk
A skill is a folder.
Required:
- `SKILL.md` (or `skill.md`)
Optional:
- any supporting *text-based* files (see “Allowed files”)
- `.clawhubignore` (ignore patterns for publish/sync, legacy `.clawdhubignore`)
- `.gitignore` (also honored)
Local install metadata (written by the CLI):
- `<skill>/.clawhub/origin.json` (legacy `.clawdhub`)
Workdir install state (written by the CLI):
- `<workdir>/.clawhub/lock.json` (legacy `.clawdhub`)
## `SKILL.md`
- Markdown with optional YAML frontmatter.
- The server extracts metadata from frontmatter during publish.
- `description` is used as the skill summary in the UI/search.
## Allowed files
Only “text-based” files are accepted by publish.
- Extension allowlist is in `packages/schema/src/textFiles.ts` (`TEXT_FILE_EXTENSIONS`).
- Content types starting with `text/` are treated as text; plus a small allowlist (JSON/YAML/TOML/JS/TS/Markdown/SVG).
Limits (server-side):
- Total bundle size: 50MB.
- Embedding text includes `SKILL.md` + up to ~40 non-`.md` files (best-effort cap).
## Slugs
- Derived from folder name by default.
- Must be lowercase and URL-safe: `^[a-z0-9][a-z0-9-]*$`.
## Versioning + tags
- Each publish creates a new version (semver).
- Tags are string pointers to a version; `latest` is commonly used.
+37
View File
@@ -0,0 +1,37 @@
---
summary: 'Soul bundle format, required files, limits.'
read_when:
- Publishing souls
- Debugging soul publish failures
---
# Soul format
## On disk
A soul is a single file:
- `SOUL.md` (or `soul.md`)
For now, onlycrabs.ai rejects any extra files.
## `SOUL.md`
- Markdown with optional YAML frontmatter.
- The server extracts metadata from frontmatter during publish.
- `description` is used as the soul summary in the UI/search.
## Limits
- Total bundle size: 50MB.
- Embedding text includes `SOUL.md` only.
## Slugs
- Derived from folder name by default.
- Must be lowercase and URL-safe: `^[a-z0-9][a-z0-9-]*$`.
## Versioning + tags
- Each publish creates a new version (semver).
- Tags are string pointers to a version; `latest` is commonly used.
+60 -13
View File
@@ -1,17 +1,18 @@
---
summary: "ClawdHub spec: skills registry, versioning, vector search, moderation"
summary: "OpenClaw spec: skills registry, versioning, vector search, moderation"
read_when:
- Bootstrapping ClawdHub
- Bootstrapping OpenClaw
- Implementing schema/auth/search/versioning
- Reviewing API and upload/download flows
---
# ClawdHub — product + implementation spec (v1)
# OpenClaw — product + implementation spec (v1)
## Goals
- onlycrabs.ai mode for sharing `SOUL.md` bundles (host-based entry point).
- Minimal, fast SPA for browsing and publishing agent skills.
- Skills stored in Convex (files + metadata + versions + stats).
- GitHub OAuth login; optional GitHub App repo sync later.
- GitHub OAuth login; GitHub App backs up skills to `clawdbot/skills`.
- Vector-based search over skill text + metadata.
- Versioning, tags (`latest` + user tags), changelog, rollback (tag movement).
- Public read access; upload requires auth.
@@ -19,7 +20,7 @@ read_when:
## Non-goals (v1)
- Paid features, private skills, or binary assets.
- GitHub App sync (future phase).
- GitHub App sync beyond backups (future phase).
## Core objects
@@ -28,7 +29,7 @@ read_when:
- `handle` (GitHub login)
- `name`, `bio`
- `avatarUrl` (GitHub, fallback gravatar)
- `role`: `admin | moderator | user`
- `role`: `admin | moderator | user` (moderators can soft-delete and flag; admins can hard-delete + change owners)
- `createdAt`, `updatedAt`
### Skill
@@ -39,9 +40,14 @@ read_when:
- `latestVersionId`
- `latestTagVersionId` (for `latest` tag)
- `tags` map: `{ tag -> versionId }`
- `badges`: `{ redactionApproved?: { byUserId, at } }`
- `badges`: `{ redactionApproved?: { byUserId, at }, highlighted?: { byUserId, at }, official?: { byUserId, at }, deprecated?: { byUserId, at } }`
- `official` marks admin-verified/official skills.
- `deprecated` marks skills that should not be used for new integrations.
- `moderationStatus`: `active | hidden | removed`
- `moderationFlags`: `string[]` (automatic detection)
- `moderationNotes`, `moderationReason`
- `hiddenAt`, `hiddenBy`, `lastReviewedAt`, `reportCount`
- `stats`: `{ downloads, stars, versions, comments }`
- `status`: `active` only (soft-delete on version/comment only)
- `createdAt`, `updatedAt`
### SkillVersion
@@ -60,7 +66,44 @@ read_when:
From SKILL.md frontmatter + AgentSkills + Clawdis extensions:
- `name`, `description`, `homepage`, `website`, `url`, `emoji`
- `metadata.clawdis`: `always`, `skillKey`, `primaryEnv`, `emoji`, `homepage`, `os`,
`requires` (`bins`, `anyBins`, `env`, `config`), `install[]`
`requires` (`bins`, `anyBins`, `env`, `config`), `install[]`, `nix` (`plugin`, `systems`),
`config` (`requiredEnv`, `stateDirs`, `example`), `cliHelp` (string; `cli --help` output)
- `metadata.clawdbot`: alias of `metadata.clawdis` (preferred for nix-clawdbot plugin pointers)
- Nix plugins are different from regular skills; they bundle the skill pack, the CLI binary, and config flags/requirements together.
- `metadata` in frontmatter is YAML (object) preferred; legacy JSON-string accepted.
### Soul
- `slug` (unique)
- `displayName`
- `ownerUserId`
- `summary` (from SOUL.md frontmatter `description`)
- `latestVersionId`
- `tags` map: `{ tag -> versionId }`
- `stats`: `{ downloads, stars, versions, comments }`
- `status`: `active` only (soft-delete on version/comment only)
- `createdAt`, `updatedAt`
### SoulVersion
- `soulId`
- `version` (semver string)
- `tag` (string, optional; `latest` always maintained separately)
- `changelog` (required)
- `files`: list of file metadata (SOUL.md only)
- `path`, `size`, `storageId`, `sha256`
- `parsed` (metadata extracted from SOUL.md)
- `vectorDocId` (if using RAG component) OR `embeddingId`
- `createdBy`, `createdAt`
- `softDeletedAt` (nullable)
### SoulComment
- `soulId`, `userId`, `body`
- `softDeletedAt`, `deletedBy`
- `createdAt`
### SoulStar
- `soulId`, `userId`, `createdAt`
### Comment
- `skillId`, `userId`, `body`
@@ -80,7 +123,8 @@ From SKILL.md frontmatter + AgentSkills + Clawdis extensions:
## Auth + roles
- Convex Auth with GitHub OAuth App.
- Default role `user`; bootstrap `steipete` to `admin` on first login.
- Admin UI to promote/demote roles; all changes logged.
- Management console: moderators can hide/restore skills + mark duplicates; admins can change owners, approve badges, and hard-delete.
- Role changes are admin-only and audited.
## Upload flow (50MB per version)
1) Client requests upload session.
@@ -93,6 +137,9 @@ From SKILL.md frontmatter + AgentSkills + Clawdis extensions:
- version uniqueness
5) Server stores files + metadata, sets `latest` tag, updates stats.
Soul upload flow: same as skills, but only `SOUL.md` is allowed in the bundle.
Seed data lives in `convex/seed.ts` for local dev.
## Versioning + tags
- Each upload is a new `SkillVersion`.
- `latest` tag always points to most recent version unless user re-tags.
@@ -100,7 +147,7 @@ From SKILL.md frontmatter + AgentSkills + Clawdis extensions:
- Changelog is optional.
## Search
- Vector search over: SKILL.md + other text files + metadata summary.
- Vector search over: SKILL.md + other text files + metadata summary (souls index SOUL.md).
- Convex embeddings + vector index.
- Filters: tag, owner, `redactionApproved` only, min stars, updatedAt.
@@ -110,7 +157,7 @@ From SKILL.md frontmatter + AgentSkills + Clawdis extensions:
- Soft-delete versions; downloads remain for non-deleted versions only.
## UI (SPA)
- Home: search + filters + trending/featured + “Highlighted” batch.
- Home: search + filters + trending/featured + “Highlighted” badge.
- Skill detail: README render, files list, version history, tags, stats, badges.
- Upload/edit: file picker + version + tag + changelog.
- Account settings: name + delete account (soft delete).
@@ -121,7 +168,7 @@ From SKILL.md frontmatter + AgentSkills + Clawdis extensions:
- Lint: Biome + Oxlint (type-aware).
## Vercel
- Env vars: Convex deployment URLs + GitHub OAuth client + OpenAI key (if used).
- Env vars: Convex deployment URLs + GitHub OAuth client + OpenAI key (if used) + GitHub App backup credentials.
- SPA feel: client-side transitions, prefetching, optimistic UI.
## Open questions (carry forward)
+91
View File
@@ -0,0 +1,91 @@
---
summary: 'Install telemetry collected via `clawhub sync` + opt-out.'
read_when:
- Working on telemetry / privacy controls
- Questions about what data is collected
---
# Telemetry
OpenClaw uses **minimal telemetry** to compute **install counts** (whats actually in use) and to power better sorting/filtering.
This is based on the CLI `clawhub sync` command.
## When telemetry is collected
Telemetry is only sent when:
- You are **logged in** in the CLI (we already require auth for sync/publish flows).
- You run `clawhub sync`.
- Telemetry is **not disabled** (see “How to disable” below).
If you are not logged in, nothing is reported.
## What we collect
On each `clawhub sync`, the CLI reports a **full snapshot** of what it found, grouped by scan root (“folder/root”).
For each root we store:
- `rootId`: a **SHA-256 hash** of the canonical root path (server never sees the raw path).
- `label`: a human-readable label derived from the last two path segments (home paths are shown with `~`).
- `firstSeenAt`, `lastSeenAt`, optional `expiredAt`.
For each skill found under a root we store:
- `skillId` (resolved by slug; only skills that exist in the registry are tracked).
- `firstSeenAt`, `lastSeenAt`.
- `lastVersion` (best-effort; currently the registry-matched version if known).
- optional `removedAt` when a previously-reported install disappears from a root.
### What we do *not* collect
- No raw absolute folder paths (only hashed `rootId` + a short display label).
- No file contents.
- No per-run logs, prompts, or other CLI output.
- No tracking for skills that arent uploaded to the registry (unknown slugs are ignored).
## Install counts
We maintain two counters per skill:
- `installsCurrent`: unique users who currently have the skill installed in at least one active root.
- `installsAllTime`: unique users who have ever reported the skill installed.
### Multiple roots
If you sync from multiple folders, we treat each scan root independently. A skill is “currently installed” if it exists in **any** active root.
### Uninstall detection
Because `sync` reports the full set per root:
- If a skill disappears from a root on the next sync, we mark it removed for that root.
- If the skill is removed from all of your roots, it no longer counts toward `installsCurrent`.
- `installsAllTime` never decreases unless you delete telemetry (see below).
### Staleness (120 days)
Roots that dont report telemetry for **120 days** are marked stale and their installs stop counting toward `installsCurrent`.
This is evaluated lazily (on the next telemetry report) to avoid background jobs.
## Transparency + user controls
OpenClaw provides a private “Installed” tab on your own profile:
- Shows the exact roots + installed skills we store.
- Includes a **JSON export** view.
- Includes a **Delete telemetry** action to remove all stored telemetry for your account.
Everyone else only sees **aggregated install counters**; no one else can see your roots/folders.
Deleting your account also deletes your telemetry data.
## How to disable telemetry
Set the environment variable:
```bash
export CLAWHUB_DISABLE_TELEMETRY=1
```
With this set, the CLI will not send telemetry during `clawhub sync`.
+49
View File
@@ -0,0 +1,49 @@
---
summary: 'Common setup/runtime issues (CLI + backend) and fixes.'
read_when:
- Something is broken and you need a fix-fast checklist
---
# Troubleshooting
## `clawhub login` opens browser but never completes
- Ensure your browser can reach `http://127.0.0.1:<port>/callback` (local firewalls/VPNs can interfere).
- Use headless mode:
- create a token in the web UI (Settings → API tokens)
- `clawhub login --token clh_...`
## `whoami` / `publish` returns `Unauthorized` (401)
- Token missing or revoked: check your config file (`CLAWHUB_CONFIG_PATH` override?).
- Ensure requests include `Authorization: Bearer ...` (CLI does this automatically).
## `publish` fails with `OPENAI_API_KEY is not configured`
- Set `OPENAI_API_KEY` in the Convex environment (not only locally).
- Re-run `bunx convex dev` / `bunx convex deploy` after setting env.
## `sync` says “No skills found”
- `sync` looks for folders containing `SKILL.md` (or `skill.md`).
- It scans:
- workdir first
- then fallback roots (legacy `~/clawdis/skills`, `~/clawdbot/skills`, etc.)
- Provide explicit roots:
```bash
clawhub sync --root /path/to/skills
```
## `update` refuses due to “local changes (no match)”
- Your local files dont match any published fingerprint.
- Options:
- keep local edits; skip updating
- overwrite: `clawhub update <slug> --force`
- publish as fork: copy to new folder/slug then `clawhub publish ... --fork-of upstream@version`
## `GET /api/*` works locally but not on Vercel
- Check `vercel.json` rewrite destination points at your Convex site URL.
- Ensure `VITE_CONVEX_SITE_URL` and `CONVEX_SITE_URL` match your deployment.
+51
View File
@@ -0,0 +1,51 @@
---
summary: 'Discord webhook events/payloads for skill publish + highlight.'
read_when:
- Working on webhooks/integrations
---
# Webhooks (Discord)
OpenClaw can post Discord embeds when skills are published or highlighted.
## Setup
Set the webhook URL in the Convex environment:
- `DISCORD_WEBHOOK_URL` (required): Discord webhook URL.
- `DISCORD_WEBHOOK_HIGHLIGHTED_ONLY` (optional): `true` to only send for highlighted skills.
- `SITE_URL` (optional): Base site URL for links (default `https://clawhub.ai`).
## Events
- `skill.publish`: fires on every publish (new or updated version).
- `skill.highlighted`: fires when a skill is newly highlighted.
### Highlight-only filter
When `DISCORD_WEBHOOK_HIGHLIGHTED_ONLY=true`:
- `skill.publish` only sends if the skill is highlighted.
- `skill.highlighted` always sends.
## Payload (Discord)
Discord receives a JSON payload with a single embed:
```json
{
"embeds": [
{
"title": "Demo Skill",
"description": "Nice skill",
"url": "https://clawhub.ai/owner/demo-skill",
"fields": [
{ "name": "Version", "value": "v1.2.3", "inline": true },
{ "name": "Owner", "value": "@owner", "inline": true },
{ "name": "Tags", "value": "latest, discord", "inline": false }
],
"footer": { "text": "OpenClaw" }
}
]
}
```
+154 -58
View File
@@ -5,23 +5,51 @@ import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import {
ApiCliWhoamiResponseSchema,
ApiRoutes,
ApiSearchResponseSchema,
ApiV1SearchResponseSchema,
ApiV1WhoamiResponseSchema,
parseArk,
} from 'clawdhub-schema'
} from 'clawhub-schema'
import { unzipSync } from 'fflate'
import { Agent, setGlobalDispatcher } from 'undici'
import { describe, expect, it } from 'vitest'
import { readGlobalConfig } from '../packages/clawdhub/src/config'
const REQUEST_TIMEOUT_MS = 15_000
try {
setGlobalDispatcher(
new Agent({
allowH2: true,
connect: { timeout: REQUEST_TIMEOUT_MS },
}),
)
} catch {
// ignore dispatcher setup failures
}
function mustGetToken() {
const fromEnv = process.env.CLAWDHUB_E2E_TOKEN?.trim()
const fromEnv = process.env.CLAWHUB_E2E_TOKEN?.trim() || process.env.CLAWDHUB_E2E_TOKEN?.trim()
if (fromEnv) return fromEnv
return null
}
function getRegistry() {
return (
process.env.CLAWHUB_REGISTRY?.trim() ||
process.env.CLAWDHUB_REGISTRY?.trim() ||
'https://clawhub.ai'
)
}
function getSite() {
return (
process.env.CLAWHUB_SITE?.trim() || process.env.CLAWDHUB_SITE?.trim() || 'https://clawhub.ai'
)
}
async function makeTempConfig(registry: string, token: string | null) {
const dir = await mkdtemp(join(tmpdir(), 'clawdhub-e2e-'))
const dir = await mkdtemp(join(tmpdir(), 'clawhub-e2e-'))
const path = join(dir, 'config.json')
await writeFile(
path,
@@ -31,9 +59,19 @@ async function makeTempConfig(registry: string, token: string | null) {
return { dir, path }
}
describe('clawdhub e2e', () => {
async function fetchWithTimeout(input: RequestInfo | URL, init?: RequestInit) {
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort('Timeout'), REQUEST_TIMEOUT_MS)
try {
return await fetch(input, { ...init, signal: controller.signal })
} finally {
clearTimeout(timeout)
}
}
describe('clawhub e2e', () => {
it('prints CLI version via --cli-version', async () => {
const result = spawnSync('bun', ['clawdhub', '--cli-version'], {
const result = spawnSync('bun', ['clawhub', '--cli-version'], {
cwd: process.cwd(),
encoding: 'utf8',
})
@@ -42,30 +80,32 @@ describe('clawdhub e2e', () => {
})
it('search endpoint returns a results array (schema parse)', async () => {
const registry = process.env.CLAWDHUB_REGISTRY?.trim() || 'https://clawdhub.com'
const registry = getRegistry()
const url = new URL(ApiRoutes.search, registry)
url.searchParams.set('q', 'gif')
url.searchParams.set('limit', '5')
const response = await fetch(url.toString(), { headers: { Accept: 'application/json' } })
const response = await fetchWithTimeout(url.toString(), {
headers: { Accept: 'application/json' },
})
expect(response.ok).toBe(true)
const json = (await response.json()) as unknown
const parsed = parseArk(ApiSearchResponseSchema, json, 'API response')
const parsed = parseArk(ApiV1SearchResponseSchema, json, 'API response')
expect(Array.isArray(parsed.results)).toBe(true)
})
it('cli search does not error on multi-result responses', async () => {
const registry = process.env.CLAWDHUB_REGISTRY?.trim() || 'https://clawdhub.com'
const site = process.env.CLAWDHUB_SITE?.trim() || 'https://clawdhub.com'
const registry = getRegistry()
const site = getSite()
const token = mustGetToken() ?? (await readGlobalConfig())?.token ?? null
const cfg = await makeTempConfig(registry, token)
try {
const workdir = await mkdtemp(join(tmpdir(), 'clawdhub-e2e-workdir-'))
const workdir = await mkdtemp(join(tmpdir(), 'clawhub-e2e-workdir-'))
const result = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'search',
'gif',
'--limit',
@@ -79,7 +119,7 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
@@ -93,22 +133,22 @@ describe('clawdhub e2e', () => {
})
it('assumes a logged-in user (whoami succeeds)', async () => {
const registry = process.env.CLAWDHUB_REGISTRY?.trim() || 'https://clawdhub.com'
const site = process.env.CLAWDHUB_SITE?.trim() || 'https://clawdhub.com'
const registry = getRegistry()
const site = getSite()
const token = mustGetToken() ?? (await readGlobalConfig())?.token ?? null
if (!token) {
throw new Error('Missing token. Set CLAWDHUB_E2E_TOKEN or run: bun clawdhub auth login')
throw new Error('Missing token. Set CLAWHUB_E2E_TOKEN or run: bun clawhub auth login')
}
const cfg = await makeTempConfig(registry, token)
try {
const whoamiUrl = new URL(ApiRoutes.cliWhoami, registry)
const whoamiRes = await fetch(whoamiUrl.toString(), {
const whoamiUrl = new URL(ApiRoutes.whoami, registry)
const whoamiRes = await fetchWithTimeout(whoamiUrl.toString(), {
headers: { Accept: 'application/json', Authorization: `Bearer ${token}` },
})
expect(whoamiRes.ok).toBe(true)
const whoami = parseArk(
ApiCliWhoamiResponseSchema,
ApiV1WhoamiResponseSchema,
(await whoamiRes.json()) as unknown,
'Whoami',
)
@@ -116,10 +156,10 @@ describe('clawdhub e2e', () => {
const result = spawnSync(
'bun',
['clawdhub', 'whoami', '--site', site, '--registry', registry],
['clawhub', 'whoami', '--site', site, '--registry', registry],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
@@ -131,15 +171,15 @@ describe('clawdhub e2e', () => {
})
it('sync dry-run finds skills from an explicit root', async () => {
const registry = process.env.CLAWDHUB_REGISTRY?.trim() || 'https://clawdhub.com'
const site = process.env.CLAWDHUB_SITE?.trim() || 'https://clawdhub.com'
const registry = getRegistry()
const site = getSite()
const token = mustGetToken() ?? (await readGlobalConfig())?.token ?? null
if (!token) {
throw new Error('Missing token. Set CLAWDHUB_E2E_TOKEN or run: bun clawdhub auth login')
throw new Error('Missing token. Set CLAWHUB_E2E_TOKEN or run: bun clawhub auth login')
}
const cfg = await makeTempConfig(registry, token)
const root = await mkdtemp(join(tmpdir(), 'clawdhub-e2e-sync-'))
const root = await mkdtemp(join(tmpdir(), 'clawhub-e2e-sync-'))
try {
const skillDir = join(root, 'cool-skill')
await mkdir(skillDir, { recursive: true })
@@ -148,7 +188,7 @@ describe('clawdhub e2e', () => {
const result = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'sync',
'--dry-run',
'--all',
@@ -161,7 +201,7 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
@@ -174,17 +214,72 @@ describe('clawdhub e2e', () => {
}
})
it('publishes, deletes, and undeletes a skill (logged-in)', async () => {
const registry = process.env.CLAWDHUB_REGISTRY?.trim() || 'https://clawdhub.com'
const site = process.env.CLAWDHUB_SITE?.trim() || 'https://clawdhub.com'
it('sync dry-run finds skills from clawdbot.json roots', async () => {
const registry = getRegistry()
const site = getSite()
const token = mustGetToken() ?? (await readGlobalConfig())?.token ?? null
if (!token) {
throw new Error('Missing token. Set CLAWDHUB_E2E_TOKEN or run: bun clawdhub auth login')
throw new Error('Missing token. Set CLAWHUB_E2E_TOKEN or run: bun clawhub auth login')
}
const cfg = await makeTempConfig(registry, token)
const workdir = await mkdtemp(join(tmpdir(), 'clawdhub-e2e-publish-'))
const installWorkdir = await mkdtemp(join(tmpdir(), 'clawdhub-e2e-install-'))
const root = await mkdtemp(join(tmpdir(), 'clawhub-e2e-clawdbot-'))
const stateDir = join(root, 'state')
const configPath = join(root, 'clawdbot.json')
const workspace = join(root, 'clawd-work')
const skillsRoot = join(workspace, 'skills')
const skillDir = join(skillsRoot, 'auto-skill')
try {
await mkdir(skillDir, { recursive: true })
await writeFile(join(skillDir, 'SKILL.md'), '# Skill\n', 'utf8')
const config = `{
// JSON5-style comments + trailing commas
routing: {
agents: {
work: { name: 'Work', workspace: '${workspace}', },
},
},
}`
await writeFile(configPath, config, 'utf8')
const result = spawnSync(
'bun',
['clawhub', 'sync', '--dry-run', '--all', '--site', site, '--registry', registry],
{
cwd: process.cwd(),
env: {
...process.env,
CLAWHUB_CONFIG_PATH: cfg.path,
CLAWHUB_DISABLE_TELEMETRY: '1',
CLAWDBOT_CONFIG_PATH: configPath,
CLAWDBOT_STATE_DIR: stateDir,
},
encoding: 'utf8',
},
)
expect(result.status).toBe(0)
expect(result.stderr).not.toMatch(/error:/i)
expect(result.stdout).toMatch(/Dry run/i)
expect(result.stdout).toMatch(/auto-skill/i)
} finally {
await rm(root, { recursive: true, force: true })
await rm(cfg.dir, { recursive: true, force: true })
}
})
it('publishes, deletes, and undeletes a skill (logged-in)', async () => {
const registry = getRegistry()
const site = getSite()
const token = mustGetToken() ?? (await readGlobalConfig())?.token ?? null
if (!token) {
throw new Error('Missing token. Set CLAWHUB_E2E_TOKEN or run: bun clawhub auth login')
}
const cfg = await makeTempConfig(registry, token)
const workdir = await mkdtemp(join(tmpdir(), 'clawhub-e2e-publish-'))
const installWorkdir = await mkdtemp(join(tmpdir(), 'clawhub-e2e-install-'))
const slug = `e2e-${Date.now()}`
const skillDir = join(workdir, slug)
@@ -195,7 +290,7 @@ describe('clawdhub e2e', () => {
const publish1 = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'publish',
skillDir,
'--slug',
@@ -215,7 +310,7 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
@@ -225,7 +320,7 @@ describe('clawdhub e2e', () => {
const publish2 = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'publish',
skillDir,
'--slug',
@@ -245,7 +340,7 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
@@ -255,7 +350,7 @@ describe('clawdhub e2e', () => {
const downloadUrl = new URL(ApiRoutes.download, registry)
downloadUrl.searchParams.set('slug', slug)
downloadUrl.searchParams.set('version', '1.0.1')
const zipRes = await fetch(downloadUrl.toString())
const zipRes = await fetchWithTimeout(downloadUrl.toString())
expect(zipRes.ok).toBe(true)
const zipBytes = new Uint8Array(await zipRes.arrayBuffer())
const unzipped = unzipSync(zipBytes)
@@ -264,7 +359,7 @@ describe('clawdhub e2e', () => {
const install = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'install',
slug,
'--version',
@@ -279,7 +374,7 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
@@ -287,10 +382,10 @@ describe('clawdhub e2e', () => {
const list = spawnSync(
'bun',
['clawdhub', 'list', '--site', site, '--registry', registry, '--workdir', installWorkdir],
['clawhub', 'list', '--site', site, '--registry', registry, '--workdir', installWorkdir],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
@@ -300,7 +395,7 @@ describe('clawdhub e2e', () => {
const update = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'update',
slug,
'--force',
@@ -313,21 +408,22 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
expect(update.status).toBe(0)
const metaUrl = new URL(ApiRoutes.skill, registry)
metaUrl.searchParams.set('slug', slug)
const metaRes = await fetch(metaUrl.toString(), { headers: { Accept: 'application/json' } })
const metaUrl = new URL(`${ApiRoutes.skills}/${slug}`, registry)
const metaRes = await fetchWithTimeout(metaUrl.toString(), {
headers: { Accept: 'application/json' },
})
expect(metaRes.status).toBe(200)
const del = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'delete',
slug,
'--yes',
@@ -340,24 +436,24 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
expect(del.status).toBe(0)
const metaAfterDelete = await fetch(metaUrl.toString(), {
const metaAfterDelete = await fetchWithTimeout(metaUrl.toString(), {
headers: { Accept: 'application/json' },
})
expect(metaAfterDelete.status).toBe(404)
const downloadAfterDelete = await fetch(downloadUrl.toString())
const downloadAfterDelete = await fetchWithTimeout(downloadUrl.toString())
expect(downloadAfterDelete.status).toBe(404)
const undelete = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'undelete',
slug,
'--yes',
@@ -370,13 +466,13 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
expect(undelete.status).toBe(0)
const metaAfterUndelete = await fetch(metaUrl.toString(), {
const metaAfterUndelete = await fetchWithTimeout(metaUrl.toString(), {
headers: { Accept: 'application/json' },
})
expect(metaAfterUndelete.status).toBe(200)
@@ -384,7 +480,7 @@ describe('clawdhub e2e', () => {
const cleanup = spawnSync(
'bun',
[
'clawdhub',
'clawhub',
'delete',
slug,
'--yes',
@@ -397,7 +493,7 @@ describe('clawdhub e2e', () => {
],
{
cwd: process.cwd(),
env: { ...process.env, CLAWDHUB_CONFIG_PATH: cfg.path },
env: { ...process.env, CLAWHUB_CONFIG_PATH: cfg.path, CLAWHUB_DISABLE_TELEMETRY: '1' },
encoding: 'utf8',
},
)
+49
View File
@@ -0,0 +1,49 @@
import { expect, test } from '@playwright/test'
const navLabels = ['Skills', 'Upload', 'Import', 'Search']
test('skills loads without error', async ({ page }) => {
await page.goto('/skills', { waitUntil: 'domcontentloaded' })
await expect(page.locator('text=Something went wrong!')).toHaveCount(0)
await expect(page.locator('h1', { hasText: 'Skills' })).toBeVisible()
})
test('souls loads without error', async ({ page }) => {
await page.goto('/souls', { waitUntil: 'domcontentloaded' })
await expect(page.locator('text=Something went wrong!')).toHaveCount(0)
await expect(page.locator('h1', { hasText: 'Souls' })).toBeVisible()
})
test('header menu routes render', async ({ page }) => {
await page.goto('/', { waitUntil: 'domcontentloaded' })
for (const label of navLabels) {
const link = page.getByRole('link', { name: label }).first()
await expect(link).toBeVisible()
await link.click()
if (label === 'Skills') {
await expect(page).toHaveURL(/\/skills/)
await expect(page.locator('h1', { hasText: 'Skills' })).toBeVisible()
}
if (label === 'Upload') {
await expect(page).toHaveURL(/\/upload/)
const heading = page.locator('h1.section-title', { hasText: /^Publish a /i })
const signInCard = page.locator('text=Sign in to upload')
await expect(heading.or(signInCard)).toBeVisible()
}
if (label === 'Import') {
await expect(page).toHaveURL(/\/import/)
const heading = page.getByRole('heading', { name: 'Import from GitHub' })
const signInCard = page.locator('text=Sign in to import and publish skills.')
await expect(heading.or(signInCard)).toBeVisible()
}
if (label === 'Search') {
await expect(page).toHaveURL(/\/?(\?|$)/)
await expect(page.locator('h1', { hasText: 'OpenClaw' })).toBeVisible()
}
}
})
+97
View File
@@ -0,0 +1,97 @@
import { expect, test } from '@playwright/test'
test('skills search paginates exact results', async ({ page }) => {
await page.addInitScript(() => {
const makeSearchResults = (count: number) =>
Array.from({ length: count }, (_, index) => ({
score: 0.9,
skill: {
_id: `skill_${index}`,
slug: `skill-${index}`,
displayName: `Skill ${index}`,
summary: `Summary ${index}`,
tags: {},
stats: {
downloads: 0,
installsCurrent: 0,
installsAllTime: 0,
stars: 0,
versions: 1,
comments: 0,
},
createdAt: 0,
updatedAt: 0,
},
version: null,
}))
class MockWebSocket {
url: string
readyState = 0
onopen?: () => void
onmessage?: (event: { data: string }) => void
onclose?: (event: { code: number; reason: string }) => void
onerror?: () => void
constructor(url: string) {
this.url = url
window.setTimeout(() => {
this.readyState = 1
this.onopen?.()
}, 0)
}
send(data: string) {
try {
const message = JSON.parse(data) as {
type?: string
requestId?: number
udfPath?: string
args?: Array<Record<string, unknown>>
}
if (message.type === 'Action' && message.udfPath?.includes('searchSkills')) {
const [args] = message.args ?? []
const limit = typeof args?.limit === 'number' ? args.limit : 10
const limits = (window as typeof window & { __searchLimits: number[] }).__searchLimits
limits.push(limit)
const response = {
type: 'ActionResponse',
requestId: message.requestId,
success: true,
result: makeSearchResults(limit),
logLines: [],
}
window.setTimeout(() => {
this.onmessage?.({ data: JSON.stringify(response) })
}, 0)
}
} catch {
this.onerror?.()
}
}
close(code = 1000, reason = 'closed') {
this.readyState = 3
this.onclose?.({ code, reason })
}
}
;(window as typeof window & { __searchLimits: number[] }).__searchLimits = []
window.WebSocket = MockWebSocket as unknown as typeof WebSocket
})
await page.goto('/skills', { waitUntil: 'domcontentloaded' })
await expect(page.getByRole('heading', { name: 'Skills' })).toBeVisible()
const input = page.getByPlaceholder('Filter by name, slug, or summary…')
await input.fill('remind')
await expect(page.getByText('Skill 0')).toBeVisible()
await expect(page.getByText('Scroll to load more')).toBeVisible()
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight))
await expect(page.getByText('Skill 75')).toBeVisible()
const limits = await page.evaluate(
() => (window as typeof window & { __searchLimits: number[] }).__searchLimits,
)
expect(limits).toEqual([50, 100])
})
+32 -18
View File
@@ -1,17 +1,22 @@
{
"name": "clawdhub",
"name": "clawhub",
"private": true,
"type": "module",
"workspaces": [
"packages/*"
],
"scripts": {
"preinstall": "bunx only-allow bun",
"dev": "bun --bun vite dev --port 3000",
"build": "bun --bun vite build",
"preview": "bun --bun vite preview",
"docs:list": "bun scripts/docs-list.ts",
"check:peers": "bun scripts/check-peer-deps.ts",
"test": "vitest run",
"test:watch": "vitest",
"test:e2e": "vitest run -c vitest.e2e.config.ts",
"test:e2e:local": "bash scripts/run-playwright-local.sh",
"test:pw": "playwright test",
"coverage": "vitest run --coverage",
"convex:deploy": "bunx convex deploy --typecheck=disable --yes",
"lint": "bun run lint:biome && bun run lint:oxlint",
@@ -20,24 +25,30 @@
"format": "biome format --write ."
},
"dependencies": {
"@auth/core": "^0.41.1",
"clawdhub-schema": "^0.0.2",
"@auth/core": "^0.37.4",
"@convex-dev/auth": "^0.0.90",
"@fontsource/bricolage-grotesque": "^5.2.10",
"@fontsource/ibm-plex-mono": "^5.2.7",
"@fontsource/manrope": "^5.2.8",
"@monaco-editor/react": "^4.7.0",
"@radix-ui/react-dropdown-menu": "^2.1.16",
"@radix-ui/react-toggle-group": "^1.1.11",
"@resvg/resvg-wasm": "^2.6.2",
"@tailwindcss/vite": "^4.1.18",
"@tanstack/react-devtools": "^0.9.0",
"@tanstack/react-router": "^1.144.0",
"@tanstack/react-router-devtools": "^1.144.0",
"@tanstack/react-start": "^1.145.3",
"@tanstack/router-plugin": "^1.145.2",
"@tanstack/react-devtools": "^0.9.2",
"@tanstack/react-router": "^1.151.6",
"@tanstack/react-router-devtools": "^1.151.6",
"@tanstack/react-start": "^1.152.0",
"@tanstack/router-plugin": "^1.151.6",
"@vercel/analytics": "^1.6.1",
"clawhub-schema": "workspace:*",
"clsx": "^2.1.1",
"convex": "^1.31.2",
"convex": "^1.31.6",
"convex-helpers": "^0.1.111",
"fflate": "^0.8.2",
"h3": "2.0.1-rc.8",
"lucide-react": "^0.562.0",
"monaco-editor": "^0.55.1",
"nitro": "^3.0.1-alpha.1",
"react": "^19.2.3",
"react-dom": "^19.2.3",
@@ -46,24 +57,27 @@
"semver": "^7.7.3",
"tailwind-merge": "^3.4.0",
"tailwindcss": "^4.1.18",
"vite-tsconfig-paths": "^6.0.3"
"vite-tsconfig-paths": "^6.0.4",
"yaml": "^2.8.2"
},
"devDependencies": {
"@biomejs/biome": "^2.3.11",
"@tanstack/devtools-vite": "^0.4.0",
"@playwright/test": "^1.57.0",
"@tanstack/devtools-vite": "^0.4.1",
"@testing-library/dom": "^10.4.1",
"@testing-library/react": "^16.3.1",
"@types/node": "^25.0.3",
"@types/react": "^19.2.7",
"@types/node": "^25.0.9",
"@types/react": "^19.2.8",
"@types/react-dom": "^19.2.3",
"@types/semver": "^7.7.1",
"@vitejs/plugin-react": "^5.1.2",
"@vitest/coverage-v8": "^4.0.16",
"@vitest/coverage-v8": "^4.0.17",
"jsdom": "^27.4.0",
"oxlint": "^1.36.0",
"oxlint-tsgolint": "^0.10.1",
"only-allow": "^1.2.1",
"oxlint": "^1.39.0",
"oxlint-tsgolint": "^0.11.1",
"typescript": "^5.9.3",
"vite": "^7.3.0",
"vitest": "^4.0.16"
"vite": "^7.3.1",
"vitest": "^4.0.17"
}
}
+20 -20
View File
@@ -1,57 +1,57 @@
# `clawdhub`
# `clawhub`
ClawdHub CLI — install, update, search, and publish agent skills as folders.
OpenClaw CLI — install, update, search, and publish agent skills as folders.
## Install
```bash
# From this repo (shortcut script at repo root)
bun clawdhub --help
bun clawhub --help
# Once published to npm
# npm i -g clawdhub
# npm i -g clawhub
```
## Auth (publish)
```bash
clawdhub login
clawhub login
# or
clawdhub auth login
clawhub auth login
# Headless / token paste
# or (token paste / headless)
clawdhub login --token clh_...
clawhub login --token clh_...
```
Notes:
- Browser login opens `https://clawdhub.com/cli/auth` and completes via a loopback callback.
- Token stored in `~/Library/Application Support/clawdhub/config.json` on macOS (override via `CLAWDHUB_CONFIG_PATH`).
- Browser login opens `https://clawhub.ai/cli/auth` and completes via a loopback callback.
- Token stored in `~/Library/Application Support/clawhub/config.json` on macOS (override via `CLAWHUB_CONFIG_PATH`, legacy `CLAWDHUB_CONFIG_PATH`).
## Examples
```bash
clawdhub search "postgres backups"
clawdhub install my-skill-pack
clawdhub update --all
clawdhub update --all --no-input --force
clawdhub publish ./my-skill-pack --slug my-skill-pack --name "My Skill Pack" --version 1.2.0 --changelog "Fixes + docs"
clawhub search "postgres backups"
clawhub install my-skill-pack
clawhub update --all
clawhub update --all --no-input --force
clawhub publish ./my-skill-pack --slug my-skill-pack --name "My Skill Pack" --version 1.2.0 --changelog "Fixes + docs"
```
## Sync (upload local skills)
```bash
# Start anywhere; scans workdir first, then legacy Clawdis/Clawd locations.
clawdhub sync
# Start anywhere; scans workdir first, then legacy Clawdis/Clawd/OpenClaw/Moltbot locations.
clawhub sync
# Explicit roots + non-interactive dry-run
clawdhub sync --root ../clawdis/skills --all --dry-run
clawhub sync --root ../clawdis/skills --all --dry-run
```
## Defaults
- Site: `https://clawdhub.com` (override via `--site` or `CLAWDHUB_SITE`)
- Registry: discovered from `/.well-known/clawdhub.json` on the site (override via `--registry` or `CLAWDHUB_REGISTRY`)
- Workdir: current directory (override via `--workdir`)
- Site: `https://clawhub.ai` (override via `--site` or `CLAWHUB_SITE`, legacy `CLAWDHUB_SITE`)
- Registry: discovered from `/.well-known/clawhub.json` on the site (legacy `/.well-known/clawdhub.json`; override via `--registry` or `CLAWHUB_REGISTRY`)
- Workdir: current directory (falls back to Clawdbot workspace if configured; override via `--workdir` or `CLAWHUB_WORKDIR`)
- Install dir: `./skills` under workdir (override via `--dir`)
View File
+8 -5
View File
@@ -1,10 +1,11 @@
{
"name": "clawdhub",
"version": "0.0.3",
"description": "ClawdHub CLI — install, update, search, and publish agent skills.",
"name": "clawhub",
"version": "0.4.0",
"description": "OpenClaw CLI \\u2014 install, update, search, and publish agent skills.",
"license": "MIT",
"type": "module",
"bin": {
"clawhub": "bin/clawdhub.js",
"clawdhub": "bin/clawdhub.js"
},
"files": [
@@ -24,13 +25,15 @@
"commander": "^14.0.2",
"fflate": "^0.8.2",
"ignore": "^7.0.5",
"json5": "^2.2.3",
"mime": "^4.1.0",
"ora": "^9.0.0",
"p-retry": "^7.1.1",
"semver": "^7.7.3"
"semver": "^7.7.3",
"undici": "^7.16.0"
},
"devDependencies": {
"@types/node": "^25.0.3",
"@types/node": "^25.0.9",
"typescript": "^5.9.3"
},
"engines": {

Some files were not shown because too many files have changed in this diff Show More