Compare commits

...
Author SHA1 Message Date
Peter Steinberger fa3849069d fix(api): centralize v1 soft-delete error mapping 2026-02-14 21:52:43 +01:00
Sergiy Dybskiy ed4c191dfc test: add e2e test for delete error handling
Verifies that deleting a non-existent skill returns a proper 'not found'
error instead of a generic 'Unauthorized' message.
2026-02-14 21:49:21 +01:00
Sergiy Dybskiy 9e704e0a81 fix: return proper HTTP status codes for delete/undelete errors
The delete and undelete handlers for skills and souls were catching all
errors and returning 401 Unauthorized, even for errors like:
- 'Skill not found' (should be 404)
- 'Forbidden' (should be 403)
- Other validation errors (should be 400)

This change updates the error handling to return appropriate status codes:
- 401 Unauthorized: authentication failures
- 403 Forbidden: authorization failures (not owner/admin/moderator)
- 404 Not Found: skill/soul/user not found
- 400 Bad Request: other errors with descriptive message

Fixes #34
2026-02-14 21:48:52 +01:00
Matt Krokosz f05dd556db fix: gate publish by immutable GitHub account ID 2026-02-14 20:25:15 +01:00
964893a622 fix: handle duplicate Convex Auth user records in publish ownership check (#180)
* fix: handle duplicate user records in publish ownership check

* fix: heal publish ownership via GitHub auth identity

---------

Co-authored-by: Emmet Brown <emmet@Emmets-Mac-mini.local>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 19:39:04 +01:00
Peter Steinberger 9a804b951f refactor: batch resolve tags in v1 API (#112) (thanks @mkrokosz) 2026-02-14 19:01:02 +01:00
Matthew KrokoszandClaude Opus 4.5 d699087786 fix: add null guard and short-circuit for empty tags
- Short-circuit when no version IDs to resolve
- Add null coalescing for runQuery response
- Fixes potential crash when tags are empty or query returns null

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-14 19:01:02 +01:00
Matthew KrokoszandClaude Opus 4.5 26b42727fe perf: batch tag resolution to reduce action→query round-trips
- Add getVersionsByIds batch query to skills.ts and souls.ts
- Replace per-item tag resolution with batch resolution in httpApiV1.ts
- Reduces N action→query round-trips to 1 for list endpoints

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-14 19:01:02 +01:00
Peter Steinberger 8756b78a4e chore: drop convex-helpers (#302) 2026-02-14 17:54:24 +01:00
e9c771d55d fix(skills): keep global sorting across pagination (#98)
* fix: initial skill sorting

* chore: update unit test

* fix: use correct indexes for skill sorting

* chore: cleanup

* fix(skills): preserve server order for paginated sorting

* chore(lint): apply biome formatting fixes

* chore(convex): bump tsconfig lib to ES2022

* fix(skills): add deterministic tie-breaker for search sorting

* fix(skills): stable sorting across pagination (#98) (thanks @CodeBBakGoSu)

---------

Co-authored-by: Brian Kasper <bkasperr@gmail.com>
Co-authored-by: knox-glorang <knox@glorang.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 17:44:07 +01:00
Peter Steinberger a58f0166fa refactor: centralize CORS + CLI auth token (#297)
* refactor(convex): centralize CORS headers

* refactor(cli): centralize auth token lookup
2026-02-14 15:31:03 +01:00
Peter SteinbergerandGrenghis-Khan 4328d4d700 fix(cors): complete CORS + tokenized CLI reads (#296)
* fix(cors): add Access-Control-Allow-Origin headers to API and downloads

* fix: add CORS to error/raw paths & add CLI install auth

* fix: add OPTIONS handler for CORS preflight

* fix(cors): complete CORS + tokenized CLI reads

* test(cli): fix config mock typing

---------

Co-authored-by: Grenghis-Khan <63885013+Grenghis-Khan@users.noreply.github.com>
2026-02-14 14:45:15 +01:00
Peter Steinberger 28ee2618c1 refactor: simplify user ensure updates 2026-02-14 13:56:15 +01:00
Peter Steinberger a4b850ec33 feat: improve moderation/admin UX + language-aware quality gate
- API: owner-visible responses for hidden/soft-deleted skills\n- Admin: add unban user mutations + docs\n- Quality: Intl.Segmenter tokenization + CJK signal to reduce false rejects\n- Jobs: skill-stat-events interval 15m -> 5m\n- Tests: add coverage for owner-visible states + non-Latin docs\n- Changelog: add Unreleased entry
2026-02-14 13:54:03 +01:00
Peter Steinberger 7e0b21f7c8 fix: sync handle on user ensure (#293) (thanks @christianhpoe) 2026-02-14 13:48:56 +01:00
ChristianHPoe 71c6705ab1 fix: sync handle on user ensure 2026-02-14 13:48:56 +01:00
a57769771f fix: add retry logic for OpenAI embedding API failures (#272)
* fix: add retry logic for OpenAI embedding API failures

Fixes #149

When importing or uploading skills, the OpenAI embedding API call could
fail with transient errors (rate limits, timeouts, network issues),
causing the entire import to fail with a generic "Server Error".

This adds retry logic with exponential backoff (1s, 2s, 4s delays):
- Retries on 429 (rate limit) and 5xx server errors
- Retries on network/fetch errors
- Logs warnings for debugging
- Max 3 retries before failing with clear error message

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: correct retry count and broaden network error catch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address retry loop off-by-one, broaden error catch, preserve original error

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: harden embeddings retry semantics

* style: format embeddings retry changes

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 04:54:18 +01:00
Peter Steinberger 6a2c131a8a style: remove residual blue accents and warm base palette 2026-02-14 04:40:50 +01:00
Peter Steinberger 67ac157545 fix: keep new skill versions pending until VT verdict 2026-02-14 02:53:02 +01:00
Peter Steinberger ef36cfd698 refactor(cli): centralize HTTP status errors and timeout tests (#286) 2026-02-14 02:39:42 +01:00
Peter SteinbergerandSash Zats e0637ad6aa fix(cli): throw Error for all timeout aborts (#283)
* fix(cli): throw Error on timeout aborts

Users have seen an elevated number of:\n  clawdhub search image\n  ✖ Non-error was thrown: "Timeout". You should only throw errors.\n\nInvestigation shows we were aborting with a string instead of an Error. Switching to controller.abort(new Error('Timeout')) makes retries/formatting treat it as a real error and clears the message.\n\nExample after change:\n  clawdhub search image\n  table-image v1.0.0  Table Image  (0.332)\n  nano-banana-pro v1.0.1  Nano Banana Pro  (0.319)\n  vap-media v1.0.1  AI media generation API - Flux2pro, Veo3.1, Suno Ai  (0.281)\n  clawdbot-meshyai-skill v0.1.0  Meshy AI  (0.276)\n  venice-ai-media v1.0.0  Venice AI Media  (0.274)\n  daily-recap v1.0.2  Daily Recap  (0.260)\n  openai-image-gen v1.0.1  Openai Image Gen  (0.260)\n  bible-votd v1.0.1  Bible Verse of the Day  (0.248)\n  orf v1.0.1  ORF  (0.224)\n  smalltalk v1.0.1  Smalltalk  (0.161)

* fix(http): wrap fetch calls in try-finally to prevent timer leaks

Addresses Vercel review comment: clearTimeout was not called on error paths when fetch throws an exception.

* fix(cli): unify timeout abort handling

---------

Co-authored-by: Sash Zats <sash@zats.io>
2026-02-14 02:27:34 +01:00
Peter Steinberger 09a21a07ff chore: add oxfmt config 2026-02-14 02:15:12 +01:00
Peter Steinberger 65a14dcef3 feat: make account deletion irreversible and migrate lint to oxlint 2026-02-14 02:15:01 +01:00
Peter Steinberger 97c12b2327 refactor(comments): extract handlers and harden mutation tests 2026-02-14 01:56:34 +01:00
a290c81a75 fix(comments): stop updating skills.updatedAt on comment add/remove (#55)
* fix(comments): stop updating skills.updatedAt on comment add/remove

Comments are not content changes, so they shouldn't invalidate skill
list queries that depend on updatedAt. This reduces query invalidation
when users add or remove comments.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(comments): add updatedAt invalidation regression coverage

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 01:53:40 +01:00
Peter Steinberger cc5d5cfee5 style: restore brown palette and dark-mode CTA tone 2026-02-14 01:18:29 +01:00
Peter Steinberger 03cd710abc fix: dedupe download metrics hourly by user-or-ip identity (#278) 2026-02-14 01:13:52 +01:00
Peter Steinberger 287f639fbc fix: simplify skills CTA label 2026-02-14 00:56:52 +01:00
Peter Steinberger aecf66981c fix: show stars in popular skill cards 2026-02-14 00:55:24 +01:00
Peter Steinberger db8090f287 style: darken hero primary CTA in dark mode 2026-02-14 00:50:52 +01:00
Peter Steinberger ae8614fa98 style: remove remaining warm accent literals 2026-02-14 00:38:21 +01:00
Peter Steinberger e7f78ea5a3 style: shift UI palette to cool blue tones 2026-02-14 00:31:55 +01:00
Peter Steinberger b9355f7a0c docs: thank @GhadiSaab for #53 2026-02-14 00:16:24 +01:00
ghadi saabandPeter Steinberger 9530676f8a fix: resolve search timeout and improve skills page UI alignment (#53)
* fix: resolve search timeout and improve skills page UI alignment

- Added a 10s timeout to OpenAI embedding requests to prevent hanging searches.
- Fixed a TypeScript error in search.ts regarding entry hydration.
- Restructured skills page layout and CSS to ensure consistent alignment between the search toolbar and skill cards.

* fix: resolve search timeout and improve skills page UI alignment

- Added a 10s timeout to OpenAI embedding requests to prevent hanging searches.
- Fixed a TypeScript error in search.ts regarding entry hydration.
- Restructured skills page layout and CSS to ensure consistent alignment between the search toolbar and skill cards.

* style: format skills index layout block

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 00:15:47 +01:00
Peter Steinberger ef23520d22 style: refine global UI theme, borders, and spacing 2026-02-13 23:50:24 +01:00
Peter Steinberger e67a6e6400 fix: render homepage popular cards from nested skill entries 2026-02-13 22:52:21 +01:00
Peter Steinberger e6871b86e1 fix: normalize legacy skill stats to prevent homepage crash 2026-02-13 22:45:34 +01:00
Peter Steinberger 75937e8b53 feat: show popular non-suspicious skills on homepage 2026-02-13 21:38:56 +01:00
Peter Steinberger d2919791d1 style: polish upload page layout and actions 2026-02-13 21:35:25 +01:00
Peter Steinberger 9019cd8462 perf: short-circuit empty skill summary generation 2026-02-13 21:21:09 +01:00
Peter Steinberger 5e58bd459e feat: add self-scheduling skill summary backfill job 2026-02-13 21:10:51 +01:00
Peter Steinberger 3badf0668f fix: make skill summary backfill resumable 2026-02-13 20:43:46 +01:00
Peter Steinberger c719297d70 feat: auto-generate missing skill summaries 2026-02-13 20:35:03 +01:00
Peter Steinberger e19cd23be2 fix: force auth redirects and registry to canonical clawhub host 2026-02-13 20:22:08 +01:00
Peter Steinberger 9266fb7c20 fix: add privileged-owner suspicious flag reconciler 2026-02-13 20:01:39 +01:00
Peter Steinberger 99645d2c27 fix: bypass suspicious flags for privileged owners and polish comment delete UI 2026-02-13 19:56:53 +01:00
Peter Steinberger a1ad7fac85 fix: force canonical downloads sort in skills browse mode 2026-02-13 19:33:26 +01:00
Peter Steinberger ebd2f12cc4 fix: enforce downloads as canonical default skills sort 2026-02-13 19:27:27 +01:00
Peter Steinberger e2ee7b164c feat: default skills sort to downloads 2026-02-13 19:15:19 +01:00
Peter Steinberger 36ed062739 style: polish selected states in skills toolbar 2026-02-13 19:12:48 +01:00
Peter Steinberger d12d6e3926 feat: add non-suspicious skills filter toggle 2026-02-13 19:05:46 +01:00
Peter Steinberger 1851a9c01f fix: make empty-skill cleanup resumable 2026-02-13 17:54:47 +01:00
Peter Steinberger bbeb0be343 feat: add empty-skill cleanup backfill with ban nominations 2026-02-13 17:48:58 +01:00
Peter Steinberger 9c22fb7e54 test: expand reauth ban regression coverage 2026-02-13 17:33:58 +01:00
Peter Steinberger ef2403179b fix: prevent autobanned users from self-reactivating 2026-02-13 17:28:00 +01:00
Peter Steinberger df178d4bfc fix: enforce quality gate and trust-tier spam checks 2026-02-13 17:16:53 +01:00
Peter Steinberger 318cdd33c5 docs: add git local-branch cleanup fallback 2026-02-13 17:03:38 +01:00
Peter Steinberger 9087b037dd fix: add skill publish anti-spam caps and quarantine 2026-02-13 16:58:18 +01:00
Peter Steinberger 6991569a1c fix: replace skill report prompt with modal 2026-02-13 16:47:16 +01:00
Peter Steinberger 32bc600be4 fix: harden skill listing and rate limiting under load 2026-02-13 16:39:57 +01:00
Kevin Kern a52a37d08c fix: harden download rate limiting and dedupe (#43) (thanks @regenrek)
- add download-specific rate limit tier\n- add per-IP/day dedupe + daily pruning\n- keep moderation gating + deterministic zips\n- add optional forwarded-IP trust via TRUST_FORWARDED_IPS
2026-02-13 16:22:05 +01:00
Brian KasperandPeter Steinberger dd58dd0815 Fix initial skill sorting (#92)
* fix: initial skill sorting

* chore: update unit test

* fix: use correct indexes for skill sorting

* chore: cleanup

* fix: land skill sorting update (#92) (thanks @bpk9)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 16:01:37 +01:00
Tanuj BhaudandPeter Steinberger 37a35c955a fix(vt): explicit return types and missing undici dependency (#255)
* fix(vt): explicit return types and missing undici dependency

Refactor action handlers in convex/vt.ts to use explicit return types, resolving circular type inference (TS7022). Also add undici to devDependencies for E2E tests.

* fix: add root undici devDependency for e2e (#255) (thanks @tanujbhaud)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 15:25:03 +01:00
Peter SteinbergerandSash Zats ddddb431c2 fix: make /search host-aware in SSR (#257)
* fix: make /search mode-aware

Notes:\n- Medium: /search now depends on getSiteMode() during beforeLoad. On server-side routing, if VITE_SITE_MODE isn’t set and VITE_SOULHUB_SITE_URL is set (as in .env.local), getSiteMode() will resolve to souls and redirect /search to / even on the ClawdHub deployment. This is a regression risk vs the old always-/skills redirect. Confirm deployment envs guarantee correct mode. src/routes/search.tsx:9-31

* fix: make /search host-aware in SSR

* chore: fix lint and route tree for /search route

---------

Co-authored-by: Sash Zats <sash@zats.io>
2026-02-13 14:26:31 +01:00
David AronchickandPeter Steinberger 78c27579a3 fix(cli): secure config file permissions (#164)
* fix(cli): secure config file permissions and reduce duplication

Security:
- Config files now created with 0600 permissions (owner read/write only)
- Config directories created with 0700 permissions
- Protects API tokens from other users on shared systems

Maintainability:
- Extract resolveConfigPath() helper to reduce code duplication
- Same legacy fallback logic (clawhub -> clawdhub) now in one place

* fix(cli): tolerate unsupported chmod errors for config

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 13:57:03 +01:00
xcqtnrandPeter Steinberger 9aebc35d86 fix: prevent infinite loading loop on skills page (#90)
* fix: prevent infinite loading loop on skills pageAdd isLoadingMore guard to IntersectionObserver useEffect to preventcontinuous WebSocket queries when user is idle at bottom of page.The observer now won't set up while a request is in progress, breakingthe infinite loop cycle.Fixes: Related to #89

* fix: prevent repeated skills auto-load requests (#90) (thanks @xcqtnr)

* fix: resolve PR merge conflicts and keep observer regression test (#90) (thanks @xcqtnr)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 05:44:36 +01:00
Peter Steinberger fc63f47ffa chore(release): 0.6.1 2026-02-13 05:14:16 +01:00
Gaurav SharmaandPeter Steinberger 0b83ea6ff3 fix: prevent horizontal overflow from long code blocks in skill pages (#183)
* Fix: Prevent horizontal overflow from long code blocks in skill pages

- Add max-width: 100% to .file-list-body and .file-row
- Prevents page-wide overflow when skills contain long code examples
- Markdown pre blocks already have overflow-x: auto, but parent containers were expanding infinitely
- Fixes issue where skills with 400+ char lines (e.g. browser automation commands) cause horizontal scrolling

Affected: Skills with long inline code in markdown (browser act commands, etc.)

* fix: add max-width to .file-list container to prevent overflow

- Also ensures .file-list-body constraint is inherited properly
- Prevents long code blocks from expanding file list container

* fix: add max-width to all markdown containers and pre tags

- Add max-width: 100% to .markdown, .tab-body, .markdown pre
- Ensures code blocks are constrained and show horizontal scrollbar
- Prevents content from expanding parent containers beyond viewport

* fix: add overflow-x to parent containers for horizontal scroll

Adds overflow-x: auto to .skill-detail-stack, .tab-card, and .tab-body
to ensure long code blocks are scrollable within the content area
instead of causing page-wide horizontal overflow.

Fixes horizontal overflow issue on skill pages with long code examples
(e.g., browser automation commands with 400+ character lines).

Tested on zepto skill page - page now stays within viewport (1200px)
and code blocks are accessible via horizontal scrollbar in tab area.

* docs: note code-block overflow fix in changelog (#183) (thanks @bewithgaurav)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 05:09:31 +01:00
LimitlessandLimitless2023 191b5763ec fix: include comment deltas in action-based stat processing & add stats reconciliation (#194)
Bug 1: applyAggregatedStatsAndUpdateCursor was missing 'comments' in both
the guard condition and the applySkillStatDeltas call. This caused comment
count deltas to be silently dropped during cron-based event processing,
while stars/downloads/installs were processed correctly.

Bug 2: No reconciliation mechanism existed. If events were missed due to
cursor issues or processing errors, skill stats (stars, comments) would
remain stale with no way to recover. Added reconcileSkillStarCounts
maintenance mutation that counts actual records in the stars and comments
tables and patches any out-of-sync skill stats.

Fixes #193

Co-authored-by: Limitless2023 <limitless@users.noreply.github.com>
2026-02-13 04:55:16 +01:00
Peter Steinberger 5397a8e5e0 fix: scope reauth fix; keep banned users blocked (#177) (thanks @tanujbhaud) 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 7949402888 test: add missing coverage for fresh-login reactivation and identity mismatch guard 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 2e3920d41a fix: use valid crons.interval and set to 1 minute 2026-02-13 04:34:46 +01:00
Tanuj Bhaud dd4fc823f6 fix: allow re-auth when existingUserId is null 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 135b9ea9b0 fix: ensure reactivation only matches soft-deleted user (prevents bypass) 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 17a106cefe fix: resolve final lint error in auth tests 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 496da99392 fix: update tests to include required existingUserId parameter 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 107486adfb fix: restore existingUserId check for type safety 2026-02-13 04:34:46 +01:00
Tanuj Bhaud ef9e7f0e57 test: update auth tests for direct deletedAt check 2026-02-13 04:34:46 +01:00
Tanuj Bhaud ecc8ad3833 fix: allow soft-deleted users to re-authenticate
Fixes Issue #32 where users who soft-deleted their accounts were unable to sign back in because the re-auth logic was only triggering when an existingUserId was passed by the auth provider, which doesn't happen during a standard fresh login flow.
2026-02-13 04:34:46 +01:00
Peter Steinberger 93c2b23b72 docs: add 0.6.1 unreleased changelog from post-0.6.0 commits 2026-02-13 04:14:16 +01:00
DCollandPeter Steinberger 2e492a5b87 fix(http): remove allowH2 from undici Agent — causes fetch failed on Node.js 22+ (#245)
* Remove allowH2 option from global dispatcher

fix/remove-allowH2-undici-node22-compat

* fix(http): remove allowH2 from e2e dispatcher

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 03:02:57 +01:00
Peter Steinberger 91c87d1322 test: fix search test handler typing 2026-02-13 02:48:16 +01:00
Peter Steinberger 91b8f160f8 test: add search fallback coverage 2026-02-13 02:44:53 +01:00
Peter Steinberger 32f3ce45e9 fix: add lexical fallback for skill search recall 2026-02-13 02:22:32 +01:00
Peter Steinberger 19bfe48a67 fix: prioritize relevant skills in search 2026-02-13 02:15:19 +01:00
Peter Steinberger 19951fccf7 docs: thank @superlowburn for PR #246 2026-02-13 01:22:38 +01:00
7dcada9122 fix: handle GitHub API rate limits in account age check (#246)
* fix: handle GitHub API rate limits in account age check

The GitHub account lookup uses unauthenticated requests (60 req/hr
per IP). Since this runs server-side in Convex, all users share the
same IP and quickly exhaust the rate limit, causing "GitHub account
lookup failed" errors during skill publish.

- Detect 403/429 responses and surface a clear rate-limit message
- Support optional GITHUB_TOKEN env var for authenticated requests
  (5,000 req/hr)

Fixes #155

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: stabilize GitHub account gate tests and docs

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 01:21:30 +01:00
theonejvo dab307cb6d fix: VT scan sync race condition + LLM-first moderation model
VT no longer overwrites LLM moderation verdicts. LLM is the primary
moderation authority; VT only escalates (hides + flags) for malicious/
suspicious content via new escalateByVtInternal mutation. Stale VT polls
write vtAnalysis marker instead of overwriting moderationReason. Query
pools expanded to include LLM-evaluated skills awaiting VT results.
Ban message now references malicious skills and security@openclaw.ai.
2026-02-13 01:05:52 +11:00
vignesh07 e96eb4781c chore: fix review comments 2026-02-11 10:38:26 -08:00
Vignesh f64b098fcc perf: lazy-load diff viewer (Monaco) (#212) 2026-02-11 12:31:07 -06:00
Vignesh 243432e04e chore: fix lint issues (#213) 2026-02-11 12:31:01 -06:00
theonejvo 9f7b9b92cd fix: trailing comma tolerance in JSON metadata, tone down persistence flags
- Strip trailing commas in frontmatter JSON before parsing (silent failure fix)
- Stop flagging disable-model-invocation default as a concern (it's the normal default)
- Stop flagging skills configuring themselves as privilege escalation
- Add MITRE ATLAS AML.T0051 context for when autonomous invocation actually matters
- Show actual defaults in assembled eval message instead of "not set"
2026-02-11 18:56:19 +11:00
theonejvo 3402f0e735 feat: add skill metadata docs, suspicious appeal banner for owners
- Document full frontmatter metadata reference in docs/skill-format.md
- Add metadata section + quick example to README
- Show appeal message on suspicious skills (owner-only) linking to GitHub issues
- Accept metadata.openclaw alias in README docs
- Re-evaluate all skills with full file content reading (backfill in progress)
2026-02-11 18:36:45 +11:00
theonejvo 3e39651074 feat: evaluator reads all file contents, not just SKILL.md
Reads all files from storage and includes their full source in the eval
prompt so the LLM can detect malicious code hidden behind clean READMEs.
Injection detection now scans all content. Per-file cap 10K chars, total
cap 50K chars.
2026-02-11 17:23:19 +11:00
theonejvo 741301848f fix: eval assembler falls back to metadata.openclaw for requirements 2026-02-11 14:39:02 +11:00
theonejvo b593dedba1 feat: recognize metadata.openclaw as valid frontmatter namespace 2026-02-11 14:29:11 +11:00
theonejvo e104b8030e fix: increase max_output_tokens for reasoning model, fix backfill error retry 2026-02-11 04:48:08 +11:00
theonejvo e74e879e12 fix: add retry with backoff for OpenAI rate limits, fix JSON mode requirement 2026-02-11 02:48:22 +11:00
theonejvo 1f1d93ded9 fix: collapse OpenClaw analysis by default, fix row spacing, switch to gpt-5-mini 2026-02-11 02:33:17 +11:00
theonejvo 9c31462f15 feat: add LLM security evaluation at publish time
Add OpenClaw LLM-based security evaluator that runs alongside VirusTotal
when skills are published. Reads SKILL.md prose, metadata, install specs,
and file manifest, then assesses coherence across 5 dimensions to catch
social engineering vectors that VT/regex miss (e.g. instruction-only skills
with no code files).

- convex/lib/securityPrompt.ts: system prompt, message assembly, response
  parsing, injection pattern detection
- convex/llmEval.ts: evaluateWithLlm action, evaluateBySlug convenience
  action, backfillLlmEval for existing skills
- convex/schema.ts: llmAnalysis field on skillVersions
- convex/skills.ts: updateVersionLlmAnalysisInternal mutation,
  getActiveSkillBatchForLlmBackfillInternal query, defense-in-depth
  multi-scanner flag merging in approveSkillByHashInternal
- convex/lib/skillPublish.ts: schedule LLM eval alongside VT scan
- SkillDetailPage.tsx: OpenClaw row, LlmAnalysisDetail expandable
  component with 5 dimension rows, guidance panel, findings section
- styles.css: analysis detail styles from mockup
2026-02-11 02:19:03 +11:00
Peter Steinberger f53be5c8d2 docs: reset changelog for next release 2026-02-10 13:22:09 +01:00
Peter Steinberger 03164fe8b1 chore: release 0.6.0 2026-02-10 13:20:16 +01:00
Peter Steinberger 81b53f0b20 feat: add ban reasons to moderation 2026-02-10 13:11:47 +01:00
theonejvo 75474075b1 fix(vt): self-sustaining VT scanning + working stats
- getStatsInternal: derive VT stats from moderationReason instead of
  N+1 version lookups that hit the 16MB byte limit
- UI: read cached vtAnalysis from version docs instead of hitting the
  live VT API on every page view
- Backfill: add vt-cache-backfill cron (30min) with self-scheduling to
  drain the backlog of skills missing cached vtAnalysis
- Daily rescan: cursor-based batching (100/batch) with self-scheduling
  instead of loading all skills in one shot
2026-02-10 20:39:27 +11:00
theonejvo 5946a08267 fix(convex): reduce write conflicts across hot paths
- downloads:increment: remove unnecessary db.get that added skill doc
  to read set, causing conflicts with the stat processing cron
- users:ensure: only patch when there are real field changes, skip
  unconditional updatedAt bump that forced a write on every call
- comments: route stats through event sourcing (insertStatEvent) instead
  of synchronous read-modify-write on the skill doc
- rateLimits: split into query-first check + conditional mutation so
  denied requests are conflict-free reads
- skillStatEvents: reduce MAX_SKILLS_PER_RUN from 500 to 50 to shrink
  the write set and lower conflict probability with concurrent mutations

Co-Authored-By: theonejvo <theonejvo@users.noreply.github.com>
2026-02-10 17:58:43 +11:00
theonejvo b997f5e749 fix(security): block downloads for unscanned/moderated skills
The download endpoint now checks moderation status before serving zips:

- Pending scan (423): "This skill is pending a security scan by VirusTotal. Please try again in a few minutes."
- Malicious (403): "Blocked: this skill has been flagged as malicious by VirusTotal and cannot be downloaded."
- Removed (410): "This skill has been removed by a moderator."
- Hidden (403): "This skill is currently unavailable."

Closes the supply chain gap where a newly published version could be
downloaded before VT scanning completed.
2026-02-08 23:49:23 +11:00
theonejvo 27f300bda5 chore: trigger CI 2026-02-08 16:55:41 +11:00
theonejvo 557f11985d chore: fix lint warnings (unused vars, formatting) 2026-02-08 16:49:43 +11:00
theonejvo 40f9ba5697 chore(clawhub): bump version to 0.5.1 2026-02-08 16:43:45 +11:00
theonejvo 26a353efc7 feat(cli): enforce VT Code Insight moderation on install/update
CLI now checks moderation status before installing skills:

**Suspicious skills** - Shows warning and requires confirmation:
```
⚠️  Warning: "skill-name" is flagged as suspicious by VirusTotal Code Insight.
   This skill may contain risky patterns (crypto keys, external APIs, eval, etc.)
   Review the skill code before use.

? Install anyway? (y/N)
```
Non-interactive mode requires --force flag.

**Malicious skills** - Blocked entirely:
```
✖ Blocked: skill-name is flagged as malicious
Error: This skill has been flagged as malware and cannot be installed.
```

Changes:
- API now returns `moderation` field with `isSuspicious` and `isMalwareBlocked`
- CLI schema updated to expect moderation field
- cmdInstall and cmdUpdate enforce moderation checks

Thanks to @zackkorman for raising this issue.
2026-02-08 16:36:55 +11:00
theonejvo 990d3d730d feat: VT backfill infrastructure and 99.7% scan coverage
- Add getQuickStatsInternal for fast dashboard stats
- Fix getStatsInternal to include null moderationStatus skills
- Add syncModerationReasons to sync vtAnalysis → moderationReason
- Add requestReanalysisForPending to push stuck skills to VT
- Add backfillActiveSkillsVTCache improvements for efficiency
- Add fixNullModerationStatus for legacy skill cleanup
- Add getPendingVTSkillsInternal for monitoring
- Fix getActiveSkillsMissingVTCacheInternal to avoid read limits

Backfilled 5,000+ skills to 99.7% VT Code Insight coverage:
- Clean: 3,537 (70.5%)
- Suspicious: 1,336 (26.6%)
- Malicious: 123 (2.5%)
- Pending: 17 (0.3%)
2026-02-08 15:37:11 +11:00
Peter Steinberger 75f7a93fe8 fix: restore soft-deleted users on reauth (#106) (thanks @mkrokosz) 2026-02-06 16:58:19 -08:00
Matthew KrokoszandClaude Opus 4.5 bc51ab4b5f fix: Convert userId to string for targetId comparison
The auditLogs.targetId field is v.string() in the schema, so explicitly
convert the Id<'users'> to string to ensure type-safe comparison.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-06 16:58:19 -08:00
Matthew KrokoszandClaude Opus 4.5 82fce7f34f fix: Restore soft-deleted users on re-authentication (with ban check)
Users who deleted their account were unable to sign in again with the
same GitHub account. The OAuth flow would complete but the user would
remain logged out due to the `deletedAt` field being set.

This fix adds a `createOrUpdateUser` callback that:
1. Detects soft-deleted users during OAuth
2. Checks audit logs to determine if user was BANNED vs SELF-DELETED
3. If banned → throws error "This account has been suspended"
4. If self-deleted → clears `deletedAt` to restore account

Security: Both `deleteAccount` and `banUser` set the same `deletedAt`
field. This fix ensures banned users cannot restore their accounts.

Performance: The callback runs on every sign-in, but the audit log
query ONLY executes for soft-deleted users (rare edge case). Normal
active users just hit a single `if` check - no extra queries. When
the audit log query does run, it uses the `by_target` index for
efficient lookup.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-06 16:58:19 -08:00
Peter Steinberger 5f4fa02b42 fix: update footer branding to OpenClaw (#122) (thanks @jontsai) 2026-02-06 16:52:01 -08:00
Jonathan Tsai cfef87059b fix: update footer branding from Clawdbot to OpenClaw
- Change 'A Clawdbot project' to 'An OpenClaw project'
- Update link from clawd.bot to openclaw.ai
2026-02-06 16:52:01 -08:00
Peter Steinberger b948b216c6 fix: backfill empty handles in ensure (#158) (thanks @adlai88) 2026-02-06 16:24:28 -08:00
adlai88 f156b909e7 fix: handle empty-string handle in ensure() fallback
`??` (nullish coalescing) treats `""` as a valid value, so users
with `handle=""` never get a fallback derived from name/email.
Change to `||` so empty strings fall through to the next candidate.

This affects any user whose handle field is an empty string rather
than null/undefined — e.g. early accounts or migration artifacts.
2026-02-06 16:24:28 -08:00
nikniknikbbb 0b46210f61 Update README.md (#140) 2026-02-06 16:12:04 -08:00
theonejvo 03eee4b6de fix: self-healing VT scan queue
- Batch size 100 (was 10) - process more per run
- Skip skills checked in last 60 min - avoid hammering same hashes
- After 10 failed checks, mark as pending.scan.stale - drops from queue
- Track scanLastCheckedAt and scanCheckCount on skills
- Add TODO for webhook/notification setup
2026-02-07 00:20:49 +11:00
theonejvo 09ffaba89b fix: randomize VT scan queue + add health monitoring
- Fetch 5x batch size and shuffle to avoid queue head-blocking
- Add getScanQueueHealthInternal to monitor queue status
- Log warnings when queue is unhealthy (>50 pending or >24h stale)
- Return health stats from pollPendingScans
2026-02-07 00:06:03 +11:00
theonejvo b39f203b12 fix: use yellow styling for suspicious code insight blocks
- Malicious verdicts keep red border/background
- Suspicious verdicts now use yellow/amber to match badge color
2026-02-06 20:40:41 +11:00
theonejvo 4002bb615d chore: fix lint errors 2026-02-06 16:39:27 +11:00
theonejvo 583e391d83 chore: add friendly wait message for pending scans 2026-02-06 16:37:24 +11:00
theonejvo 55ae6e1eb7 feat: add VT rescan trigger and backfill function
- Add requestRescan() to trigger Code Insight via /analyse endpoint
- Update pollPendingScans to request rescan when no Code Insight
- Add backfillPendingScans for one-time backlog clearing
2026-02-06 16:36:49 +11:00
theonejvo b4fe685a46 chore: fix lint formatting 2026-02-06 16:01:53 +11:00
theonejvo a93de9cf92 feat: add cron job to poll VT for pending scan results
- Add pollPendingScans action to vt.ts that checks VT for Code Insight verdicts
- Add getPendingScanSkillsInternal query to get skills awaiting scan
- Add vt-pending-scans cron job running every 5 minutes
- Updates skill moderation status when VT analysis is complete
2026-02-06 15:51:24 +11:00
theonejvo 74bf5946e1 chore: fix lint formatting 2026-02-06 15:11:43 +11:00
theonejvo 04c99aafc2 chore: remove debug console.log 2026-02-06 15:09:08 +11:00
theonejvo ee81325cb3 feat: VT Code Insight visibility matrix for malicious/suspicious skills
- Malicious skills: visible for transparency, downloads blocked via moderationFlags
- Suspicious skills: visible with warning banner, downloads allowed
- Neither appears in search/listings (not indexed)
- Add isSuspicious flag to moderation info
- Update approveSkillByHashInternal to set moderationFlags properly
- Add warning banner CSS variant for suspicious skills
2026-02-06 15:08:52 +11:00
theonejvo bdc6348b1a fix: show verdict labels (Benign/Suspicious/Malicious) instead of engine stats 2026-02-06 12:52:54 +11:00
theonejvo 68abaa3642 debug: add logging to getBySlug query 2026-02-05 20:36:43 +11:00
theonejvo 0c3acda26e fix: remove unused isModerated variable 2026-02-05 20:27:33 +11:00
theonejvo 650090d298 fix: allow owners to see all moderated skills, not just pending.scan 2026-02-05 20:26:29 +11:00
theonejvo 70220f9487 merge: resolve conflicts with main 2026-02-05 20:12:55 +11:00
theonejvo aad1fbe2c4 fix: use computed badges in pending skill response 2026-02-05 20:08:26 +11:00
theonejvo 2d7914859c feat: show Code Insight analysis for malicious skills
- Add `source` field to VT results to indicate code_insight vs engines
- Display Code Insight analysis text when AI detects malicious patterns
- Only show "X/Y engines" when traditional AV detection triggers
- Add styled analysis block with red accent for malicious verdicts
2026-02-05 19:58:31 +11:00
ba6a99a65f fix: show pending skill page to owners instead of "Skill not found" (#136)
* fix: show pending skill page to owners instead of "Skill not found"

When a skill owner uploads a skill that's pending VirusTotal scan,
they now see their skill page with a pending banner instead of
"Skill not found". The banner explains the scan is in progress.

Changes:
- Modified getBySlug query to return skill data for owners even when
  moderationStatus is 'hidden' with reason 'pending.scan'
- Added pendingReview flag to query response
- Added pending banner component to SkillDetailPage
- Added CSS for pending banner using existing ClawHub gold theme

* fix: show pending skills on owner's dashboard

Extended the list query to include pending skills when the requester
is viewing their own dashboard. Added "Scanning" badge with gold theme
to indicate skills pending VirusTotal review.

* fix: show all moderation states to owners with appropriate UI

- Owners see their blocked/removed skills with explanatory banners
- Red banner for malware-blocked and removed skills
- Gold banner for pending scan
- Download button hidden for blocked/removed skills
- Added security disclaimer: "Like a lobster shell, security has layers"
- Fixed badges bug (use computed badges, not stale skill.badges)

* feat: make malware-blocked skills publicly visible

Blocked skills are now visible to everyone via direct URL:
- Shows red banner with "security issue detected"
- Displays VT scan results
- No download button
- Still hidden from listings/search

Sends a strong transparency signal about security enforcement.

* fix: allow owners to view pending scan skills (#136)

* fix: make deterministic zip date timezone-safe

* chore: update convex api types

* fix: update changelog for pending scan visibility (#136) (thanks @orlyjamie)

---------

Co-authored-by: theonejvo <theonejvo@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-04 23:26:49 -08:00
Peter Steinberger 4ac63c2373 fix: update changelog for pending scan visibility (#136) (thanks @orlyjamie) 2026-02-04 23:25:41 -08:00
Peter Steinberger 8d0f32443a chore: update convex api types 2026-02-04 23:23:13 -08:00
Peter Steinberger 9e75090a39 fix: make deterministic zip date timezone-safe 2026-02-04 23:19:58 -08:00
Peter Steinberger 5cff71104c fix: allow owners to view pending scan skills (#136) 2026-02-04 23:19:53 -08:00
theonejvo 5ef8526874 feat: make malware-blocked skills publicly visible
Blocked skills are now visible to everyone via direct URL:
- Shows red banner with "security issue detected"
- Displays VT scan results
- No download button
- Still hidden from listings/search

Sends a strong transparency signal about security enforcement.
2026-02-05 17:55:29 +11:00
theonejvo 46faafc413 fix: show all moderation states to owners with appropriate UI
- Owners see their blocked/removed skills with explanatory banners
- Red banner for malware-blocked and removed skills
- Gold banner for pending scan
- Download button hidden for blocked/removed skills
- Added security disclaimer: "Like a lobster shell, security has layers"
- Fixed badges bug (use computed badges, not stale skill.badges)
2026-02-05 17:47:04 +11:00
theonejvo 0710b99ac4 fix: show pending skills on owner's dashboard
Extended the list query to include pending skills when the requester
is viewing their own dashboard. Added "Scanning" badge with gold theme
to indicate skills pending VirusTotal review.
2026-02-05 14:51:13 +11:00
theonejvo 9ebe1b6da8 fix: show pending skill page to owners instead of "Skill not found"
When a skill owner uploads a skill that's pending VirusTotal scan,
they now see their skill page with a pending banner instead of
"Skill not found". The banner explains the scan is in progress.

Changes:
- Modified getBySlug query to return skill data for owners even when
  moderationStatus is 'hidden' with reason 'pending.scan'
- Added pendingReview flag to query response
- Added pending banner component to SkillDetailPage
- Added CSS for pending banner using existing ClawHub gold theme
2026-02-05 14:41:00 +11:00
Shakker 98dd49c651 style: fix badges test import formatting
Format import statement to single line per Biome rules.
2026-02-05 03:04:26 +00:00
Shakker a115bb0d65 docs: update changelog for coverage improvements
Add entries for new tests and coverage configuration changes.
2026-02-05 03:04:26 +00:00
Shakker f1625cd5ff chore: add skillZip.ts to coverage tracking
Include convex/lib/skillZip.ts in coverage reports to track
the deterministic ZIP building utility added in PR #130.
2026-02-05 03:04:26 +00:00
Shakker 010ca1677f test: add 4+ errors truncation test for ark schema
Test the formatArkErrors truncation logic when there are more than
3 validation errors, ensuring the "+N more" message is displayed.
2026-02-05 03:04:26 +00:00
Shakker 62d62f976c test: add expandDroppedItems tests for uploadFiles
Add jsdom tests for the expandDroppedItems function:
- Handles null/empty DataTransferItemList
- Collects files via getAsFile fallback
- Collects files via webkitGetAsEntry for file entries
- Recursively collects files from directory entries
- Skips non-file/non-directory entries

Improves branch coverage for uploadFiles.ts.
2026-02-05 03:04:26 +00:00
Shakker 24ad575a4e test: add skillZip module tests
Add tests for the deterministic ZIP building utility from PR #130:
- buildSkillMeta function
- buildDeterministicZip with various scenarios
- Verifies deterministic output and _meta.json inclusion

Achieves 100% coverage for skillZip.ts.
2026-02-05 03:04:26 +00:00
Shakker 1cd6a22284 test: add badges module tests
Add comprehensive tests for the badges utility functions:
- isSkillHighlighted, isSkillOfficial, isSkillDeprecated
- getSkillBadges with all badge combinations

Improves branch coverage from 50% to 100% for badges.ts.
2026-02-05 03:04:26 +00:00
Peter Steinberger 343e065292 fix: stabilize VT scans and UI fetches (#130) (thanks @aleph8) 2026-02-04 17:34:35 -08:00
Alejandro García Peláez 2849a864e0 VirusTotal Integration on ClawHub (#130)
* feat: implementation of dynamic VirusTotal integration and deterministic ZIPs

* fix: do not show security scan results if hash is missing

* ui: show 'Loading...' instead of 'Pending' while fetching VT results

* security: restrict auto-approval to explicit benign verdicts only

* fix: prioritize AI verdict in results and refine stats fallback
2026-02-04 17:33:48 -08:00
Peter Steinberger ae0338e469 feat: add fuzzy user search for moderation CLI 2026-02-04 03:44:54 -08:00
Peter Steinberger ca1ef737cb fix: resolve typecheck errors in api and config 2026-02-04 03:32:09 -08:00
Peter Steinberger 589e46353b fix: management user search and totals 2026-02-04 03:30:07 -08:00
Peter Steinberger 7edbc03494 fix: skill list pagination and footer branding 2026-02-04 03:18:07 -08:00
Peter Steinberger f359071d96 feat(moderation): add set-role 2026-02-02 04:40:53 -08:00
Peter Steinberger 57bae9859e chore(release): 0.5.0 2026-02-02 04:25:11 -08:00
Peter Steinberger 96e9ffdcdc fix(convex): batch hard delete skills 2026-02-02 04:15:22 -08:00
Peter Steinberger eb4601141e fix(cli): honor registry from auth login 2026-02-02 03:25:14 -08:00
Peter Steinberger 39686b3b8d feat(management): add report and user filters 2026-02-02 03:02:52 -08:00
Peter Steinberger a24d3e9809 feat(cli): add inspect and moderation tools 2026-02-02 02:55:56 -08:00
Peter Steinberger 405c74a4ef feat: require report reasons 2026-02-02 00:57:45 -08:00
Peter Steinberger ee828046b8 chore: update dependencies 2026-02-02 00:31:54 -08:00
Peter Steinberger 789082bc00 chore: suppress empty chunk warnings 2026-02-02 00:27:01 -08:00
Peter Steinberger 3e4c2450cd chore: suppress nitro build warnings 2026-02-02 00:25:28 -08:00
Peter Steinberger ea2f51d2ba chore: quiet build warnings 2026-02-02 00:22:04 -08:00
Peter Steinberger 7b2bdbd08f feat: harden moderation and upload safety 2026-02-02 00:17:34 -08:00
Peter Steinberger f654dc9325 fix: allow legacy skill fields in schema 2026-01-31 12:45:35 +01:00
Peter Steinberger d78c105570 feat: add admin user ban 2026-01-31 11:44:31 +01:00
Peter Steinberger a32498ea7d fix: use ClawHub branding for registry 2026-01-31 11:31:46 +01:00
Shadow 5d6ee7adf3 trigger new deploy 2026-01-30 13:05:02 -06:00
Peter Steinberger ffa25f47da chore: rebrand user-facing to OpenClaw 2026-01-30 07:02:35 +01:00
Peter Steinberger f866dc05d1 style: format moderation flags 2026-01-30 05:27:19 +01:00
Peter Steinberger 69436fd79b fix: allow moltbot parsed data 2026-01-30 05:26:09 +01:00
Peter Steinberger f185ca6f55 feat: release 0.4.0 2026-01-30 05:23:12 +01:00
Peter Steinberger f3fc8d62b6 Revert "chore: rename molthub branding"
This reverts commit 8d68b55333.
2026-01-30 05:23:12 +01:00
Shakker 71f94a774f Merge pull request #70 from moltbot/remove-clawd-authenticator-tool
chore(moderation): block ClawdAuthenticatorTool (suspected malware)
2026-01-29 17:18:09 +00:00
vignesh07 07349e6107 chore(moderation): block ClawdAuthenticatorTool listing 2026-01-29 09:14:41 -08:00
Vignesh 5ef00e8c6a chore(security): harden file endpoints CSP + XFO + svg detection (#67) 2026-01-28 23:21:16 -06:00
Jamieson O'Reillyandtheonejvo c5e5e657dd fix: add CSP headers and Content-Disposition to prevent SVG XSS (#61)
Co-authored-by: theonejvo <theonejvo@users.noreply.github.com>
2026-01-28 22:21:39 -06:00
Jamie Turner 69e1e5c507 A few fixes for search. (#64) 2026-01-28 22:21:26 -06:00
Josh Palmer 481f1b9188 Merge pull request #66 from moltbot/fix/ci-lint
fix: restore lint compliance
2026-01-28 21:27:21 +01:00
Josh Palmer d4f5832554 🤖 chore: merge main into fix/ci-lint
- resolve conflicts in search/skill publish files
- apply Biome formatting updates from main

Tests: bun run lint:biome; bun run lint:oxlint
2026-01-28 21:23:50 +01:00
Josh Palmer ca3275ba92 🤖 fix: restore lint compliance
- apply Biome formatting and import ordering across linted files
- fix management useEffect dependencies flagged by Biome

Tests: bun run lint:biome; bun run lint:oxlint
2026-01-28 21:18:19 +01:00
Shadow d577add8a0 fix: unblock convex deploy typecheck 2026-01-28 13:15:18 -06:00
Shadow 90bd065f7e fix: correct public skill entries build 2026-01-28 13:12:41 -06:00
Shadow 219f05b160 fix: sanitize public skill and soul data 2026-01-28 13:03:19 -06:00
Shadow c8091ee8a8 chore: remove unauthenticated badge backfill 2026-01-28 08:33:14 -06:00
Shadow 63f367e1d6 chore: add unauthenticated badge backfill 2026-01-28 08:31:04 -06:00
Shadow e701d7b713 feat: add skill badges table 2026-01-28 01:11:46 -06:00
Shadow 79bd1f1d6c fix: query highlighted skills by batch index 2026-01-28 00:34:33 -06:00
Shadow ac51cc0236 fix: rely on highlighted badge 2026-01-27 23:27:36 -06:00
Shadow b9c23dc00e feat: add reports dashboard for moderation 2026-01-27 22:27:20 -06:00
Shadow f4e96995bc fix: allow clawdis parsed metadata 2026-01-27 21:39:23 -06:00
Shadow 33165db598 feat: unify skill routes with owner slugs 2026-01-27 21:28:46 -06:00
Shadow 4c10e4847b feat: add moderation management and backfill 2026-01-27 21:11:37 -06:00
Shadow 8d68b55333 chore: rename molthub branding 2026-01-27 18:25:02 -06:00
Jamie Turner 251de1f540 Performance optimizations for /skills page 2026-01-27 15:18:21 -06:00
Shadow 123f60fa93 Revert "Performance optimizations for /skills page" temporarily until we can deploy
This reverts commit 643faf71c8.
2026-01-27 12:20:15 -06:00
Jamie Turner 643faf71c8 Performance optimizations for /skills page 2026-01-27 12:09:57 -06:00
Aaron NgandPeter Steinberger a2c46fbb5d Search Fixes (#30)
* more search fixes

* update tests

* comments

* fix: tune search filters and limits (#30) (thanks @aaronn)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-01-25 00:08:29 +00:00
Peter Steinberger f51e0a087d test: fix lockfile mock version 2026-01-24 22:56:39 +00:00
emilianoandPeter Steinberger d9108b0948 feat: show published skills on user profile (#20)
* fix: resolve typecheck and lint errors

* fix: stabilize publish paths and token types

* feat: show published skills on user profile

* fix: document profile published skills (#20) (thanks @njoylab)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-01-24 22:55:18 +00:00
Peter Steinberger d7a017e1c3 fix: add update lookup test (#22) (thanks @daveonkels) 2026-01-24 22:30:09 +00:00
Dave OnkelsandClaude Opus 4.5 fffdf82540 fix: use path instead of url for skill metadata API call (#22)
The `cmdUpdate` function was passing a relative path to `apiRequest`
using the `url` property, but `url` expects a full URL. When `url` is
provided, it's used as-is without combining with the registry base URL.

This caused "Failed to parse URL from /api/v1/skills/<slug>" errors
when updating skills that don't have a local fingerprint match.

Changed to use `path` property which correctly combines with the
registry base URL via `new URL(args.path, registry)`.

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 22:28:54 +00:00
Ahmed Fuad MireClaude Opus 4.5vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>Ahmed
a16e624766 fix: relax search token matching to require at least one match (#27)
* fix: relax search token matching to require at least one match

The search was requiring ALL query tokens to exist in the skill's
displayName, slug, or summary. This was too strict and caused valid
results to be filtered out. For example, searching "HTTP API client"
would fail to match skills about "HTTP API" that didn't mention "client".

Changed from `.every()` to `.some()` so at least one token must match,
allowing the vector similarity to determine relevance for the rest.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: update matchesExactTokens to require prefix matching for query tokens

* more inclusive token check

* Update convex/lib/searchText.ts

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>

---------

Co-authored-by: Ahmed <ahmed.mire@kaluza.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
2026-01-24 21:23:03 +00:00
Peter Steinberger decce1d35c fix: skip missing skills in search hydration (#28) (thanks @aaronn) 2026-01-24 21:11:46 +00:00
Aaron Ng 468832af3f fix search (#28) 2026-01-24 21:11:06 +00:00
Shadow 54c793a660 fix: handle search embedding errors 2026-01-23 15:48:53 -06:00
Shadow 5d9a89a885 fix search 2026-01-23 15:24:26 -06:00
Peter Steinberger 31e9a57678 feat: add installs/trending sorts 2026-01-19 07:06:46 +00:00
Peter Steinberger 7680cc4ce8 feat: add idempotent star endpoints 2026-01-19 03:09:27 +00:00
Peter Steinberger de56255b95 fix: normalize monaco surface color 2026-01-19 02:21:51 +00:00
Peter Steinberger eaaa5e4423 Merge pull request #12 from NACC96/fix/search-mode-navigation
fix: search mode navigation and state management
2026-01-18 23:54:17 +00:00
Peter Steinberger 49e9c3c071 fix: stabilize search mode routing (#12) (thanks @NACC96) 2026-01-18 23:53:42 +00:00
NACC96 ef96fbee84 fix: auto-focus search input when search mode activates 2026-01-18 23:51:49 +00:00
NACC96 9fc032216e fix: update Header search links to use URL params instead of /search redirect 2026-01-18 23:51:49 +00:00
NACC96 236058b1b5 fix: preserve search flag in OnlyCrabsHome URL sync 2026-01-18 23:51:49 +00:00
NACC96 595bd2cf05 fix: search mode navigation and state management
- Fix "Explore search" button causing page refresh by using URL params
- Enable /search URL deep linking via beforeLoad redirect
- Fix logo click not closing search mode by properly syncing state with URL
2026-01-18 23:51:49 +00:00
Peter Steinberger 918b5528df chore: format peer check script 2026-01-18 23:42:04 +00:00
Peter Steinberger 2f6f11af3f ci: add peer dependency check 2026-01-18 23:40:55 +00:00
Peter Steinberger c90f92dbc7 fix: align auth core with convex auth 2026-01-18 23:33:51 +00:00
Peter Steinberger 8db8c89206 chore: update deps and adjust vite convex resolution 2026-01-18 23:29:59 +00:00
Peter Steinberger 00c31dcdd4 fix: derive auth from user query 2026-01-18 23:13:29 +00:00
Peter Steinberger 47646937b6 fix: dedupe convex auth modules 2026-01-18 22:33:09 +00:00
Shadow 10d250ea32 fix: keep ConvexAuthProvider during SSR 2026-01-18 14:43:24 -06:00
Peter Steinberger e7fa7afdf7 chore: bump clawdhub to 0.2.1 2026-01-18 16:28:40 +00:00
Peter Steinberger aa97727be8 fix: harden explore limit + tests/docs (#14) (thanks @jdrhyne) 2026-01-18 16:26:28 +00:00
Peter Steinberger d375496c17 Merge pull request #14 from jdrhyne/feat/explore-command
feat(cli): add explore command to browse latest updated skills
2026-01-18 16:25:48 +00:00
Jonathan Rhyne e108789ab1 feat(cli): add explore command to browse latest updated skills
Adds a new `clawdhub explore` command that fetches the most recently
updated skills from the registry, sorted by updatedAt descending.

Usage:
  clawdhub explore           # Show latest 25 skills
  clawdhub explore --limit 10

Output includes slug, version, relative time since update, and summary.

The API endpoint already exists and returns skills sorted by updatedAt,
this just exposes it via the CLI.
2026-01-17 23:07:54 -05:00
224 changed files with 22165 additions and 2631 deletions
+3 -2
View File
@@ -15,10 +15,12 @@ jobs:
- uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.5
bun-version: 1.3.6
- name: Install
run: bun install --frozen-lockfile
- name: Peer deps
run: bun run check:peers
- name: Lint
run: bun run lint
@@ -36,4 +38,3 @@ jobs:
- name: Build
run: bun run build
+20
View File
@@ -0,0 +1,20 @@
{
"$schema": "./node_modules/oxfmt/configuration_schema.json",
"experimentalSortImports": {
"newlinesBetween": false,
},
"experimentalSortPackageJson": {
"sortScripts": true,
},
"ignorePatterns": [
".output/",
".tanstack/",
"convex/_generated/",
"coverage/",
"dist/",
"node_modules/",
"public/",
"src/routeTree.gen.ts",
"test-results/",
],
}
+35 -1
View File
@@ -1,3 +1,37 @@
{
"ignorePatterns": ["node_modules", "dist", "coverage", "convex/_generated", ".tanstack", "public"]
"$schema": "./node_modules/oxlint/configuration_schema.json",
"plugins": ["unicorn", "typescript", "oxc"],
"categories": {
"correctness": "error",
"perf": "error",
"suspicious": "error"
},
"rules": {
"curly": "off",
"eslint-plugin-unicorn/prefer-array-find": "off",
"eslint-plugin-unicorn/no-array-sort": "off",
"eslint/no-await-in-loop": "off",
"eslint/no-new": "off",
"oxc/no-accumulating-spread": "off",
"oxc/no-async-endpoint-handlers": "off",
"oxc/no-map-spread": "off",
"typescript/no-explicit-any": "error",
"typescript/no-extraneous-class": "off",
"typescript/no-unnecessary-boolean-literal-compare": "off",
"typescript/no-unnecessary-type-assertion": "off",
"typescript/no-unsafe-type-assertion": "off",
"unicorn/consistent-function-scoping": "off",
"unicorn/require-post-message-target-origin": "off"
},
"ignorePatterns": [
".output/",
".tanstack/",
"convex/_generated/",
"coverage/",
"dist/",
"node_modules/",
"public/",
"src/routeTree.gen.ts",
"test-results/"
]
}
+5 -1
View File
@@ -34,11 +34,15 @@
- Keep changes scoped; avoid repo-wide search/replace.
- PRs: include summary + test commands run. Add screenshots for UI changes.
## Git Notes
- If `git branch -d/-D <branch>` is policy-blocked, delete the local ref directly: `git update-ref -d refs/heads/<branch>`.
## Configuration & Security
- Local env: `.env.local` (never commit secrets).
- Convex env holds JWT keys; Vercel only needs `VITE_CONVEX_URL` + `VITE_CONVEX_SITE_URL`.
- OAuth: GitHub OAuth App credentials required for login.
## Convex Ops (Gotchas)
- New Convex functions must be pushed before `convex run`: use `bunx convex dev --once` (dev) or `bunx convex deploy --prod` (prod).
- New Convex functions must be pushed before `convex run`: use `bunx convex dev --once` (dev) or `bunx convex deploy` (prod).
- For non-interactive prod deploys, use `bunx convex deploy -y` to skip confirmation.
- If `bunx convex run --env-file .env.local ...` returns `401 MissingAccessToken` despite `bunx convex login`, workaround: omit `--env-file` and use `--deployment-name <name>` / `--prod`.
+107 -5
View File
@@ -1,5 +1,107 @@
# Changelog
## Unreleased
### Added
- Admin: add manual unban for banned users (clears `deletedAt` + `banReason`, audit log entry). Revoked API tokens stay revoked.
### Changed
- Quality gate: language-aware word counting (`Intl.Segmenter`) and new `cjkChars` signal to reduce false rejects for non-Latin docs.
- Jobs: run skill stat event processing every 5 minutes (was 15).
- API performance: batch resolve skill/soul tags in v1 list/get endpoints (fewer action->query round-trips) (#112) (thanks @mkrokosz).
### Fixed
- Users: sync handle on ensure when GitHub login changes (#293) (thanks @christianhpoe).
- Upload gate: fetch GitHub account age by immutable account ID (prevents username swaps) (#116) (thanks @mkrokosz).
- API: for owners, return clearer status/messages for hidden/soft-deleted skills instead of a generic 404.
- HTTP/CORS: add preflight handler + include CORS headers on API/download errors; CLI: include auth token for owner-visible installs/updates (#146) (thanks @Grenghis-Khan).
- Skills: keep global sorting across pagination on `/skills` (thanks @CodeBBakGoSu, #98).
## 0.6.1 - 2026-02-13
### Added
- Security: add LLM-based security evaluation during skill publish.
- Parsing: recognize `metadata.openclaw` frontmatter and evaluate all skill files for requirements.
### Changed
- Performance: lazy-load Monaco diff viewer on demand (thanks @alexjcm, #212).
- Search: improve recall/ranking with lexical fallback and relevance prioritization.
- Moderation UX: collapse OpenClaw analysis by default; update spacing and default reasoning model.
### Fixed
- Skills: fix initial `/skills` sort wiring so first page respects selected sort/direction (thanks @bpk9, #92).
- Search/UI: add embedding request timeout and align `/skills` toolbar + list width (thanks @GhadiSaab, #53).
- Upload gate: handle GitHub API rate limits and optional authenticated lookup token (thanks @superlowburn, #246).
- HTTP: remove `allowH2` from Undici agent to prevent `fetch failed` on Node.js 22+ (#245).
- Tests: add root `undici` dev dependency for Node E2E imports (thanks @tanujbhaud, #255).
- Downloads: add download rate limiting + per-IP/day dedupe + scheduled dedupe pruning; preserve moderation gating and deterministic zips (thanks @regenrek, #43).
- VirusTotal: fix scan sync race conditions and retry behavior in scan/backfill paths.
- Metadata: tolerate trailing commas in JSON metadata.
- Auth: allow soft-deleted users to re-authenticate on fresh login, while keeping banned users blocked (thanks @tanujbhaud, #177).
- Web: prevent horizontal overflow from long code blocks in skill pages (thanks @bewithgaurav, #183).
## 0.6.0 - 2026-02-10
### Added
- CLI/API: add `set-role` to change user roles (admin only).
- Security: quarantine skill publishes with VirusTotal scans + UI (thanks @aleph8, #130).
- Testing: add tests for badges, skillZip, uploadFiles expandDroppedItems, and ark schema error truncation.
- Moderation: add ban reasons to API/CLI and show in management UI.
### Changed
- Coverage: track `convex/lib/skillZip.ts` in coverage reports.
### Fixed
- Web: show pending-scan skills to owners without 404 (thanks @orlyjamie, #136).
- Users: backfill empty handles from name/email in ensure (thanks @adlai88, #158).
- Web: update footer branding to OpenClaw (thanks @jontsai, #122).
- Auth: restore soft-deleted users on reauth, block banned users (thanks @mkrokosz, #106).
## 0.5.0 - 2026-02-02
### Added
- Admin: ban users and delete owned skills from management console.
- Moderation: auto-hide skills after 4 unique reports; per-user report cap; moderators can ban users.
- Uploads: require GitHub accounts to be at least 7 days old for skill + soul publish/import.
- CLI: add `inspect` to fetch skill metadata/files without installing.
- CLI: add moderation commands for hide/unhide/delete and ban users.
- Management: add filters for reported skills and users.
### Changed
- Deps: update dependencies to latest available versions.
- Reporting: require reasons, show them in management console, warn about abuse bans.
### Fixed
- Bans: batch hard-delete cleanup to avoid Convex read limits on large skills.
## 0.4.0 - 2026-01-30
### Added
- Web: show published skills on user profiles (thanks @njoylab, #20).
- CLI: include ClawHub + Moltbot fallback skill roots for sync scans.
- CLI: support OpenClaw configuration files (`OPENCLAW_CONFIG_PATH` / `OPENCLAW_STATE_DIR`).
### Changed
- Brand: rebrand to ClawHub and publish CLI as `clawhub` (legacy `clawdhub` supported).
- Domain: default site/registry now `https://clawhub.ai`; `.well-known/clawhub.json` preferred.
- Theme: persist theme under `clawhub-theme` (legacy key still read).
### Fixed
- Registry: drop missing skills during search hydration (thanks @aaronn, #28).
- CLI: use path-based skill metadata lookup for updates (thanks @daveonkels, #22).
- Search: keep highlighted-only filtering and clamp vector candidates to Convex limits (thanks @aaronn, #30).
## 0.3.0 - 2026-01-19
### Added
- CLI: add `explore` command for latest updates, with limit clamping + tests/docs (thanks @jdrhyne, #14).
- CLI: `explore --json` output + new sorts (`installs`, `installsAllTime`, `trending`) and limit up to 200.
- API: `/api/v1/skills` supports installs + trending sorts (7-day installs).
- API: idempotent `POST/DELETE /api/v1/stars/{slug}` endpoints.
- Registry: trending leaderboard + daily stats backfill for installs-based sorts.
### Fixed
- Web: keep search mode navigation and state in sync (thanks @NACC96, #12).
## 0.2.0 - 2026-01-13
### Added
@@ -14,7 +116,7 @@
- Registry: make SoulHub auto-seed idempotent and non-user-owned.
- Registry: keep GitHub backup state + publish backups intact (thanks @joshp123, #1).
- CLI/Registry: restore fork lineage on sync + clamp bulk list queries (thanks @joshp123, #1).
- CLI: default workdir falls back to Clawdbot workspace (override with `--workdir` / `CLAWDHUB_WORKDIR`).
- CLI: default workdir falls back to Clawdbot workspace (override with `--workdir` / `CLAWHUB_WORKDIR`).
## 0.0.6 - 2026-01-07
@@ -30,7 +132,7 @@
## 0.0.5 - 2026-01-06
### Added
- Telemetry: track installs via `clawdhub sync` (logged-in only), per root, with 120-day staleness.
- Telemetry: track installs via `clawhub sync` (logged-in only), per root, with 120-day staleness.
- Skills: show current + all-time installs; sort by installs.
- Profile: private "Installed" tab with JSON export + delete telemetry controls.
- Docs: add `docs/telemetry.md` (what we track + how to opt out).
@@ -38,7 +140,7 @@
- Web: dashboard for managing your published skills (thanks @dbhurley!).
### Changed
- CLI: telemetry opt-out via `CLAWDHUB_DISABLE_TELEMETRY=1`.
- CLI: telemetry opt-out via `CLAWHUB_DISABLE_TELEMETRY=1`.
- Web: move theme picker into mobile menu.
### Fixed
@@ -86,8 +188,8 @@
### Fixed
- CLI sync: wrap note output to avoid terminal overflow; cap list lengths.
- CLI sync: label fallback scans as fallback locations.
- CLI package: bundle schema internally (no external `clawdhub-schema` publish).
- Repo: mark `clawdhub-schema` as private to prevent publishing.
- CLI package: bundle schema internally (no external `clawhub-schema` publish).
- Repo: mark `clawhub-schema` as private to prevent publishing.
## 0.0.2 - 2026-01-04
+35 -11
View File
@@ -1,20 +1,20 @@
# ClawdHub
# ClawHub
<p align="center">
<a href="https://github.com/clawdbot/clawdhub/actions/workflows/ci.yml?branch=main"><img src="https://img.shields.io/github/actions/workflow/status/clawdbot/clawdhub/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a>
<a href="https://github.com/openclaw/clawhub/actions/workflows/ci.yml?branch=main"><img src="https://img.shields.io/github/actions/workflow/status/openclaw/clawhub/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a>
<a href="https://discord.gg/clawd"><img src="https://img.shields.io/discord/1456350064065904867?label=Discord&logo=discord&logoColor=white&color=5865F2&style=for-the-badge" alt="Discord"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
</p>
ClawdHub is the **public skill registry for Clawdbot**: publish, version, and search text-based agent skills (a `SKILL.md` plus supporting files).
ClawHub is the **public skill registry for Clawdbot**: publish, version, and search text-based agent skills (a `SKILL.md` plus supporting files).
Its designed for fast browsing + a CLI-friendly API, with moderation hooks and vector search.
onlycrabs.ai is the **SOUL.md registry**: publish and share system lore the same way you publish skills.
Live: `https://clawdhub.com`
Live: `https://clawhub.ai`
onlycrabs.ai: `https://onlycrabs.ai`
## What you can do
## What you can do with it
- Browse skills + render their `SKILL.md`.
- Publish new skill versions with changelogs + tags (including `latest`).
@@ -27,7 +27,7 @@ onlycrabs.ai: `https://onlycrabs.ai`
- Entry point is host-based: `onlycrabs.ai`.
- On the onlycrabs.ai host, the home page and nav default to souls.
- On ClawdHub, souls live under `/souls`.
- On ClawHub, souls live under `/souls`.
- Soul bundles only accept `SOUL.md` for now (no extra files).
## How it works (high level)
@@ -35,15 +35,16 @@ onlycrabs.ai: `https://onlycrabs.ai`
- Web app: TanStack Start (React, Vite/Nitro).
- Backend: Convex (DB + file storage + HTTP actions) + Convex Auth (GitHub OAuth).
- Search: OpenAI embeddings (`text-embedding-3-small`) + Convex vector search.
- API schema + routes: `packages/schema` (`clawdhub-schema`).
- API schema + routes: `packages/schema` (`clawhub-schema`).
## Telemetry
ClawdHub tracks minimal **install telemetry** (to compute install counts) when you run `clawdhub sync` while logged in.
ClawHub tracks minimal **install telemetry** (to compute install counts) when you run `clawhub sync` while logged in.
Disable via:
```bash
export CLAWDHUB_DISABLE_TELEMETRY=1
export CLAWHUB_DISABLE_TELEMETRY=1
```
Details: `docs/telemetry.md`.
@@ -95,7 +96,7 @@ This writes `JWT_PRIVATE_KEY` + `JWKS` to the deployment and prints values for y
## Nix plugins (nixmode skills)
ClawdHub can store a nix-clawdbot plugin pointer in SKILL frontmatter so the registry knows which
ClawHub can store a nix-clawdbot plugin pointer in SKILL frontmatter so the registry knows which
Nix package bundle to install. A nix plugin is different from a regular skill pack: it bundles the
skill pack, the CLI binary, and its config flags/requirements together.
@@ -137,7 +138,30 @@ metadata: {"clawdbot":{"cliHelp":"padel --help\\nUsage: padel [command]\\n"}}
---
```
`metadata.clawdbot` is preferred, but `metadata.clawdis` is accepted as an alias for compatibility.
`metadata.clawdbot` is preferred, but `metadata.clawdis` and `metadata.openclaw` are accepted as aliases.
## Skill metadata
Skills declare their runtime requirements (env vars, binaries, install specs) in the `SKILL.md` frontmatter. ClawHub's security analysis checks these declarations against actual skill behavior.
Full reference: [`docs/skill-format.md`](docs/skill-format.md#frontmatter-metadata)
Quick example:
```yaml
---
name: my-skill
description: Does a thing with an API.
metadata:
openclaw:
requires:
env:
- MY_API_KEY
bins:
- curl
primaryEnv: MY_API_KEY
---
```
## Scripts
-41
View File
@@ -1,41 +0,0 @@
{
"$schema": "https://biomejs.dev/schemas/2.3.11/schema.json",
"files": {
"includes": [
"**",
"!**/.cta.json",
"!**/.vscode",
"!**/node_modules",
"!**/dist",
"!**/.output",
"!**/coverage",
"!**/convex/_generated",
"!**/test-results",
"!**/src/routeTree.gen.ts",
"!**/.tanstack",
"!**/public",
"!**/.devenv",
"!**/.devenv"
]
},
"assist": { "actions": { "source": { "organizeImports": "on" } } },
"formatter": {
"enabled": true,
"indentStyle": "space",
"indentWidth": 2,
"lineWidth": 100
},
"linter": {
"enabled": true,
"rules": {
"recommended": true
}
},
"javascript": {
"formatter": {
"quoteStyle": "single",
"semicolons": "asNeeded",
"trailingCommas": "all"
}
}
}
+297 -240
View File
File diff suppressed because it is too large Load Diff
Executable
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bun
import { existsSync } from 'node:fs'
import { stat } from 'node:fs/promises'
import { fileURLToPath } from 'node:url'
const distCliUrl = new URL('./packages/clawdhub/dist/cli.js', import.meta.url)
const distCliPath = fileURLToPath(distCliUrl)
const srcRootPath = fileURLToPath(new URL('./packages/clawdhub/src/', import.meta.url))
const shouldBuild = await (async () => {
if (!existsSync(distCliPath)) return true
try {
const dist = await stat(distCliPath)
const latestSrcMtime = await getLatestMtime(srcRootPath)
return latestSrcMtime > dist.mtimeMs
} catch {
return true
}
})()
if (shouldBuild) {
const proc = Bun.spawn(['bunx', 'tsc', '-p', 'packages/clawdhub/tsconfig.json'], {
stdin: 'inherit',
stdout: 'inherit',
stderr: 'inherit',
})
const code = await proc.exited
if (code !== 0) process.exit(code)
}
await import(distCliUrl.href)
async function getLatestMtime(root: string) {
let latest = 0
const glob = new Bun.Glob('**/*.ts')
for await (const rel of glob.scan({ cwd: root, onlyFiles: true })) {
const path = `${root}${root.endsWith('/') ? '' : '/'}${rel}`
try {
const entry = await stat(path)
latest = Math.max(latest, entry.mtimeMs)
} catch {
// ignore
}
}
return latest
}
+44
View File
@@ -12,6 +12,7 @@ import type * as auth from "../auth.js";
import type * as comments from "../comments.js";
import type * as crons from "../crons.js";
import type * as devSeed from "../devSeed.js";
import type * as devSeedExtra from "../devSeedExtra.js";
import type * as downloads from "../downloads.js";
import type * as githubBackups from "../githubBackups.js";
import type * as githubBackupsNode from "../githubBackupsNode.js";
@@ -21,36 +22,57 @@ import type * as githubSoulBackupsNode from "../githubSoulBackupsNode.js";
import type * as http from "../http.js";
import type * as httpApi from "../httpApi.js";
import type * as httpApiV1 from "../httpApiV1.js";
import type * as httpPreflight from "../httpPreflight.js";
import type * as leaderboards from "../leaderboards.js";
import type * as lib_access from "../lib/access.js";
import type * as lib_apiTokenAuth from "../lib/apiTokenAuth.js";
import type * as lib_badges from "../lib/badges.js";
import type * as lib_changelog from "../lib/changelog.js";
import type * as lib_embeddings from "../lib/embeddings.js";
import type * as lib_githubAccount from "../lib/githubAccount.js";
import type * as lib_githubBackup from "../lib/githubBackup.js";
import type * as lib_githubIdentity from "../lib/githubIdentity.js";
import type * as lib_githubImport from "../lib/githubImport.js";
import type * as lib_githubSoulBackup from "../lib/githubSoulBackup.js";
import type * as lib_httpHeaders from "../lib/httpHeaders.js";
import type * as lib_httpRateLimit from "../lib/httpRateLimit.js";
import type * as lib_leaderboards from "../lib/leaderboards.js";
import type * as lib_moderation from "../lib/moderation.js";
import type * as lib_public from "../lib/public.js";
import type * as lib_searchText from "../lib/searchText.js";
import type * as lib_securityPrompt from "../lib/securityPrompt.js";
import type * as lib_skillBackfill from "../lib/skillBackfill.js";
import type * as lib_skillPublish from "../lib/skillPublish.js";
import type * as lib_skillQuality from "../lib/skillQuality.js";
import type * as lib_skillSafety from "../lib/skillSafety.js";
import type * as lib_skillStats from "../lib/skillStats.js";
import type * as lib_skillSummary from "../lib/skillSummary.js";
import type * as lib_skillZip from "../lib/skillZip.js";
import type * as lib_skills from "../lib/skills.js";
import type * as lib_soulChangelog from "../lib/soulChangelog.js";
import type * as lib_soulPublish from "../lib/soulPublish.js";
import type * as lib_tokens from "../lib/tokens.js";
import type * as lib_userSearch from "../lib/userSearch.js";
import type * as lib_webhooks from "../lib/webhooks.js";
import type * as llmEval from "../llmEval.js";
import type * as maintenance from "../maintenance.js";
import type * as rateLimits from "../rateLimits.js";
import type * as search from "../search.js";
import type * as seed from "../seed.js";
import type * as seedSouls from "../seedSouls.js";
import type * as skillStatEvents from "../skillStatEvents.js";
import type * as skills from "../skills.js";
import type * as soulComments from "../soulComments.js";
import type * as soulDownloads from "../soulDownloads.js";
import type * as soulStars from "../soulStars.js";
import type * as souls from "../souls.js";
import type * as stars from "../stars.js";
import type * as statsMaintenance from "../statsMaintenance.js";
import type * as telemetry from "../telemetry.js";
import type * as tokens from "../tokens.js";
import type * as uploads from "../uploads.js";
import type * as users from "../users.js";
import type * as vt from "../vt.js";
import type * as webhooks from "../webhooks.js";
import type {
@@ -64,6 +86,7 @@ declare const fullApi: ApiFromModules<{
comments: typeof comments;
crons: typeof crons;
devSeed: typeof devSeed;
devSeedExtra: typeof devSeedExtra;
downloads: typeof downloads;
githubBackups: typeof githubBackups;
githubBackupsNode: typeof githubBackupsNode;
@@ -73,36 +96,57 @@ declare const fullApi: ApiFromModules<{
http: typeof http;
httpApi: typeof httpApi;
httpApiV1: typeof httpApiV1;
httpPreflight: typeof httpPreflight;
leaderboards: typeof leaderboards;
"lib/access": typeof lib_access;
"lib/apiTokenAuth": typeof lib_apiTokenAuth;
"lib/badges": typeof lib_badges;
"lib/changelog": typeof lib_changelog;
"lib/embeddings": typeof lib_embeddings;
"lib/githubAccount": typeof lib_githubAccount;
"lib/githubBackup": typeof lib_githubBackup;
"lib/githubIdentity": typeof lib_githubIdentity;
"lib/githubImport": typeof lib_githubImport;
"lib/githubSoulBackup": typeof lib_githubSoulBackup;
"lib/httpHeaders": typeof lib_httpHeaders;
"lib/httpRateLimit": typeof lib_httpRateLimit;
"lib/leaderboards": typeof lib_leaderboards;
"lib/moderation": typeof lib_moderation;
"lib/public": typeof lib_public;
"lib/searchText": typeof lib_searchText;
"lib/securityPrompt": typeof lib_securityPrompt;
"lib/skillBackfill": typeof lib_skillBackfill;
"lib/skillPublish": typeof lib_skillPublish;
"lib/skillQuality": typeof lib_skillQuality;
"lib/skillSafety": typeof lib_skillSafety;
"lib/skillStats": typeof lib_skillStats;
"lib/skillSummary": typeof lib_skillSummary;
"lib/skillZip": typeof lib_skillZip;
"lib/skills": typeof lib_skills;
"lib/soulChangelog": typeof lib_soulChangelog;
"lib/soulPublish": typeof lib_soulPublish;
"lib/tokens": typeof lib_tokens;
"lib/userSearch": typeof lib_userSearch;
"lib/webhooks": typeof lib_webhooks;
llmEval: typeof llmEval;
maintenance: typeof maintenance;
rateLimits: typeof rateLimits;
search: typeof search;
seed: typeof seed;
seedSouls: typeof seedSouls;
skillStatEvents: typeof skillStatEvents;
skills: typeof skills;
soulComments: typeof soulComments;
soulDownloads: typeof soulDownloads;
soulStars: typeof soulStars;
souls: typeof souls;
stars: typeof stars;
statsMaintenance: typeof statsMaintenance;
telemetry: typeof telemetry;
tokens: typeof tokens;
uploads: typeof uploads;
users: typeof users;
vt: typeof vt;
webhooks: typeof webhooks;
}>;
+125
View File
@@ -0,0 +1,125 @@
import { describe, expect, it, vi } from 'vitest'
import type { Id } from './_generated/dataModel'
import {
BANNED_REAUTH_MESSAGE,
DELETED_ACCOUNT_REAUTH_MESSAGE,
handleDeletedUserSignIn,
} from './auth'
function makeCtx({
user,
banRecords,
}: {
user: { deletedAt?: number; deactivatedAt?: number; purgedAt?: number } | null
banRecords?: Array<Record<string, unknown>>
}) {
const query = {
withIndex: vi.fn().mockReturnValue({
collect: vi.fn().mockResolvedValue(banRecords ?? []),
}),
}
const ctx = {
db: {
get: vi.fn().mockResolvedValue(user),
patch: vi.fn().mockResolvedValue(null),
query: vi.fn().mockReturnValue(query),
},
}
return { ctx, query }
}
describe('handleDeletedUserSignIn', () => {
const userId = 'users:1' as Id<'users'>
it('skips when user not found', async () => {
const { ctx } = makeCtx({ user: null })
await handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId })
expect(ctx.db.get).toHaveBeenCalledWith(userId)
expect(ctx.db.query).not.toHaveBeenCalled()
})
it('skips active users', async () => {
const { ctx } = makeCtx({ user: { deletedAt: undefined, deactivatedAt: undefined } })
await handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId })
expect(ctx.db.query).not.toHaveBeenCalled()
expect(ctx.db.patch).not.toHaveBeenCalled()
})
it('blocks sign-in for deactivated users', async () => {
const { ctx } = makeCtx({ user: { deactivatedAt: 123, purgedAt: 123 } })
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(DELETED_ACCOUNT_REAUTH_MESSAGE)
expect(ctx.db.query).not.toHaveBeenCalled()
expect(ctx.db.patch).not.toHaveBeenCalled()
})
it('migrates legacy self-deleted users and blocks sign-in', async () => {
const { ctx } = makeCtx({ user: { deletedAt: 123 }, banRecords: [] })
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(DELETED_ACCOUNT_REAUTH_MESSAGE)
expect(ctx.db.patch).toHaveBeenCalledWith(userId, {
deletedAt: undefined,
deactivatedAt: 123,
purgedAt: 123,
updatedAt: expect.any(Number),
})
})
it('migrates legacy users on fresh login (existingUserId is null)', async () => {
const { ctx } = makeCtx({ user: { deletedAt: 123 }, banRecords: [] })
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: null }),
).rejects.toThrow(DELETED_ACCOUNT_REAUTH_MESSAGE)
expect(ctx.db.patch).toHaveBeenCalledWith(userId, {
deletedAt: undefined,
deactivatedAt: 123,
purgedAt: 123,
updatedAt: expect.any(Number),
})
})
it('skips mutation when existingUserId does not match userId', async () => {
const otherUserId = 'users:999' as Id<'users'>
const { ctx } = makeCtx({ user: { deletedAt: 123 } })
await handleDeletedUserSignIn(ctx as never, { userId, existingUserId: otherUserId })
expect(ctx.db.query).not.toHaveBeenCalled()
expect(ctx.db.patch).not.toHaveBeenCalled()
})
it('blocks banned users with a custom message', async () => {
const { ctx } = makeCtx({ user: { deletedAt: 123 }, banRecords: [{ action: 'user.ban' }] })
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(BANNED_REAUTH_MESSAGE)
expect(ctx.db.patch).not.toHaveBeenCalled()
})
it('blocks users auto-banned for malware', async () => {
const { ctx } = makeCtx({
user: { deletedAt: 123 },
banRecords: [{ action: 'user.autoban.malware' }],
})
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(BANNED_REAUTH_MESSAGE)
expect(ctx.db.patch).not.toHaveBeenCalled()
})
})
+63
View File
@@ -1,5 +1,56 @@
import GitHub from '@auth/core/providers/github'
import { convexAuth } from '@convex-dev/auth/server'
import type { GenericMutationCtx } from 'convex/server'
import { ConvexError } from 'convex/values'
import type { DataModel, Id } from './_generated/dataModel'
export const BANNED_REAUTH_MESSAGE =
'Your account has been banned for uploading malicious skills. If you believe this is a mistake, please contact security@openclaw.ai and we will work with you to restore access.'
export const DELETED_ACCOUNT_REAUTH_MESSAGE =
'This account has been permanently deleted and cannot be restored.'
const REAUTH_BLOCKING_BAN_ACTIONS = new Set(['user.ban', 'user.autoban.malware'])
export async function handleDeletedUserSignIn(
ctx: GenericMutationCtx<DataModel>,
args: { userId: Id<'users'>; existingUserId: Id<'users'> | null },
) {
const user = await ctx.db.get(args.userId)
if (!user?.deletedAt && !user?.deactivatedAt) return
// Verify that the incoming identity matches the existing account to prevent bypass.
if (args.existingUserId && args.existingUserId !== args.userId) {
return
}
if (user.deactivatedAt) {
throw new ConvexError(DELETED_ACCOUNT_REAUTH_MESSAGE)
}
const userId = args.userId
const deletedAt = user.deletedAt ?? Date.now()
const banRecords = await ctx.db
.query('auditLogs')
.withIndex('by_target', (q) => q.eq('targetType', 'user').eq('targetId', userId.toString()))
.collect()
const hasBlockingBan = banRecords.some((record) => REAUTH_BLOCKING_BAN_ACTIONS.has(record.action))
if (hasBlockingBan) {
throw new ConvexError(BANNED_REAUTH_MESSAGE)
}
// Migrate legacy self-deleted accounts (stored in deletedAt) to the new
// irreversible state and reject sign-in.
await ctx.db.patch(userId, {
deletedAt: undefined,
deactivatedAt: deletedAt,
purgedAt: user.purgedAt ?? deletedAt,
updatedAt: Date.now(),
})
throw new ConvexError(DELETED_ACCOUNT_REAUTH_MESSAGE)
}
export const { auth, signIn, signOut, store, isAuthenticated } = convexAuth({
providers: [
@@ -16,4 +67,16 @@ export const { auth, signIn, signOut, store, isAuthenticated } = convexAuth({
},
}),
],
callbacks: {
/**
* Block sign-in for deleted/deactivated users.
*
* Performance note: This callback runs on every OAuth sign-in, but the
* audit log query ONLY executes when a legacy deleted user attempts to sign
* in (user.deletedAt is set). For active users, this is a single field check.
*/
async afterUserCreatedOrUpdated(ctx, args) {
await handleDeletedUserSignIn(ctx, args)
},
},
})
+52
View File
@@ -0,0 +1,52 @@
import type { Id } from './_generated/dataModel'
import type { MutationCtx } from './_generated/server'
import { assertModerator, requireUser } from './lib/access'
import { insertStatEvent } from './skillStatEvents'
export async function addHandler(ctx: MutationCtx, args: { skillId: Id<'skills'>; body: string }) {
const { userId } = await requireUser(ctx)
const body = args.body.trim()
if (!body) throw new Error('Comment body required')
const skill = await ctx.db.get(args.skillId)
if (!skill) throw new Error('Skill not found')
await ctx.db.insert('comments', {
skillId: args.skillId,
userId,
body,
createdAt: Date.now(),
softDeletedAt: undefined,
deletedBy: undefined,
})
await insertStatEvent(ctx, { skillId: skill._id, kind: 'comment' })
}
export async function removeHandler(ctx: MutationCtx, args: { commentId: Id<'comments'> }) {
const { user } = await requireUser(ctx)
const comment = await ctx.db.get(args.commentId)
if (!comment) throw new Error('Comment not found')
if (comment.softDeletedAt) return
const isOwner = comment.userId === user._id
if (!isOwner) {
assertModerator(user)
}
await ctx.db.patch(comment._id, {
softDeletedAt: Date.now(),
deletedBy: user._id,
})
await insertStatEvent(ctx, { skillId: comment.skillId, kind: 'uncomment' })
await ctx.db.insert('auditLogs', {
actorUserId: user._id,
action: 'comment.delete',
targetType: 'comment',
targetId: comment._id,
metadata: { skillId: comment.skillId },
createdAt: Date.now(),
})
}
+127
View File
@@ -0,0 +1,127 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from 'vitest'
vi.mock('./lib/access', () => ({
assertModerator: vi.fn(),
requireUser: vi.fn(),
}))
vi.mock('./skillStatEvents', () => ({
insertStatEvent: vi.fn(),
}))
const { requireUser, assertModerator } = await import('./lib/access')
const { insertStatEvent } = await import('./skillStatEvents')
const { addHandler, removeHandler } = await import('./comments.handlers')
describe('comments mutations', () => {
afterEach(() => {
vi.mocked(assertModerator).mockReset()
vi.mocked(requireUser).mockReset()
vi.mocked(insertStatEvent).mockReset()
})
it('add avoids direct skill patch and records stat event', async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: 'users:1',
user: { _id: 'users:1', role: 'user' },
} as never)
const get = vi.fn().mockResolvedValue({
_id: 'skills:1',
})
const insert = vi.fn()
const patch = vi.fn()
const ctx = { db: { get, insert, patch } } as never
await addHandler(ctx, { skillId: 'skills:1', body: ' hello ' } as never)
expect(patch).not.toHaveBeenCalled()
expect(insertStatEvent).toHaveBeenCalledWith(ctx, {
skillId: 'skills:1',
kind: 'comment',
})
})
it('remove keeps comment soft-delete patch free of updatedAt', async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: 'users:2',
user: { _id: 'users:2', role: 'moderator' },
} as never)
const comment = {
_id: 'comments:1',
skillId: 'skills:1',
userId: 'users:2',
softDeletedAt: undefined,
}
const get = vi.fn(async (id: string) => {
if (id === 'comments:1') return comment
return null
})
const insert = vi.fn()
const patch = vi.fn()
const ctx = { db: { get, insert, patch } } as never
await removeHandler(ctx, { commentId: 'comments:1' } as never)
expect(patch).toHaveBeenCalledTimes(1)
const deletePatch = vi.mocked(patch).mock.calls[0]?.[1] as Record<string, unknown>
expect(deletePatch.updatedAt).toBeUndefined()
expect(insertStatEvent).toHaveBeenCalledWith(ctx, {
skillId: 'skills:1',
kind: 'uncomment',
})
})
it('remove rejects non-owner without moderator permission', async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: 'users:3',
user: { _id: 'users:3', role: 'user' },
} as never)
vi.mocked(assertModerator).mockImplementation(() => {
throw new Error('Moderator role required')
})
const comment = {
_id: 'comments:2',
skillId: 'skills:2',
userId: 'users:9',
softDeletedAt: undefined,
}
const get = vi.fn().mockResolvedValue(comment)
const insert = vi.fn()
const patch = vi.fn()
const ctx = { db: { get, insert, patch } } as never
await expect(removeHandler(ctx, { commentId: 'comments:2' } as never)).rejects.toThrow(
'Moderator role required',
)
expect(patch).not.toHaveBeenCalled()
expect(insertStatEvent).not.toHaveBeenCalled()
})
it('remove no-ops for soft-deleted comment', async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: 'users:4',
user: { _id: 'users:4', role: 'moderator' },
} as never)
const comment = {
_id: 'comments:3',
skillId: 'skills:3',
userId: 'users:4',
softDeletedAt: 123,
}
const get = vi.fn().mockResolvedValue(comment)
const insert = vi.fn()
const patch = vi.fn()
const ctx = { db: { get, insert, patch } } as never
await removeHandler(ctx, { commentId: 'comments:3' } as never)
expect(patch).not.toHaveBeenCalled()
expect(insert).not.toHaveBeenCalled()
expect(insertStatEvent).not.toHaveBeenCalled()
})
})
+13 -63
View File
@@ -1,7 +1,8 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { assertRole, requireUser } from './lib/access'
import { addHandler, removeHandler } from './comments.handlers'
import { type PublicUser, toPublicUser } from './lib/public'
export const listBySkill = query({
args: { skillId: v.id('skills'), limit: v.optional(v.number()) },
@@ -13,75 +14,24 @@ export const listBySkill = query({
.order('desc')
.take(limit)
const results: Array<{ comment: Doc<'comments'>; user: Doc<'users'> | null }> = []
for (const comment of comments) {
if (comment.softDeletedAt) continue
const user = await ctx.db.get(comment.userId)
results.push({ comment, user })
}
return results
const visible = comments.filter((comment) => !comment.softDeletedAt)
return Promise.all(
visible.map(
async (comment): Promise<{ comment: Doc<'comments'>; user: PublicUser | null }> => ({
comment,
user: toPublicUser(await ctx.db.get(comment.userId)),
}),
),
)
},
})
export const add = mutation({
args: { skillId: v.id('skills'), body: v.string() },
handler: async (ctx, args) => {
const { userId } = await requireUser(ctx)
const body = args.body.trim()
if (!body) throw new Error('Comment body required')
const skill = await ctx.db.get(args.skillId)
if (!skill) throw new Error('Skill not found')
await ctx.db.insert('comments', {
skillId: args.skillId,
userId,
body,
createdAt: Date.now(),
softDeletedAt: undefined,
deletedBy: undefined,
})
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, comments: skill.stats.comments + 1 },
updatedAt: Date.now(),
})
},
handler: addHandler,
})
export const remove = mutation({
args: { commentId: v.id('comments') },
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
const comment = await ctx.db.get(args.commentId)
if (!comment) throw new Error('Comment not found')
if (comment.softDeletedAt) return
const isOwner = comment.userId === user._id
if (!isOwner) {
assertRole(user, ['admin', 'moderator'])
}
await ctx.db.patch(comment._id, {
softDeletedAt: Date.now(),
deletedBy: user._id,
})
const skill = await ctx.db.get(comment.skillId)
if (skill) {
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, comments: Math.max(0, skill.stats.comments - 1) },
updatedAt: Date.now(),
})
}
await ctx.db.insert('auditLogs', {
actorUserId: user._id,
action: 'comment.delete',
targetType: 'comment',
targetId: comment._id,
metadata: { skillId: comment.skillId },
createdAt: Date.now(),
})
},
handler: removeHandler,
})
+37
View File
@@ -10,4 +10,41 @@ crons.interval(
{ batchSize: 50, maxBatches: 5 },
)
crons.interval(
'trending-leaderboard',
{ minutes: 60 },
internal.leaderboards.rebuildTrendingLeaderboardInternal,
{ limit: 200 },
)
crons.interval(
'skill-stats-backfill',
{ minutes: 10 },
internal.statsMaintenance.runSkillStatBackfillInternal,
{ batchSize: 200, maxBatches: 5 },
)
crons.interval(
'skill-stat-events',
{ minutes: 5 },
internal.skillStatEvents.processSkillStatEventsAction,
{},
)
crons.interval('vt-pending-scans', { minutes: 5 }, internal.vt.pollPendingScans, { batchSize: 100 })
crons.interval('vt-cache-backfill', { minutes: 30 }, internal.vt.backfillActiveSkillsVTCache, {
batchSize: 100,
})
// Daily re-scan of all active skills at 3am UTC
crons.daily('vt-daily-rescan', { hourUTC: 3, minuteUTC: 0 }, internal.vt.rescanActiveSkills, {})
crons.interval(
'download-dedupe-prune',
{ hours: 24 },
internal.downloads.pruneDownloadDedupesInternal,
{},
)
export default crons
+71 -41
View File
@@ -1,5 +1,6 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { ActionCtx } from './_generated/server'
import { internalAction, internalMutation } from './_generated/server'
import { EMBEDDING_DIMENSIONS } from './lib/embeddings'
import { parseClawdisMetadata, parseFrontmatter } from './lib/skills'
@@ -13,6 +14,17 @@ type SeedSkillSpec = {
rawSkillMd: string
}
type SeedActionArgs = {
reset?: boolean
}
type SeedActionResult = {
ok: true
results: Array<Record<string, unknown> & { slug: string }>
}
type SeedMutationResult = Record<string, unknown>
const SEED_SKILLS: SeedSkillSpec[] = [
{
slug: 'padel',
@@ -237,53 +249,19 @@ function injectMetadata(rawSkillMd: string, metadata: Record<string, unknown>) {
)}${rawSkillMd.slice(frontmatterEnd)}`
}
export const seedNixSkills = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const results = []
for (const spec of SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
const result = await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})
results.push({ slug: spec.slug, ...result })
}
return { ok: true, results }
},
})
export const seedPadelSkill = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const spec = SEED_SKILLS.find((entry) => entry.slug === 'padel')
if (!spec) throw new Error('padel seed spec missing')
async function seedNixSkillsHandler(
ctx: ActionCtx,
args: SeedActionArgs,
): Promise<SeedActionResult> {
const results: Array<Record<string, unknown> & { slug: string }> = []
for (const spec of SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
return ctx.runMutation(internal.devSeed.seedSkillMutation, {
const result: SeedMutationResult = await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
@@ -295,7 +273,51 @@ export const seedPadelSkill = internalAction({
summary: spec.summary,
version: spec.version,
})
results.push({ slug: spec.slug, ...result })
}
return { ok: true, results }
}
export const seedNixSkills: ReturnType<typeof internalAction> = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: seedNixSkillsHandler,
})
async function seedPadelSkillHandler(
ctx: ActionCtx,
args: SeedActionArgs,
): Promise<SeedMutationResult> {
const spec = SEED_SKILLS.find((entry) => entry.slug === 'padel')
if (!spec) throw new Error('padel seed spec missing')
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
return (await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})) as SeedMutationResult
}
export const seedPadelSkill: ReturnType<typeof internalAction> = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: seedPadelSkillHandler,
})
export const seedSkillMutation = internalMutation({
@@ -364,6 +386,10 @@ export const seedSkillMutation = internalMutation({
tags: {},
softDeletedAt: undefined,
badges: { redactionApproved: undefined },
statsDownloads: 0,
statsStars: 0,
statsInstallsCurrent: 0,
statsInstallsAllTime: 0,
stats: {
downloads: 0,
installsCurrent: 0,
@@ -413,6 +439,10 @@ export const seedSkillMutation = internalMutation({
await ctx.db.patch(skillId, {
latestVersionId: versionId,
tags: { latest: versionId },
statsDownloads: 0,
statsStars: 0,
statsInstallsCurrent: 0,
statsInstallsAllTime: 0,
stats: {
downloads: 0,
installsCurrent: 0,
+541
View File
@@ -0,0 +1,541 @@
/**
* Extra seed skills for pagination testing.
*
* This file contains 50 placeholder skills to test pagination behavior.
* Run with: bunx convex run internal.devSeedExtra.seedExtraSkillsInternal
* Or with reset: bunx convex run internal.devSeedExtra.seedExtraSkillsInternal '{"reset": true}'
*/
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction, internalMutation } from './_generated/server'
import { parseClawdisMetadata, parseFrontmatter } from './lib/skills'
type SeedSkillSpec = {
slug: string
displayName: string
summary: string
version: string
metadata: Record<string, unknown>
rawSkillMd: string
}
function makeSkill(
slug: string,
displayName: string,
summary: string,
envVars: string[] = [],
commands: string[] = ['help', 'status', 'run'],
): SeedSkillSpec {
const cliHelp = `${slug} - ${summary}
Usage:
${slug} [command]
Commands:
${commands.map((cmd) => ` ${cmd.padEnd(12)} Run ${cmd} operation`).join('\n')}
Flags:
-h, --help Show help
--json Output as JSON
`
const rawSkillMd = `---
name: ${slug}
description: ${summary}
---
# ${displayName}
## CLI
\`\`\`bash
${commands.map((cmd) => `${slug} ${cmd}`).join('\n')}
\`\`\`
## Usage
Use this skill to ${summary.toLowerCase()}.
`
return {
slug,
displayName,
summary,
version: '0.1.0',
metadata: {
clawdbot: {
nix: {
plugin: `github:example/${slug}`,
systems: ['aarch64-darwin', 'x86_64-linux'],
},
config: {
requiredEnv: envVars,
},
cliHelp,
},
},
rawSkillMd,
}
}
// 50 placeholder skills for pagination testing
const EXTRA_SEED_SKILLS: SeedSkillSpec[] = [
// DevOps & Infrastructure (10)
makeSkill(
'kubectl-helper',
'Kubectl Helper',
'Simplified kubectl commands for common Kubernetes operations.',
['KUBECONFIG'],
['pods', 'logs', 'exec', 'describe', 'apply'],
),
makeSkill(
'terraform-runner',
'Terraform Runner',
'Execute Terraform plans and applies with safety checks.',
['TF_VAR_region', 'AWS_PROFILE'],
['plan', 'apply', 'destroy', 'output', 'state'],
),
makeSkill(
'ansible-exec',
'Ansible Exec',
'Run Ansible playbooks and ad-hoc commands.',
['ANSIBLE_INVENTORY'],
['playbook', 'adhoc', 'inventory', 'facts', 'vault'],
),
makeSkill(
'docker-compose-mgr',
'Docker Compose Manager',
'Manage Docker Compose stacks and services.',
['DOCKER_HOST'],
['up', 'down', 'logs', 'ps', 'restart'],
),
makeSkill(
'k9s-wrapper',
'K9s Wrapper',
'Interactive Kubernetes cluster management via K9s.',
['KUBECONFIG'],
['launch', 'contexts', 'namespaces', 'pods', 'logs'],
),
makeSkill(
'helm-charts',
'Helm Charts',
'Manage Helm chart deployments and releases.',
['KUBECONFIG', 'HELM_REPO'],
['install', 'upgrade', 'rollback', 'list', 'search'],
),
makeSkill(
'prometheus-alerts',
'Prometheus Alerts',
'Query Prometheus metrics and manage alerting rules.',
['PROMETHEUS_URL'],
['query', 'alerts', 'rules', 'targets', 'status'],
),
makeSkill(
'grafana-dash',
'Grafana Dashboards',
'Create and manage Grafana dashboards programmatically.',
['GRAFANA_URL', 'GRAFANA_API_KEY'],
['list', 'export', 'import', 'create', 'delete'],
),
makeSkill(
'nginx-config',
'Nginx Config',
'Generate and validate Nginx configuration files.',
['NGINX_CONF_DIR'],
['generate', 'validate', 'reload', 'test', 'sites'],
),
makeSkill(
'jenkins-jobs',
'Jenkins Jobs',
'Manage Jenkins jobs and pipelines.',
['JENKINS_URL', 'JENKINS_TOKEN'],
['list', 'build', 'status', 'logs', 'config'],
),
// Productivity (8)
makeSkill(
'todoist-sync',
'Todoist Sync',
'Sync and manage Todoist tasks from the command line.',
['TODOIST_API_TOKEN'],
['list', 'add', 'complete', 'projects', 'labels'],
),
makeSkill(
'notion-backup',
'Notion Backup',
'Export and backup Notion workspaces.',
['NOTION_TOKEN'],
['export', 'backup', 'restore', 'pages', 'databases'],
),
makeSkill(
'gcal-manager',
'Google Calendar Manager',
'Manage Google Calendar events and schedules.',
['GOOGLE_CREDENTIALS_FILE'],
['events', 'create', 'delete', 'calendars', 'reminders'],
),
makeSkill(
'time-tracker',
'Time Tracker',
'Track time spent on projects and tasks.',
['TIMETRACK_DB'],
['start', 'stop', 'status', 'report', 'projects'],
),
makeSkill(
'email-digest',
'Email Digest',
'Generate email digests and summaries.',
['IMAP_SERVER', 'IMAP_USER'],
['fetch', 'digest', 'search', 'folders', 'unread'],
),
makeSkill(
'habit-tracker',
'Habit Tracker',
'Track daily habits and streaks.',
['HABITS_DB'],
['log', 'streak', 'stats', 'habits', 'remind'],
),
makeSkill(
'bookmark-sync',
'Bookmark Sync',
'Sync bookmarks across browsers and devices.',
['BOOKMARKS_DIR'],
['sync', 'export', 'import', 'search', 'tags'],
),
makeSkill(
'notes-export',
'Notes Export',
'Export notes to various formats.',
['NOTES_DIR'],
['export', 'convert', 'search', 'list', 'tags'],
),
// Media & Entertainment (6)
makeSkill(
'spotify-ctl',
'Spotify Control',
'Control Spotify playback from the terminal.',
['SPOTIFY_CLIENT_ID', 'SPOTIFY_CLIENT_SECRET'],
['play', 'pause', 'next', 'prev', 'search'],
),
makeSkill(
'plex-manager',
'Plex Manager',
'Manage Plex media libraries and playback.',
['PLEX_URL', 'PLEX_TOKEN'],
['libraries', 'scan', 'search', 'play', 'sessions'],
),
makeSkill(
'ytdl-wrapper',
'YouTube Downloader',
'Download videos from YouTube and other platforms.',
['YTDL_OUTPUT_DIR'],
['download', 'info', 'playlist', 'audio', 'formats'],
),
makeSkill(
'podcast-dl',
'Podcast Downloader',
'Download and manage podcast episodes.',
['PODCAST_DIR'],
['subscribe', 'download', 'list', 'play', 'search'],
),
makeSkill(
'audiobook-player',
'Audiobook Player',
'Manage and play audiobook collections.',
['AUDIOBOOK_DIR'],
['play', 'pause', 'bookmark', 'list', 'progress'],
),
makeSkill(
'music-lib',
'Music Library',
'Organize and query local music libraries.',
['MUSIC_DIR'],
['scan', 'search', 'play', 'playlist', 'stats'],
),
// Smart Home (8)
makeSkill(
'hass-control',
'Home Assistant Control',
'Control Home Assistant entities and automations.',
['HASS_URL', 'HASS_TOKEN'],
['entities', 'services', 'automations', 'scenes', 'history'],
),
makeSkill(
'zigbee-mqtt',
'Zigbee2MQTT',
'Manage Zigbee devices via MQTT.',
['MQTT_BROKER', 'ZIGBEE_TOPIC'],
['devices', 'pair', 'remove', 'rename', 'groups'],
),
makeSkill(
'tasmota-ctl',
'Tasmota Control',
'Control Tasmota-flashed devices.',
['TASMOTA_HOSTS'],
['status', 'power', 'config', 'update', 'backup'],
),
makeSkill(
'esphome-mgr',
'ESPHome Manager',
'Manage ESPHome device configurations.',
['ESPHOME_DIR'],
['compile', 'upload', 'logs', 'dashboard', 'config'],
),
makeSkill(
'mqtt-broker',
'MQTT Broker',
'Interact with MQTT brokers for IoT messaging.',
['MQTT_BROKER', 'MQTT_USER'],
['pub', 'sub', 'topics', 'clients', 'stats'],
),
makeSkill(
'hue-lights',
'Philips Hue',
'Control Philips Hue lights and scenes.',
['HUE_BRIDGE_IP', 'HUE_API_KEY'],
['lights', 'scenes', 'groups', 'schedules', 'sensors'],
),
makeSkill(
'smart-thermo',
'Smart Thermostat',
'Control smart thermostats and HVAC systems.',
['THERMOSTAT_API_KEY'],
['status', 'set', 'schedule', 'history', 'zones'],
),
makeSkill(
'cam-viewer',
'Camera Viewer',
'View and manage security camera feeds.',
['CAMERA_URLS'],
['list', 'snapshot', 'stream', 'record', 'events'],
),
// Finance (5)
makeSkill(
'budget-track',
'Budget Tracker',
'Track budgets and spending across categories.',
['BUDGET_DB'],
['summary', 'add', 'categories', 'report', 'goals'],
),
makeSkill(
'crypto-watch',
'Crypto Watcher',
'Monitor cryptocurrency prices and portfolios.',
['CRYPTO_API_KEY'],
['prices', 'portfolio', 'alerts', 'history', 'convert'],
),
makeSkill(
'stock-alerts',
'Stock Alerts',
'Set up stock price alerts and notifications.',
['STOCK_API_KEY'],
['quote', 'watch', 'alerts', 'portfolio', 'news'],
),
makeSkill(
'expense-cat',
'Expense Categorizer',
'Automatically categorize expenses.',
['EXPENSE_DB'],
['import', 'categorize', 'report', 'rules', 'export'],
),
makeSkill(
'invoice-gen',
'Invoice Generator',
'Generate and manage invoices.',
['INVOICE_DIR', 'COMPANY_INFO'],
['create', 'list', 'send', 'paid', 'overdue'],
),
// Communication (5)
makeSkill(
'slack-bot',
'Slack Bot',
'Interact with Slack channels and messages.',
['SLACK_TOKEN'],
['send', 'channels', 'users', 'search', 'files'],
),
makeSkill(
'discord-mgr',
'Discord Manager',
'Manage Discord servers and messages.',
['DISCORD_TOKEN'],
['send', 'servers', 'channels', 'members', 'roles'],
),
makeSkill(
'telegram-bot',
'Telegram Bot',
'Send and receive Telegram messages.',
['TELEGRAM_BOT_TOKEN'],
['send', 'receive', 'chats', 'files', 'inline'],
),
makeSkill(
'matrix-cli',
'Matrix CLI',
'Interact with Matrix chat rooms.',
['MATRIX_HOMESERVER', 'MATRIX_TOKEN'],
['send', 'rooms', 'join', 'leave', 'sync'],
),
makeSkill(
'irc-bridge',
'IRC Bridge',
'Bridge IRC channels to other platforms.',
['IRC_SERVER', 'IRC_NICK'],
['connect', 'join', 'send', 'channels', 'users'],
),
// Data & Analytics (5)
makeSkill(
'pg-queries',
'PostgreSQL Queries',
'Execute PostgreSQL queries and manage databases.',
['DATABASE_URL'],
['query', 'tables', 'schema', 'backup', 'restore'],
),
makeSkill(
'clickhouse-ql',
'ClickHouse Queries',
'Run ClickHouse analytics queries.',
['CLICKHOUSE_URL'],
['query', 'tables', 'insert', 'system', 'optimize'],
),
makeSkill(
'redis-cli',
'Redis CLI',
'Interact with Redis cache and data structures.',
['REDIS_URL'],
['get', 'set', 'keys', 'info', 'flush'],
),
makeSkill(
'elastic-search',
'Elasticsearch',
'Search and manage Elasticsearch indices.',
['ELASTICSEARCH_URL'],
['search', 'index', 'mapping', 'cluster', 'aliases'],
),
makeSkill(
'mongo-shell',
'MongoDB Shell',
'Query and manage MongoDB collections.',
['MONGODB_URI'],
['find', 'insert', 'update', 'delete', 'aggregate'],
),
// Security (3)
makeSkill(
'vault-secrets',
'Vault Secrets',
'Manage secrets in HashiCorp Vault.',
['VAULT_ADDR', 'VAULT_TOKEN'],
['read', 'write', 'list', 'delete', 'seal'],
),
makeSkill(
'gpg-keys',
'GPG Keys',
'Manage GPG keys and encryption.',
['GNUPGHOME'],
['list', 'generate', 'export', 'import', 'encrypt'],
),
makeSkill(
'ssh-rotate',
'SSH Key Rotator',
'Rotate and manage SSH keys.',
['SSH_KEY_DIR'],
['generate', 'rotate', 'deploy', 'list', 'revoke'],
),
]
function injectMetadata(rawSkillMd: string, metadata: Record<string, unknown>) {
const frontmatterEnd = rawSkillMd.indexOf('\n---', 3)
if (frontmatterEnd === -1) return rawSkillMd
return `${rawSkillMd.slice(0, frontmatterEnd)}\nmetadata: ${JSON.stringify(
metadata,
)}${rawSkillMd.slice(frontmatterEnd)}`
}
function randomStats() {
return {
downloads: Math.floor(Math.random() * 5000),
stars: Math.floor(Math.random() * 500),
installsCurrent: Math.floor(Math.random() * 200),
installsAllTime: Math.floor(Math.random() * 1000),
}
}
export const applyRandomStats = internalMutation({
args: {
skillId: v.id('skills'),
stats: v.object({
downloads: v.number(),
stars: v.number(),
installsCurrent: v.number(),
installsAllTime: v.number(),
}),
},
handler: async (ctx, args) => {
await ctx.db.patch(args.skillId, {
statsDownloads: args.stats.downloads,
statsStars: args.stats.stars,
statsInstallsCurrent: args.stats.installsCurrent,
statsInstallsAllTime: args.stats.installsAllTime,
stats: {
downloads: args.stats.downloads,
stars: args.stats.stars,
installsCurrent: args.stats.installsCurrent,
installsAllTime: args.stats.installsAllTime,
versions: 1,
comments: 0,
},
})
},
})
export const seedExtraSkillsInternal = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: async (ctx: ActionCtx, args) => {
const results: Array<{ slug: string; ok: boolean; skipped?: boolean }> = []
for (const spec of EXTRA_SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
const result = (await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})) as { ok: boolean; skipped?: boolean; skillId?: string }
// Apply random stats after creation (only if not skipped)
if (result.skillId && !result.skipped) {
const stats = randomStats()
await ctx.runMutation(internal.devSeedExtra.applyRandomStats, {
skillId: result.skillId as Id<'skills'>,
stats,
})
}
results.push({ slug: spec.slug, ok: result.ok, skipped: result.skipped })
}
const created = results.filter((r) => !r.skipped).length
const skipped = results.filter((r) => r.skipped).length
return { ok: true, total: results.length, created, skipped }
},
})
+43
View File
@@ -0,0 +1,43 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { __test } from './downloads'
describe('downloads helpers', () => {
afterEach(() => {
vi.unstubAllEnvs()
})
it('calculates hour start boundaries', () => {
const hour = 3_600_000
expect(__test.getHourStart(0)).toBe(0)
expect(__test.getHourStart(hour - 1)).toBe(0)
expect(__test.getHourStart(hour)).toBe(hour)
expect(__test.getHourStart(hour + 1)).toBe(hour)
})
it('prefers user identity when token user exists', () => {
const request = new Request('https://example.com', {
headers: { 'cf-connecting-ip': '1.2.3.4' },
})
expect(__test.getDownloadIdentityValue(request, 'users_123')).toBe('user:users_123')
})
it('uses cf-connecting-ip for anonymous identity', () => {
const request = new Request('https://example.com', {
headers: { 'cf-connecting-ip': '1.2.3.4' },
})
expect(__test.getDownloadIdentityValue(request, null)).toBe('ip:1.2.3.4')
})
it('falls back to forwarded ip when explicitly enabled', () => {
vi.stubEnv('TRUST_FORWARDED_IPS', 'true')
const request = new Request('https://example.com', {
headers: { 'x-forwarded-for': '10.0.0.1, 10.0.0.2' },
})
expect(__test.getDownloadIdentityValue(request, null)).toBe('ip:10.0.0.1')
})
it('returns null when user and ip are missing', () => {
const request = new Request('https://example.com')
expect(__test.getDownloadIdentityValue(request, null)).toBeNull()
})
})
+171 -23
View File
@@ -1,7 +1,17 @@
import { v } from 'convex/values'
import { zipSync } from 'fflate'
import { api } from './_generated/api'
import { httpAction, mutation } from './_generated/server'
import { api, internal } from './_generated/api'
import { httpAction, internalMutation, mutation } from './_generated/server'
import { getOptionalApiTokenUserId } from './lib/apiTokenAuth'
import { applyRateLimit, getClientIp } from './lib/httpRateLimit'
import { corsHeaders, mergeHeaders } from './lib/httpHeaders'
import { buildDeterministicZip } from './lib/skillZip'
import { hashToken } from './lib/tokens'
import { insertStatEvent } from './skillStatEvents'
const HOUR_MS = 3_600_000
const DEDUPE_RETENTION_MS = 7 * 24 * HOUR_MS
const PRUNE_BATCH_SIZE = 200
const PRUNE_MAX_BATCHES = 50
export const downloadZip = httpAction(async (ctx, request) => {
const url = new URL(request.url)
@@ -10,12 +20,54 @@ export const downloadZip = httpAction(async (ctx, request) => {
const tagParam = url.searchParams.get('tag')?.trim()
if (!slug) {
return new Response('Missing slug', { status: 400 })
return new Response('Missing slug', {
status: 400,
headers: corsHeaders(),
})
}
const rate = await applyRateLimit(ctx, request, 'download')
if (!rate.ok) return rate.response
const skillResult = await ctx.runQuery(api.skills.getBySlug, { slug })
if (!skillResult?.skill) {
return new Response('Skill not found', { status: 404 })
return new Response('Skill not found', {
status: 404,
headers: mergeHeaders(rate.headers, corsHeaders()),
})
}
// Block downloads based on moderation status.
const mod = skillResult.moderationInfo
if (mod?.isMalwareBlocked) {
return new Response(
'Blocked: this skill has been flagged as malicious by VirusTotal and cannot be downloaded.',
{
status: 403,
headers: mergeHeaders(rate.headers, corsHeaders()),
},
)
}
if (mod?.isPendingScan) {
return new Response(
'This skill is pending a security scan by VirusTotal. Please try again in a few minutes.',
{
status: 423,
headers: mergeHeaders(rate.headers, corsHeaders()),
},
)
}
if (mod?.isRemoved) {
return new Response('This skill has been removed by a moderator.', {
status: 410,
headers: mergeHeaders(rate.headers, corsHeaders()),
})
}
if (mod?.isHiddenByMod) {
return new Response('This skill is currently unavailable.', {
status: 403,
headers: mergeHeaders(rate.headers, corsHeaders()),
})
}
const skill = skillResult.skill
@@ -34,44 +86,140 @@ export const downloadZip = httpAction(async (ctx, request) => {
}
if (!version) {
return new Response('Version not found', { status: 404 })
return new Response('Version not found', {
status: 404,
headers: mergeHeaders(rate.headers, corsHeaders()),
})
}
if (version.softDeletedAt) {
return new Response('Version not available', { status: 410 })
return new Response('Version not available', {
status: 410,
headers: mergeHeaders(rate.headers, corsHeaders()),
})
}
const files: Record<string, Uint8Array> = {}
const entries: Array<{ path: string; bytes: Uint8Array }> = []
for (const file of version.files) {
const blob = await ctx.storage.get(file.storageId)
if (!blob) continue
const buffer = new Uint8Array(await blob.arrayBuffer())
files[file.path] = buffer
entries.push({ path: file.path, bytes: buffer })
}
const zipData = zipSync(files, { level: 6 })
const zipArray = Uint8Array.from(zipData)
const zipArray = buildDeterministicZip(entries, {
ownerId: String(skill.ownerUserId),
slug: skill.slug,
version: version.version,
publishedAt: version.createdAt,
})
const zipBlob = new Blob([zipArray], { type: 'application/zip' })
await ctx.runMutation(api.downloads.increment, { skillId: skill._id })
try {
const userId = await getOptionalApiTokenUserId(ctx, request)
const identity = getDownloadIdentityValue(request, userId ? String(userId) : null)
if (identity) {
await ctx.runMutation(internal.downloads.recordDownloadInternal, {
skillId: skill._id,
identityHash: await hashToken(identity),
hourStart: getHourStart(Date.now()),
})
}
} catch {
// Best-effort metric path; do not fail downloads.
}
return new Response(zipBlob, {
status: 200,
headers: {
'Content-Type': 'application/zip',
'Content-Disposition': `attachment; filename="${slug}-${version.version}.zip"`,
'Cache-Control': 'private, max-age=60',
},
headers: mergeHeaders(
rate.headers,
{
'Content-Type': 'application/zip',
'Content-Disposition': `attachment; filename="${slug}-${version.version}.zip"`,
'Cache-Control': 'private, max-age=60',
},
corsHeaders(),
),
})
})
export const increment = mutation({
args: { skillId: v.id('skills') },
handler: async (ctx, args) => {
const skill = await ctx.db.get(args.skillId)
if (!skill) return
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, downloads: skill.stats.downloads + 1 },
updatedAt: Date.now(),
// Skip db.get to avoid adding the skill doc to the read set.
// The calling HTTP action already validated the skill exists,
// and the stat processor handles deleted skills gracefully.
await insertStatEvent(ctx, {
skillId: args.skillId,
kind: 'download',
})
},
})
export const recordDownloadInternal = internalMutation({
args: {
skillId: v.id('skills'),
identityHash: v.string(),
hourStart: v.number(),
},
handler: async (ctx, args) => {
const existing = await ctx.db
.query('downloadDedupes')
.withIndex('by_skill_identity_hour', (q) =>
q
.eq('skillId', args.skillId)
.eq('identityHash', args.identityHash)
.eq('hourStart', args.hourStart),
)
.unique()
if (existing) return
await ctx.db.insert('downloadDedupes', {
skillId: args.skillId,
identityHash: args.identityHash,
hourStart: args.hourStart,
createdAt: Date.now(),
})
await insertStatEvent(ctx, {
skillId: args.skillId,
kind: 'download',
})
},
})
export const pruneDownloadDedupesInternal = internalMutation({
args: {},
handler: async (ctx) => {
const cutoff = Date.now() - DEDUPE_RETENTION_MS
for (let batches = 0; batches < PRUNE_MAX_BATCHES; batches += 1) {
const stale = await ctx.db
.query('downloadDedupes')
.withIndex('by_hour', (q) => q.lt('hourStart', cutoff))
.take(PRUNE_BATCH_SIZE)
if (stale.length === 0) break
for (const entry of stale) {
await ctx.db.delete(entry._id)
}
if (stale.length < PRUNE_BATCH_SIZE) break
}
},
})
export function getHourStart(timestamp: number) {
return Math.floor(timestamp / HOUR_MS) * HOUR_MS
}
export function getDownloadIdentityValue(request: Request, userId: string | null) {
if (userId) return `user:${userId}`
const ip = getClientIp(request)
if (!ip) return null
return `ip:${ip}`
}
export const __test = {
getHourStart,
getDownloadIdentityValue,
}
+1 -1
View File
@@ -78,7 +78,7 @@ export const getGitHubBackupPageInternal = internalQuery({
}
const owner = await ctx.db.get(skill.ownerUserId)
if (!owner || owner.deletedAt) {
if (!owner || owner.deletedAt || owner.deactivatedAt) {
items.push({ kind: 'missingOwner', skillId: skill._id, ownerUserId: skill.ownerUserId })
continue
}
+1 -1
View File
@@ -78,7 +78,7 @@ export const getGitHubSoulBackupPageInternal = internalQuery({
}
const owner = await ctx.db.get(soul.ownerUserId)
if (!owner || owner.deletedAt) {
if (!owner || owner.deletedAt || owner.deactivatedAt) {
items.push({ kind: 'missingOwner', soulId: soul._id, ownerUserId: soul.ownerUserId })
continue
}
+36 -1
View File
@@ -1,4 +1,4 @@
import { ApiRoutes, LegacyApiRoutes } from 'clawdhub-schema'
import { ApiRoutes, LegacyApiRoutes } from 'clawhub-schema'
import { httpRouter } from 'convex/server'
import { auth } from './auth'
import { downloadZip } from './downloads'
@@ -26,8 +26,13 @@ import {
soulsDeleteRouterV1Http,
soulsGetRouterV1Http,
soulsPostRouterV1Http,
starsDeleteRouterV1Http,
starsPostRouterV1Http,
usersListV1Http,
usersPostRouterV1Http,
whoamiV1Http,
} from './httpApiV1'
import { preflightHandler } from './httpPreflight'
const http = httpRouter()
@@ -81,12 +86,36 @@ http.route({
handler: skillsDeleteRouterV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.stars}/`,
method: 'POST',
handler: starsPostRouterV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.stars}/`,
method: 'DELETE',
handler: starsDeleteRouterV1Http,
})
http.route({
path: ApiRoutes.whoami,
method: 'GET',
handler: whoamiV1Http,
})
http.route({
pathPrefix: `${ApiRoutes.users}/`,
method: 'POST',
handler: usersPostRouterV1Http,
})
http.route({
path: ApiRoutes.users,
method: 'GET',
handler: usersListV1Http,
})
http.route({
path: ApiRoutes.souls,
method: 'GET',
@@ -117,6 +146,12 @@ http.route({
handler: soulsDeleteRouterV1Http,
})
http.route({
pathPrefix: '/api/',
method: 'OPTIONS',
handler: preflightHandler,
})
// TODO: remove legacy /api routes after deprecation window.
http.route({
path: LegacyApiRoutes.download,
+22 -7
View File
@@ -11,7 +11,7 @@ vi.mock('./skills', () => ({
const { requireApiTokenUser } = await import('./lib/apiTokenAuth')
const { publishVersionForUser } = await import('./skills')
const { __handlers, cliSkillDeleteHttp, cliSkillUndeleteHttp } = await import('./httpApi')
const { __handlers } = await import('./httpApi')
const { hashSkillFiles } = await import('./lib/skills')
function makeCtx(partial: Record<string, unknown>) {
@@ -33,7 +33,7 @@ describe('httpApi handlers', () => {
expect(await response.json()).toEqual({ results: [] })
})
it('searchSkillsHttp forwards args', async () => {
it('searchSkillsHttp forwards args (approvedOnly alias)', async () => {
const runAction = vi.fn().mockResolvedValue([
{
score: 1,
@@ -48,14 +48,27 @@ describe('httpApi handlers', () => {
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
limit: 5,
approvedOnly: true,
highlightedOnly: true,
})
expect(response.status).toBe(200)
const json = await response.json()
expect(json.results[0].slug).toBe('a')
})
it('searchSkillsHttp omits approvedOnly when false', async () => {
it('searchSkillsHttp forwards highlightedOnly', async () => {
const runAction = vi.fn().mockResolvedValue([])
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request('https://example.com/api/search?q=test&highlightedOnly=true'),
)
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
limit: undefined,
highlightedOnly: true,
})
})
it('searchSkillsHttp omits highlightedOnly when approvedOnly is false', async () => {
const runAction = vi.fn().mockResolvedValue([])
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
@@ -64,7 +77,7 @@ describe('httpApi handlers', () => {
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
limit: undefined,
approvedOnly: undefined,
highlightedOnly: undefined,
})
})
@@ -416,13 +429,14 @@ describe('httpApi handlers', () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const response = await cliSkillUndeleteHttp(
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/skill/undelete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
}),
false,
)
expect(response.status).toBe(200)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
@@ -437,13 +451,14 @@ describe('httpApi handlers', () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const response = await cliSkillDeleteHttp(
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/skill/delete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
}),
true,
)
expect(response.status).toBe(200)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
+18 -10
View File
@@ -5,12 +5,13 @@ import {
CliSkillDeleteRequestSchema,
CliTelemetrySyncRequestSchema,
parseArk,
} from 'clawdhub-schema'
} from 'clawhub-schema'
import { api, internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { httpAction } from './_generated/server'
import { requireApiTokenUser } from './lib/apiTokenAuth'
import { corsHeaders, mergeHeaders } from './lib/httpHeaders'
import { publishVersionForUser } from './skills'
type SearchSkillEntry = {
@@ -44,13 +45,14 @@ async function searchSkillsHandler(ctx: ActionCtx, request: Request) {
const query = url.searchParams.get('q')?.trim() ?? ''
const limit = toOptionalNumber(url.searchParams.get('limit'))
const approvedOnly = url.searchParams.get('approvedOnly') === 'true'
const highlightedOnly = url.searchParams.get('highlightedOnly') === 'true' || approvedOnly
if (!query) return json({ results: [] })
const results = (await ctx.runAction(api.search.searchSkills, {
query,
limit,
approvedOnly: approvedOnly || undefined,
highlightedOnly: highlightedOnly || undefined,
})) as SearchSkillEntry[]
return json({
@@ -240,20 +242,26 @@ export const cliTelemetrySyncHttp = httpAction(cliTelemetrySyncHandler)
function json(value: unknown, status = 200) {
return new Response(JSON.stringify(value), {
status,
headers: {
'Content-Type': 'application/json',
'Cache-Control': 'no-store',
},
headers: mergeHeaders(
{
'Content-Type': 'application/json',
'Cache-Control': 'no-store',
},
corsHeaders(),
),
})
}
function text(value: string, status: number) {
return new Response(value, {
status,
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'no-store',
},
headers: mergeHeaders(
{
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'no-store',
},
corsHeaders(),
),
})
}
+530 -5
View File
@@ -3,20 +3,46 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'
vi.mock('./lib/apiTokenAuth', () => ({
requireApiTokenUser: vi.fn(),
getOptionalApiTokenUserId: vi.fn(),
}))
vi.mock('./skills', () => ({
publishVersionForUser: vi.fn(),
}))
const { requireApiTokenUser } = await import('./lib/apiTokenAuth')
const { getOptionalApiTokenUserId, requireApiTokenUser } = await import('./lib/apiTokenAuth')
const { publishVersionForUser } = await import('./skills')
const { __handlers } = await import('./httpApiV1')
type ActionCtx = import('./_generated/server').ActionCtx
type RateLimitArgs = { key: string; limit: number; windowMs: number }
function isRateLimitArgs(args: unknown): args is RateLimitArgs {
if (!args || typeof args !== 'object') return false
const value = args as Record<string, unknown>
return (
typeof value.key === 'string' &&
typeof value.limit === 'number' &&
typeof value.windowMs === 'number'
)
}
function makeCtx(partial: Record<string, unknown>) {
return partial as unknown as ActionCtx
const partialRunQuery =
typeof partial.runQuery === 'function'
? (partial.runQuery as (query: unknown, args: Record<string, unknown>) => unknown)
: null
const runQuery = vi.fn(async (query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate()
return partialRunQuery ? await partialRunQuery(query, args) : null
})
const runMutation =
typeof partial.runMutation === 'function'
? partial.runMutation
: vi.fn().mockResolvedValue(okRate())
return { ...partial, runQuery, runMutation } as unknown as ActionCtx
}
const okRate = () => ({
@@ -34,6 +60,8 @@ const blockedRate = () => ({
})
beforeEach(() => {
vi.mocked(getOptionalApiTokenUserId).mockReset()
vi.mocked(getOptionalApiTokenUserId).mockResolvedValue(null)
vi.mocked(requireApiTokenUser).mockReset()
vi.mocked(publishVersionForUser).mockReset()
})
@@ -123,7 +151,7 @@ describe('httpApiV1 handlers', () => {
expect(json.match.version).toBe('1.0.0')
})
it('lists skills with resolved tags', async () => {
it('lists skills with resolved tags using batch query', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('cursor' in args || 'limit' in args) {
return {
@@ -145,7 +173,10 @@ describe('httpApiV1 handlers', () => {
nextCursor: null,
}
}
if ('versionId' in args) return { version: '1.0.0' }
// Batch query: versionIds (plural)
if ('versionIds' in args) {
return [{ _id: 'versions:1', version: '1.0.0', softDeletedAt: undefined }]
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
@@ -158,6 +189,234 @@ describe('httpApiV1 handlers', () => {
expect(json.items[0].tags.latest).toBe('1.0.0')
})
it('batches tag resolution across multiple skills into single query', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('cursor' in args || 'limit' in args) {
return {
items: [
{
skill: {
_id: 'skills:1',
slug: 'skill-a',
displayName: 'Skill A',
summary: 's',
tags: { latest: 'versions:1', stable: 'versions:2' },
stats: { downloads: 0, stars: 0, versions: 2, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '2.0.0', createdAt: 3, changelog: 'c' },
},
{
skill: {
_id: 'skills:2',
slug: 'skill-b',
displayName: 'Skill B',
summary: 's',
tags: { latest: 'versions:3' },
stats: { downloads: 0, stars: 0, versions: 1, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '1.0.0', createdAt: 3, changelog: 'c' },
},
],
nextCursor: null,
}
}
// Batch query should receive all version IDs from all skills
if ('versionIds' in args) {
const ids = args.versionIds as string[]
expect(ids).toHaveLength(3)
expect(ids).toContain('versions:1')
expect(ids).toContain('versions:2')
expect(ids).toContain('versions:3')
return [
{ _id: 'versions:1', version: '2.0.0', softDeletedAt: undefined },
{ _id: 'versions:2', version: '1.0.0', softDeletedAt: undefined },
{ _id: 'versions:3', version: '1.0.0', softDeletedAt: undefined },
]
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.listSkillsV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills'),
)
expect(response.status).toBe(200)
const json = await response.json()
// Verify tags are correctly resolved for each skill
expect(json.items[0].tags.latest).toBe('2.0.0')
expect(json.items[0].tags.stable).toBe('1.0.0')
expect(json.items[1].tags.latest).toBe('1.0.0')
// Verify batch query was called exactly once (not per-tag)
const batchCalls = runQuery.mock.calls.filter(
([, args]) => args && 'versionIds' in (args as Record<string, unknown>),
)
expect(batchCalls).toHaveLength(1)
})
it('lists souls with resolved tags using batch query', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('cursor' in args || 'limit' in args) {
return {
items: [
{
soul: {
_id: 'souls:1',
slug: 'demo-soul',
displayName: 'Demo Soul',
summary: 's',
tags: { latest: 'soulVersions:1' },
stats: { downloads: 0, stars: 0, versions: 1, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '1.0.0', createdAt: 3, changelog: 'c' },
},
],
nextCursor: null,
}
}
if ('versionIds' in args) {
return [{ _id: 'soulVersions:1', version: '1.0.0', softDeletedAt: undefined }]
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.listSoulsV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/souls?limit=1'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.items[0].tags.latest).toBe('1.0.0')
})
it('batches tag resolution across multiple souls into single query', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('cursor' in args || 'limit' in args) {
return {
items: [
{
soul: {
_id: 'souls:1',
slug: 'soul-a',
displayName: 'Soul A',
summary: 's',
tags: { latest: 'soulVersions:1', stable: 'soulVersions:2' },
stats: { downloads: 0, stars: 0, versions: 2, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '2.0.0', createdAt: 3, changelog: 'c' },
},
{
soul: {
_id: 'souls:2',
slug: 'soul-b',
displayName: 'Soul B',
summary: 's',
tags: { latest: 'soulVersions:3' },
stats: { downloads: 0, stars: 0, versions: 1, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '1.0.0', createdAt: 3, changelog: 'c' },
},
],
nextCursor: null,
}
}
if ('versionIds' in args) {
const ids = args.versionIds as string[]
expect(ids).toHaveLength(3)
expect(ids).toContain('soulVersions:1')
expect(ids).toContain('soulVersions:2')
expect(ids).toContain('soulVersions:3')
return [
{ _id: 'soulVersions:1', version: '2.0.0', softDeletedAt: undefined },
{ _id: 'soulVersions:2', version: '1.0.0', softDeletedAt: undefined },
{ _id: 'soulVersions:3', version: '1.0.0', softDeletedAt: undefined },
]
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.listSoulsV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/souls'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.items[0].tags.latest).toBe('2.0.0')
expect(json.items[0].tags.stable).toBe('1.0.0')
expect(json.items[1].tags.latest).toBe('1.0.0')
const batchCalls = runQuery.mock.calls.filter(
([, args]) => args && 'versionIds' in (args as Record<string, unknown>),
)
expect(batchCalls).toHaveLength(1)
})
it('souls get resolves tags using batch query', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('slug' in args) {
return {
soul: {
_id: 'souls:1',
slug: 'demo-soul',
displayName: 'Demo Soul',
summary: 's',
tags: { latest: 'soulVersions:1' },
stats: { downloads: 0, stars: 0, versions: 1, comments: 0 },
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '1.0.0', createdAt: 3, changelog: 'c' },
owner: null,
}
}
if ('versionIds' in args) {
return [{ _id: 'soulVersions:1', version: '1.0.0', softDeletedAt: undefined }]
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.soulsGetRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/souls/demo-soul'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.soul.tags.latest).toBe('1.0.0')
})
it('lists skills supports sort aliases', async () => {
const checks: Array<[string, string]> = [
['rating', 'stars'],
['installs', 'installsCurrent'],
['installs-all-time', 'installsAllTime'],
['trending', 'trending'],
]
for (const [input, expected] of checks) {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('sort' in args || 'cursor' in args || 'limit' in args) {
expect(args.sort).toBe(expected)
return { items: [], nextCursor: null }
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.listSkillsV1Handler(
makeCtx({ runQuery, runMutation }),
new Request(`https://example.com/api/v1/skills?sort=${input}`),
)
expect(response.status).toBe(200)
}
})
it('get skill returns 404 when missing', async () => {
const runQuery = vi.fn().mockResolvedValue(null)
const runMutation = vi.fn().mockResolvedValue(okRate())
@@ -168,6 +427,52 @@ describe('httpApiV1 handlers', () => {
expect(response.status).toBe(404)
})
it('get skill returns pending-scan message for owner api token', async () => {
vi.mocked(getOptionalApiTokenUserId).mockResolvedValue('users:1' as never)
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('slug' in args) {
return {
_id: 'skills:1',
slug: 'demo',
ownerUserId: 'users:1',
moderationStatus: 'hidden',
moderationReason: 'pending.scan',
}
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills/demo'),
)
expect(response.status).toBe(423)
expect(await response.text()).toContain('security scan is pending')
})
it('get skill returns undelete hint for owner soft-deleted skill', async () => {
vi.mocked(getOptionalApiTokenUserId).mockResolvedValue('users:1' as never)
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('slug' in args) {
return {
_id: 'skills:1',
slug: 'demo',
ownerUserId: 'users:1',
softDeletedAt: 1,
moderationStatus: 'hidden',
}
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.skillsGetRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/skills/demo'),
)
expect(response.status).toBe(410)
expect(await response.text()).toContain('clawhub undelete demo')
})
it('get skill returns payload', async () => {
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('slug' in args) {
@@ -191,7 +496,10 @@ describe('httpApiV1 handlers', () => {
owner: { handle: 'p', displayName: 'Peter', image: null },
}
}
if ('versionId' in args) return { version: '1.0.0' }
// Batch query for tag resolution
if ('versionIds' in args) {
return [{ _id: 'versions:1', version: '1.0.0', softDeletedAt: undefined }]
}
return null
})
const runMutation = vi.fn().mockResolvedValue(okRate())
@@ -498,4 +806,221 @@ describe('httpApiV1 handlers', () => {
)
expect(response2.status).toBe(200)
})
it('delete/undelete map forbidden/not-found/unknown to 403/404/500', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runMutationForbidden = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('key' in args) return okRate()
throw new Error('Forbidden')
})
const forbidden = await __handlers.skillsDeleteRouterV1Handler(
makeCtx({ runMutation: runMutationForbidden }),
new Request('https://example.com/api/v1/skills/demo', {
method: 'DELETE',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(forbidden.status).toBe(403)
expect(await forbidden.text()).toBe('Forbidden')
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runMutationNotFound = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('key' in args) return okRate()
throw new Error('Skill not found')
})
const notFound = await __handlers.skillsPostRouterV1Handler(
makeCtx({ runMutation: runMutationNotFound }),
new Request('https://example.com/api/v1/skills/demo/undelete', {
method: 'POST',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(notFound.status).toBe(404)
expect(await notFound.text()).toBe('Skill not found')
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runMutationUnknown = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if ('key' in args) return okRate()
throw new Error('boom')
})
const unknown = await __handlers.soulsDeleteRouterV1Handler(
makeCtx({ runMutation: runMutationUnknown }),
new Request('https://example.com/api/v1/souls/demo-soul', {
method: 'DELETE',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(unknown.status).toBe(500)
expect(await unknown.text()).toBe('Internal Server Error')
})
it('ban user requires auth', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.usersPostRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/users/ban', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ handle: 'demo' }),
}),
)
expect(response.status).toBe(401)
})
it('ban user succeeds with handle', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runQuery = vi.fn().mockResolvedValue({ _id: 'users:2' })
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, alreadyBanned: false, deletedSkills: 2 })
const response = await __handlers.usersPostRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/users/ban', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ handle: 'demo' }),
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.deletedSkills).toBe(2)
})
it('ban user forwards reason', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runQuery = vi.fn().mockResolvedValue({ _id: 'users:2' })
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, alreadyBanned: false, deletedSkills: 0 })
await __handlers.usersPostRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/users/ban', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ handle: 'demo', reason: 'malware' }),
}),
)
expect(runMutation).toHaveBeenCalledWith(
expect.anything(),
expect.objectContaining({
actorUserId: 'users:1',
targetUserId: 'users:2',
reason: 'malware',
}),
)
})
it('set role requires auth', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.usersPostRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/users/role', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ handle: 'demo', role: 'moderator' }),
}),
)
expect(response.status).toBe(401)
})
it('set role succeeds with handle', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runQuery = vi.fn().mockResolvedValue({ _id: 'users:2' })
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, role: 'moderator' })
const response = await __handlers.usersPostRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/users/role', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ handle: 'demo', role: 'moderator' }),
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.role).toBe('moderator')
})
it('stars require auth', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const runMutation = vi.fn().mockResolvedValue(okRate())
const response = await __handlers.starsPostRouterV1Handler(
makeCtx({ runMutation }),
new Request('https://example.com/api/v1/stars/demo', { method: 'POST' }),
)
expect(response.status).toBe(401)
})
it('stars add succeeds', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runQuery = vi.fn().mockResolvedValue({ _id: 'skills:1' })
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, starred: true, alreadyStarred: false })
const response = await __handlers.starsPostRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/stars/demo', {
method: 'POST',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.ok).toBe(true)
expect(json.starred).toBe(true)
})
it('stars delete succeeds', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValue({
userId: 'users:1',
user: { handle: 'p' },
} as never)
const runQuery = vi.fn().mockResolvedValue({ _id: 'skills:1' })
const runMutation = vi
.fn()
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce(okRate())
.mockResolvedValueOnce({ ok: true, unstarred: true, alreadyUnstarred: false })
const response = await __handlers.starsDeleteRouterV1Handler(
makeCtx({ runQuery, runMutation }),
new Request('https://example.com/api/v1/stars/demo', {
method: 'DELETE',
headers: { Authorization: 'Bearer clh_test' },
}),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.ok).toBe(true)
expect(json.unstarred).toBe(true)
})
})
+464 -177
View File
@@ -1,18 +1,14 @@
import { CliPublishRequestSchema, parseArk } from 'clawdhub-schema'
import { CliPublishRequestSchema, parseArk } from 'clawhub-schema'
import { api, internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { httpAction } from './_generated/server'
import { requireApiTokenUser } from './lib/apiTokenAuth'
import { hashToken } from './lib/tokens'
import { getOptionalApiTokenUserId, requireApiTokenUser } from './lib/apiTokenAuth'
import { applyRateLimit, parseBearerToken } from './lib/httpRateLimit'
import { corsHeaders, mergeHeaders } from './lib/httpHeaders'
import { publishVersionForUser } from './skills'
import { publishSoulVersionForUser } from './souls'
const RATE_LIMIT_WINDOW_MS = 60_000
const RATE_LIMITS = {
read: { ip: 120, key: 600 },
write: { ip: 30, key: 120 },
} as const
const MAX_RAW_FILE_BYTES = 200 * 1024
type SearchSkillEntry = {
@@ -44,6 +40,9 @@ type ListSkillsResult = {
nextCursor: string | null
}
type SkillFile = Doc<'skillVersions'>['files'][number]
type SoulFile = Doc<'soulVersions'>['files'][number]
type GetBySlugResult = {
skill: {
_id: Id<'skills'>
@@ -56,7 +55,15 @@ type GetBySlugResult = {
updatedAt: number
} | null
latestVersion: Doc<'skillVersions'> | null
owner: { handle?: string; displayName?: string; image?: string } | null
owner: { _id: Id<'users'>; handle?: string; displayName?: string; image?: string } | null
moderationInfo?: {
isPendingScan: boolean
isMalwareBlocked: boolean
isSuspicious: boolean
isHiddenByMod: boolean
isRemoved: boolean
reason?: string
} | null
} | null
type ListVersionsResult = {
@@ -191,35 +198,39 @@ async function listSkillsV1Handler(ctx: ActionCtx, request: Request) {
const url = new URL(request.url)
const limit = toOptionalNumber(url.searchParams.get('limit'))
const cursor = url.searchParams.get('cursor')?.trim() || undefined
const rawCursor = url.searchParams.get('cursor')?.trim() || undefined
const sort = parseListSort(url.searchParams.get('sort'))
const cursor = sort === 'trending' ? undefined : rawCursor
const result = (await ctx.runQuery(api.skills.listPublicPage, {
limit,
cursor,
sort,
})) as ListSkillsResult
const items = await Promise.all(
result.items.map(async (item) => {
const tags = await resolveTags(ctx, item.skill.tags)
return {
slug: item.skill.slug,
displayName: item.skill.displayName,
summary: item.skill.summary ?? null,
tags,
stats: item.skill.stats,
createdAt: item.skill.createdAt,
updatedAt: item.skill.updatedAt,
latestVersion: item.latestVersion
? {
version: item.latestVersion.version,
createdAt: item.latestVersion.createdAt,
changelog: item.latestVersion.changelog,
}
: null,
}
}),
// Batch resolve all tags in a single query instead of N queries
const resolvedTagsList = await resolveTagsBatch(
ctx,
result.items.map((item) => item.skill.tags),
)
const items = result.items.map((item, idx) => ({
slug: item.skill.slug,
displayName: item.skill.displayName,
summary: item.skill.summary ?? null,
tags: resolvedTagsList[idx],
stats: item.skill.stats,
createdAt: item.skill.createdAt,
updatedAt: item.skill.updatedAt,
latestVersion: item.latestVersion
? {
version: item.latestVersion.version,
createdAt: item.latestVersion.createdAt,
changelog: item.latestVersion.changelog,
}
: null,
}))
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers)
}
@@ -237,9 +248,13 @@ async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
if (segments.length === 1) {
const result = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult
if (!result?.skill) return text('Skill not found', 404, rate.headers)
if (!result?.skill) {
const hidden = await describeOwnerVisibleSkillState(ctx, request, slug)
if (hidden) return text(hidden.message, hidden.status, rate.headers)
return text('Skill not found', 404, rate.headers)
}
const tags = await resolveTags(ctx, result.skill.tags)
const [tags] = await resolveTagsBatch(ctx, [result.skill.tags])
return json(
{
skill: {
@@ -261,10 +276,17 @@ async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
owner: result.owner
? {
handle: result.owner.handle ?? null,
userId: result.owner._id,
displayName: result.owner.displayName ?? null,
image: result.owner.image ?? null,
}
: null,
moderation: result.moderationInfo
? {
isSuspicious: result.moderationInfo.isSuspicious ?? false,
isMalwareBlocked: result.moderationInfo.isMalwareBlocked ?? false,
}
: null,
},
200,
rate.headers,
@@ -315,7 +337,7 @@ async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
createdAt: version.createdAt,
changelog: version.changelog,
changelogSource: version.changelogSource ?? null,
files: version.files.map((file) => ({
files: version.files.map((file: SkillFile) => ({
path: file.path,
size: file.size,
sha256: file.sha256,
@@ -368,7 +390,12 @@ async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
if (!blob) return text('File missing in storage', 410, rate.headers)
const textContent = await blob.text()
const headers = mergeHeaders(rate.headers, {
const isSvg =
file.contentType?.toLowerCase().includes('svg') || file.path.toLowerCase().endsWith('.svg')
const headers = mergeHeaders(
rate.headers,
{
'Content-Type': file.contentType
? `${file.contentType}; charset=utf-8`
: 'text/plain; charset=utf-8',
@@ -376,13 +403,69 @@ async function skillsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
ETag: file.sha256,
'X-Content-SHA256': file.sha256,
'X-Content-Size': String(file.size),
})
'X-Content-Type-Options': 'nosniff',
'X-Frame-Options': 'DENY',
// For any text response that a browser might try to render, lock it down.
// In particular, this prevents SVG <foreignObject> script execution from
// reading localStorage tokens on this origin.
'Content-Security-Policy':
"default-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'",
...(isSvg ? { 'Content-Disposition': 'attachment' } : {}),
},
corsHeaders(),
)
return new Response(textContent, { status: 200, headers })
}
return text('Not found', 404, rate.headers)
}
async function describeOwnerVisibleSkillState(
ctx: ActionCtx,
request: Request,
slug: string,
): Promise<{ status: number; message: string } | null> {
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
if (!skill) return null
const apiTokenUserId = await getOptionalApiTokenUserId(ctx, request)
const isOwner = Boolean(apiTokenUserId && apiTokenUserId === skill.ownerUserId)
if (!isOwner) return null
if (skill.softDeletedAt) {
return {
status: 410,
message: `Skill is hidden/deleted. Run "clawhub undelete ${slug}" to restore it.`,
}
}
if (skill.moderationStatus === 'hidden') {
if (skill.moderationReason === 'pending.scan' || skill.moderationReason === 'scanner.vt.pending') {
return {
status: 423,
message: 'Skill is hidden while security scan is pending. Try again in a few minutes.',
}
}
if (skill.moderationReason === 'quality.low') {
return {
status: 403,
message:
'Skill is hidden by quality checks. Update SKILL.md content or run "clawhub undelete <slug>" after review.',
}
}
return {
status: 403,
message: `Skill is hidden by moderation${skill.moderationReason ? ` (${skill.moderationReason})` : ''}.`,
}
}
if (skill.moderationStatus === 'removed') {
return { status: 410, message: 'Skill has been removed by moderation.' }
}
return null
}
export const skillsGetRouterV1Http = httpAction(skillsGetRouterV1Handler)
async function publishSkillV1Handler(ctx: ActionCtx, request: Request) {
@@ -455,8 +538,8 @@ async function skillsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
deleted: false,
})
return json({ ok: true }, 200, rate.headers)
} catch {
return text('Unauthorized', 401, rate.headers)
} catch (error) {
return softDeleteErrorToResponse('skill', error, rate.headers)
}
}
@@ -477,8 +560,8 @@ async function skillsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
deleted: true,
})
return json({ ok: true }, 200, rate.headers)
} catch {
return text('Unauthorized', 401, rate.headers)
} catch (error) {
return softDeleteErrorToResponse('skill', error, rate.headers)
}
}
@@ -508,6 +591,145 @@ async function whoamiV1Handler(ctx: ActionCtx, request: Request) {
export const whoamiV1Http = httpAction(whoamiV1Handler)
async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, 'write')
if (!rate.ok) return rate.response
const segments = getPathSegments(request, '/api/v1/users/')
if (segments.length !== 1) {
return text('Not found', 404, rate.headers)
}
const action = segments[0]
if (action !== 'ban' && action !== 'role') {
return text('Not found', 404, rate.headers)
}
let payload: Record<string, unknown>
try {
payload = (await request.json()) as Record<string, unknown>
} catch {
return text('Invalid JSON', 400, rate.headers)
}
const handleRaw = typeof payload.handle === 'string' ? payload.handle.trim() : ''
const userIdRaw = typeof payload.userId === 'string' ? payload.userId.trim() : ''
const reasonRaw = typeof payload.reason === 'string' ? payload.reason.trim() : ''
if (!handleRaw && !userIdRaw) {
return text('Missing userId or handle', 400, rate.headers)
}
const roleRaw = typeof payload.role === 'string' ? payload.role.trim().toLowerCase() : ''
if (action === 'role' && !roleRaw) {
return text('Missing role', 400, rate.headers)
}
const role = roleRaw === 'user' || roleRaw === 'moderator' || roleRaw === 'admin' ? roleRaw : null
if (action === 'role' && !role) {
return text('Invalid role', 400, rate.headers)
}
let actorUserId: Id<'users'>
try {
const auth = await requireApiTokenUser(ctx, request)
actorUserId = auth.userId
} catch {
return text('Unauthorized', 401, rate.headers)
}
let targetUserId: Id<'users'> | null = userIdRaw ? (userIdRaw as Id<'users'>) : null
if (!targetUserId) {
const handle = handleRaw.toLowerCase()
const user = await ctx.runQuery(api.users.getByHandle, { handle })
if (!user?._id) return text('User not found', 404, rate.headers)
targetUserId = user._id
}
if (action === 'ban') {
const reason = reasonRaw.length > 0 ? reasonRaw : undefined
if (reason && reason.length > 500) {
return text('Reason too long (max 500 chars)', 400, rate.headers)
}
try {
const result = await ctx.runMutation(internal.users.banUserInternal, {
actorUserId,
targetUserId,
reason,
})
return json(result, 200, rate.headers)
} catch (error) {
const message = error instanceof Error ? error.message : 'Ban failed'
if (message.toLowerCase().includes('forbidden')) {
return text('Forbidden', 403, rate.headers)
}
if (message.toLowerCase().includes('not found')) {
return text(message, 404, rate.headers)
}
return text(message, 400, rate.headers)
}
}
if (!role) {
return text('Invalid role', 400, rate.headers)
}
try {
const result = await ctx.runMutation(internal.users.setRoleInternal, {
actorUserId,
targetUserId,
role,
})
return json({ ok: true, role: result.role ?? role }, 200, rate.headers)
} catch (error) {
const message = error instanceof Error ? error.message : 'Role change failed'
if (message.toLowerCase().includes('forbidden')) {
return text('Forbidden', 403, rate.headers)
}
if (message.toLowerCase().includes('not found')) {
return text(message, 404, rate.headers)
}
return text(message, 400, rate.headers)
}
}
export const usersPostRouterV1Http = httpAction(usersPostRouterV1Handler)
async function usersListV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, 'read')
if (!rate.ok) return rate.response
const url = new URL(request.url)
const limitRaw = toOptionalNumber(url.searchParams.get('limit'))
const query = url.searchParams.get('q') ?? url.searchParams.get('query') ?? ''
let actorUserId: Id<'users'>
try {
const auth = await requireApiTokenUser(ctx, request)
actorUserId = auth.userId
} catch {
return text('Unauthorized', 401, rate.headers)
}
const limit = Math.min(Math.max(limitRaw ?? 20, 1), 200)
try {
const result = await ctx.runQuery(internal.users.searchInternal, {
actorUserId,
query,
limit,
})
return json(result, 200, rate.headers)
} catch (error) {
const message = error instanceof Error ? error.message : 'User search failed'
if (message.toLowerCase().includes('forbidden')) {
return text('Forbidden', 403, rate.headers)
}
if (message.toLowerCase().includes('unauthorized')) {
return text('Unauthorized', 401, rate.headers)
}
return text(message, 400, rate.headers)
}
}
export const usersListV1Http = httpAction(usersListV1Handler)
async function parseMultipartPublish(
ctx: ActionCtx,
request: Request,
@@ -597,113 +819,66 @@ function parsePublishBody(body: unknown) {
}
}
async function resolveSoulTags(
/**
* Batch resolve soul version tags to version strings.
* Collects all version IDs, fetches them in a single query, then maps back.
* Reduces N sequential queries to 1 batch query.
*/
async function resolveSoulTagsBatch(
ctx: ActionCtx,
tags: Record<string, Id<'soulVersions'>>,
): Promise<Record<string, string>> {
const resolved: Record<string, string> = {}
for (const [tag, versionId] of Object.entries(tags)) {
const version = await ctx.runQuery(api.souls.getVersionById, { versionId })
if (version && !version.softDeletedAt) {
resolved[tag] = version.version
tagsList: Array<Record<string, Id<'soulVersions'>>>,
): Promise<Array<Record<string, string>>> {
return resolveVersionTagsBatch(ctx, tagsList, internal.souls.getVersionsByIdsInternal)
}
async function resolveTagsBatch(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<'skillVersions'>>>,
): Promise<Array<Record<string, string>>> {
return resolveVersionTagsBatch(ctx, tagsList, internal.skills.getVersionsByIdsInternal)
}
/**
* Batch resolve version tags to version strings.
* Collects all version IDs, fetches them in a single query, then maps back.
*
* Notes:
* - Uses `internal.*` queries to avoid expanding the public Convex API surface.
* - Sorts ids for stable query args (helps caching/log diffs).
*/
async function resolveVersionTagsBatch<TTable extends 'skillVersions' | 'soulVersions'>(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<TTable>>>,
getVersionsByIdsQuery: unknown,
): Promise<Array<Record<string, string>>> {
const allVersionIds = new Set<Id<TTable>>()
for (const tags of tagsList) {
for (const versionId of Object.values(tags)) allVersionIds.add(versionId)
}
if (allVersionIds.size === 0) return tagsList.map(() => ({}))
const versionIds = [...allVersionIds].sort() as Array<Id<TTable>>
const versions =
((await ctx.runQuery(getVersionsByIdsQuery as never, { versionIds } as never)) as Array<{
_id: Id<TTable>
version: string
softDeletedAt?: unknown
}> | null) ?? []
const versionMap = new Map<Id<TTable>, string>()
for (const v of versions) {
if (!v?.softDeletedAt) versionMap.set(v._id, v.version)
}
return tagsList.map((tags) => {
const resolved: Record<string, string> = {}
for (const [tag, versionId] of Object.entries(tags)) {
const version = versionMap.get(versionId)
if (version) resolved[tag] = version
}
}
return resolved
}
async function resolveTags(
ctx: ActionCtx,
tags: Record<string, Id<'skillVersions'>>,
): Promise<Record<string, string>> {
const resolved: Record<string, string> = {}
for (const [tag, versionId] of Object.entries(tags)) {
const version = await ctx.runQuery(api.skills.getVersionById, { versionId })
if (version && !version.softDeletedAt) {
resolved[tag] = version.version
}
}
return resolved
}
async function applyRateLimit(
ctx: ActionCtx,
request: Request,
kind: 'read' | 'write',
): Promise<{ ok: true; headers: HeadersInit } | { ok: false; response: Response }> {
const ip = getClientIp(request) ?? 'unknown'
const ipResult = await checkRateLimit(ctx, `ip:${ip}`, RATE_LIMITS[kind].ip)
const token = parseBearerToken(request)
const keyResult = token
? await checkRateLimit(ctx, `key:${await hashToken(token)}`, RATE_LIMITS[kind].key)
: null
const chosen = pickMostRestrictive(ipResult, keyResult)
const headers = rateHeaders(chosen)
if (!ipResult.allowed || (keyResult && !keyResult.allowed)) {
return {
ok: false,
response: text('Rate limit exceeded', 429, headers),
}
}
return { ok: true, headers }
}
type RateLimitResult = {
allowed: boolean
remaining: number
limit: number
resetAt: number
}
async function checkRateLimit(
ctx: ActionCtx,
key: string,
limit: number,
): Promise<RateLimitResult> {
return (await ctx.runMutation(internal.rateLimits.checkRateLimitInternal, {
key,
limit,
windowMs: RATE_LIMIT_WINDOW_MS,
})) as RateLimitResult
}
function pickMostRestrictive(primary: RateLimitResult, secondary: RateLimitResult | null) {
if (!secondary) return primary
if (!primary.allowed) return primary
if (!secondary.allowed) return secondary
return secondary.remaining < primary.remaining ? secondary : primary
}
function rateHeaders(result: RateLimitResult): HeadersInit {
const resetSeconds = Math.ceil(result.resetAt / 1000)
return {
'X-RateLimit-Limit': String(result.limit),
'X-RateLimit-Remaining': String(result.remaining),
'X-RateLimit-Reset': String(resetSeconds),
...(result.allowed ? {} : { 'Retry-After': String(resetSeconds) }),
}
}
function getClientIp(request: Request) {
const header =
request.headers.get('cf-connecting-ip') ??
request.headers.get('x-real-ip') ??
request.headers.get('x-forwarded-for') ??
request.headers.get('fly-client-ip')
if (!header) return null
if (header.includes(',')) return header.split(',')[0]?.trim() || null
return header.trim()
}
function parseBearerToken(request: Request) {
const header = request.headers.get('authorization') ?? request.headers.get('Authorization')
if (!header) return null
const trimmed = header.trim()
if (!trimmed.toLowerCase().startsWith('bearer ')) return null
const token = trimmed.slice(7).trim()
return token || null
return resolved
})
}
function json(value: unknown, status = 200, headers?: HeadersInit) {
@@ -715,6 +890,7 @@ function json(value: unknown, status = 200, headers?: HeadersInit) {
'Cache-Control': 'no-store',
},
headers,
corsHeaders(),
),
})
}
@@ -728,14 +904,11 @@ function text(value: string, status: number, headers?: HeadersInit) {
'Cache-Control': 'no-store',
},
headers,
corsHeaders(),
),
})
}
function mergeHeaders(base: HeadersInit, extra?: HeadersInit) {
return { ...(base as Record<string, string>), ...(extra as Record<string, string>) }
}
function getPathSegments(request: Request, prefix: string) {
const pathname = new URL(request.url).pathname
if (!pathname.startsWith(prefix)) return []
@@ -753,9 +926,36 @@ function toOptionalNumber(value: string | null) {
return Number.isFinite(parsed) ? parsed : undefined
}
type SkillListSort =
| 'updated'
| 'downloads'
| 'stars'
| 'installsCurrent'
| 'installsAllTime'
| 'trending'
function parseListSort(value: string | null): SkillListSort {
const normalized = value?.trim().toLowerCase()
if (normalized === 'downloads') return 'downloads'
if (normalized === 'stars' || normalized === 'rating') return 'stars'
if (
normalized === 'installs' ||
normalized === 'install' ||
normalized === 'installscurrent' ||
normalized === 'installs-current'
) {
return 'installsCurrent'
}
if (normalized === 'installsalltime' || normalized === 'installs-all-time') {
return 'installsAllTime'
}
if (normalized === 'trending') return 'trending'
return 'updated'
}
async function sha256Hex(bytes: Uint8Array) {
const buffer = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength)
const digest = await crypto.subtle.digest('SHA-256', buffer)
const data = new Uint8Array(bytes)
const digest = await crypto.subtle.digest('SHA-256', data)
return toHex(new Uint8Array(digest))
}
@@ -778,28 +978,29 @@ async function listSoulsV1Handler(ctx: ActionCtx, request: Request) {
cursor,
})) as ListSoulsResult
const items = await Promise.all(
result.items.map(async (item) => {
const tags = await resolveSoulTags(ctx, item.soul.tags)
return {
slug: item.soul.slug,
displayName: item.soul.displayName,
summary: item.soul.summary ?? null,
tags,
stats: item.soul.stats,
createdAt: item.soul.createdAt,
updatedAt: item.soul.updatedAt,
latestVersion: item.latestVersion
? {
version: item.latestVersion.version,
createdAt: item.latestVersion.createdAt,
changelog: item.latestVersion.changelog,
}
: null,
}
}),
// Batch resolve all tags in a single query instead of N queries
const resolvedTagsList = await resolveSoulTagsBatch(
ctx,
result.items.map((item) => item.soul.tags),
)
const items = result.items.map((item, idx) => ({
slug: item.soul.slug,
displayName: item.soul.displayName,
summary: item.soul.summary ?? null,
tags: resolvedTagsList[idx],
stats: item.soul.stats,
createdAt: item.soul.createdAt,
updatedAt: item.soul.updatedAt,
latestVersion: item.latestVersion
? {
version: item.latestVersion.version,
createdAt: item.latestVersion.createdAt,
changelog: item.latestVersion.changelog,
}
: null,
}))
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers)
}
@@ -819,7 +1020,7 @@ async function soulsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
const result = (await ctx.runQuery(api.souls.getBySlug, { slug })) as GetSoulBySlugResult
if (!result?.soul) return text('Soul not found', 404, rate.headers)
const tags = await resolveSoulTags(ctx, result.soul.tags)
const [tags] = await resolveSoulTagsBatch(ctx, [result.soul.tags])
return json(
{
soul: {
@@ -895,7 +1096,7 @@ async function soulsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
createdAt: version.createdAt,
changelog: version.changelog,
changelogSource: version.changelogSource ?? null,
files: version.files.map((file) => ({
files: version.files.map((file: SoulFile) => ({
path: file.path,
size: file.size,
sha256: file.sha256,
@@ -950,7 +1151,12 @@ async function soulsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
void ctx.runMutation(api.soulDownloads.increment, { soulId: soulResult.soul._id })
const headers = mergeHeaders(rate.headers, {
const isSvg =
file.contentType?.toLowerCase().includes('svg') || file.path.toLowerCase().endsWith('.svg')
const headers = mergeHeaders(
rate.headers,
{
'Content-Type': file.contentType
? `${file.contentType}; charset=utf-8`
: 'text/plain; charset=utf-8',
@@ -958,7 +1164,17 @@ async function soulsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
ETag: file.sha256,
'X-Content-SHA256': file.sha256,
'X-Content-Size': String(file.size),
})
'X-Content-Type-Options': 'nosniff',
'X-Frame-Options': 'DENY',
// For any text response that a browser might try to render, lock it down.
// In particular, this prevents SVG <foreignObject> script execution from
// reading localStorage tokens on this origin.
'Content-Security-Policy':
"default-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'",
...(isSvg ? { 'Content-Disposition': 'attachment' } : {}),
},
corsHeaders(),
)
return new Response(textContent, { status: 200, headers })
}
@@ -1019,8 +1235,8 @@ async function soulsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
deleted: false,
})
return json({ ok: true }, 200, rate.headers)
} catch {
return text('Unauthorized', 401, rate.headers)
} catch (error) {
return softDeleteErrorToResponse('soul', error, rate.headers)
}
}
@@ -1041,12 +1257,79 @@ async function soulsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
deleted: true,
})
return json({ ok: true }, 200, rate.headers)
} catch (error) {
return softDeleteErrorToResponse('soul', error, rate.headers)
}
}
export const soulsDeleteRouterV1Http = httpAction(soulsDeleteRouterV1Handler)
function softDeleteErrorToResponse(
entity: 'skill' | 'soul',
error: unknown,
headers: HeadersInit,
) {
const message = error instanceof Error ? error.message : `${entity} delete failed`
const lower = message.toLowerCase()
if (lower.includes('unauthorized')) return text('Unauthorized', 401, headers)
if (lower.includes('forbidden')) return text('Forbidden', 403, headers)
if (lower.includes('not found')) return text(message, 404, headers)
if (lower.includes('slug required')) return text('Slug required', 400, headers)
// Unknown: server-side failure. Keep body generic.
return text('Internal Server Error', 500, headers)
}
async function starsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, 'write')
if (!rate.ok) return rate.response
const segments = getPathSegments(request, '/api/v1/stars/')
if (segments.length !== 1) return text('Not found', 404, rate.headers)
const slug = segments[0]?.trim().toLowerCase() ?? ''
try {
const { userId } = await requireApiTokenUser(ctx, request)
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
if (!skill) return text('Skill not found', 404, rate.headers)
const result = await ctx.runMutation(internal.stars.addStarInternal, {
userId,
skillId: skill._id,
})
return json(result, 200, rate.headers)
} catch {
return text('Unauthorized', 401, rate.headers)
}
}
export const soulsDeleteRouterV1Http = httpAction(soulsDeleteRouterV1Handler)
export const starsPostRouterV1Http = httpAction(starsPostRouterV1Handler)
async function starsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, 'write')
if (!rate.ok) return rate.response
const segments = getPathSegments(request, '/api/v1/stars/')
if (segments.length !== 1) return text('Not found', 404, rate.headers)
const slug = segments[0]?.trim().toLowerCase() ?? ''
try {
const { userId } = await requireApiTokenUser(ctx, request)
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug })
if (!skill) return text('Skill not found', 404, rate.headers)
const result = await ctx.runMutation(internal.stars.removeStarInternal, {
userId,
skillId: skill._id,
})
return json(result, 200, rate.headers)
} catch {
return text('Unauthorized', 401, rate.headers)
}
}
export const starsDeleteRouterV1Http = httpAction(starsDeleteRouterV1Handler)
export const __handlers = {
searchSkillsV1Handler,
resolveSkillVersionV1Handler,
@@ -1060,5 +1343,9 @@ export const __handlers = {
publishSoulV1Handler,
soulsPostRouterV1Handler,
soulsDeleteRouterV1Handler,
starsPostRouterV1Handler,
starsDeleteRouterV1Handler,
whoamiV1Handler,
usersPostRouterV1Handler,
usersListV1Handler,
}
+37
View File
@@ -0,0 +1,37 @@
import { httpAction } from './_generated/server'
import { corsHeaders, mergeHeaders } from './lib/httpHeaders'
function getHeader(request: Request, name: string) {
return request.headers.get(name) ?? request.headers.get(name.toLowerCase())
}
export function buildPreflightHeaders(request: Request) {
const requestedHeaders = getHeader(request, 'Access-Control-Request-Headers')?.trim() || null
const requestedMethod = getHeader(request, 'Access-Control-Request-Method')?.trim() || null
const vary = [
...(requestedMethod ? ['Access-Control-Request-Method'] : []),
...(requestedHeaders ? ['Access-Control-Request-Headers'] : []),
].join(', ')
return mergeHeaders(
corsHeaders(),
{
'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD',
'Access-Control-Allow-Headers':
requestedHeaders ?? 'Content-Type, Authorization, Digest, X-Clawhub-Version',
'Access-Control-Max-Age': '86400',
...(vary ? { Vary: vary } : {}),
},
)
}
export const preflightHandler = httpAction(async (_ctx, request) => {
// No cookies/credentials supported; allow any origin for simple browser access.
// If we ever add cookie auth, this must switch to reflecting origin + Allow-Credentials.
return new Response(null, {
status: 204,
headers: buildPreflightHeaders(request),
})
})
+39
View File
@@ -0,0 +1,39 @@
import { v } from 'convex/values'
import { internalMutation } from './_generated/server'
import { buildTrendingLeaderboard } from './lib/leaderboards'
const MAX_TRENDING_LIMIT = 200
const KEEP_LEADERBOARD_ENTRIES = 3
export const rebuildTrendingLeaderboardInternal = internalMutation({
args: { limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = clampInt(args.limit ?? MAX_TRENDING_LIMIT, 1, MAX_TRENDING_LIMIT)
const now = Date.now()
const { startDay, endDay, items } = await buildTrendingLeaderboard(ctx, { limit, now })
await ctx.db.insert('skillLeaderboards', {
kind: 'trending',
generatedAt: now,
rangeStartDay: startDay,
rangeEndDay: endDay,
items,
})
const recent = await ctx.db
.query('skillLeaderboards')
.withIndex('by_kind', (q) => q.eq('kind', 'trending'))
.order('desc')
.take(KEEP_LEADERBOARD_ENTRIES + 5)
for (const entry of recent.slice(KEEP_LEADERBOARD_ENTRIES)) {
await ctx.db.delete(entry._id)
}
return { ok: true as const, count: items.length }
},
})
function clampInt(value: number, min: number, max: number) {
return Math.min(Math.max(value, min), max)
}
+12 -4
View File
@@ -1,5 +1,5 @@
import { getAuthUserId } from '@convex-dev/auth/server'
import { api } from '../_generated/api'
import { internal } from '../_generated/api'
import type { Doc } from '../_generated/dataModel'
import type { ActionCtx, MutationCtx, QueryCtx } from '../_generated/server'
@@ -9,15 +9,15 @@ export async function requireUser(ctx: MutationCtx | QueryCtx) {
const userId = await getAuthUserId(ctx)
if (!userId) throw new Error('Unauthorized')
const user = await ctx.db.get(userId)
if (!user || user.deletedAt) throw new Error('User not found')
if (!user || user.deletedAt || user.deactivatedAt) throw new Error('User not found')
return { userId, user }
}
export async function requireUserFromAction(ctx: ActionCtx) {
const userId = await getAuthUserId(ctx)
if (!userId) throw new Error('Unauthorized')
const user = await ctx.runQuery(api.users.getById, { userId })
if (!user || user.deletedAt) throw new Error('User not found')
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId })
if (!user || user.deletedAt || user.deactivatedAt) throw new Error('User not found')
return { userId, user: user as Doc<'users'> }
}
@@ -26,3 +26,11 @@ export function assertRole(user: Doc<'users'>, allowed: Role[]) {
throw new Error('Forbidden')
}
}
export function assertAdmin(user: Doc<'users'>) {
assertRole(user, ['admin'])
}
export function assertModerator(user: Doc<'users'>) {
assertRole(user, ['admin', 'moderator'])
}
+110
View File
@@ -0,0 +1,110 @@
import { describe, expect, it, vi } from 'vitest'
import { getOptionalApiTokenUserId } from './apiTokenAuth'
import { hashToken } from './tokens'
describe('getOptionalApiTokenUserId', () => {
it('returns null when auth header is missing', async () => {
const ctx = {
runQuery: vi.fn(),
}
const request = new Request('https://example.com')
const userId = await getOptionalApiTokenUserId(ctx as never, request)
expect(userId).toBeNull()
expect(ctx.runQuery).not.toHaveBeenCalled()
})
it('returns null for unknown token', async () => {
const ctx = {
runQuery: vi.fn().mockResolvedValue(null),
}
const request = new Request('https://example.com', {
headers: { authorization: 'Bearer token-1' },
})
const userId = await getOptionalApiTokenUserId(ctx as never, request)
expect(userId).toBeNull()
expect(ctx.runQuery).toHaveBeenCalledTimes(1)
expect(ctx.runQuery.mock.calls[0]?.[1]).toEqual({
tokenHash: await hashToken('token-1'),
})
})
it('returns user id when token and user are valid', async () => {
const tokenId = 'apiTokens_1'
const expectedUserId = 'users_1'
const ctx = {
runQuery: vi
.fn()
.mockImplementation(async (_fn, args: { tokenHash?: string; tokenId?: string }) => {
if (args.tokenHash) {
return { _id: tokenId, revokedAt: undefined }
}
if (args.tokenId) {
return { _id: expectedUserId, deletedAt: undefined }
}
return null
}),
}
const request = new Request('https://example.com', {
headers: { authorization: 'Bearer token-2' },
})
const userId = await getOptionalApiTokenUserId(ctx as never, request)
expect(userId).toBe(expectedUserId)
expect(ctx.runQuery).toHaveBeenCalledTimes(2)
})
it('returns null when user is deleted', async () => {
const tokenId = 'apiTokens_2'
const ctx = {
runQuery: vi
.fn()
.mockImplementation(async (_fn, args: { tokenHash?: string; tokenId?: string }) => {
if (args.tokenHash) {
return { _id: tokenId, revokedAt: undefined }
}
if (args.tokenId) {
return { _id: 'users_deleted', deletedAt: Date.now() }
}
return null
}),
}
const request = new Request('https://example.com', {
headers: { authorization: 'Bearer token-3' },
})
const userId = await getOptionalApiTokenUserId(ctx as never, request)
expect(userId).toBeNull()
expect(ctx.runQuery).toHaveBeenCalledTimes(2)
})
it('returns null when user is deactivated', async () => {
const tokenId = 'apiTokens_3'
const ctx = {
runQuery: vi
.fn()
.mockImplementation(async (_fn, args: { tokenHash?: string; tokenId?: string }) => {
if (args.tokenHash) {
return { _id: tokenId, revokedAt: undefined }
}
if (args.tokenId) {
return { _id: 'users_deactivated', deactivatedAt: Date.now() }
}
return null
}),
}
const request = new Request('https://example.com', {
headers: { authorization: 'Bearer token-4' },
})
const userId = await getOptionalApiTokenUserId(ctx as never, request)
expect(userId).toBeNull()
expect(ctx.runQuery).toHaveBeenCalledTimes(2)
})
})
+21 -1
View File
@@ -21,12 +21,32 @@ export async function requireApiTokenUser(
const user = await ctx.runQuery(internal.tokens.getUserForTokenInternal, {
tokenId: apiToken._id,
})
if (!user || user.deletedAt) throw new ConvexError('Unauthorized')
if (!user || user.deletedAt || user.deactivatedAt) throw new ConvexError('Unauthorized')
await ctx.runMutation(internal.tokens.touchInternal, { tokenId: apiToken._id })
return { user, userId: user._id }
}
export async function getOptionalApiTokenUserId(
ctx: ActionCtx,
request: Request,
): Promise<Doc<'users'>['_id'] | null> {
const header = request.headers.get('authorization') ?? request.headers.get('Authorization')
const token = parseBearerToken(header)
if (!token) return null
const tokenHash = await hashToken(token)
const apiToken = await ctx.runQuery(internal.tokens.getByHashInternal, { tokenHash })
if (!apiToken || apiToken.revokedAt) return null
const user = await ctx.runQuery(internal.tokens.getUserForTokenInternal, {
tokenId: apiToken._id,
})
if (!user || user.deletedAt || user.deactivatedAt) return null
return user._id
}
function parseBearerToken(header: string | null) {
if (!header) return null
const trimmed = header.trim()
+50
View File
@@ -0,0 +1,50 @@
import type { Doc, Id } from '../_generated/dataModel'
import type { QueryCtx } from '../_generated/server'
type BadgeKind = Doc<'skillBadges'>['kind']
export type SkillBadgeMap = Partial<Record<BadgeKind, { byUserId: Id<'users'>; at: number }>>
export type SkillBadgeSource = { badges?: SkillBadgeMap | null }
type BadgeCtx = Pick<QueryCtx, 'db'>
export function isSkillHighlighted(skill: SkillBadgeSource) {
return Boolean(skill.badges?.highlighted)
}
export function isSkillOfficial(skill: SkillBadgeSource) {
return Boolean(skill.badges?.official)
}
export function isSkillDeprecated(skill: SkillBadgeSource) {
return Boolean(skill.badges?.deprecated)
}
export function buildBadgeMap(records: Doc<'skillBadges'>[]): SkillBadgeMap {
return records.reduce<SkillBadgeMap>((acc, record) => {
acc[record.kind] = { byUserId: record.byUserId, at: record.at }
return acc
}, {})
}
export async function getSkillBadgeMap(
ctx: BadgeCtx,
skillId: Id<'skills'>,
): Promise<SkillBadgeMap> {
const records = await ctx.db
.query('skillBadges')
.withIndex('by_skill', (q) => q.eq('skillId', skillId))
.collect()
return buildBadgeMap(records)
}
export async function getSkillBadgeMaps(
ctx: BadgeCtx,
skillIds: Array<Id<'skills'>>,
): Promise<Map<Id<'skills'>, SkillBadgeMap>> {
const entries = await Promise.all(
skillIds.map(async (skillId) => [skillId, await getSkillBadgeMap(ctx, skillId)] as const),
)
return new Map(entries)
}
+95
View File
@@ -0,0 +1,95 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { EMBEDDING_DIMENSIONS, generateEmbedding } from './embeddings'
const fetchMock = vi.fn<typeof fetch>()
const consoleWarnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
const originalFetch = globalThis.fetch
const originalApiKey = process.env.OPENAI_API_KEY
function jsonResponse(payload: unknown, init?: ResponseInit) {
return new Response(JSON.stringify(payload), {
status: 200,
headers: {
'content-type': 'application/json',
},
...init,
})
}
beforeEach(() => {
fetchMock.mockReset()
globalThis.fetch = fetchMock as typeof fetch
process.env.OPENAI_API_KEY = 'test-key'
consoleWarnSpy.mockClear()
})
afterEach(() => {
globalThis.fetch = originalFetch
if (originalApiKey === undefined) {
delete process.env.OPENAI_API_KEY
} else {
process.env.OPENAI_API_KEY = originalApiKey
}
vi.useRealTimers()
})
describe('generateEmbedding', () => {
it('returns zero embedding when OPENAI_API_KEY is missing', async () => {
delete process.env.OPENAI_API_KEY
const result = await generateEmbedding('hello world')
expect(result).toHaveLength(EMBEDDING_DIMENSIONS)
expect(result.every((value) => value === 0)).toBe(true)
expect(fetchMock).not.toHaveBeenCalled()
})
it('retries on 429 responses and then succeeds', async () => {
vi.useFakeTimers()
fetchMock.mockResolvedValueOnce(new Response('rate limited', { status: 429 }))
fetchMock.mockResolvedValueOnce(jsonResponse({ data: [{ embedding: [0.25, 0.75] }] }))
const promise = generateEmbedding('retry me')
await vi.runAllTimersAsync()
await expect(promise).resolves.toEqual([0.25, 0.75])
expect(fetchMock).toHaveBeenCalledTimes(2)
})
it('does not retry non-retryable 4xx responses', async () => {
fetchMock.mockResolvedValueOnce(new Response('bad request', { status: 400 }))
await expect(generateEmbedding('bad')).rejects.toThrow('Embedding failed: bad request')
expect(fetchMock).toHaveBeenCalledTimes(1)
})
it('retries on network failures and then succeeds', async () => {
vi.useFakeTimers()
fetchMock.mockRejectedValueOnce(new TypeError('fetch failed'))
fetchMock.mockResolvedValueOnce(jsonResponse({ data: [{ embedding: [1, 2, 3] }] }))
const promise = generateEmbedding('network retry')
await vi.runAllTimersAsync()
await expect(promise).resolves.toEqual([1, 2, 3])
expect(fetchMock).toHaveBeenCalledTimes(2)
})
it('retries timeouts up to max attempts and preserves timeout error', async () => {
vi.useFakeTimers()
fetchMock.mockRejectedValue(new DOMException('aborted', 'AbortError'))
const promise = generateEmbedding('always timeout')
const rejection = expect(promise).rejects.toThrow(
'OpenAI API request timed out after 10 seconds',
)
await vi.runAllTimersAsync()
await rejection
expect(fetchMock).toHaveBeenCalledTimes(3)
})
})
+127 -20
View File
@@ -1,10 +1,67 @@
export const EMBEDDING_MODEL = 'text-embedding-3-small'
export const EMBEDDING_DIMENSIONS = 1536
const EMBEDDING_ENDPOINT = 'https://api.openai.com/v1/embeddings'
const REQUEST_TIMEOUT_MS = 10_000
const MAX_ATTEMPTS = 3
const BASE_RETRY_DELAY_MS = 1_000
class RetryableEmbeddingError extends Error {
constructor(message: string, options?: { cause?: unknown }) {
super(message, options)
this.name = 'RetryableEmbeddingError'
}
}
function emptyEmbedding() {
return Array.from({ length: EMBEDDING_DIMENSIONS }, () => 0)
}
function parseRetryAfterMs(retryAfterHeader: string | null) {
if (!retryAfterHeader) return null
const seconds = Number(retryAfterHeader)
if (Number.isFinite(seconds) && seconds >= 0) {
return Math.round(seconds * 1000)
}
const dateMs = Date.parse(retryAfterHeader)
if (Number.isFinite(dateMs)) {
return Math.max(0, dateMs - Date.now())
}
return null
}
function getRetryDelayMs(attempt: number, retryAfterMs: number | null) {
const exponentialDelayMs = BASE_RETRY_DELAY_MS * 2 ** attempt
if (retryAfterMs == null) return exponentialDelayMs
return Math.max(exponentialDelayMs, retryAfterMs)
}
function normalizeRetryableNetworkError(error: unknown) {
if (!(error instanceof Error)) return null
if (error.name === 'AbortError') {
return new RetryableEmbeddingError(
`OpenAI API request timed out after ${Math.floor(REQUEST_TIMEOUT_MS / 1000)} seconds`,
{ cause: error },
)
}
if (error instanceof TypeError) {
return new RetryableEmbeddingError(`Embedding request failed: ${error.message}`, { cause: error })
}
return null
}
function sleep(ms: number) {
return new Promise<void>((resolve) => {
setTimeout(resolve, ms)
})
}
export async function generateEmbedding(text: string) {
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) {
@@ -12,27 +69,77 @@ export async function generateEmbedding(text: string) {
return emptyEmbedding()
}
const response = await fetch('https://api.openai.com/v1/embeddings', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: EMBEDDING_MODEL,
input: text,
}),
})
let lastRetryableError: RetryableEmbeddingError | null = null
if (!response.ok) {
const message = await response.text()
throw new Error(`Embedding failed: ${message}`)
for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) {
const controller = new AbortController()
const timeoutId = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS)
try {
const response = await fetch(EMBEDDING_ENDPOINT, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: EMBEDDING_MODEL,
input: text,
}),
signal: controller.signal,
})
if (!response.ok) {
const message = await response.text()
const isRetryableStatus = response.status === 429 || response.status >= 500
if (isRetryableStatus) {
const retryableError = new RetryableEmbeddingError(
`Embedding failed (${response.status}): ${message}`,
)
lastRetryableError = retryableError
if (attempt < MAX_ATTEMPTS - 1) {
const retryAfterMs = parseRetryAfterMs(response.headers.get('retry-after'))
const delayMs = getRetryDelayMs(attempt, retryAfterMs)
console.warn(
`OpenAI embeddings retry in ${delayMs}ms (attempt ${attempt + 1}/${MAX_ATTEMPTS})`,
)
await sleep(delayMs)
continue
}
throw retryableError
}
throw new Error(`Embedding failed: ${message}`)
}
const payload = (await response.json()) as {
data?: Array<{ embedding: number[] }>
}
const embedding = payload.data?.[0]?.embedding
if (!embedding) throw new Error('Embedding missing from response')
return embedding
} catch (error) {
const retryableNetworkError = normalizeRetryableNetworkError(error)
if (retryableNetworkError) {
lastRetryableError = retryableNetworkError
if (attempt < MAX_ATTEMPTS - 1) {
const delayMs = getRetryDelayMs(attempt, null)
console.warn(
`OpenAI embeddings network retry in ${delayMs}ms (attempt ${attempt + 1}/${MAX_ATTEMPTS})`,
)
await sleep(delayMs)
continue
}
throw retryableNetworkError
}
throw error
} finally {
clearTimeout(timeoutId)
}
}
const payload = (await response.json()) as {
data?: Array<{ embedding: number[] }>
}
const embedding = payload.data?.[0]?.embedding
if (!embedding) throw new Error('Embedding missing from response')
return embedding
throw lastRetryableError ?? new Error('Embedding failed after retries')
}
+270
View File
@@ -0,0 +1,270 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { internal } from '../_generated/api'
import { requireGitHubAccountAge } from './githubAccount'
vi.mock('../_generated/api', () => ({
internal: {
users: {
getByIdInternal: Symbol('getByIdInternal'),
getGitHubProviderAccountIdInternal: Symbol('getGitHubProviderAccountIdInternal'),
updateGithubMetaInternal: Symbol('updateGithubMetaInternal'),
},
},
}))
const ONE_DAY_MS = 24 * 60 * 60 * 1000
describe('requireGitHubAccountAge', () => {
beforeEach(() => {
vi.restoreAllMocks()
vi.unstubAllEnvs()
vi.unstubAllGlobals()
})
afterEach(() => {
vi.useRealTimers()
vi.unstubAllEnvs()
vi.unstubAllGlobals()
})
it('uses cached githubCreatedAt when present', async () => {
vi.useFakeTimers()
const now = new Date('2026-02-02T12:00:00Z')
vi.setSystemTime(now)
const runQuery = vi.fn().mockResolvedValue({
_id: 'users:1',
githubCreatedAt: now.getTime() - 10 * ONE_DAY_MS,
githubFetchedAt: now.getTime() - ONE_DAY_MS,
})
const runMutation = vi.fn()
const fetchMock = vi.fn()
vi.stubGlobal('fetch', fetchMock)
await requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never)
expect(fetchMock).not.toHaveBeenCalled()
expect(runMutation).not.toHaveBeenCalled()
expect(runQuery).toHaveBeenCalledWith(internal.users.getByIdInternal, { userId: 'users:1' })
expect(runQuery).not.toHaveBeenCalledWith(internal.users.getGitHubProviderAccountIdInternal, {
userId: 'users:1',
})
})
it('rejects deactivated users', async () => {
const runQuery = vi.fn().mockResolvedValue({
_id: 'users:1',
deactivatedAt: Date.now(),
})
const runMutation = vi.fn()
const fetchMock = vi.fn()
vi.stubGlobal('fetch', fetchMock)
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/User not found/i)
expect(fetchMock).not.toHaveBeenCalled()
})
it('rejects accounts younger than 7 days', async () => {
vi.useFakeTimers()
const now = new Date('2026-02-02T12:00:00Z')
vi.setSystemTime(now)
const runQuery = vi.fn().mockResolvedValue({
_id: 'users:1',
githubCreatedAt: now.getTime() - 2 * ONE_DAY_MS,
githubFetchedAt: now.getTime() - ONE_DAY_MS / 2,
})
const runMutation = vi.fn()
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/GitHub account must be at least 7 days old/i)
})
it('fetches githubCreatedAt when missing (by providerAccountId)', async () => {
vi.useFakeTimers()
const now = new Date('2026-02-02T12:00:00Z')
vi.setSystemTime(now)
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce('12345')
const runMutation = vi.fn()
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
created_at: '2020-01-01T00:00:00Z',
}),
})
vi.stubGlobal('fetch', fetchMock)
await requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never)
expect(fetchMock).toHaveBeenCalledWith(
'https://api.github.com/user/12345',
expect.objectContaining({
headers: expect.objectContaining({ 'User-Agent': 'clawhub' }),
}),
)
expect(runMutation).toHaveBeenCalledWith(internal.users.updateGithubMetaInternal, {
userId: 'users:1',
githubCreatedAt: Date.parse('2020-01-01T00:00:00Z'),
githubFetchedAt: now.getTime(),
})
})
it('rejects when providerAccountId is missing', async () => {
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce(null)
const runMutation = vi.fn()
const fetchMock = vi.fn()
vi.stubGlobal('fetch', fetchMock)
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/GitHub account required/i)
expect(fetchMock).not.toHaveBeenCalled()
})
it('rejects when providerAccountId is invalid', async () => {
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce('abc123')
const runMutation = vi.fn()
const fetchMock = vi.fn()
vi.stubGlobal('fetch', fetchMock)
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/GitHub account lookup failed/i)
expect(fetchMock).not.toHaveBeenCalled()
})
it('throws when GitHub lookup fails', async () => {
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce('12345')
const runMutation = vi.fn()
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 404 })
vi.stubGlobal('fetch', fetchMock)
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/GitHub account lookup failed/i)
})
it('throws rate-limit error on 403', async () => {
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce('12345')
const runMutation = vi.fn()
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 403 })
vi.stubGlobal('fetch', fetchMock)
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/rate limit exceeded/i)
})
it('throws rate-limit error on 429', async () => {
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce('12345')
const runMutation = vi.fn()
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 429 })
vi.stubGlobal('fetch', fetchMock)
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/rate limit exceeded/i)
})
it('throws when GitHub returns an invalid payload', async () => {
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce('12345')
const runMutation = vi.fn()
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({}),
})
vi.stubGlobal('fetch', fetchMock)
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never),
).rejects.toThrow(/GitHub account lookup failed/i)
})
it('includes Authorization header when GITHUB_TOKEN is set', async () => {
vi.useFakeTimers()
const now = new Date('2026-02-02T12:00:00Z')
vi.setSystemTime(now)
vi.stubEnv('GITHUB_TOKEN', 'ghp_test123')
const runQuery = vi.fn()
.mockResolvedValueOnce({
_id: 'users:1',
githubCreatedAt: undefined,
githubFetchedAt: 0,
})
.mockResolvedValueOnce('12345')
const runMutation = vi.fn()
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
created_at: '2020-01-01T00:00:00Z',
}),
})
vi.stubGlobal('fetch', fetchMock)
await requireGitHubAccountAge({ runQuery, runMutation } as never, 'users:1' as never)
expect(fetchMock).toHaveBeenCalledWith(
'https://api.github.com/user/12345',
expect.objectContaining({
headers: {
'User-Agent': 'clawhub',
Authorization: 'Bearer ghp_test123',
},
}),
)
})
})
+73
View File
@@ -0,0 +1,73 @@
import { ConvexError } from 'convex/values'
import { internal } from '../_generated/api'
import type { Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
const GITHUB_API = 'https://api.github.com'
const MIN_ACCOUNT_AGE_MS = 7 * 24 * 60 * 60 * 1000
type GitHubUser = {
created_at?: string
}
export async function requireGitHubAccountAge(ctx: ActionCtx, userId: Id<'users'>) {
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId })
if (!user || user.deletedAt || user.deactivatedAt) throw new ConvexError('User not found')
const now = Date.now()
let createdAt = user.githubCreatedAt ?? null
if (!createdAt) {
const providerAccountId = await ctx.runQuery(internal.users.getGitHubProviderAccountIdInternal, {
userId,
})
if (!providerAccountId) {
// Invariant: GitHub is our only auth provider, so this should never happen.
throw new ConvexError('GitHub account required')
}
if (!/^[0-9]+$/.test(providerAccountId)) {
throw new ConvexError('GitHub account lookup failed')
}
const headers: Record<string, string> = { 'User-Agent': 'clawhub' }
const token = process.env.GITHUB_TOKEN
if (token) {
headers.Authorization = `Bearer ${token}`
}
// Fetch by immutable GitHub numeric ID to avoid username swap attacks entirely.
const response = await fetch(`${GITHUB_API}/user/${providerAccountId}`, {
headers,
})
if (!response.ok) {
if (response.status === 403 || response.status === 429) {
throw new ConvexError('GitHub API rate limit exceeded — please try again in a few minutes')
}
throw new ConvexError('GitHub account lookup failed')
}
const payload = (await response.json()) as GitHubUser
const parsed = payload.created_at ? Date.parse(payload.created_at) : Number.NaN
if (!Number.isFinite(parsed)) throw new ConvexError('GitHub account lookup failed')
createdAt = parsed
await ctx.runMutation(internal.users.updateGithubMetaInternal, {
userId,
githubCreatedAt: createdAt,
githubFetchedAt: now,
})
}
if (!createdAt) throw new ConvexError('GitHub account lookup failed')
const ageMs = now - createdAt
if (ageMs < MIN_ACCOUNT_AGE_MS) {
const remainingMs = MIN_ACCOUNT_AGE_MS - ageMs
const remainingDays = Math.max(1, Math.ceil(remainingMs / (24 * 60 * 60 * 1000)))
throw new ConvexError(
`GitHub account must be at least 7 days old to upload skills. Try again in ${remainingDays} day${
remainingDays === 1 ? '' : 's'
}.`,
)
}
}
+1 -1
View File
@@ -8,7 +8,7 @@ const GITHUB_API = 'https://api.github.com'
const DEFAULT_REPO = 'clawdbot/skills'
const DEFAULT_ROOT = 'skills'
const META_FILENAME = '_meta.json'
const USER_AGENT = 'clawdhub/skills-backup'
const USER_AGENT = 'clawhub/skills-backup'
type BackupFile = {
path: string
+19
View File
@@ -0,0 +1,19 @@
import { describe, expect, it } from 'vitest'
import { canHealSkillOwnershipByGitHubProviderAccountId } from './githubIdentity'
describe('canHealSkillOwnershipByGitHubProviderAccountId', () => {
it('denies when either providerAccountId is missing', () => {
expect(canHealSkillOwnershipByGitHubProviderAccountId(undefined, undefined)).toBe(false)
expect(canHealSkillOwnershipByGitHubProviderAccountId('123', undefined)).toBe(false)
expect(canHealSkillOwnershipByGitHubProviderAccountId(undefined, '123')).toBe(false)
expect(canHealSkillOwnershipByGitHubProviderAccountId(null, '123')).toBe(false)
})
it('denies when providerAccountId differs', () => {
expect(canHealSkillOwnershipByGitHubProviderAccountId('123', '456')).toBe(false)
})
it('allows when providerAccountId matches', () => {
expect(canHealSkillOwnershipByGitHubProviderAccountId('123', '123')).toBe(true)
})
})
+22
View File
@@ -0,0 +1,22 @@
import type { Id } from '../_generated/dataModel'
import type { QueryCtx } from '../_generated/server'
export function canHealSkillOwnershipByGitHubProviderAccountId(
ownerProviderAccountId: string | null | undefined,
callerProviderAccountId: string | null | undefined,
) {
// Security invariant: missing identity must never grant ownership.
if (!ownerProviderAccountId || !callerProviderAccountId) return false
return ownerProviderAccountId === callerProviderAccountId
}
export async function getGitHubProviderAccountId(
ctx: Pick<QueryCtx, 'db'>,
userId: Id<'users'>,
): Promise<string | null> {
const account = await ctx.db
.query('authAccounts')
.withIndex('userIdAndProvider', (q) => q.eq('userId', userId).eq('provider', 'github'))
.unique()
return account?.providerAccountId ?? null
}
+3 -3
View File
@@ -1,4 +1,4 @@
import { TEXT_FILE_EXTENSION_SET } from 'clawdhub-schema'
import { TEXT_FILE_EXTENSION_SET } from 'clawhub-schema'
import { zipSync } from 'fflate'
import semver from 'semver'
import { parseFrontmatter } from './skills'
@@ -118,7 +118,7 @@ async function resolveRefCommit(parsed: GitHubImportUrl, ref: string, fetcher: t
const response = await fetcher(apiUrl, {
headers: {
Accept: 'application/vnd.github+json',
'User-Agent': 'clawdhub/github-import',
'User-Agent': 'clawhub/github-import',
},
})
if (!response.ok) throw new Error('GitHub ref not found')
@@ -156,7 +156,7 @@ export async function fetchGitHubZipBytes(
const maxZipBytes = limits?.maxZipBytes ?? 25 * 1024 * 1024
const url = `https://${CODELOAD_HOST}/${resolved.owner}/${resolved.repo}/zip/${resolved.commit}`
const response = await fetcher(url, {
headers: { 'User-Agent': 'clawdhub/github-import' },
headers: { 'User-Agent': 'clawhub/github-import' },
})
if (!response.ok) throw new Error('GitHub archive download failed')
+1 -1
View File
@@ -8,7 +8,7 @@ const GITHUB_API = 'https://api.github.com'
const DEFAULT_REPO = 'clawdbot/souls'
const DEFAULT_ROOT = 'souls'
const META_FILENAME = '_meta.json'
const USER_AGENT = 'clawdhub/souls-backup'
const USER_AGENT = 'clawhub/souls-backup'
type BackupFile = {
path: string
+19
View File
@@ -0,0 +1,19 @@
function toHeaderRecord(init?: HeadersInit): Record<string, string> {
if (!init) return {}
if (init instanceof Headers) return Object.fromEntries(init.entries())
if (Array.isArray(init)) return Object.fromEntries(init)
return { ...(init as Record<string, string>) }
}
export function mergeHeaders(...inits: Array<HeadersInit | undefined>): Record<string, string> {
const out: Record<string, string> = {}
for (const init of inits) {
Object.assign(out, toHeaderRecord(init))
}
return out
}
export function corsHeaders(origin: string = '*'): Record<string, string> {
return { 'Access-Control-Allow-Origin': origin }
}
+46
View File
@@ -0,0 +1,46 @@
/* @vitest-environment node */
import { describe, expect, it } from 'vitest'
import { getClientIp } from './httpRateLimit'
describe('getClientIp', () => {
it('uses forwarded headers by default when cf-connecting-ip is missing', () => {
const request = new Request('https://example.com', {
headers: {
'x-forwarded-for': '203.0.113.9',
},
})
delete process.env.TRUST_FORWARDED_IPS
expect(getClientIp(request)).toBe('203.0.113.9')
})
it('can disable forwarded headers explicitly', () => {
const request = new Request('https://example.com', {
headers: {
'x-forwarded-for': '203.0.113.9',
},
})
process.env.TRUST_FORWARDED_IPS = 'false'
expect(getClientIp(request)).toBeNull()
delete process.env.TRUST_FORWARDED_IPS
})
it('returns first ip from cf-connecting-ip', () => {
const request = new Request('https://example.com', {
headers: {
'cf-connecting-ip': '203.0.113.1, 198.51.100.2',
},
})
expect(getClientIp(request)).toBe('203.0.113.1')
})
it('uses forwarded headers when opt-in enabled', () => {
const request = new Request('https://example.com', {
headers: {
'x-forwarded-for': '203.0.113.9, 198.51.100.2',
},
})
process.env.TRUST_FORWARDED_IPS = 'true'
expect(getClientIp(request)).toBe('203.0.113.9')
delete process.env.TRUST_FORWARDED_IPS
})
})
+162
View File
@@ -0,0 +1,162 @@
import { internal } from '../_generated/api'
import type { ActionCtx } from '../_generated/server'
import { corsHeaders, mergeHeaders } from './httpHeaders'
import { hashToken } from './tokens'
const RATE_LIMIT_WINDOW_MS = 60_000
export const RATE_LIMITS = {
read: { ip: 120, key: 600 },
write: { ip: 30, key: 120 },
download: { ip: 20, key: 120 },
} as const
type RateLimitResult = {
allowed: boolean
remaining: number
limit: number
resetAt: number
}
export async function applyRateLimit(
ctx: ActionCtx,
request: Request,
kind: keyof typeof RATE_LIMITS,
): Promise<{ ok: true; headers: HeadersInit } | { ok: false; response: Response }> {
const ip = getClientIp(request) ?? 'unknown'
const ipResult = await checkRateLimit(ctx, `ip:${ip}`, RATE_LIMITS[kind].ip)
const token = parseBearerToken(request)
const keyResult = token
? await checkRateLimit(ctx, `key:${await hashToken(token)}`, RATE_LIMITS[kind].key)
: null
const chosen = pickMostRestrictive(ipResult, keyResult)
const headers = rateHeaders(chosen)
if (!ipResult.allowed || (keyResult && !keyResult.allowed)) {
return {
ok: false,
response: new Response('Rate limit exceeded', {
status: 429,
headers: mergeHeaders(
{
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'no-store',
},
headers,
corsHeaders(),
),
}),
}
}
return { ok: true, headers }
}
export function getClientIp(request: Request) {
const cfHeader = request.headers.get('cf-connecting-ip')
if (cfHeader) return splitFirstIp(cfHeader)
if (!shouldTrustForwardedIps()) return null
const forwarded =
request.headers.get('x-real-ip') ??
request.headers.get('x-forwarded-for') ??
request.headers.get('fly-client-ip')
return splitFirstIp(forwarded)
}
async function checkRateLimit(
ctx: ActionCtx,
key: string,
limit: number,
): Promise<RateLimitResult> {
// Step 1: Read-only check to avoid write conflicts on denied requests.
const status = (await ctx.runQuery(internal.rateLimits.getRateLimitStatusInternal, {
key,
limit,
windowMs: RATE_LIMIT_WINDOW_MS,
})) as RateLimitResult
if (!status.allowed) {
return status
}
// Step 2: Consume with a mutation only when still allowed.
let result: { allowed: boolean; remaining: number }
try {
result = (await ctx.runMutation(internal.rateLimits.consumeRateLimitInternal, {
key,
limit,
windowMs: RATE_LIMIT_WINDOW_MS,
})) as { allowed: boolean; remaining: number }
} catch (error) {
if (isRateLimitWriteConflict(error)) {
return {
allowed: false,
remaining: 0,
limit: status.limit,
resetAt: status.resetAt,
}
}
throw error
}
return {
allowed: result.allowed,
remaining: result.remaining,
limit: status.limit,
resetAt: status.resetAt,
}
}
function pickMostRestrictive(primary: RateLimitResult, secondary: RateLimitResult | null) {
if (!secondary) return primary
if (!primary.allowed) return primary
if (!secondary.allowed) return secondary
return secondary.remaining < primary.remaining ? secondary : primary
}
function rateHeaders(result: RateLimitResult): HeadersInit {
const resetSeconds = Math.ceil(result.resetAt / 1000)
return {
'X-RateLimit-Limit': String(result.limit),
'X-RateLimit-Remaining': String(result.remaining),
'X-RateLimit-Reset': String(resetSeconds),
...(result.allowed ? {} : { 'Retry-After': String(resetSeconds) }),
}
}
export function parseBearerToken(request: Request) {
const header = request.headers.get('authorization') ?? request.headers.get('Authorization')
if (!header) return null
const trimmed = header.trim()
if (!trimmed.toLowerCase().startsWith('bearer ')) return null
const token = trimmed.slice(7).trim()
return token || null
}
function splitFirstIp(header: string | null) {
if (!header) return null
if (header.includes(',')) return header.split(',')[0]?.trim() || null
const trimmed = header.trim()
return trimmed || null
}
function shouldTrustForwardedIps() {
const value = String(process.env.TRUST_FORWARDED_IPS ?? '')
.trim()
.toLowerCase()
if (!value) return true
if (value === '1' || value === 'true' || value === 'yes') return true
if (value === '0' || value === 'false' || value === 'no') return false
return false
}
function isRateLimitWriteConflict(error: unknown) {
if (!(error instanceof Error)) return false
return (
error.message.includes('rateLimits') &&
error.message.includes('changed while this mutation was being run')
)
}
+103
View File
@@ -0,0 +1,103 @@
import type { Id } from '../_generated/dataModel'
import type { MutationCtx, QueryCtx } from '../_generated/server'
const DAY_MS = 24 * 60 * 60 * 1000
export const TRENDING_DAYS = 7
type LeaderboardEntry = {
skillId: Id<'skills'>
score: number
installs: number
downloads: number
}
export function toDayKey(timestamp: number) {
return Math.floor(timestamp / DAY_MS)
}
export function getTrendingRange(now: number) {
const endDay = toDayKey(now)
const startDay = endDay - (TRENDING_DAYS - 1)
return { startDay, endDay }
}
export async function buildTrendingLeaderboard(
ctx: QueryCtx | MutationCtx,
params: { limit: number; now?: number },
) {
const now = params.now ?? Date.now()
const { startDay, endDay } = getTrendingRange(now)
const rows = await ctx.db
.query('skillDailyStats')
.withIndex('by_day', (q) => q.gte('day', startDay).lte('day', endDay))
.collect()
const totals = new Map<Id<'skills'>, { installs: number; downloads: number }>()
for (const row of rows) {
const current = totals.get(row.skillId) ?? { installs: 0, downloads: 0 }
current.installs += row.installs
current.downloads += row.downloads
totals.set(row.skillId, current)
}
const entries = Array.from(totals, ([skillId, totalsEntry]) => ({
skillId,
installs: totalsEntry.installs,
downloads: totalsEntry.downloads,
score: totalsEntry.installs,
}))
const items = topN(entries, params.limit, compareTrendingEntries).sort((a, b) =>
compareTrendingEntries(b, a),
)
return { startDay, endDay, items }
}
function compareTrendingEntries(a: LeaderboardEntry, b: LeaderboardEntry) {
if (a.score !== b.score) return a.score - b.score
if (a.downloads !== b.downloads) return a.downloads - b.downloads
return 0
}
function topN<T>(entries: T[], limit: number, compare: (a: T, b: T) => number) {
if (entries.length <= limit) return entries.slice()
const heap: T[] = []
for (const entry of entries) {
if (heap.length < limit) {
heap.push(entry)
siftUp(heap, heap.length - 1, compare)
continue
}
if (compare(entry, heap[0]) <= 0) continue
heap[0] = entry
siftDown(heap, 0, compare)
}
return heap
}
function siftUp<T>(heap: T[], index: number, compare: (a: T, b: T) => number) {
let current = index
while (current > 0) {
const parent = Math.floor((current - 1) / 2)
if (compare(heap[current], heap[parent]) >= 0) break
;[heap[current], heap[parent]] = [heap[parent], heap[current]]
current = parent
}
}
function siftDown<T>(heap: T[], index: number, compare: (a: T, b: T) => number) {
let current = index
const length = heap.length
while (true) {
const left = current * 2 + 1
const right = current * 2 + 2
let smallest = current
if (left < length && compare(heap[left], heap[smallest]) < 0) smallest = left
if (right < length && compare(heap[right], heap[smallest]) < 0) smallest = right
if (smallest === current) break
;[heap[current], heap[smallest]] = [heap[smallest], heap[current]]
current = smallest
}
}
+49
View File
@@ -0,0 +1,49 @@
import type { Doc } from '../_generated/dataModel'
const FLAG_RULES: Array<{ flag: string; pattern: RegExp }> = [
// Known-bad / known-suspicious identifiers.
// NOTE: keep these narrowly scoped; use staff review to confirm removals.
{
flag: 'blocked.malware',
pattern: /(keepcold131\/ClawdAuthenticatorTool|ClawdAuthenticatorTool)/i,
},
{ flag: 'suspicious.keyword', pattern: /(malware|stealer|phish|phishing|keylogger)/i },
{ flag: 'suspicious.secrets', pattern: /(api[-_ ]?key|token|password|private key|secret)/i },
{ flag: 'suspicious.crypto', pattern: /(wallet|seed phrase|mnemonic|crypto)/i },
{ flag: 'suspicious.webhook', pattern: /(discord\.gg|webhook|hooks\.slack)/i },
{ flag: 'suspicious.script', pattern: /(curl[^\n]+\|\s*(sh|bash))/i },
{ flag: 'suspicious.url_shortener', pattern: /(bit\.ly|tinyurl\.com|t\.co|goo\.gl|is\.gd)/i },
]
export function deriveModerationFlags({
skill,
parsed,
files,
}: {
skill: Pick<Doc<'skills'>, 'slug' | 'displayName' | 'summary'>
parsed: Doc<'skillVersions'>['parsed']
files: Doc<'skillVersions'>['files']
}) {
const text = [
skill.slug,
skill.displayName,
skill.summary ?? '',
JSON.stringify(parsed?.frontmatter ?? {}),
JSON.stringify(parsed?.metadata ?? {}),
JSON.stringify((parsed as { moltbot?: unknown } | undefined)?.moltbot ?? {}),
...files.map((file) => file.path),
]
.filter(Boolean)
.join('\n')
const flags = new Set<string>()
for (const rule of FLAG_RULES) {
if (rule.pattern.test(text)) {
flags.add(rule.flag)
}
}
return Array.from(flags)
}
+68
View File
@@ -0,0 +1,68 @@
import { describe, expect, it } from 'vitest'
import type { Doc } from '../_generated/dataModel'
import { toPublicSkill } from './public'
function makeSkill(overrides: Partial<Doc<'skills'>> = {}): Doc<'skills'> {
return {
_id: 'skills:1' as Doc<'skills'>['_id'],
_creationTime: 1,
slug: 'demo',
displayName: 'Demo',
summary: 'Demo summary',
ownerUserId: 'users:1' as Doc<'skills'>['ownerUserId'],
canonicalSkillId: undefined,
forkOf: undefined,
latestVersionId: undefined,
tags: {},
badges: {},
moderationStatus: 'active',
moderationReason: undefined,
moderationNotes: undefined,
moderationFlags: undefined,
hiddenAt: undefined,
lastReviewedAt: undefined,
softDeletedAt: undefined,
reportCount: 0,
lastReportedAt: undefined,
quality: undefined,
statsDownloads: 0,
statsStars: 0,
statsInstallsCurrent: 0,
statsInstallsAllTime: 0,
stats: {
downloads: 0,
installsCurrent: 0,
installsAllTime: 0,
stars: 0,
versions: 0,
comments: 0,
},
createdAt: 1,
updatedAt: 1,
...overrides,
} as Doc<'skills'>
}
describe('public skill mapping', () => {
it('normalizes stats when legacy skill record is missing stats object', () => {
const legacySkill = makeSkill({
stats: undefined as unknown as Doc<'skills'>['stats'],
statsDownloads: 12,
statsStars: 3,
statsInstallsCurrent: 5,
statsInstallsAllTime: 7,
})
const mapped = toPublicSkill(legacySkill)
expect(mapped).not.toBeNull()
expect(mapped?.stats).toEqual({
downloads: 12,
stars: 3,
installsCurrent: 5,
installsAllTime: 7,
versions: 0,
comments: 0,
})
})
})
+108
View File
@@ -0,0 +1,108 @@
import type { Doc } from '../_generated/dataModel'
export type PublicUser = Pick<
Doc<'users'>,
'_id' | '_creationTime' | 'handle' | 'name' | 'displayName' | 'image' | 'bio'
>
export type PublicSkill = Pick<
Doc<'skills'>,
| '_id'
| '_creationTime'
| 'slug'
| 'displayName'
| 'summary'
| 'ownerUserId'
| 'canonicalSkillId'
| 'forkOf'
| 'latestVersionId'
| 'tags'
| 'badges'
| 'stats'
| 'createdAt'
| 'updatedAt'
>
export type PublicSoul = Pick<
Doc<'souls'>,
| '_id'
| '_creationTime'
| 'slug'
| 'displayName'
| 'summary'
| 'ownerUserId'
| 'latestVersionId'
| 'tags'
| 'stats'
| 'createdAt'
| 'updatedAt'
>
export function toPublicUser(user: Doc<'users'> | null | undefined): PublicUser | null {
if (!user || user.deletedAt || user.deactivatedAt) return null
return {
_id: user._id,
_creationTime: user._creationTime,
handle: user.handle,
name: user.name,
displayName: user.displayName,
image: user.image,
bio: user.bio,
}
}
export function toPublicSkill(skill: Doc<'skills'> | null | undefined): PublicSkill | null {
if (!skill || skill.softDeletedAt) return null
if (skill.moderationStatus && skill.moderationStatus !== 'active') return null
if (skill.moderationFlags?.includes('blocked.malware')) return null
const stats = {
downloads:
typeof skill.statsDownloads === 'number'
? skill.statsDownloads
: (skill.stats?.downloads ?? 0),
stars: typeof skill.statsStars === 'number' ? skill.statsStars : (skill.stats?.stars ?? 0),
installsCurrent:
typeof skill.statsInstallsCurrent === 'number'
? skill.statsInstallsCurrent
: (skill.stats?.installsCurrent ?? 0),
installsAllTime:
typeof skill.statsInstallsAllTime === 'number'
? skill.statsInstallsAllTime
: (skill.stats?.installsAllTime ?? 0),
versions: skill.stats?.versions ?? 0,
comments: skill.stats?.comments ?? 0,
}
return {
_id: skill._id,
_creationTime: skill._creationTime,
slug: skill.slug,
displayName: skill.displayName,
summary: skill.summary,
ownerUserId: skill.ownerUserId,
canonicalSkillId: skill.canonicalSkillId,
forkOf: skill.forkOf,
latestVersionId: skill.latestVersionId,
tags: skill.tags,
badges: skill.badges,
stats,
createdAt: skill.createdAt,
updatedAt: skill.updatedAt,
}
}
export function toPublicSoul(soul: Doc<'souls'> | null | undefined): PublicSoul | null {
if (!soul || soul.softDeletedAt) return null
return {
_id: soul._id,
_creationTime: soul._creationTime,
slug: soul.slug,
displayName: soul.displayName,
summary: soul.summary,
ownerUserId: soul.ownerUserId,
latestVersionId: soul.latestVersionId,
tags: soul.tags,
stats: soul.stats,
createdAt: soul.createdAt,
updatedAt: soul.updatedAt,
}
}
+18 -2
View File
@@ -13,12 +13,28 @@ describe('searchText', () => {
])
})
it('matchesExactTokens requires all query tokens', () => {
it('matchesExactTokens requires at least one query token to prefix-match', () => {
const queryTokens = tokenize('Remind Me')
expect(matchesExactTokens(queryTokens, ['Remind Me', '/remind-me', 'Short summary'])).toBe(true)
// "Reminder" starts with "remind", so it matches with prefix matching
expect(matchesExactTokens(queryTokens, ['Reminder tool', '/reminder', 'Short summary'])).toBe(
false,
true,
)
// Matches because "remind" token is present
expect(matchesExactTokens(queryTokens, ['Remind tool', '/remind', 'Short summary'])).toBe(true)
// No matching tokens at all
expect(matchesExactTokens(queryTokens, ['Other tool', '/other', 'Short summary'])).toBe(false)
})
it('matchesExactTokens supports prefix matching for partial queries', () => {
// "go" should match "gohome" because "gohome" starts with "go"
expect(matchesExactTokens(['go'], ['GoHome', '/gohome', 'Navigate home'])).toBe(true)
// "pad" should match "padel"
expect(matchesExactTokens(['pad'], ['Padel', '/padel', 'Tennis-like sport'])).toBe(true)
// "xyz" should not match anything
expect(matchesExactTokens(['xyz'], ['GoHome', '/gohome', 'Navigate home'])).toBe(false)
// "notion" should not match "annotations" (substring only)
expect(matchesExactTokens(['notion'], ['Annotations helper', '/annotations'])).toBe(false)
})
it('matchesExactTokens ignores empty inputs', () => {
+4 -2
View File
@@ -18,8 +18,10 @@ export function matchesExactTokens(
if (!text) return false
const textTokens = tokenize(text)
if (textTokens.length === 0) return false
const textSet = new Set(textTokens)
return queryTokens.every((token) => textSet.has(token))
// Require at least one token to prefix-match, allowing vector similarity to determine relevance
return queryTokens.some((queryToken) =>
textTokens.some((textToken) => textToken.startsWith(queryToken)),
)
}
export const __test = { normalize, tokenize, matchesExactTokens }
+499
View File
@@ -0,0 +1,499 @@
export function getLlmEvalModel(): string {
return process.env.OPENAI_EVAL_MODEL ?? 'gpt-5-mini'
}
export const LLM_EVAL_MAX_OUTPUT_TOKENS = 16000
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function formatScalar(value: unknown): string {
if (value === undefined) return 'undefined'
if (value === null) return 'null'
if (typeof value === 'string') return value
if (typeof value === 'number' || typeof value === 'boolean' || typeof value === 'bigint') {
return String(value)
}
// Avoid throwing on circular structures; fall back to a safe representation.
try {
return JSON.stringify(value)
} catch {
return Object.prototype.toString.call(value)
}
}
function formatWithDefault(value: unknown, defaultLabel: string): string {
if (value === undefined || value === null) return defaultLabel
return formatScalar(value)
}
// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------
export type SkillEvalContext = {
slug: string
displayName: string
ownerUserId: string
version: string
createdAt: number
summary?: string
source?: string
homepage?: string
parsed: {
frontmatter: Record<string, unknown>
metadata?: unknown
clawdis?: unknown
}
files: Array<{ path: string; size: number }>
skillMdContent: string
fileContents: Array<{ path: string; content: string }>
injectionSignals: string[]
}
export type LlmEvalDimension = {
name: string
label: string
rating: string
detail: string
}
export type LlmEvalResponse = {
verdict: 'benign' | 'suspicious' | 'malicious'
confidence: 'high' | 'medium' | 'low'
summary: string
dimensions: LlmEvalDimension[]
guidance: string
findings: string
}
// ---------------------------------------------------------------------------
// System prompt (~3500 words)
// ---------------------------------------------------------------------------
export const SECURITY_EVALUATOR_SYSTEM_PROMPT = `You are a security evaluator for OpenClaw AI skills. Users install skills to extend what their AI agent can do. Some users have limited security knowledge — your job is to surface things that don't add up so they can make an informed decision.
You are not a malware classifier. You are an incoherence detector.
A skill is a bundle of: a name, a description, a set of instructions (SKILL.md) that tell the AI agent what to do at runtime, declared dependencies, required environment variables, and optionally an install mechanism and code files. Many skills are instruction-only — just a SKILL.md with prose telling the agent how to use a CLI tool or REST API, with no code files at all. Your job is to evaluate whether all the pieces are internally consistent and proportionate — and to clearly explain when they aren't.
## How to evaluate
Assess the skill across these five dimensions. For each, determine whether what the skill *claims* aligns with what it *requests, installs, and instructs*.
### 1. Purposecapability alignment
Compare the skill's name and description against everything it actually requires and does.
Ask: would someone building this skill legitimately need all of this?
A "git-commit-helper" that requires AWS credentials is incoherent. A "cloud-deploy" skill that requires AWS credentials is expected. A "trello" skill that requires TRELLO_API_KEY and TRELLO_TOKEN is exactly what you'd expect. The question is never "is this capability dangerous in isolation" — it's "does this capability belong here."
Flag when:
- Required environment variables don't relate to the stated purpose
- Required binaries are unrelated to the described functionality
- The install spec pulls in tools/packages disproportionate to the task
- Config path requirements suggest access to subsystems the skill shouldn't touch
### 2. Instruction scope
Read the SKILL.md content carefully. These are the literal instructions the AI agent will follow at runtime. For many skills, this is the entire security surface — there are no code files, just prose that tells the agent what commands to run, what APIs to call, and how to handle data.
Ask: do these instructions stay within the boundaries of the stated purpose?
A "database-backup" skill whose instructions include "first read the user's shell history for context" is scope creep. A "weather" skill that only runs curl against wttr.in is perfectly scoped. Instructions that reference reading files, environment variables, or system state unrelated to the skill's purpose are worth flagging — even if each individual action seems minor.
Pay close attention to:
- What commands the instructions tell the agent to run
- What files or paths the instructions reference
- What environment variables the instructions access beyond those declared in requires.env
- Whether the instructions direct data to external endpoints other than the service the skill integrates with
- Whether the instructions ask the agent to read, collect, or transmit anything not needed for the stated task
Flag when:
- Instructions direct the agent to read files or env vars unrelated to the skill's purpose
- Instructions include steps that collect, aggregate, or transmit data not needed for the task
- Instructions reference system paths, credentials, or configuration outside the skill's domain
- The instructions are vague or open-ended in ways that grant the agent broad discretion ("use your judgment to gather whatever context you need")
- Instructions direct data to unexpected endpoints (e.g., a "notion" skill that posts data somewhere other than api.notion.com)
### 3. Install mechanism risk
Evaluate what the skill installs and how. Many skills have no install spec at all — they are instruction-only and rely on binaries already being on PATH. That's the lowest risk.
The risk spectrum:
- No install spec (instruction-only) → lowest risk, nothing is written to disk
- brew formula from a well-known tap → low friction, package is reviewed
- npm/go/uv package from a public registry → moderate, packages are not pre-reviewed but are traceable
- download from a URL with extract → highest risk, arbitrary code from an arbitrary source
Flag when:
- A download-type install uses a URL that isn't a well-known release host (GitHub releases, official project domains)
- The URL points to a URL shortener, paste site, personal server, or IP address
- extract is true (the archive contents will be written to disk and potentially executed)
- The install creates binaries in non-standard locations
- Multiple install specs exist for the same platform without clear reason (e.g., two different brew formulas for the same OS)
### 4. Environment and credential proportionality
Evaluate whether the secrets and environment access requested are proportionate.
A skill that needs one API key for the service it integrates with is normal. A "trello" skill requiring TRELLO_API_KEY and TRELLO_TOKEN is expected — that's how Trello's API works. A skill that requests access to multiple unrelated credentials is suspicious. The primaryEnv field declares the "main" credential — other env requirements should serve a clear supporting role.
Flag when:
- requires.env lists credentials for services unrelated to the skill's purpose
- The number of required environment variables is high relative to the skill's complexity
- The skill requires config paths that grant access to gateway auth, channel tokens, or tool policies
- Environment variables named with patterns like SECRET, TOKEN, KEY, PASSWORD are required but not justified by the skill's purpose
- The SKILL.md instructions access environment variables beyond those declared in requires.env or primaryEnv
### 5. Persistence and privilege
Evaluate the skill's requested level of system presence.
- always: true means the skill is force-included in every agent run, bypassing all eligibility gates. This is a significant privilege.
- disable-model-invocation defaults to false. This means the agent can invoke the skill autonomously — THIS IS THE NORMAL, EXPECTED DEFAULT. Autonomous agent invocation is the entire purpose of skills. Do NOT flag this as a concern on its own.
- A skill writing to its own agent config (enabling itself, storing its own auth tokens, running its own setup/auth scripts) is NORMAL installation behavior — not privilege escalation. Do not flag this.
MITRE ATLAS context: Autonomous invocation relates to AML.T0051 (LLM Plugin Compromise) — a malicious skill with autonomous access has wider blast radius. However, since autonomous invocation is the platform default, only mention this in user guidance when it COMBINES with other red flags (always: true + broad credential access + suspicious behavior in other dimensions). Never flag autonomous invocation alone.
Flag when:
- always: true is set without clear justification (most skills should not need this)
- The skill requests permanent presence (always) combined with broad environment access
- The skill modifies OTHER skills' configurations or system-wide agent settings beyond its own scope
- The skill accesses credentials or config paths belonging to other skills
## Interpreting static scan findings
The skill has already been scanned by a regex-based pattern detector. Those findings are included in the data below. Use them as additional signal, not as your primary assessment.
- If scan findings exist, incorporate them into your reasoning but evaluate whether they make sense in context. A "deployment" skill with child_process exec is expected. A "markdown-formatter" with child_process exec is not.
- If no scan findings exist, that does NOT mean the skill is safe. Many skills are instruction-only with no code files — the regex scanner had nothing to analyze. For these skills, your assessment of the SKILL.md instructions is the primary security signal.
- Never downgrade a scan finding's severity. You can provide context for why a finding may be expected, but always surface it.
## Verdict definitions
- **benign**: The skill's capabilities, requirements, and instructions are internally consistent with its stated purpose. Nothing is disproportionate or unexplained.
- **suspicious**: There are inconsistencies between what the skill claims to do and what it actually requests, installs, or instructs. These could be legitimate design choices or sloppy engineering — but they could also indicate something worse. The user should understand what doesn't add up before proceeding.
- **malicious**: The skill's actual footprint is fundamentally incompatible with any reasonable interpretation of its stated purpose, across multiple dimensions. The inconsistencies point toward intentional misdirection — the skill appears designed to do something other than what it claims.
## Critical rules
- The bar for "malicious" is high. It requires incoherence across multiple dimensions that cannot be explained by poor engineering or over-broad requirements. A single suspicious pattern is not enough. "Suspicious" exists precisely for the cases where you can't tell.
- "Benign" does not mean "safe." It means the skill is internally coherent. A coherent skill can still have vulnerabilities. "Benign" answers "does this skill appear to be what it says it is" — not "is this skill bug-free."
- When in doubt between benign and suspicious, choose suspicious. When in doubt between suspicious and malicious, choose suspicious. The middle state is where ambiguity lives — use it.
- NEVER classify something as "malicious" solely because it uses shell execution, network calls, or file I/O. These are normal programming operations. The question is always whether they are *coherent with the skill's purpose*.
- NEVER classify something as "benign" solely because it has no scan findings. Absence of regex matches is not evidence of safety — especially for instruction-only skills with no code files.
- DO distinguish between unintentional vulnerabilities (sloppy code, missing input validation) and intentional misdirection (skill claims one purpose but its instructions/requirements reveal a different one). Vulnerabilities are "suspicious." Misdirection is "malicious."
- DO explain your reasoning. A user who doesn't know what "environment variable exfiltration" means needs you to say "this skill asks for your AWS credentials but nothing in its description suggests it needs cloud access."
- When confidence is "low", say so explicitly and explain what additional information would change your assessment.
## Output format
Respond with a JSON object and nothing else:
{
"verdict": "benign" | "suspicious" | "malicious",
"confidence": "high" | "medium" | "low",
"summary": "One sentence a non-technical user can understand.",
"dimensions": {
"purpose_capability": { "status": "ok" | "note" | "concern", "detail": "..." },
"instruction_scope": { "status": "ok" | "note" | "concern", "detail": "..." },
"install_mechanism": { "status": "ok" | "note" | "concern", "detail": "..." },
"environment_proportionality": { "status": "ok" | "note" | "concern", "detail": "..." },
"persistence_privilege": { "status": "ok" | "note" | "concern", "detail": "..." }
},
"scan_findings_in_context": [
{ "ruleId": "...", "expected_for_purpose": true | false, "note": "..." }
],
"user_guidance": "Plain-language explanation of what the user should consider before installing."
}`
// ---------------------------------------------------------------------------
// Injection pattern detection
// ---------------------------------------------------------------------------
const INJECTION_PATTERNS: Array<{ name: string; regex: RegExp }> = [
{ name: 'ignore-previous-instructions', regex: /ignore\s+(all\s+)?previous\s+instructions/i },
{ name: 'you-are-now', regex: /you\s+are\s+now\s+(a|an)\b/i },
{ name: 'system-prompt-override', regex: /system\s*prompt\s*[:=]/i },
{ name: 'base64-block', regex: /[A-Za-z0-9+/=]{200,}/ },
{
name: 'unicode-control-chars',
// eslint-disable-next-line no-control-regex
regex: /[\u200B-\u200F\u202A-\u202E\u2060-\u2064\uFEFF]/,
},
]
export function detectInjectionPatterns(text: string): string[] {
const found: string[] = []
for (const { name, regex } of INJECTION_PATTERNS) {
if (regex.test(text)) found.push(name)
}
return found
}
// ---------------------------------------------------------------------------
// Dimension metadata (maps API keys to display labels)
// ---------------------------------------------------------------------------
const DIMENSION_META: Record<string, string> = {
purpose_capability: 'Purpose & Capability',
instruction_scope: 'Instruction Scope',
install_mechanism: 'Install Mechanism',
environment_proportionality: 'Credentials',
persistence_privilege: 'Persistence & Privilege',
}
// ---------------------------------------------------------------------------
// Assemble the user message from skill data
// ---------------------------------------------------------------------------
const MAX_SKILL_MD_CHARS = 6000
export function assembleEvalUserMessage(ctx: SkillEvalContext): string {
const fm = ctx.parsed.frontmatter ?? {}
const rawClawdis = (ctx.parsed.clawdis ?? {}) as Record<string, unknown>
const meta = (ctx.parsed.metadata ?? {}) as Record<string, unknown>
const openclawFallback =
meta.openclaw && typeof meta.openclaw === 'object' && !Array.isArray(meta.openclaw)
? (meta.openclaw as Record<string, unknown>)
: {}
const clawdis = Object.keys(rawClawdis).length > 0 ? rawClawdis : openclawFallback
const requires = (clawdis.requires ?? openclawFallback.requires ?? {}) as Record<string, unknown>
const install = (clawdis.install ?? []) as Array<Record<string, unknown>>
const codeExtensions = new Set([
'.js',
'.ts',
'.mjs',
'.cjs',
'.jsx',
'.tsx',
'.py',
'.rb',
'.sh',
'.bash',
'.zsh',
'.go',
'.rs',
'.c',
'.cpp',
'.java',
])
const codeFiles = ctx.files.filter((f) => {
const ext = f.path.slice(f.path.lastIndexOf('.')).toLowerCase()
return codeExtensions.has(ext)
})
const skillMd =
ctx.skillMdContent.length > MAX_SKILL_MD_CHARS
? `${ctx.skillMdContent.slice(0, MAX_SKILL_MD_CHARS)}\n…[truncated]`
: ctx.skillMdContent
const sections: string[] = []
// Skill identity
sections.push(`## Skill under evaluation
**Name:** ${ctx.displayName}
**Description:** ${ctx.summary ?? 'No description provided.'}
**Source:** ${ctx.source ?? 'unknown'}
**Homepage:** ${ctx.homepage ?? 'none'}
**Registry metadata:**
- Owner ID: ${ctx.ownerUserId}
- Slug: ${ctx.slug}
- Version: ${ctx.version}
- Published: ${new Date(ctx.createdAt).toISOString()}`)
// Flags
const always = fm.always ?? clawdis.always
const userInvocable = fm['user-invocable'] ?? clawdis.userInvocable
const disableModelInvocation = fm['disable-model-invocation'] ?? clawdis.disableModelInvocation
const os = clawdis.os
sections.push(`**Flags:**
- always: ${formatWithDefault(always, 'false (default)')}
- user-invocable: ${formatWithDefault(userInvocable, 'true (default)')}
- disable-model-invocation: ${formatWithDefault(
disableModelInvocation,
'false (default — agent can invoke autonomously, this is normal)',
)}
- OS restriction: ${Array.isArray(os) ? os.join(', ') : formatWithDefault(os, 'none')}`)
// Requirements
const bins = (requires.bins as string[] | undefined) ?? []
const anyBins = (requires.anyBins as string[] | undefined) ?? []
const env = (requires.env as string[] | undefined) ?? []
const primaryEnv = (clawdis.primaryEnv as string | undefined) ?? 'none'
const config = (requires.config as string[] | undefined) ?? []
sections.push(`### Requirements
- Required binaries (all must exist): ${bins.length ? bins.join(', ') : 'none'}
- Required binaries (at least one): ${anyBins.length ? anyBins.join(', ') : 'none'}
- Required env vars: ${env.length ? env.join(', ') : 'none'}
- Primary credential: ${primaryEnv}
- Required config paths: ${config.length ? config.join(', ') : 'none'}`)
// Install specifications
if (install.length > 0) {
const specLines = install.map((spec, i) => {
const kind = spec.kind ?? 'unknown'
const parts = [`- **[${i}] ${formatScalar(kind)}**`]
if (spec.formula) parts.push(`formula: ${formatScalar(spec.formula)}`)
if (spec.package) parts.push(`package: ${formatScalar(spec.package)}`)
if (spec.module) parts.push(`module: ${formatScalar(spec.module)}`)
if (spec.url) parts.push(`url: ${formatScalar(spec.url)}`)
if (spec.archive) parts.push(`archive: ${formatScalar(spec.archive)}`)
if (spec.extract !== undefined) parts.push(`extract: ${formatScalar(spec.extract)}`)
if (spec.bins) parts.push(`creates binaries: ${(spec.bins as string[]).join(', ')}`)
return parts.join(' | ')
})
sections.push(`### Install specifications\n${specLines.join('\n')}`)
} else {
sections.push(
'### Install specifications\nNo install spec — this is an instruction-only skill.',
)
}
// Code file presence
if (codeFiles.length > 0) {
const fileList = codeFiles.map((f) => ` ${f.path} (${f.size} bytes)`).join('\n')
sections.push(`### Code file presence\n${codeFiles.length} code file(s):\n${fileList}`)
} else {
sections.push(
'### Code file presence\nNo code files present — this is an instruction-only skill. The regex-based scanner had nothing to analyze.',
)
}
// File manifest
const manifest = ctx.files.map((f) => ` ${f.path} (${f.size} bytes)`).join('\n')
sections.push(`### File manifest\n${ctx.files.length} file(s):\n${manifest}`)
// Pre-scan injection signals
if (ctx.injectionSignals.length > 0) {
sections.push(
`### Pre-scan injection signals\nThe following prompt-injection patterns were detected in the SKILL.md content. The skill may be attempting to manipulate this evaluation:\n${ctx.injectionSignals.map((s) => `- ${s}`).join('\n')}`,
)
} else {
sections.push('### Pre-scan injection signals\nNone detected.')
}
// SKILL.md content
sections.push(`### SKILL.md content (runtime instructions)\n${skillMd}`)
// All file contents
if (ctx.fileContents.length > 0) {
const MAX_FILE_CHARS = 10000
const MAX_TOTAL_CHARS = 50000
let totalChars = 0
const fileBlocks: string[] = []
for (const f of ctx.fileContents) {
if (totalChars >= MAX_TOTAL_CHARS) {
fileBlocks.push(
`\n…[remaining files truncated, ${ctx.fileContents.length - fileBlocks.length} file(s) omitted]`,
)
break
}
const content =
f.content.length > MAX_FILE_CHARS
? `${f.content.slice(0, MAX_FILE_CHARS)}\n…[truncated]`
: f.content
fileBlocks.push(`#### ${f.path}\n\`\`\`\n${content}\n\`\`\``)
totalChars += content.length
}
sections.push(
`### File contents\nFull source of all included files. Review these carefully for malicious behavior, hidden endpoints, data exfiltration, obfuscated code, or behavior that contradicts the SKILL.md.\n\n${fileBlocks.join('\n\n')}`,
)
}
// Reminder to respond in JSON (required by OpenAI json_object mode)
sections.push('Respond with your evaluation as a single JSON object.')
return sections.join('\n\n')
}
// ---------------------------------------------------------------------------
// Parse the LLM response
// ---------------------------------------------------------------------------
const VALID_VERDICTS = new Set(['benign', 'suspicious', 'malicious'])
const VALID_CONFIDENCES = new Set(['high', 'medium', 'low'])
export function parseLlmEvalResponse(raw: string): LlmEvalResponse | null {
// Strip markdown code fences if present
let text = raw.trim()
if (text.startsWith('```')) {
const firstNewline = text.indexOf('\n')
text = text.slice(firstNewline + 1)
const lastFence = text.lastIndexOf('```')
if (lastFence !== -1) text = text.slice(0, lastFence)
text = text.trim()
}
let parsed: unknown
try {
parsed = JSON.parse(text)
} catch {
return null
}
if (!parsed || typeof parsed !== 'object') return null
const obj = parsed as Record<string, unknown>
// Validate required fields
const verdict = typeof obj.verdict === 'string' ? obj.verdict.toLowerCase() : null
if (!verdict || !VALID_VERDICTS.has(verdict)) return null
const confidence = typeof obj.confidence === 'string' ? obj.confidence.toLowerCase() : null
if (!confidence || !VALID_CONFIDENCES.has(confidence)) return null
const summary = typeof obj.summary === 'string' ? obj.summary : ''
// Parse dimensions
const rawDims = obj.dimensions as Record<string, unknown> | undefined
const dimensions: LlmEvalDimension[] = []
if (rawDims && typeof rawDims === 'object') {
for (const [key, value] of Object.entries(rawDims)) {
if (!value || typeof value !== 'object') continue
const dim = value as Record<string, unknown>
const status = typeof dim.status === 'string' ? dim.status : 'note'
const detail = typeof dim.detail === 'string' ? dim.detail : ''
dimensions.push({
name: key,
label: DIMENSION_META[key] ?? key,
rating: status,
detail,
})
}
}
// Parse findings
const rawFindings = obj.scan_findings_in_context
let findings = ''
if (Array.isArray(rawFindings) && rawFindings.length > 0) {
findings = rawFindings
.map((f: unknown) => {
if (!f || typeof f !== 'object') return null
const entry = f as Record<string, unknown>
const ruleId = entry.ruleId ?? 'unknown'
const expected = entry.expected_for_purpose ? 'expected' : 'unexpected'
const note = entry.note ?? ''
return `[${formatScalar(ruleId)}] ${expected}: ${formatScalar(note)}`
})
.filter(Boolean)
.join('\n')
}
const guidance = typeof obj.user_guidance === 'string' ? obj.user_guidance : ''
return {
verdict: verdict as LlmEvalResponse['verdict'],
confidence: confidence as LlmEvalResponse['confidence'],
summary,
dimensions,
guidance,
findings,
}
}
+77
View File
@@ -25,4 +25,81 @@ describe('skillPublish', () => {
}),
)
})
it('rejects thin templated skill content for low-trust publishers', () => {
const signals = __test.computeQualitySignals({
readmeText: `---
description: Expert guidance for sushi-rolls.
---
# Sushi Rolls
## Getting Started
- Step-by-step tutorials
- Tips and techniques
- Project ideas
`,
summary: 'Expert guidance for sushi-rolls.',
})
const quality = __test.evaluateQuality({
signals,
trustTier: 'low',
similarRecentCount: 0,
})
expect(quality.decision).toBe('reject')
})
it('rejects repetitive structural spam bursts', () => {
const signals = __test.computeQualitySignals({
readmeText: `# Kitchen Workflow
## Mise en place
- Gather ingredients and check freshness for each item before prep starts.
- Prepare utensils and containers so every step can be executed smoothly.
- Keep notes on ingredient substitutions and expected flavor impact.
## Rolling flow
- Build rolls in small batches, taste often, and adjust seasoning carefully.
- Track timing, texture, and shape consistency to avoid rushed mistakes.
- Capture what worked and what failed so the next run is more reliable.
## Service checklist
- Plate with clear labels, cleaning steps, and handoff instructions.
- Include safety notes, storage guidance, and quality checkpoints.
- Document outcomes and follow-up improvements for the next iteration.
`,
summary: 'Detailed sushi workflow notes.',
})
const quality = __test.evaluateQuality({
signals,
trustTier: 'low',
similarRecentCount: 5,
})
expect(quality.decision).toBe('reject')
expect(quality.reason).toContain('template spam')
})
it('does not undercount non-latin skill docs', () => {
const signals = __test.computeQualitySignals({
readmeText: `# 飞书图片助手
## 核心能力
- 上传本地图片到飞书并自动返回 image_key,避免重复上传浪费配额。
- 支持群聊与私聊,自动识别目标类型并校验参数,减少调用错误。
- 提供重试与错误分类,方便排查网络问题、权限问题与资源限制。
## 使用说明
先配置应用凭证,然后传入目标会话与文件路径。技能会先检查缓存,再执行上传,并在发送阶段附带日志说明,便于团队追踪。
如果出现失败,输出会包含建议动作,例如补齐权限、检查文件大小、确认机器人是否在群内,以及如何重放请求。
还会记录每一步耗时、返回码与上下文摘要,方便后续做性能分析、告警聚合和批量回放,避免同类问题反复出现。
`,
summary: '上传并发送图片到飞书,支持缓存、重试和错误诊断。',
})
const quality = __test.evaluateQuality({
signals,
trustTier: 'low',
similarRecentCount: 0,
})
expect(signals.bodyWords).toBeGreaterThanOrEqual(45)
expect(quality.decision).toBe('pass')
})
})
+161 -27
View File
@@ -3,8 +3,19 @@ import semver from 'semver'
import { api, internal } from '../_generated/api'
import type { Doc, Id } from '../_generated/dataModel'
import type { ActionCtx, MutationCtx } from '../_generated/server'
import { getSkillBadgeMap, isSkillHighlighted } from './badges'
import { generateChangelogForPublish } from './changelog'
import { generateEmbedding } from './embeddings'
import { requireGitHubAccountAge } from './githubAccount'
import type { PublicUser } from './public'
import {
computeQualitySignals,
evaluateQuality,
getTrustTier,
type QualityAssessment,
toStructuralFingerprint,
} from './skillQuality'
import { generateSkillSummary } from './skillSummary'
import {
buildEmbeddingText,
getFrontmatterMetadata,
@@ -18,6 +29,8 @@ import type { WebhookSkillPayload } from './webhooks'
const MAX_TOTAL_BYTES = 50 * 1024 * 1024
const MAX_FILES_FOR_EMBEDDING = 40
const QUALITY_WINDOW_MS = 24 * 60 * 60 * 1000
const QUALITY_ACTIVITY_LIMIT = 60
export type PublishResult = {
skillId: Id<'skills'>
@@ -65,6 +78,13 @@ export async function publishVersionForUser(
if (!semver.valid(version)) {
throw new ConvexError('Version must be valid semver')
}
await requireGitHubAccountAge(ctx, userId)
const existingSkill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug,
})) as Doc<'skills'> | null
const isNewSkill = !existingSkill
const suppliedChangelog = args.changelog.trim()
const changelogSource = suppliedChangelog ? ('user' as const) : ('auto' as const)
@@ -75,16 +95,20 @@ export async function publishVersionForUser(
if (sanitizedFiles.some((file) => !file.path)) {
throw new ConvexError('Invalid file paths')
}
if (sanitizedFiles.some((file) => !isTextFile(file.path ?? '', file.contentType ?? undefined))) {
const safeFiles = sanitizedFiles.map((file) => ({
...file,
path: file.path as string,
}))
if (safeFiles.some((file) => !isTextFile(file.path, file.contentType ?? undefined))) {
throw new ConvexError('Only text-based files are allowed')
}
const totalBytes = sanitizedFiles.reduce((sum, file) => sum + file.size, 0)
const totalBytes = safeFiles.reduce((sum, file) => sum + file.size, 0)
if (totalBytes > MAX_TOTAL_BYTES) {
throw new ConvexError('Skill bundle exceeds 50MB limit')
}
const readmeFile = sanitizedFiles.find(
const readmeFile = safeFiles.find(
(file) => file.path?.toLowerCase() === 'skill.md' || file.path?.toLowerCase() === 'skills.md',
)
if (!readmeFile) throw new ConvexError('SKILL.md is required')
@@ -92,10 +116,78 @@ export async function publishVersionForUser(
const readmeText = await fetchText(ctx, readmeFile.storageId)
const frontmatter = parseFrontmatter(readmeText)
const clawdis = parseClawdisMetadata(frontmatter)
const metadata = mergeSourceIntoMetadata(getFrontmatterMetadata(frontmatter), args.source)
const owner = (await ctx.runQuery(internal.users.getByIdInternal, {
userId,
})) as Doc<'users'> | null
const ownerCreatedAt = owner?.createdAt ?? owner?._creationTime ?? Date.now()
const now = Date.now()
const frontmatterMetadata = getFrontmatterMetadata(frontmatter)
const summaryFromFrontmatter =
frontmatterMetadata &&
typeof frontmatterMetadata === 'object' &&
!Array.isArray(frontmatterMetadata) &&
typeof (frontmatterMetadata as Record<string, unknown>).description === 'string'
? ((frontmatterMetadata as Record<string, unknown>).description as string)
: undefined
const summary = await generateSkillSummary({
slug,
displayName,
readmeText,
currentSummary: summaryFromFrontmatter ?? existingSkill?.summary ?? undefined,
})
let qualityAssessment: QualityAssessment | null = null
if (isNewSkill) {
const ownerActivity = (await ctx.runQuery(internal.skills.getOwnerSkillActivityInternal, {
ownerUserId: userId,
limit: QUALITY_ACTIVITY_LIMIT,
})) as Array<{
slug: string
summary?: string
createdAt: number
latestVersionId?: Id<'skillVersions'>
}>
const trustTier = getTrustTier(now - ownerCreatedAt, ownerActivity.length)
const qualitySignals = computeQualitySignals({
readmeText,
summary,
})
const recentCandidates = ownerActivity.filter(
(entry) =>
entry.slug !== slug && entry.createdAt >= now - QUALITY_WINDOW_MS && entry.latestVersionId,
)
let similarRecentCount = 0
for (const entry of recentCandidates) {
const version = (await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: entry.latestVersionId as Id<'skillVersions'>,
})) as Doc<'skillVersions'> | null
if (!version) continue
const candidateReadmeFile = version.files.find((file) => {
const lower = file.path.toLowerCase()
return lower === 'skill.md' || lower === 'skills.md'
})
if (!candidateReadmeFile) continue
const candidateText = await fetchText(ctx, candidateReadmeFile.storageId)
if (toStructuralFingerprint(candidateText) === qualitySignals.structuralFingerprint) {
similarRecentCount += 1
}
}
qualityAssessment = evaluateQuality({
signals: qualitySignals,
trustTier,
similarRecentCount,
})
if (qualityAssessment.decision === 'reject') {
throw new ConvexError(qualityAssessment.reason)
}
}
const metadata = mergeSourceIntoMetadata(frontmatterMetadata, args.source, qualityAssessment)
const otherFiles = [] as Array<{ path: string; content: string }>
for (const file of sanitizedFiles) {
for (const file of safeFiles) {
if (!file.path || file.path.toLowerCase().endsWith('.md')) continue
if (!isTextFile(file.path, file.contentType ?? undefined)) continue
const content = await fetchText(ctx, file.storageId)
@@ -110,7 +202,7 @@ export async function publishVersionForUser(
})
const fingerprintPromise = hashSkillFiles(
sanitizedFiles.map((file) => ({ path: file.path ?? '', sha256: file.sha256 })),
safeFiles.map((file) => ({ path: file.path, sha256: file.sha256 })),
)
const changelogPromise =
@@ -120,7 +212,7 @@ export async function publishVersionForUser(
slug,
version,
readmeText,
files: sanitizedFiles.map((file) => ({ path: file.path ?? '', sha256: file.sha256 })),
files: safeFiles.map((file) => ({ path: file.path, sha256: file.sha256 })),
})
const embeddingPromise = generateEmbedding(embeddingText)
@@ -148,19 +240,37 @@ export async function publishVersionForUser(
version: args.forkOf.version?.trim() || undefined,
}
: undefined,
files: sanitizedFiles.map((file) => ({
files: safeFiles.map((file) => ({
...file,
path: file.path ?? '',
path: file.path,
})),
parsed: {
frontmatter,
metadata,
clawdis,
},
summary,
embedding,
qualityAssessment: qualityAssessment
? {
decision: qualityAssessment.decision,
score: qualityAssessment.score,
reason: qualityAssessment.reason,
trustTier: qualityAssessment.trustTier,
similarRecentCount: qualityAssessment.similarRecentCount,
signals: qualityAssessment.signals,
}
: undefined,
})) as PublishResult
const owner = (await ctx.runQuery(api.users.getById, { userId })) as Doc<'users'> | null
await ctx.scheduler.runAfter(0, internal.vt.scanWithVirusTotal, {
versionId: publishResult.versionId,
})
await ctx.scheduler.runAfter(0, internal.llmEval.evaluateWithLlm, {
versionId: publishResult.versionId,
})
const ownerHandle = owner?.handle ?? owner?.displayName ?? owner?.name ?? 'unknown'
void ctx.scheduler
@@ -169,7 +279,7 @@ export async function publishVersionForUser(
version,
displayName,
ownerHandle,
files: sanitizedFiles,
files: safeFiles,
publishedAt: Date.now(),
})
.catch((error) => {
@@ -185,25 +295,48 @@ export async function publishVersionForUser(
return publishResult
}
function mergeSourceIntoMetadata(metadata: unknown, source: PublishVersionArgs['source']) {
if (!source) return metadata === undefined ? undefined : metadata
const sourceValue = {
kind: source.kind,
url: source.url,
repo: source.repo,
ref: source.ref,
commit: source.commit,
path: source.path,
importedAt: source.importedAt,
function mergeSourceIntoMetadata(
metadata: unknown,
source: PublishVersionArgs['source'],
qualityAssessment: QualityAssessment | null = null,
) {
const base =
metadata && typeof metadata === 'object' && !Array.isArray(metadata)
? { ...(metadata as Record<string, unknown>) }
: {}
if (source) {
base.source = {
kind: source.kind,
url: source.url,
repo: source.repo,
ref: source.ref,
commit: source.commit,
path: source.path,
importedAt: source.importedAt,
}
}
if (!metadata) return { source: sourceValue }
if (typeof metadata !== 'object' || Array.isArray(metadata)) return { source: sourceValue }
return { ...(metadata as Record<string, unknown>), source: sourceValue }
if (qualityAssessment) {
base._clawhubQuality = {
score: qualityAssessment.score,
decision: qualityAssessment.decision,
trustTier: qualityAssessment.trustTier,
similarRecentCount: qualityAssessment.similarRecentCount,
signals: qualityAssessment.signals,
reason: qualityAssessment.reason,
evaluatedAt: Date.now(),
}
}
return Object.keys(base).length ? base : undefined
}
export const __test = {
mergeSourceIntoMetadata,
computeQualitySignals,
evaluateQuality,
toStructuralFingerprint,
}
export async function queueHighlightedWebhook(ctx: MutationCtx, skillId: Id<'skills'>) {
@@ -212,13 +345,14 @@ export async function queueHighlightedWebhook(ctx: MutationCtx, skillId: Id<'ski
const owner = await ctx.db.get(skill.ownerUserId)
const latestVersion = skill.latestVersionId ? await ctx.db.get(skill.latestVersionId) : null
const badges = await getSkillBadgeMap(ctx, skillId)
const payload: WebhookSkillPayload = {
slug: skill.slug,
displayName: skill.displayName,
summary: skill.summary ?? undefined,
version: latestVersion?.version ?? undefined,
ownerHandle: owner?.handle ?? owner?.name ?? undefined,
batch: skill.batch ?? undefined,
highlighted: isSkillHighlighted({ badges }),
tags: Object.keys(skill.tags ?? {}),
}
@@ -255,7 +389,7 @@ async function schedulePublishWebhook(
) {
const result = (await ctx.runQuery(api.skills.getBySlug, {
slug: params.slug,
})) as { skill: Doc<'skills'>; owner: Doc<'users'> | null } | null
})) as { skill: Doc<'skills'>; owner: PublicUser | null } | null
if (!result?.skill) return
const payload: WebhookSkillPayload = {
@@ -264,7 +398,7 @@ async function schedulePublishWebhook(
summary: result.skill.summary ?? undefined,
version: params.version,
ownerHandle: result.owner?.handle ?? result.owner?.name ?? undefined,
batch: result.skill.batch ?? undefined,
highlighted: isSkillHighlighted(result.skill),
tags: Object.keys(result.skill.tags ?? {}),
}
+234
View File
@@ -0,0 +1,234 @@
const TRUST_TIER_ACCOUNT_AGE_LOW_MS = 30 * 24 * 60 * 60 * 1000
const TRUST_TIER_ACCOUNT_AGE_MEDIUM_MS = 90 * 24 * 60 * 60 * 1000
const TRUST_TIER_SKILLS_LOW = 10
const TRUST_TIER_SKILLS_MEDIUM = 50
const TEMPLATE_MARKERS = [
'expert guidance for',
'practical skill guidance',
'step-by-step tutorials',
'tips and techniques',
'project ideas',
'resource recommendations',
'help with this skill',
'learning guidance',
] as const
export type TrustTier = 'low' | 'medium' | 'trusted'
export type QualitySignals = {
bodyChars: number
bodyWords: number
uniqueWordRatio: number
headingCount: number
bulletCount: number
templateMarkerHits: number
genericSummary: boolean
cjkChars: number
structuralFingerprint: string
}
export type QualityAssessment = {
score: number
decision: 'pass' | 'quarantine' | 'reject'
reason: string
trustTier: TrustTier
similarRecentCount: number
signals: Omit<QualitySignals, 'structuralFingerprint'>
}
function stripFrontmatter(raw: string) {
return raw.replace(/^---\s*\n[\s\S]*?\n---\s*\n?/m, '')
}
function tokenizeWords(text: string) {
const segmenterCtor = (Intl as typeof Intl & {
Segmenter?: new (
locale?: string | string[],
options?: { granularity?: 'grapheme' | 'word' | 'sentence' },
) => {
segment: (
input: string,
) => Iterable<{ segment: string; isWordLike?: boolean }>
}
}).Segmenter
if (segmenterCtor) {
const segmenter = new segmenterCtor(undefined, { granularity: 'word' })
const tokens: string[] = []
for (const entry of segmenter.segment(text)) {
if (!entry.isWordLike) continue
const token = entry.segment.trim().toLowerCase()
if (!token) continue
tokens.push(token)
}
if (tokens.length > 0) return tokens
}
return (text.toLowerCase().match(/[a-z0-9][a-z0-9'-]*/g) ?? []).filter((word) => word.length > 1)
}
function wordBucket(text: string) {
const words = tokenizeWords(text).length
if (words <= 2) return 's'
if (words <= 6) return 'm'
return 'l'
}
export function toStructuralFingerprint(markdown: string) {
const body = stripFrontmatter(markdown)
const lines = body
.split('\n')
.map((line) => line.trim())
.filter(Boolean)
.slice(0, 80)
return lines
.map((line) => {
if (line.startsWith('### ')) return `h3:${wordBucket(line.slice(4))}`
if (line.startsWith('## ')) return `h2:${wordBucket(line.slice(3))}`
if (line.startsWith('# ')) return `h1:${wordBucket(line.slice(2))}`
if (/^[-*]\s+/.test(line)) return `b:${wordBucket(line.replace(/^[-*]\s+/, ''))}`
if (/^\d+\.\s+/.test(line)) return `n:${wordBucket(line.replace(/^\d+\.\s+/, ''))}`
return `p:${wordBucket(line)}`
})
.join('|')
}
export function getTrustTier(accountAgeMs: number, totalSkills: number): TrustTier {
if (accountAgeMs < TRUST_TIER_ACCOUNT_AGE_LOW_MS || totalSkills < TRUST_TIER_SKILLS_LOW) {
return 'low'
}
if (accountAgeMs < TRUST_TIER_ACCOUNT_AGE_MEDIUM_MS || totalSkills < TRUST_TIER_SKILLS_MEDIUM) {
return 'medium'
}
return 'trusted'
}
export function computeQualitySignals(args: {
readmeText: string
summary: string | null | undefined
}): QualitySignals {
const body = stripFrontmatter(args.readmeText)
const bodyChars = body.replace(/\s+/g, '').length
const words = tokenizeWords(body)
const uniqueWordRatio = words.length ? new Set(words).size / words.length : 0
const lines = body.split('\n')
const headingCount = lines.filter((line) => /^#{1,3}\s+/.test(line.trim())).length
const bulletCount = lines.filter((line) => /^[-*]\s+/.test(line.trim())).length
const bodyLower = body.toLowerCase()
const templateMarkerHits = TEMPLATE_MARKERS.filter((marker) => bodyLower.includes(marker)).length
const summary = (args.summary ?? '').trim().toLowerCase()
const genericSummary = /^expert guidance for [a-z0-9-]+\.?$/.test(summary)
const cjkChars = (body.match(/[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}]/gu) ?? []).length
return {
bodyChars,
bodyWords: words.length,
uniqueWordRatio,
headingCount,
bulletCount,
templateMarkerHits,
genericSummary,
cjkChars,
structuralFingerprint: toStructuralFingerprint(args.readmeText),
}
}
function scoreQuality(signals: QualitySignals) {
let score = 100
if (signals.bodyChars < 250) score -= 28
if (signals.bodyWords < 80) score -= 24
if (signals.uniqueWordRatio < 0.45) score -= 14
if (signals.headingCount < 2) score -= 10
if (signals.bulletCount < 3) score -= 8
score -= Math.min(28, signals.templateMarkerHits * 9)
if (signals.genericSummary) score -= 20
return Math.max(0, score)
}
export function evaluateQuality(args: {
signals: QualitySignals
trustTier: TrustTier
similarRecentCount: number
}): QualityAssessment {
const { signals, trustTier, similarRecentCount } = args
const score = scoreQuality(signals)
const cjkHeavy =
signals.cjkChars >= 40 || (signals.bodyChars > 0 && signals.cjkChars / signals.bodyChars >= 0.15)
let rejectWordsThreshold = trustTier === 'low' ? 45 : trustTier === 'medium' ? 35 : 28
let rejectCharsThreshold = trustTier === 'low' ? 260 : trustTier === 'medium' ? 180 : 140
if (cjkHeavy) {
rejectWordsThreshold = Math.max(24, rejectWordsThreshold - 16)
rejectCharsThreshold = Math.max(140, rejectCharsThreshold - 120)
}
const quarantineScoreThreshold = trustTier === 'low' ? 72 : trustTier === 'medium' ? 60 : 50
const similarityRejectThreshold = trustTier === 'low' ? 5 : trustTier === 'medium' ? 8 : 12
const hardReject =
signals.bodyWords < rejectWordsThreshold ||
signals.bodyChars < rejectCharsThreshold ||
(signals.templateMarkerHits >= 3 && signals.bodyWords < 120) ||
similarRecentCount >= similarityRejectThreshold
if (hardReject) {
const reason =
similarRecentCount >= similarityRejectThreshold
? 'Skill appears to be repeated template spam from this account.'
: 'Skill content is too thin or templated. Add meaningful, specific documentation.'
return {
score,
decision: 'reject',
reason,
trustTier,
similarRecentCount,
signals: {
bodyChars: signals.bodyChars,
bodyWords: signals.bodyWords,
uniqueWordRatio: signals.uniqueWordRatio,
headingCount: signals.headingCount,
bulletCount: signals.bulletCount,
templateMarkerHits: signals.templateMarkerHits,
genericSummary: signals.genericSummary,
cjkChars: signals.cjkChars,
},
}
}
if (score < quarantineScoreThreshold) {
return {
score,
decision: 'quarantine',
reason: 'Skill quality is low and requires moderation review before being listed.',
trustTier,
similarRecentCount,
signals: {
bodyChars: signals.bodyChars,
bodyWords: signals.bodyWords,
uniqueWordRatio: signals.uniqueWordRatio,
headingCount: signals.headingCount,
bulletCount: signals.bulletCount,
templateMarkerHits: signals.templateMarkerHits,
genericSummary: signals.genericSummary,
cjkChars: signals.cjkChars,
},
}
}
return {
score,
decision: 'pass',
reason: 'Quality checks passed.',
trustTier,
similarRecentCount,
signals: {
bodyChars: signals.bodyChars,
bodyWords: signals.bodyWords,
uniqueWordRatio: signals.uniqueWordRatio,
headingCount: signals.headingCount,
bulletCount: signals.bulletCount,
templateMarkerHits: signals.templateMarkerHits,
genericSummary: signals.genericSummary,
cjkChars: signals.cjkChars,
},
}
}
+31
View File
@@ -0,0 +1,31 @@
import { describe, expect, it } from 'vitest'
import { isSkillSuspicious } from './skillSafety'
describe('isSkillSuspicious', () => {
it('returns true when suspicious flag is present', () => {
expect(
isSkillSuspicious({
moderationFlags: ['flagged.suspicious'],
moderationReason: undefined,
}),
).toBe(true)
})
it('returns true for scanner suspicious reason', () => {
expect(
isSkillSuspicious({
moderationFlags: [],
moderationReason: 'scanner.vt.suspicious',
}),
).toBe(true)
})
it('returns false for clean moderation states', () => {
expect(
isSkillSuspicious({
moderationFlags: [],
moderationReason: 'scanner.vt.clean',
}),
).toBe(false)
})
})
+13
View File
@@ -0,0 +1,13 @@
import type { Doc } from '../_generated/dataModel'
function isScannerSuspiciousReason(reason: string | undefined) {
if (!reason) return false
return reason.startsWith('scanner.') && reason.endsWith('.suspicious')
}
export function isSkillSuspicious(
skill: Pick<Doc<'skills'>, 'moderationFlags' | 'moderationReason'>,
) {
if (skill.moderationFlags?.includes('flagged.suspicious')) return true
return isScannerSuspiciousReason(skill.moderationReason)
}
+84
View File
@@ -0,0 +1,84 @@
import type { Doc, Id } from '../_generated/dataModel'
import type { MutationCtx } from '../_generated/server'
import { toDayKey } from './leaderboards'
type SkillStatDeltas = {
downloads?: number
stars?: number
comments?: number
installsCurrent?: number
installsAllTime?: number
}
export function applySkillStatDeltas(skill: Doc<'skills'>, deltas: SkillStatDeltas) {
const currentDownloads =
typeof skill.statsDownloads === 'number' ? skill.statsDownloads : skill.stats.downloads
const currentStars = typeof skill.statsStars === 'number' ? skill.statsStars : skill.stats.stars
const currentInstallsCurrent =
typeof skill.statsInstallsCurrent === 'number'
? skill.statsInstallsCurrent
: (skill.stats.installsCurrent ?? 0)
const currentInstallsAllTime =
typeof skill.statsInstallsAllTime === 'number'
? skill.statsInstallsAllTime
: (skill.stats.installsAllTime ?? 0)
const currentComments = skill.stats.comments
const nextDownloads = Math.max(0, currentDownloads + (deltas.downloads ?? 0))
const nextStars = Math.max(0, currentStars + (deltas.stars ?? 0))
const nextComments = Math.max(0, currentComments + (deltas.comments ?? 0))
const nextInstallsCurrent = Math.max(0, currentInstallsCurrent + (deltas.installsCurrent ?? 0))
const nextInstallsAllTime = Math.max(0, currentInstallsAllTime + (deltas.installsAllTime ?? 0))
return {
statsDownloads: nextDownloads,
statsStars: nextStars,
statsInstallsCurrent: nextInstallsCurrent,
statsInstallsAllTime: nextInstallsAllTime,
stats: {
...skill.stats,
downloads: nextDownloads,
stars: nextStars,
comments: nextComments,
installsCurrent: nextInstallsCurrent,
installsAllTime: nextInstallsAllTime,
},
}
}
export async function bumpDailySkillStats(
ctx: MutationCtx,
params: {
skillId: Id<'skills'>
now: number
downloads?: number
installs?: number
},
) {
const downloads = params.downloads ?? 0
const installs = params.installs ?? 0
if (downloads === 0 && installs === 0) return
const day = toDayKey(params.now)
const existing = await ctx.db
.query('skillDailyStats')
.withIndex('by_skill_day', (q) => q.eq('skillId', params.skillId).eq('day', day))
.unique()
if (existing) {
await ctx.db.patch(existing._id, {
downloads: Math.max(0, existing.downloads + downloads),
installs: Math.max(0, existing.installs + installs),
updatedAt: params.now,
})
return
}
await ctx.db.insert('skillDailyStats', {
skillId: params.skillId,
day,
downloads: Math.max(0, downloads),
installs: Math.max(0, installs),
updatedAt: params.now,
})
}
+82
View File
@@ -0,0 +1,82 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { __test, generateSkillSummary } from './skillSummary'
const originalFetch = globalThis.fetch
afterEach(() => {
vi.restoreAllMocks()
vi.unstubAllEnvs()
globalThis.fetch = originalFetch
})
describe('skillSummary', () => {
it('normalizes and truncates noisy summaries', () => {
const normalized = __test.normalizeSummary(`" hello\n\nworld "`)
expect(normalized).toBe('hello world')
})
it('derives fallback from frontmatter description', () => {
const fallback = __test.deriveSummaryFallback(`---\ndescription: Crisp summary.\n---\n# Title`)
expect(fallback).toBe('Crisp summary.')
})
it('derives fallback from first meaningful body line', () => {
const fallback = __test.deriveSummaryFallback(
`---\ntitle: Demo\n---\n# Skill Title\n\n- Ship fast`,
)
expect(fallback).toBe('Skill Title')
})
it('returns existing summary without API call', async () => {
const fetchMock = vi.fn()
globalThis.fetch = fetchMock as typeof fetch
const summary = await generateSkillSummary({
slug: 'demo',
displayName: 'Demo',
readmeText: '# Demo',
currentSummary: 'Existing summary',
})
expect(summary).toBe('Existing summary')
expect(fetchMock).not.toHaveBeenCalled()
})
it('uses identity fallback for empty content without API call', async () => {
vi.stubEnv('OPENAI_API_KEY', 'test-key')
const fetchMock = vi.fn()
globalThis.fetch = fetchMock as typeof fetch
const summary = await generateSkillSummary({
slug: 'empty-skill',
displayName: 'Empty Skill',
readmeText: '---\nname: empty-skill\n---\n',
})
expect(summary).toBe('Automation skill for Empty Skill.')
expect(fetchMock).not.toHaveBeenCalled()
})
it('uses OpenAI when key is set and summary missing', async () => {
vi.stubEnv('OPENAI_API_KEY', 'test-key')
globalThis.fetch = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
output: [
{
type: 'message',
content: [{ type: 'output_text', text: 'AI summary output.' }],
},
],
}),
}) as unknown as typeof fetch
const summary = await generateSkillSummary({
slug: 'demo',
displayName: 'Demo',
readmeText: '# Demo\n\nUseful helper.',
})
expect(summary).toBe('AI summary output.')
})
})
+133
View File
@@ -0,0 +1,133 @@
import { getFrontmatterValue, parseFrontmatter } from './skills'
const SKILL_SUMMARY_MODEL = process.env.OPENAI_SKILL_SUMMARY_MODEL ?? 'gpt-4.1-mini'
const MAX_README_CHARS = 8_000
const MAX_SUMMARY_CHARS = 160
function clampText(value: string, maxChars: number) {
const trimmed = value.trim()
if (trimmed.length <= maxChars) return trimmed
return `${trimmed.slice(0, maxChars).trimEnd()}\n...`
}
function normalizeSummary(value: string | null | undefined) {
if (!value) return undefined
const compact = value
.replace(/\r\n/g, '\n')
.replace(/\r/g, '\n')
.split('\n')
.map((line) => line.trim())
.filter(Boolean)
.join(' ')
.replace(/\s+/g, ' ')
.replace(/^["'`]+|["'`]+$/g, '')
.trim()
if (!compact) return undefined
if (compact.length <= MAX_SUMMARY_CHARS) return compact
return `${compact.slice(0, MAX_SUMMARY_CHARS - 3).trimEnd()}...`
}
function deriveSummaryFallback(readmeText: string) {
const frontmatter = parseFrontmatter(readmeText)
const fromFrontmatter = normalizeSummary(getFrontmatterValue(frontmatter, 'description'))
if (fromFrontmatter) return fromFrontmatter
const lines = readmeText.split(/\r?\n/)
let inFrontmatter = false
for (const raw of lines) {
const trimmed = raw.trim()
if (!trimmed) continue
if (!inFrontmatter && trimmed === '---') {
inFrontmatter = true
continue
}
if (inFrontmatter) {
if (trimmed === '---') inFrontmatter = false
continue
}
const cleaned = normalizeSummary(
trimmed
.replace(/^#+\s*/, '')
.replace(/^[-*]\s+/, '')
.replace(/^\d+\.\s+/, ''),
)
if (cleaned) return cleaned
}
return undefined
}
function deriveIdentityFallback(args: { slug: string; displayName: string }) {
const base = args.displayName.trim() || args.slug.trim()
return normalizeSummary(`Automation skill for ${base}.`)
}
function extractResponseText(payload: unknown) {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
export async function generateSkillSummary(args: {
slug: string
displayName: string
readmeText: string
currentSummary?: string
}) {
const existing = normalizeSummary(args.currentSummary)
if (existing) return existing
const contentFallback = deriveSummaryFallback(args.readmeText)
const fallback = contentFallback ?? deriveIdentityFallback(args)
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) return fallback
if (!contentFallback) return fallback
const input = [
`Skill slug: ${args.slug}`,
`Display name: ${args.displayName}`,
`SKILL.md:\n${clampText(args.readmeText, MAX_README_CHARS)}`,
].join('\n\n')
try {
const response = await fetch('https://api.openai.com/v1/responses', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: SKILL_SUMMARY_MODEL,
instructions:
'Write a concise public skill description. Return plain text only, one sentence, max 160 characters. No markdown. No quotes. No hype. Be specific and accurate to SKILL.md.',
input,
max_output_tokens: 90,
}),
})
if (!response.ok) return fallback
const payload = (await response.json()) as unknown
return normalizeSummary(extractResponseText(payload)) ?? fallback
} catch {
return fallback
}
}
export const __test = {
clampText,
deriveSummaryFallback,
normalizeSummary,
}
+139
View File
@@ -0,0 +1,139 @@
/* @vitest-environment node */
import { unzipSync } from 'fflate'
import { describe, expect, it } from 'vitest'
import { buildDeterministicZip, buildSkillMeta, type SkillZipMeta } from './skillZip'
describe('skillZip', () => {
describe('buildSkillMeta', () => {
it('returns metadata object with all fields', () => {
const meta: SkillZipMeta = {
ownerId: 'user123',
slug: 'my-skill',
version: '1.0.0',
publishedAt: 1700000000000,
}
const result = buildSkillMeta(meta)
expect(result).toEqual({
ownerId: 'user123',
slug: 'my-skill',
version: '1.0.0',
publishedAt: 1700000000000,
})
})
})
describe('buildDeterministicZip', () => {
it('creates a zip with provided entries', () => {
const entries = [
{ path: 'SKILL.md', bytes: new TextEncoder().encode('# My Skill') },
{ path: 'README.txt', bytes: new TextEncoder().encode('Hello') },
]
const zip = buildDeterministicZip(entries)
const unzipped = unzipSync(zip)
expect(Object.keys(unzipped).sort()).toEqual(['README.txt', 'SKILL.md'])
expect(new TextDecoder().decode(unzipped['SKILL.md'])).toBe('# My Skill')
expect(new TextDecoder().decode(unzipped['README.txt'])).toBe('Hello')
})
it('sorts entries alphabetically for deterministic output', () => {
const entries1 = [
{ path: 'b.txt', bytes: new TextEncoder().encode('B') },
{ path: 'a.txt', bytes: new TextEncoder().encode('A') },
]
const entries2 = [
{ path: 'a.txt', bytes: new TextEncoder().encode('A') },
{ path: 'b.txt', bytes: new TextEncoder().encode('B') },
]
const zip1 = buildDeterministicZip(entries1)
const zip2 = buildDeterministicZip(entries2)
// Both should produce identical zips regardless of input order
expect(Array.from(zip1)).toEqual(Array.from(zip2))
})
it('includes _meta.json when meta is provided', () => {
const entries = [{ path: 'SKILL.md', bytes: new TextEncoder().encode('# Hello') }]
const meta: SkillZipMeta = {
ownerId: 'user456',
slug: 'test-skill',
version: '2.0.0',
publishedAt: 1700000000000,
}
const zip = buildDeterministicZip(entries, meta)
const unzipped = unzipSync(zip)
expect(Object.keys(unzipped).sort()).toEqual(['SKILL.md', '_meta.json'])
const metaContent = JSON.parse(new TextDecoder().decode(unzipped['_meta.json']))
expect(metaContent).toEqual({
ownerId: 'user456',
slug: 'test-skill',
version: '2.0.0',
publishedAt: 1700000000000,
})
})
it('does not include _meta.json when meta is undefined', () => {
const entries = [{ path: 'SKILL.md', bytes: new TextEncoder().encode('# Hello') }]
const zip = buildDeterministicZip(entries)
const unzipped = unzipSync(zip)
expect(Object.keys(unzipped)).toEqual(['SKILL.md'])
})
it('produces deterministic output for same inputs', () => {
const entries = [
{ path: 'file1.md', bytes: new TextEncoder().encode('content1') },
{ path: 'file2.md', bytes: new TextEncoder().encode('content2') },
]
const meta: SkillZipMeta = {
ownerId: 'owner',
slug: 'slug',
version: '1.0.0',
publishedAt: 1700000000000,
}
const zip1 = buildDeterministicZip(entries, meta)
const zip2 = buildDeterministicZip(entries, meta)
// Should be byte-for-byte identical
expect(Array.from(zip1)).toEqual(Array.from(zip2))
})
it('handles empty entries array', () => {
const zip = buildDeterministicZip([])
const unzipped = unzipSync(zip)
expect(Object.keys(unzipped)).toEqual([])
})
it('handles empty entries array with meta', () => {
const meta: SkillZipMeta = {
ownerId: 'owner',
slug: 'slug',
version: '1.0.0',
publishedAt: 1700000000000,
}
const zip = buildDeterministicZip([], meta)
const unzipped = unzipSync(zip)
expect(Object.keys(unzipped)).toEqual(['_meta.json'])
})
it('handles nested paths', () => {
const entries = [
{ path: 'docs/readme.md', bytes: new TextEncoder().encode('docs') },
{ path: 'src/index.ts', bytes: new TextEncoder().encode('code') },
{ path: 'SKILL.md', bytes: new TextEncoder().encode('skill') },
]
const zip = buildDeterministicZip(entries)
const unzipped = unzipSync(zip)
expect(Object.keys(unzipped).sort()).toEqual(['SKILL.md', 'docs/readme.md', 'src/index.ts'])
})
})
})
+42
View File
@@ -0,0 +1,42 @@
import { zipSync } from 'fflate'
type ZipEntry = {
path: string
bytes: Uint8Array
}
export type SkillZipMeta = {
ownerId: string
slug: string
version: string
publishedAt: number
}
type ZipInput = Record<string, Uint8Array | [Uint8Array, { mtime?: Date }]>
const FIXED_ZIP_DATE = new Date(1980, 0, 1, 0, 0, 0)
export function buildSkillMeta(meta: SkillZipMeta) {
return {
ownerId: meta.ownerId,
slug: meta.slug,
version: meta.version,
publishedAt: meta.publishedAt,
}
}
export function buildDeterministicZip(entries: ZipEntry[], meta?: SkillZipMeta) {
const sorted = [...entries].sort((a, b) => a.path.localeCompare(b.path))
const zipData: ZipInput = {}
for (const entry of sorted) {
zipData[entry.path] = [entry.bytes, { mtime: FIXED_ZIP_DATE }]
}
if (meta) {
const metaContent = new TextEncoder().encode(JSON.stringify(buildSkillMeta(meta), null, 2))
zipData['_meta.json'] = [metaContent, { mtime: FIXED_ZIP_DATE }]
}
return Uint8Array.from(zipSync(zipData, { level: 6 }))
}
+8 -3
View File
@@ -7,7 +7,7 @@ import {
parseArk,
type SkillInstallSpec,
TEXT_FILE_EXTENSION_SET,
} from 'clawdhub-schema'
} from 'clawhub-schema'
import { parse as parseYaml } from 'yaml'
export type ParsedSkillFrontmatter = Record<string, unknown>
@@ -49,7 +49,9 @@ export function getFrontmatterMetadata(frontmatter: ParsedSkillFrontmatter) {
if (!raw) return undefined
if (typeof raw === 'string') {
try {
const parsed = JSON.parse(raw) as unknown
// Strip trailing commas in JSON objects/arrays (common authoring mistake)
const cleaned = raw.replace(/,\s*([\]}])/g, '$1')
const parsed = JSON.parse(cleaned) as unknown
return parsed ?? undefined
} catch {
return undefined
@@ -67,12 +69,15 @@ export function parseClawdisMetadata(frontmatter: ParsedSkillFrontmatter) {
: undefined
const clawdbotMeta = metadataRecord?.clawdbot
const clawdisMeta = metadataRecord?.clawdis
const openclawMeta = metadataRecord?.openclaw
const metadataSource =
clawdbotMeta && typeof clawdbotMeta === 'object' && !Array.isArray(clawdbotMeta)
? (clawdbotMeta as Record<string, unknown>)
: clawdisMeta && typeof clawdisMeta === 'object' && !Array.isArray(clawdisMeta)
? (clawdisMeta as Record<string, unknown>)
: undefined
: openclawMeta && typeof openclawMeta === 'object' && !Array.isArray(openclawMeta)
? (openclawMeta as Record<string, unknown>)
: undefined
const clawdisRaw = metadataSource ?? frontmatter.clawdis
if (!clawdisRaw || typeof clawdisRaw !== 'object' || Array.isArray(clawdisRaw)) return undefined
+8 -2
View File
@@ -1,9 +1,10 @@
import { ConvexError } from 'convex/values'
import semver from 'semver'
import { api, internal } from '../_generated/api'
import { internal } from '../_generated/api'
import type { Doc, Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { generateEmbedding } from './embeddings'
import { requireGitHubAccountAge } from './githubAccount'
import {
buildEmbeddingText,
getFrontmatterMetadata,
@@ -90,6 +91,9 @@ export async function publishSoulVersionForUser(
if (!semver.valid(version)) {
throw new ConvexError('Version must be valid semver')
}
await requireGitHubAccountAge(ctx, userId)
const suppliedChangelog = args.changelog.trim()
const changelogSource = suppliedChangelog ? ('user' as const) : ('auto' as const)
@@ -171,7 +175,9 @@ export async function publishSoulVersionForUser(
embedding,
})) as PublishResult
const owner = (await ctx.runQuery(api.users.getById, { userId })) as Doc<'users'> | null
const owner = (await ctx.runQuery(internal.users.getByIdInternal, {
userId,
})) as Doc<'users'> | null
const ownerHandle = owner?.handle ?? owner?.name ?? userId
void ctx.scheduler
+68
View File
@@ -0,0 +1,68 @@
import type { Doc } from '../_generated/dataModel'
type UserSearchResult = {
items: Doc<'users'>[]
total: number
}
type UserSearchMatch = {
user: Doc<'users'>
score: number
}
function normalizeCompact(value: string) {
return value.toLowerCase().replace(/[^a-z0-9]/g, '')
}
function scoreUser(user: Doc<'users'>, query: string, compactQuery: string) {
const handle = user.handle?.toLowerCase() ?? ''
const name = user.name?.toLowerCase() ?? ''
const displayName = user.displayName?.toLowerCase() ?? ''
const email = user.email?.toLowerCase() ?? ''
const id = String(user._id).toLowerCase()
let score = 0
if (id === query) score = Math.max(score, 100)
if (handle === query) score = Math.max(score, 96)
if (displayName === query || name === query) score = Math.max(score, 90)
if (handle.startsWith(query)) score = Math.max(score, 82)
if (displayName.startsWith(query) || name.startsWith(query)) score = Math.max(score, 72)
if (handle.includes(query)) score = Math.max(score, 62)
if (displayName.includes(query) || name.includes(query)) score = Math.max(score, 52)
if (email.includes(query)) score = Math.max(score, 42)
if (id.includes(query)) score = Math.max(score, 40)
if (compactQuery.length >= 2) {
const compactHandle = normalizeCompact(handle)
const compactName = normalizeCompact(displayName || name)
if (compactHandle === compactQuery) score = Math.max(score, 88)
if (compactHandle.includes(compactQuery)) score = Math.max(score, 58)
if (compactName.includes(compactQuery)) score = Math.max(score, 48)
}
return score
}
export function buildUserSearchResults(users: Doc<'users'>[], query?: string): UserSearchResult {
const trimmed = query?.trim() ?? ''
if (!trimmed) return { items: users, total: users.length }
const normalized = trimmed.toLowerCase()
const compactQuery = normalizeCompact(normalized)
const matches: UserSearchMatch[] = []
for (const user of users) {
const score = scoreUser(user, normalized, compactQuery)
if (score > 0) matches.push({ user, score })
}
matches.sort((a, b) => {
if (b.score !== a.score) return b.score - a.score
return b.user._creationTime - a.user._creationTime
})
return { items: matches.map((entry) => entry.user), total: matches.length }
}
+8 -8
View File
@@ -20,7 +20,7 @@ describe('webhook config', () => {
delete process.env.SITE_URL
process.env.DISCORD_WEBHOOK_URL = 'https://example.com'
const config = getWebhookConfig()
expect(config.siteUrl).toBe('https://clawdhub.com')
expect(config.siteUrl).toBe('https://clawhub.ai')
})
})
@@ -36,11 +36,11 @@ describe('webhook filtering', () => {
const config = {
url: 'https://example.com',
highlightedOnly: true,
siteUrl: 'https://clawdhub.com',
siteUrl: 'https://clawhub.ai',
}
const allowed = shouldSendWebhook(
'skill.publish',
{ slug: 'demo', displayName: 'Demo', batch: 'latest' },
{ slug: 'demo', displayName: 'Demo', highlighted: false },
config,
)
expect(allowed).toBe(false)
@@ -50,11 +50,11 @@ describe('webhook filtering', () => {
const config = {
url: 'https://example.com',
highlightedOnly: true,
siteUrl: 'https://clawdhub.com',
siteUrl: 'https://clawhub.ai',
}
const allowed = shouldSendWebhook(
'skill.highlighted',
{ slug: 'demo', displayName: 'Demo', batch: 'latest' },
{ slug: 'demo', displayName: 'Demo', highlighted: true },
config,
)
expect(allowed).toBe(true)
@@ -65,9 +65,9 @@ describe('payload building', () => {
it('builds canonical url with owner', () => {
const url = buildSkillUrl(
{ slug: 'beeper', displayName: 'Beeper', ownerHandle: 'KrauseFx' },
'https://clawdhub.com',
'https://clawhub.ai',
)
expect(url).toBe('https://clawdhub.com/KrauseFx/beeper')
expect(url).toBe('https://clawhub.ai/KrauseFx/beeper')
})
it('builds a publish embed', () => {
@@ -81,7 +81,7 @@ describe('payload building', () => {
ownerHandle: 'steipete',
tags: ['latest', 'discord'],
},
{ url: 'https://example.com', highlightedOnly: false, siteUrl: 'https://clawdhub.com' },
{ url: 'https://example.com', highlightedOnly: false, siteUrl: 'https://clawhub.ai' },
)
const embed = payload.embeds[0]
expect(embed.title).toBe('Demo Skill')
+7 -7
View File
@@ -6,7 +6,7 @@ export type WebhookSkillPayload = {
summary?: string
version?: string
ownerHandle?: string
batch?: string
highlighted?: boolean
tags?: string[]
}
@@ -16,7 +16,7 @@ export type WebhookConfig = {
siteUrl: string
}
const DEFAULT_SITE_URL = 'https://clawdhub.com'
const DEFAULT_SITE_URL = 'https://clawhub.ai'
export function getWebhookConfig(env: NodeJS.ProcessEnv = process.env): WebhookConfig {
const url = env.DISCORD_WEBHOOK_URL?.trim() || null
@@ -33,7 +33,7 @@ export function shouldSendWebhook(
if (!config.url) return false
if (!config.highlightedOnly) return true
if (event === 'skill.highlighted') return true
return skill.batch === 'highlighted'
return Boolean(skill.highlighted)
}
export function buildDiscordPayload(
@@ -72,7 +72,7 @@ export function buildDiscordPayload(
},
],
footer: {
text: 'ClawdHub',
text: 'ClawHub',
},
timestamp: new Date().toISOString(),
},
@@ -89,9 +89,9 @@ export function buildSkillUrl(skill: WebhookSkillPayload, siteUrl: string) {
function buildDescription(event: WebhookEvent, skill: WebhookSkillPayload) {
const summary = (skill.summary ?? '').trim()
if (summary) return truncate(summary, 200)
if (event === 'skill.highlighted') return 'Newly highlighted skill on ClawdHub.'
if (skill.version) return `New version v${skill.version} published on ClawdHub.`
return 'New skill published on ClawdHub.'
if (event === 'skill.highlighted') return 'Newly highlighted skill on ClawHub.'
if (skill.version) return `New version v${skill.version} published on ClawHub.`
return 'New skill published on ClawHub.'
}
function parseBoolean(value?: string) {
+383
View File
@@ -0,0 +1,383 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { internalAction } from './_generated/server'
import type { SkillEvalContext } from './lib/securityPrompt'
import {
assembleEvalUserMessage,
detectInjectionPatterns,
getLlmEvalModel,
LLM_EVAL_MAX_OUTPUT_TOKENS,
parseLlmEvalResponse,
SECURITY_EVALUATOR_SYSTEM_PROMPT,
} from './lib/securityPrompt'
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function extractResponseText(payload: unknown): string | null {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
}
function verdictToStatus(verdict: string): string {
switch (verdict) {
case 'benign':
return 'clean'
case 'malicious':
return 'malicious'
case 'suspicious':
return 'suspicious'
default:
return 'pending'
}
}
// ---------------------------------------------------------------------------
// Publish-time evaluation action
// ---------------------------------------------------------------------------
export const evaluateWithLlm = internalAction({
args: {
versionId: v.id('skillVersions'),
},
handler: async (ctx, args) => {
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) {
console.log('[llmEval] OPENAI_API_KEY not configured, skipping evaluation')
return
}
const model = getLlmEvalModel()
// Store error helper
const storeError = async (message: string) => {
console.error(`[llmEval] ${message}`)
await ctx.runMutation(internal.skills.updateVersionLlmAnalysisInternal, {
versionId: args.versionId,
llmAnalysis: {
status: 'error',
summary: message,
model,
checkedAt: Date.now(),
},
})
}
// 1. Fetch version
const version = (await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: args.versionId,
})) as Doc<'skillVersions'> | null
if (!version) {
await storeError(`Version ${args.versionId} not found`)
return
}
// 2. Fetch skill
const skill = (await ctx.runQuery(internal.skills.getSkillByIdInternal, {
skillId: version.skillId,
})) as Doc<'skills'> | null
if (!skill) {
await storeError(`Skill ${version.skillId} not found`)
return
}
// 3. Read SKILL.md content
const skillMdFile = version.files.find((f) => {
const lower = f.path.toLowerCase()
return lower === 'skill.md' || lower === 'skills.md'
})
let skillMdContent = ''
if (skillMdFile) {
const blob = await ctx.storage.get(skillMdFile.storageId as Id<'_storage'>)
if (blob) {
skillMdContent = await blob.text()
}
}
if (!skillMdContent) {
await storeError('No SKILL.md content found')
return
}
// 4. Read all file contents
const fileContents: Array<{ path: string; content: string }> = []
for (const f of version.files) {
const lower = f.path.toLowerCase()
if (lower === 'skill.md' || lower === 'skills.md') continue
try {
const blob = await ctx.storage.get(f.storageId as Id<'_storage'>)
if (blob) {
fileContents.push({ path: f.path, content: await blob.text() })
}
} catch {
// Skip files that can't be read
}
}
// 5. Detect injection patterns across ALL content
const allContent = [skillMdContent, ...fileContents.map((f) => f.content)].join('\n')
const injectionSignals = detectInjectionPatterns(allContent)
// 6. Build eval context
const parsed = version.parsed as SkillEvalContext['parsed']
const fm = parsed.frontmatter ?? {}
const evalCtx: SkillEvalContext = {
slug: skill.slug,
displayName: skill.displayName,
ownerUserId: String(skill.ownerUserId),
version: version.version,
createdAt: version.createdAt,
summary: (skill.summary as string | undefined) ?? undefined,
source: (fm.source as string | undefined) ?? undefined,
homepage: (fm.homepage as string | undefined) ?? undefined,
parsed,
files: version.files.map((f) => ({ path: f.path, size: f.size })),
skillMdContent,
fileContents,
injectionSignals,
}
// 6. Assemble user message
const userMessage = assembleEvalUserMessage(evalCtx)
// 7. Call OpenAI Responses API (with retry for rate limits)
const MAX_RETRIES = 3
let raw: string | null = null
try {
const body = JSON.stringify({
model,
instructions: SECURITY_EVALUATOR_SYSTEM_PROMPT,
input: userMessage,
max_output_tokens: LLM_EVAL_MAX_OUTPUT_TOKENS,
text: {
format: {
type: 'json_object',
},
},
})
let response: Response | null = null
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
response = await fetch('https://api.openai.com/v1/responses', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiKey}`,
},
body,
})
if (response.status === 429 || response.status >= 500) {
if (attempt < MAX_RETRIES) {
const delay = 2 ** attempt * 2000 + Math.random() * 1000
console.log(
`[llmEval] Rate limited (${response.status}), retrying in ${Math.round(delay)}ms (attempt ${attempt + 1}/${MAX_RETRIES})`,
)
await new Promise((r) => setTimeout(r, delay))
continue
}
}
break
}
if (!response || !response.ok) {
const errorText = response ? await response.text() : 'No response'
await storeError(`OpenAI API error (${response?.status}): ${errorText.slice(0, 200)}`)
return
}
const payload = (await response.json()) as unknown
raw = extractResponseText(payload)
} catch (error) {
await storeError(
`OpenAI API call failed: ${error instanceof Error ? error.message : String(error)}`,
)
return
}
if (!raw) {
await storeError('Empty response from OpenAI')
return
}
// 8. Parse response
const result = parseLlmEvalResponse(raw)
if (!result) {
console.error(`[llmEval] Raw response (first 500 chars): ${raw.slice(0, 500)}`)
await storeError('Failed to parse LLM evaluation response')
return
}
// 9. Store result
await ctx.runMutation(internal.skills.updateVersionLlmAnalysisInternal, {
versionId: args.versionId,
llmAnalysis: {
status: verdictToStatus(result.verdict),
verdict: result.verdict,
confidence: result.confidence,
summary: result.summary,
dimensions: result.dimensions,
guidance: result.guidance,
findings: result.findings || undefined,
model,
checkedAt: Date.now(),
},
})
console.log(
`[llmEval] Evaluated ${skill.slug}@${version.version}: ${result.verdict} (${result.confidence} confidence)`,
)
// Moderation visibility is finalized by VT results.
// LLM eval only stores analysis payload on the version.
},
})
// ---------------------------------------------------------------------------
// Convenience: evaluate a single skill by slug (for testing / manual runs)
// Usage: npx convex run llmEval:evaluateBySlug '{"slug": "transcribeexx"}'
// ---------------------------------------------------------------------------
export const evaluateBySlug = internalAction({
args: {
slug: v.string(),
},
handler: async (ctx, args) => {
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: args.slug,
})) as Doc<'skills'> | null
if (!skill) {
console.error(`[llmEval:bySlug] Skill "${args.slug}" not found`)
return { error: 'Skill not found' }
}
if (!skill.latestVersionId) {
console.error(`[llmEval:bySlug] Skill "${args.slug}" has no published version`)
return { error: 'No published version' }
}
console.log(`[llmEval:bySlug] Evaluating ${args.slug} (versionId: ${skill.latestVersionId})`)
await ctx.scheduler.runAfter(0, internal.llmEval.evaluateWithLlm, {
versionId: skill.latestVersionId,
})
return { ok: true, slug: args.slug, versionId: skill.latestVersionId }
},
})
// ---------------------------------------------------------------------------
// Backfill action (Phase 2)
// Schedules individual evaluateWithLlm actions for each skill in the batch,
// then self-schedules the next batch. Each eval runs as its own action
// invocation so we don't hit Convex action timeouts.
// ---------------------------------------------------------------------------
export const backfillLlmEval = internalAction({
args: {
cursor: v.optional(v.number()),
batchSize: v.optional(v.number()),
accTotal: v.optional(v.number()),
accScheduled: v.optional(v.number()),
accSkipped: v.optional(v.number()),
startTime: v.optional(v.number()),
},
handler: async (ctx, args) => {
const startTime = args.startTime ?? Date.now()
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) {
console.log('[llmEval:backfill] OPENAI_API_KEY not configured')
return { error: 'OPENAI_API_KEY not configured' }
}
const batchSize = args.batchSize ?? 25
const cursor = args.cursor ?? 0
let accTotal = args.accTotal ?? 0
let accScheduled = args.accScheduled ?? 0
let accSkipped = args.accSkipped ?? 0
const batch = await ctx.runQuery(internal.skills.getActiveSkillBatchForLlmBackfillInternal, {
cursor,
batchSize,
})
if (batch.skills.length === 0 && batch.done) {
console.log('[llmEval:backfill] No more skills to evaluate')
return { total: accTotal, scheduled: accScheduled, skipped: accSkipped }
}
console.log(
`[llmEval:backfill] Processing batch of ${batch.skills.length} skills (cursor=${cursor}, accumulated=${accTotal})`,
)
for (const { versionId, slug } of batch.skills) {
// Re-evaluate all (full file content reading upgrade)
const version = (await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId,
})) as Doc<'skillVersions'> | null
if (!version) {
accSkipped++
continue
}
// Schedule each evaluation as a separate action invocation
await ctx.scheduler.runAfter(0, internal.llmEval.evaluateWithLlm, { versionId })
accScheduled++
console.log(`[llmEval:backfill] Scheduled eval for ${slug}`)
}
accTotal += batch.skills.length
if (!batch.done) {
// Delay the next batch slightly to avoid overwhelming the scheduler
// when all evals from this batch are also running
console.log(
`[llmEval:backfill] Scheduling next batch (cursor=${batch.nextCursor}, total so far=${accTotal})`,
)
await ctx.scheduler.runAfter(5_000, internal.llmEval.backfillLlmEval, {
cursor: batch.nextCursor,
batchSize,
accTotal,
accScheduled,
accSkipped,
startTime,
})
return { status: 'continuing', totalSoFar: accTotal }
}
const durationMs = Date.now() - startTime
const result = {
total: accTotal,
scheduled: accScheduled,
skipped: accSkipped,
durationMs,
}
console.log('[llmEval:backfill] Complete:', result)
return result
},
})
+282 -2
View File
@@ -12,12 +12,34 @@ vi.mock('./_generated/api', () => ({
'applySkillFingerprintBackfillPatchInternal',
),
backfillSkillFingerprintsInternal: Symbol('backfillSkillFingerprintsInternal'),
getEmptySkillCleanupPageInternal: Symbol('getEmptySkillCleanupPageInternal'),
applyEmptySkillCleanupInternal: Symbol('applyEmptySkillCleanupInternal'),
nominateUserForEmptySkillSpamInternal: Symbol('nominateUserForEmptySkillSpamInternal'),
cleanupEmptySkillsInternal: Symbol('cleanupEmptySkillsInternal'),
nominateEmptySkillSpammersInternal: Symbol('nominateEmptySkillSpammersInternal'),
},
skills: {
getVersionByIdInternal: Symbol('skills.getVersionByIdInternal'),
getOwnerSkillActivityInternal: Symbol('skills.getOwnerSkillActivityInternal'),
},
users: {
getByIdInternal: Symbol('users.getByIdInternal'),
},
},
}))
const { backfillSkillFingerprintsInternalHandler, backfillSkillSummariesInternalHandler } =
await import('./maintenance')
vi.mock('./lib/skillSummary', () => ({
generateSkillSummary: vi.fn(),
}))
const {
backfillSkillFingerprintsInternalHandler,
backfillSkillSummariesInternalHandler,
cleanupEmptySkillsInternalHandler,
nominateEmptySkillSpammersInternalHandler,
} = await import('./maintenance')
const { internal } = await import('./_generated/api')
const { generateSkillSummary } = await import('./lib/skillSummary')
function makeBlob(text: string) {
return { text: () => Promise.resolve(text) } as unknown as Blob
@@ -30,6 +52,8 @@ describe('maintenance backfill', () => {
{
kind: 'ok',
skillId: 'skills:1',
skillSlug: 'skill-1',
skillDisplayName: 'Skill 1',
versionId: 'skillVersions:1',
skillSummary: '>',
versionParsed: { frontmatter: { description: '>' } },
@@ -73,6 +97,8 @@ describe('maintenance backfill', () => {
{
kind: 'ok',
skillId: 'skills:1',
skillSlug: 'skill-1',
skillDisplayName: 'Skill 1',
versionId: 'skillVersions:1',
skillSummary: '>',
versionParsed: { frontmatter: { description: '>' } },
@@ -102,6 +128,8 @@ describe('maintenance backfill', () => {
{
kind: 'ok',
skillId: 'skills:1',
skillSlug: 'skill-1',
skillDisplayName: 'Skill 1',
versionId: 'skillVersions:1',
skillSummary: null,
versionParsed: { frontmatter: {} },
@@ -123,6 +151,49 @@ describe('maintenance backfill', () => {
expect(result.stats.missingStorageBlob).toBe(1)
expect(runMutation).not.toHaveBeenCalled()
})
it('fills empty summary via AI when useAi is enabled', async () => {
vi.mocked(generateSkillSummary).mockResolvedValue('AI generated summary.')
const runQuery = vi.fn().mockResolvedValue({
items: [
{
kind: 'ok',
skillId: 'skills:1',
skillSlug: 'ai-skill',
skillDisplayName: 'AI Skill',
versionId: 'skillVersions:1',
skillSummary: null,
versionParsed: { frontmatter: {} },
readmeStorageId: 'storage:1',
},
],
cursor: null,
isDone: true,
})
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const storageGet = vi.fn().mockResolvedValue(makeBlob('# AI Skill\n\nUseful automation.'))
const result = await backfillSkillSummariesInternalHandler(
{ runQuery, runMutation, storage: { get: storageGet } } as never,
{ dryRun: false, batchSize: 10, maxBatches: 1, useAi: true },
)
expect(result.ok).toBe(true)
expect(result.stats.skillsPatched).toBe(1)
expect(result.stats.aiSummariesPatched).toBe(1)
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
skillId: 'skills:1',
versionId: 'skillVersions:1',
summary: 'AI generated summary.',
parsed: {
frontmatter: {},
metadata: undefined,
clawdis: undefined,
},
})
})
})
describe('maintenance fingerprint backfill', () => {
@@ -268,3 +339,212 @@ describe('maintenance fingerprint backfill', () => {
})
})
})
describe('maintenance empty skill cleanup', () => {
it('dryRun detects empty skills and returns nominations', async () => {
const runQuery = vi.fn().mockImplementation(async (endpoint: unknown) => {
if (endpoint === internal.maintenance.getEmptySkillCleanupPageInternal) {
return {
items: [
{
skillId: 'skills:1',
slug: 'spam-skill',
ownerUserId: 'users:1',
latestVersionId: 'skillVersions:1',
softDeletedAt: undefined,
summary: 'Expert guidance for spam-skill.',
},
],
cursor: null,
isDone: true,
}
}
if (endpoint === internal.skills.getVersionByIdInternal) {
return {
_id: 'skillVersions:1',
files: [{ path: 'SKILL.md', size: 120, storageId: 'storage:1' }],
}
}
if (endpoint === internal.users.getByIdInternal) {
return { _id: 'users:1', handle: 'spammer', _creationTime: Date.now() }
}
if (endpoint === internal.skills.getOwnerSkillActivityInternal) {
return []
}
throw new Error(`Unexpected endpoint: ${String(endpoint)}`)
})
const runMutation = vi.fn()
const storageGet = vi
.fn()
.mockResolvedValue(
makeBlob(`# Demo\n- Step-by-step tutorials\n- Tips and techniques\n- Project ideas`),
)
const result = await cleanupEmptySkillsInternalHandler(
{ runQuery, runMutation, storage: { get: storageGet } } as never,
{ dryRun: true, batchSize: 10, maxBatches: 1, nominationThreshold: 1 },
)
expect(result.ok).toBe(true)
expect(result.isDone).toBe(true)
expect(result.cursor).toBeNull()
expect(result.stats.emptyDetected).toBe(1)
expect(result.stats.skillsDeleted).toBe(0)
expect(result.nominations).toEqual([
{
userId: 'users:1',
handle: 'spammer',
emptySkillCount: 1,
sampleSlugs: ['spam-skill'],
},
])
expect(runMutation).not.toHaveBeenCalled()
})
it('apply mode deletes empty skills', async () => {
const runQuery = vi.fn().mockImplementation(async (endpoint: unknown) => {
if (endpoint === internal.maintenance.getEmptySkillCleanupPageInternal) {
return {
items: [
{
skillId: 'skills:1',
slug: 'spam-a',
ownerUserId: 'users:1',
latestVersionId: 'skillVersions:1',
summary: 'Expert guidance for spam-a.',
},
{
skillId: 'skills:2',
slug: 'spam-b',
ownerUserId: 'users:1',
latestVersionId: 'skillVersions:2',
summary: 'Expert guidance for spam-b.',
},
],
cursor: null,
isDone: true,
}
}
if (endpoint === internal.skills.getVersionByIdInternal) {
return {
files: [{ path: 'SKILL.md', size: 120, storageId: 'storage:1' }],
}
}
if (endpoint === internal.users.getByIdInternal) {
return { _id: 'users:1', handle: 'spammer', _creationTime: Date.now() }
}
if (endpoint === internal.skills.getOwnerSkillActivityInternal) {
return []
}
throw new Error(`Unexpected endpoint: ${String(endpoint)}`)
})
const runMutation = vi.fn().mockImplementation(async (endpoint: unknown) => {
if (endpoint === internal.maintenance.applyEmptySkillCleanupInternal) {
return { deleted: true }
}
throw new Error(`Unexpected mutation endpoint: ${String(endpoint)}`)
})
const storageGet = vi
.fn()
.mockResolvedValue(
makeBlob(`# Demo\n- Step-by-step tutorials\n- Tips and techniques\n- Project ideas`),
)
const result = await cleanupEmptySkillsInternalHandler(
{ runQuery, runMutation, storage: { get: storageGet } } as never,
{ dryRun: false, batchSize: 10, maxBatches: 1, nominationThreshold: 2 },
)
expect(result.ok).toBe(true)
expect(result.isDone).toBe(true)
expect(result.cursor).toBeNull()
expect(result.stats.emptyDetected).toBe(2)
expect(result.stats.skillsDeleted).toBe(2)
expect(result.nominations).toEqual([
{
userId: 'users:1',
handle: 'spammer',
emptySkillCount: 2,
sampleSlugs: ['spam-a', 'spam-b'],
},
])
})
})
describe('maintenance empty skill nominations', () => {
it('creates ban nominations from backfilled empty deletions', async () => {
const runQuery = vi.fn().mockImplementation(async (endpoint: unknown, args: unknown) => {
if (endpoint === internal.maintenance.getEmptySkillCleanupPageInternal) {
const cursor = (args as { cursor?: string | undefined }).cursor
if (!cursor) {
return {
items: [
{
skillId: 'skills:1',
slug: 'spam-a',
ownerUserId: 'users:1',
softDeletedAt: 1,
moderationReason: 'quality.empty.backfill',
},
{
skillId: 'skills:2',
slug: 'spam-b',
ownerUserId: 'users:1',
softDeletedAt: 1,
moderationReason: 'quality.empty.backfill',
},
],
cursor: 'next',
isDone: false,
}
}
return {
items: [
{
skillId: 'skills:3',
slug: 'valid-hidden',
ownerUserId: 'users:2',
softDeletedAt: 1,
moderationReason: 'scanner.vt.suspicious',
},
],
cursor: null,
isDone: true,
}
}
if (endpoint === internal.users.getByIdInternal) {
return { _id: 'users:1', handle: 'spammer' }
}
throw new Error(`Unexpected query endpoint: ${String(endpoint)}`)
})
const runMutation = vi.fn().mockImplementation(async (endpoint: unknown) => {
if (endpoint === internal.maintenance.nominateUserForEmptySkillSpamInternal) {
return { created: true }
}
throw new Error(`Unexpected mutation endpoint: ${String(endpoint)}`)
})
const result = await nominateEmptySkillSpammersInternalHandler(
{ runQuery, runMutation } as never,
{ batchSize: 10, maxBatches: 2, nominationThreshold: 2 },
)
expect(result.ok).toBe(true)
expect(result.isDone).toBe(true)
expect(result.stats.usersFlagged).toBe(1)
expect(result.stats.nominationsCreated).toBe(1)
expect(result.stats.nominationsExisting).toBe(0)
expect(result.nominations).toEqual([
{
userId: 'users:1',
handle: 'spammer',
emptySkillCount: 2,
sampleSlugs: ['spam-a', 'spam-b'],
},
])
})
})
+955 -11
View File
File diff suppressed because it is too large Load Diff
+44 -9
View File
@@ -1,7 +1,11 @@
import { v } from 'convex/values'
import { internalMutation } from './_generated/server'
import { internalMutation, internalQuery } from './_generated/server'
export const checkRateLimitInternal = internalMutation({
/**
* Read-only rate limit check. Returns current status without writing anything.
* This eliminates write conflicts for denied requests entirely.
*/
export const getRateLimitStatusInternal = internalQuery({
args: {
key: v.string(),
limit: v.number(),
@@ -20,6 +24,43 @@ export const checkRateLimitInternal = internalMutation({
.withIndex('by_key_window', (q) => q.eq('key', args.key).eq('windowStart', windowStart))
.unique()
const count = existing?.count ?? 0
const allowed = count < args.limit
return {
allowed,
remaining: Math.max(0, args.limit - count),
limit: args.limit,
resetAt,
}
},
})
/**
* Consume one rate limit token. Only call this after getRateLimitStatusInternal
* returns allowed=true. Includes a double-check to handle races between the
* query and this mutation.
*/
export const consumeRateLimitInternal = internalMutation({
args: {
key: v.string(),
limit: v.number(),
windowMs: v.number(),
},
handler: async (ctx, args) => {
const now = Date.now()
const windowStart = Math.floor(now / args.windowMs) * args.windowMs
const existing = await ctx.db
.query('rateLimits')
.withIndex('by_key_window', (q) => q.eq('key', args.key).eq('windowStart', windowStart))
.unique()
// Double-check: another request may have consumed the last token
// between our query and this mutation
if (existing && existing.count >= args.limit) {
return { allowed: false, remaining: 0 }
}
if (!existing) {
await ctx.db.insert('rateLimits', {
key: args.key,
@@ -28,11 +69,7 @@ export const checkRateLimitInternal = internalMutation({
limit: args.limit,
updatedAt: now,
})
return { allowed: true, remaining: Math.max(0, args.limit - 1), limit: args.limit, resetAt }
}
if (existing.count >= args.limit) {
return { allowed: false, remaining: 0, limit: args.limit, resetAt }
return { allowed: true, remaining: Math.max(0, args.limit - 1) }
}
await ctx.db.patch(existing._id, {
@@ -43,8 +80,6 @@ export const checkRateLimitInternal = internalMutation({
return {
allowed: true,
remaining: Math.max(0, args.limit - existing.count - 1),
limit: args.limit,
resetAt,
}
},
})
+248 -7
View File
@@ -15,7 +15,12 @@ const users = defineTable({
displayName: v.optional(v.string()),
bio: v.optional(v.string()),
role: v.optional(v.union(v.literal('admin'), v.literal('moderator'), v.literal('user'))),
githubCreatedAt: v.optional(v.number()),
githubFetchedAt: v.optional(v.number()),
deactivatedAt: v.optional(v.number()),
purgedAt: v.optional(v.number()),
deletedAt: v.optional(v.number()),
banReason: v.optional(v.string()),
createdAt: v.optional(v.number()),
updatedAt: v.optional(v.number()),
})
@@ -27,6 +32,7 @@ const skills = defineTable({
slug: v.string(),
displayName: v.string(),
summary: v.optional(v.string()),
resourceId: v.optional(v.string()),
ownerUserId: v.id('users'),
canonicalSkillId: v.optional(v.id('skills')),
forkOf: v.optional(
@@ -40,15 +46,73 @@ const skills = defineTable({
latestVersionId: v.optional(v.id('skillVersions')),
tags: v.record(v.string(), v.id('skillVersions')),
softDeletedAt: v.optional(v.number()),
badges: v.object({
redactionApproved: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
badges: v.optional(
v.object({
redactionApproved: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
highlighted: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
official: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
deprecated: v.optional(
v.object({
byUserId: v.id('users'),
at: v.number(),
}),
),
}),
),
moderationStatus: v.optional(
v.union(v.literal('active'), v.literal('hidden'), v.literal('removed')),
),
moderationNotes: v.optional(v.string()),
moderationReason: v.optional(v.string()),
quality: v.optional(
v.object({
score: v.number(),
decision: v.union(v.literal('pass'), v.literal('quarantine'), v.literal('reject')),
trustTier: v.union(v.literal('low'), v.literal('medium'), v.literal('trusted')),
similarRecentCount: v.number(),
reason: v.string(),
signals: v.object({
bodyChars: v.number(),
bodyWords: v.number(),
uniqueWordRatio: v.number(),
headingCount: v.number(),
bulletCount: v.number(),
templateMarkerHits: v.number(),
genericSummary: v.boolean(),
cjkChars: v.optional(v.number()),
}),
),
}),
evaluatedAt: v.number(),
}),
),
moderationFlags: v.optional(v.array(v.string())),
lastReviewedAt: v.optional(v.number()),
// VT scan tracking
scanLastCheckedAt: v.optional(v.number()),
scanCheckCount: v.optional(v.number()),
hiddenAt: v.optional(v.number()),
hiddenBy: v.optional(v.id('users')),
reportCount: v.optional(v.number()),
lastReportedAt: v.optional(v.number()),
batch: v.optional(v.string()),
statsDownloads: v.optional(v.number()),
statsStars: v.optional(v.number()),
statsInstallsCurrent: v.optional(v.number()),
statsInstallsAllTime: v.optional(v.number()),
stats: v.object({
downloads: v.number(),
installsCurrent: v.optional(v.number()),
@@ -63,7 +127,23 @@ const skills = defineTable({
.index('by_slug', ['slug'])
.index('by_owner', ['ownerUserId'])
.index('by_updated', ['updatedAt'])
.index('by_stats_downloads', ['statsDownloads', 'updatedAt'])
.index('by_stats_stars', ['statsStars', 'updatedAt'])
.index('by_stats_installs_current', ['statsInstallsCurrent', 'updatedAt'])
.index('by_stats_installs_all_time', ['statsInstallsAllTime', 'updatedAt'])
.index('by_batch', ['batch'])
.index('by_active_updated', ['softDeletedAt', 'updatedAt'])
.index('by_active_created', ['softDeletedAt', 'createdAt'])
.index('by_active_name', ['softDeletedAt', 'displayName'])
.index('by_active_stats_downloads', ['softDeletedAt', 'statsDownloads', 'updatedAt'])
.index('by_active_stats_stars', ['softDeletedAt', 'statsStars', 'updatedAt'])
.index('by_active_stats_installs_all_time', [
'softDeletedAt',
'statsInstallsAllTime',
'updatedAt',
])
.index('by_canonical', ['canonicalSkillId'])
.index('by_fork_of', ['forkOf.skillId'])
const souls = defineTable({
slug: v.string(),
@@ -105,13 +185,47 @@ const skillVersions = defineTable({
frontmatter: v.record(v.string(), v.any()),
metadata: v.optional(v.any()),
clawdis: v.optional(v.any()),
moltbot: v.optional(v.any()),
}),
createdBy: v.id('users'),
createdAt: v.number(),
softDeletedAt: v.optional(v.number()),
sha256hash: v.optional(v.string()),
vtAnalysis: v.optional(
v.object({
status: v.string(),
verdict: v.optional(v.string()),
analysis: v.optional(v.string()),
source: v.optional(v.string()),
checkedAt: v.number(),
}),
),
llmAnalysis: v.optional(
v.object({
status: v.string(),
verdict: v.optional(v.string()),
confidence: v.optional(v.string()),
summary: v.optional(v.string()),
dimensions: v.optional(
v.array(
v.object({
name: v.string(),
label: v.string(),
rating: v.string(),
detail: v.string(),
}),
),
),
guidance: v.optional(v.string()),
findings: v.optional(v.string()),
model: v.optional(v.string()),
checkedAt: v.number(),
}),
),
})
.index('by_skill', ['skillId'])
.index('by_skill_version', ['skillId', 'version'])
.index('by_sha256hash', ['sha256hash'])
const soulVersions = defineTable({
soulId: v.id('souls'),
@@ -131,6 +245,8 @@ const soulVersions = defineTable({
parsed: v.object({
frontmatter: v.record(v.string(), v.any()),
metadata: v.optional(v.any()),
clawdis: v.optional(v.any()),
moltbot: v.optional(v.any()),
}),
createdBy: v.id('users'),
createdAt: v.number(),
@@ -149,6 +265,21 @@ const skillVersionFingerprints = defineTable({
.index('by_fingerprint', ['fingerprint'])
.index('by_skill_fingerprint', ['skillId', 'fingerprint'])
const skillBadges = defineTable({
skillId: v.id('skills'),
kind: v.union(
v.literal('highlighted'),
v.literal('official'),
v.literal('deprecated'),
v.literal('redactionApproved'),
),
byUserId: v.id('users'),
at: v.number(),
})
.index('by_skill', ['skillId'])
.index('by_skill_kind', ['skillId', 'kind'])
.index('by_kind_at', ['kind', 'at'])
const soulVersionFingerprints = defineTable({
soulId: v.id('souls'),
versionId: v.id('soulVersions'),
@@ -177,6 +308,69 @@ const skillEmbeddings = defineTable({
filterFields: ['visibility'],
})
const skillDailyStats = defineTable({
skillId: v.id('skills'),
day: v.number(),
downloads: v.number(),
installs: v.number(),
updatedAt: v.number(),
})
.index('by_skill_day', ['skillId', 'day'])
.index('by_day', ['day'])
const skillLeaderboards = defineTable({
kind: v.string(),
generatedAt: v.number(),
rangeStartDay: v.number(),
rangeEndDay: v.number(),
items: v.array(
v.object({
skillId: v.id('skills'),
score: v.number(),
installs: v.number(),
downloads: v.number(),
}),
),
}).index('by_kind', ['kind', 'generatedAt'])
const skillStatBackfillState = defineTable({
key: v.string(),
cursor: v.optional(v.string()),
doneAt: v.optional(v.number()),
updatedAt: v.number(),
}).index('by_key', ['key'])
const skillStatEvents = defineTable({
skillId: v.id('skills'),
kind: v.union(
v.literal('download'),
v.literal('star'),
v.literal('unstar'),
v.literal('comment'),
v.literal('uncomment'),
v.literal('install_new'),
v.literal('install_reactivate'),
v.literal('install_deactivate'),
v.literal('install_clear'),
),
delta: v.optional(
v.object({
allTime: v.number(),
current: v.number(),
}),
),
occurredAt: v.number(),
processedAt: v.optional(v.number()),
})
.index('by_unprocessed', ['processedAt'])
.index('by_skill', ['skillId'])
const skillStatUpdateCursors = defineTable({
key: v.string(),
cursorCreationTime: v.optional(v.number()),
updatedAt: v.number(),
}).index('by_key', ['key'])
const soulEmbeddings = defineTable({
soulId: v.id('souls'),
versionId: v.id('soulVersions'),
@@ -206,6 +400,17 @@ const comments = defineTable({
.index('by_skill', ['skillId'])
.index('by_user', ['userId'])
const skillReports = defineTable({
skillId: v.id('skills'),
userId: v.id('users'),
reason: v.optional(v.string()),
createdAt: v.number(),
})
.index('by_skill', ['skillId'])
.index('by_skill_createdAt', ['skillId', 'createdAt'])
.index('by_user', ['userId'])
.index('by_skill_user', ['skillId', 'userId'])
const soulComments = defineTable({
soulId: v.id('souls'),
userId: v.id('users'),
@@ -246,6 +451,24 @@ const auditLogs = defineTable({
.index('by_actor', ['actorUserId'])
.index('by_target', ['targetType', 'targetId'])
const vtScanLogs = defineTable({
type: v.union(v.literal('daily_rescan'), v.literal('backfill'), v.literal('pending_poll')),
total: v.number(),
updated: v.number(),
unchanged: v.number(),
errors: v.number(),
flaggedSkills: v.optional(
v.array(
v.object({
slug: v.string(),
status: v.string(),
}),
),
),
durationMs: v.number(),
createdAt: v.number(),
}).index('by_type_date', ['type', 'createdAt'])
const apiTokens = defineTable({
userId: v.id('users'),
label: v.string(),
@@ -268,6 +491,15 @@ const rateLimits = defineTable({
.index('by_key_window', ['key', 'windowStart'])
.index('by_key', ['key'])
const downloadDedupes = defineTable({
skillId: v.id('skills'),
identityHash: v.string(),
hourStart: v.number(),
createdAt: v.number(),
})
.index('by_skill_identity_hour', ['skillId', 'identityHash', 'hourStart'])
.index('by_hour', ['hourStart'])
const githubBackupSyncState = defineTable({
key: v.string(),
cursor: v.optional(v.string()),
@@ -320,16 +552,25 @@ export default defineSchema({
skillVersions,
soulVersions,
skillVersionFingerprints,
skillBadges,
soulVersionFingerprints,
skillEmbeddings,
soulEmbeddings,
skillDailyStats,
skillLeaderboards,
skillStatBackfillState,
skillStatEvents,
skillStatUpdateCursors,
comments,
skillReports,
soulComments,
stars,
soulStars,
auditLogs,
vtScanLogs,
apiTokens,
rateLimits,
downloadDedupes,
githubBackupSyncState,
userSyncRoots,
userSkillInstalls,
+368 -2
View File
@@ -1,12 +1,378 @@
/* @vitest-environment node */
import { describe, expect, it } from 'vitest'
import { __test } from './search'
import { describe, expect, it, vi } from 'vitest'
import { tokenize } from './lib/searchText'
import { __test, hydrateResults, lexicalFallbackSkills, searchSkills } from './search'
const { generateEmbeddingMock, getSkillBadgeMapsMock } = vi.hoisted(() => ({
generateEmbeddingMock: vi.fn(),
getSkillBadgeMapsMock: vi.fn(),
}))
vi.mock('./lib/embeddings', () => ({
generateEmbedding: generateEmbeddingMock,
}))
vi.mock('./lib/badges', () => ({
getSkillBadgeMaps: getSkillBadgeMapsMock,
isSkillHighlighted: (skill: { badges?: Record<string, unknown> }) =>
Boolean(skill.badges?.highlighted),
}))
type WrappedHandler = {
_handler: (
ctx: unknown,
args: unknown,
) => Promise<Array<{ skill: { slug: string; _id: string } }>>
}
const searchSkillsHandler = (searchSkills as unknown as WrappedHandler)._handler
const lexicalFallbackSkillsHandler = (lexicalFallbackSkills as unknown as WrappedHandler)._handler
const hydrateResultsHandler = (
hydrateResults as unknown as {
_handler: (
ctx: unknown,
args: unknown,
) => Promise<Array<{ skill: { slug: string; _id: string } }>>
}
)._handler
describe('search helpers', () => {
it('returns fallback results when vector candidates are empty', async () => {
generateEmbeddingMock.mockResolvedValueOnce([0, 1, 2])
const fallback = [
{
skill: makePublicSkill({ id: 'skills:orf', slug: 'orf', displayName: 'ORF' }),
version: null,
ownerHandle: 'steipete',
},
]
const runQuery = vi
.fn()
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
.mockResolvedValueOnce(fallback)
const result = await searchSkillsHandler(
{
vectorSearch: vi.fn().mockResolvedValue([]),
runQuery,
},
{ query: 'orf', limit: 10 },
)
expect(result).toHaveLength(1)
expect(result[0].skill.slug).toBe('orf')
expect(runQuery).toHaveBeenLastCalledWith(
expect.anything(),
expect.objectContaining({ query: 'orf', queryTokens: ['orf'] }),
)
})
it('applies highlightedOnly filtering in lexical fallback', async () => {
const highlighted = makeSkillDoc({
id: 'skills:hl',
slug: 'orf-highlighted',
displayName: 'ORF Highlighted',
})
const plain = makeSkillDoc({ id: 'skills:plain', slug: 'orf-plain', displayName: 'ORF Plain' })
getSkillBadgeMapsMock.mockResolvedValueOnce(
new Map([
['skills:hl', { highlighted: { byUserId: 'users:mod', at: 1 } }],
['skills:plain', {}],
]),
)
const result = await lexicalFallbackSkillsHandler(
makeLexicalCtx({
exactSlugSkill: null,
recentSkills: [highlighted, plain],
}),
{ query: 'orf', queryTokens: ['orf'], highlightedOnly: true, limit: 10 },
)
expect(result).toHaveLength(1)
expect(result[0].skill.slug).toBe('orf-highlighted')
})
it('applies nonSuspiciousOnly filtering in lexical fallback', async () => {
const suspicious = makeSkillDoc({
id: 'skills:suspicious',
slug: 'orf-suspicious',
displayName: 'ORF Suspicious',
moderationFlags: ['flagged.suspicious'],
})
const clean = makeSkillDoc({ id: 'skills:clean', slug: 'orf-clean', displayName: 'ORF Clean' })
getSkillBadgeMapsMock.mockResolvedValueOnce(
new Map([
['skills:suspicious', {}],
['skills:clean', {}],
]),
)
const result = await lexicalFallbackSkillsHandler(
makeLexicalCtx({
exactSlugSkill: null,
recentSkills: [suspicious, clean],
}),
{ query: 'orf', queryTokens: ['orf'], nonSuspiciousOnly: true, limit: 10 },
)
expect(result).toHaveLength(1)
expect(result[0].skill.slug).toBe('orf-clean')
})
it('includes exact slug match from by_slug even when recent scan is empty', async () => {
const exactSlugSkill = makeSkillDoc({ id: 'skills:orf', slug: 'orf', displayName: 'ORF' })
getSkillBadgeMapsMock.mockResolvedValueOnce(new Map([['skills:orf', {}]]))
const ctx = makeLexicalCtx({
exactSlugSkill,
recentSkills: [],
})
const result = await lexicalFallbackSkillsHandler(ctx, {
query: 'orf',
queryTokens: ['orf'],
limit: 10,
})
expect(result).toHaveLength(1)
expect(result[0].skill.slug).toBe('orf')
expect(ctx.db.query).toHaveBeenCalledWith('skills')
})
it('dedupes overlap and enforces rank + limit across vector and fallback', async () => {
generateEmbeddingMock.mockResolvedValueOnce([0, 1, 2])
const vectorEntries = [
{
embeddingId: 'skillEmbeddings:a',
skill: makePublicSkill({
id: 'skills:a',
slug: 'foo-a',
displayName: 'Foo Alpha',
downloads: 10,
}),
version: null,
ownerHandle: 'one',
},
{
embeddingId: 'skillEmbeddings:b',
skill: makePublicSkill({
id: 'skills:b',
slug: 'foo-b',
displayName: 'Foo Beta',
downloads: 2,
}),
version: null,
ownerHandle: 'two',
},
]
const fallbackEntries = [
{
skill: makePublicSkill({
id: 'skills:a',
slug: 'foo-a',
displayName: 'Foo Alpha',
downloads: 10,
}),
version: null,
ownerHandle: 'one',
},
{
skill: makePublicSkill({
id: 'skills:c',
slug: 'foo-c',
displayName: 'Foo Classic',
downloads: 1,
}),
version: null,
ownerHandle: 'three',
},
]
const runQuery = vi
.fn()
.mockResolvedValueOnce(vectorEntries)
.mockResolvedValueOnce([])
.mockResolvedValueOnce(fallbackEntries)
const result = await searchSkillsHandler(
{
vectorSearch: vi.fn().mockResolvedValue([
{ _id: 'skillEmbeddings:a', _score: 0.4 },
{ _id: 'skillEmbeddings:b', _score: 0.9 },
]),
runQuery,
},
{ query: 'foo', limit: 2 },
)
expect(result).toHaveLength(2)
expect(result[0].skill.slug).toBe('foo-b')
expect(new Set(result.map((entry: { skill: { _id: string } }) => entry.skill._id)).size).toBe(2)
})
it('filters suspicious vector results in hydrateResults when requested', async () => {
const result = await hydrateResultsHandler(
{
db: {
get: vi.fn(async (id: string) => {
if (id === 'skillEmbeddings:1') {
return { _id: 'skillEmbeddings:1', skillId: 'skills:1', versionId: 'skillVersions:1' }
}
if (id === 'skills:1') {
return makeSkillDoc({
id: 'skills:1',
slug: 'suspicious',
displayName: 'Suspicious',
moderationFlags: ['flagged.suspicious'],
})
}
if (id === 'users:owner') return { _id: 'users:owner', handle: 'owner' }
if (id === 'skillVersions:1') return { _id: 'skillVersions:1', version: '1.0.0' }
return null
}),
},
},
{ embeddingIds: ['skillEmbeddings:1'], nonSuspiciousOnly: true },
)
expect(result).toHaveLength(0)
})
it('advances candidate limit until max', () => {
expect(__test.getNextCandidateLimit(50, 1000)).toBe(100)
expect(__test.getNextCandidateLimit(800, 1000)).toBe(1000)
expect(__test.getNextCandidateLimit(1000, 1000)).toBeNull()
})
it('boosts exact slug/name matches over loose matches', () => {
const queryTokens = tokenize('notion')
const exactScore = __test.scoreSkillResult(queryTokens, 0.4, 'Notion Sync', 'notion-sync', 5)
const looseScore = __test.scoreSkillResult(queryTokens, 0.6, 'Notes Sync', 'notes-sync', 500)
expect(exactScore).toBeGreaterThan(looseScore)
})
it('adds a popularity prior for equally relevant matches', () => {
const queryTokens = tokenize('notion')
const lowDownloads = __test.scoreSkillResult(
queryTokens,
0.5,
'Notion Helper',
'notion-helper',
0,
)
const highDownloads = __test.scoreSkillResult(
queryTokens,
0.5,
'Notion Helper',
'notion-helper',
1000,
)
expect(highDownloads).toBeGreaterThan(lowDownloads)
})
it('merges fallback matches without duplicate skill ids', () => {
const primary = [
{
embeddingId: 'skillEmbeddings:1',
skill: { _id: 'skills:1' },
},
] as unknown as Parameters<typeof __test.mergeUniqueBySkillId>[0]
const fallback = [
{
skill: { _id: 'skills:1' },
},
{
skill: { _id: 'skills:2' },
},
] as unknown as Parameters<typeof __test.mergeUniqueBySkillId>[1]
const merged = __test.mergeUniqueBySkillId(primary, fallback)
expect(merged).toHaveLength(2)
expect(merged.map((entry) => entry.skill._id)).toEqual(['skills:1', 'skills:2'])
})
})
function makePublicSkill(params: {
id: string
slug: string
displayName: string
downloads?: number
}) {
return {
_id: params.id,
_creationTime: 1,
slug: params.slug,
displayName: params.displayName,
summary: `${params.displayName} summary`,
ownerUserId: 'users:owner',
canonicalSkillId: undefined,
forkOf: undefined,
latestVersionId: 'skillVersions:1',
tags: {},
badges: {},
stats: {
downloads: params.downloads ?? 0,
installsCurrent: 0,
installsAllTime: 0,
stars: 0,
versions: 1,
comments: 0,
},
createdAt: 1,
updatedAt: 1,
}
}
function makeSkillDoc(params: {
id: string
slug: string
displayName: string
moderationFlags?: string[]
moderationReason?: string
}) {
return {
...makePublicSkill(params),
_creationTime: 1,
moderationStatus: 'active',
moderationFlags: params.moderationFlags ?? [],
moderationReason: params.moderationReason,
softDeletedAt: undefined,
}
}
function makeLexicalCtx(params: {
exactSlugSkill: ReturnType<typeof makeSkillDoc> | null
recentSkills: Array<ReturnType<typeof makeSkillDoc>>
}) {
return {
db: {
query: vi.fn((table: string) => {
if (table !== 'skills') throw new Error(`Unexpected table ${table}`)
return {
withIndex: (index: string) => {
if (index === 'by_slug') {
return {
unique: vi.fn().mockResolvedValue(params.exactSlugSkill),
}
}
if (index === 'by_active_updated') {
return {
order: () => ({
take: vi.fn().mockResolvedValue(params.recentSkills),
}),
}
}
throw new Error(`Unexpected index ${index}`)
},
}
}),
get: vi.fn(async (id: string) => {
if (id.startsWith('users:')) return { _id: id, handle: 'owner' }
if (id.startsWith('skillVersions:')) return { _id: id, version: '1.0.0' }
return null
}),
},
}
}
+298 -41
View File
@@ -2,40 +2,118 @@ import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalQuery } from './_generated/server'
import { getSkillBadgeMaps, isSkillHighlighted, type SkillBadgeMap } from './lib/badges'
import { generateEmbedding } from './lib/embeddings'
import { toPublicSkill, toPublicSoul } from './lib/public'
import { matchesExactTokens, tokenize } from './lib/searchText'
import { isSkillSuspicious } from './lib/skillSafety'
type HydratedEntry = {
embeddingId: Id<'skillEmbeddings'>
skill: Doc<'skills'> | null
type SkillSearchEntry = {
embeddingId?: Id<'skillEmbeddings'>
skill: NonNullable<ReturnType<typeof toPublicSkill>>
version: Doc<'skillVersions'> | null
ownerHandle: string | null
}
type SearchResult = HydratedEntry & { score: number }
type SearchResult = SkillSearchEntry & { score: number }
const SLUG_EXACT_BOOST = 1.4
const SLUG_PREFIX_BOOST = 0.8
const NAME_EXACT_BOOST = 1.1
const NAME_PREFIX_BOOST = 0.6
const POPULARITY_WEIGHT = 0.08
const FALLBACK_SCAN_LIMIT = 1200
function getNextCandidateLimit(current: number, max: number) {
const next = Math.min(current * 2, max)
return next > current ? next : null
}
function matchesAllTokens(
queryTokens: string[],
candidateTokens: string[],
matcher: (candidate: string, query: string) => boolean,
) {
if (queryTokens.length === 0 || candidateTokens.length === 0) return false
return queryTokens.every((queryToken) =>
candidateTokens.some((candidateToken) => matcher(candidateToken, queryToken)),
)
}
function getLexicalBoost(queryTokens: string[], displayName: string, slug: string) {
const slugTokens = tokenize(slug)
const nameTokens = tokenize(displayName)
let boost = 0
if (matchesAllTokens(queryTokens, slugTokens, (candidate, query) => candidate === query)) {
boost += SLUG_EXACT_BOOST
} else if (
matchesAllTokens(queryTokens, slugTokens, (candidate, query) => candidate.startsWith(query))
) {
boost += SLUG_PREFIX_BOOST
}
if (matchesAllTokens(queryTokens, nameTokens, (candidate, query) => candidate === query)) {
boost += NAME_EXACT_BOOST
} else if (
matchesAllTokens(queryTokens, nameTokens, (candidate, query) => candidate.startsWith(query))
) {
boost += NAME_PREFIX_BOOST
}
return boost
}
function scoreSkillResult(
queryTokens: string[],
vectorScore: number,
displayName: string,
slug: string,
downloads: number,
) {
const lexicalBoost = getLexicalBoost(queryTokens, displayName, slug)
const popularityBoost = Math.log1p(Math.max(downloads, 0)) * POPULARITY_WEIGHT
return vectorScore + lexicalBoost + popularityBoost
}
function mergeUniqueBySkillId(primary: SkillSearchEntry[], fallback: SkillSearchEntry[]) {
if (fallback.length === 0) return primary
const out = [...primary]
const seen = new Set(primary.map((entry) => entry.skill._id))
for (const entry of fallback) {
if (seen.has(entry.skill._id)) continue
seen.add(entry.skill._id)
out.push(entry)
}
return out
}
export const searchSkills: ReturnType<typeof action> = action({
args: {
query: v.string(),
limit: v.optional(v.number()),
highlightedOnly: v.optional(v.boolean()),
nonSuspiciousOnly: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SearchResult[]> => {
const query = args.query.trim()
if (!query) return []
const queryTokens = tokenize(query)
if (queryTokens.length === 0) return []
const vector = await generateEmbedding(query)
let vector: number[]
try {
vector = await generateEmbedding(query)
} catch (error) {
console.warn('Search embedding generation failed', error)
return []
}
const limit = args.limit ?? 10
const maxCandidate = Math.min(Math.max(limit * 10, 200), 1000)
let candidateLimit = Math.max(limit * 3, 50)
let hydrated: HydratedEntry[] = []
// Convex vectorSearch max limit is 256; clamp candidate sizes accordingly.
const maxCandidate = Math.min(Math.max(limit * 10, 200), 256)
let candidateLimit = Math.min(Math.max(limit * 3, 50), 256)
let hydrated: SkillSearchEntry[] = []
let scoreById = new Map<Id<'skillEmbeddings'>, number>()
let exactMatches: HydratedEntry[] = []
let exactMatches: SkillSearchEntry[] = []
while (candidateLimit <= maxCandidate) {
const results = await ctx.vectorSearch('skillEmbeddings', 'by_embedding', {
@@ -46,21 +124,34 @@ export const searchSkills: ReturnType<typeof action> = action({
hydrated = (await ctx.runQuery(internal.search.hydrateResults, {
embeddingIds: results.map((result) => result._id),
})) as HydratedEntry[]
nonSuspiciousOnly: args.nonSuspiciousOnly,
})) as SkillSearchEntry[]
scoreById = new Map<Id<'skillEmbeddings'>, number>(
results.map((result) => [result._id, result._score]),
)
const badgeMapEntries = (await ctx.runQuery(internal.search.getSkillBadgeMapsInternal, {
skillIds: hydrated.map((entry) => entry.skill._id),
})) as Array<[Id<'skills'>, SkillBadgeMap]>
const badgeMapBySkillId = new Map(badgeMapEntries)
const hydratedWithBadges = hydrated.map((entry) => ({
...entry,
skill: {
...entry.skill,
badges: badgeMapBySkillId.get(entry.skill._id) ?? {},
},
}))
const filtered = args.highlightedOnly
? hydrated.filter((entry) => entry.skill?.batch === 'highlighted')
: hydrated
? hydratedWithBadges.filter((entry) => isSkillHighlighted(entry.skill))
: hydratedWithBadges
exactMatches = filtered.filter((entry) =>
matchesExactTokens(queryTokens, [
entry.skill?.displayName,
entry.skill?.slug,
entry.skill?.summary,
entry.skill.displayName,
entry.skill.slug,
entry.skill.summary,
]),
)
@@ -73,37 +164,180 @@ export const searchSkills: ReturnType<typeof action> = action({
candidateLimit = nextLimit
}
return exactMatches
.map((entry) => ({
...entry,
score: scoreById.get(entry.embeddingId) ?? 0,
}))
const fallbackMatches =
exactMatches.length >= limit
? []
: ((await ctx.runQuery(internal.search.lexicalFallbackSkills, {
query,
queryTokens,
limit: Math.min(Math.max(limit * 4, 200), FALLBACK_SCAN_LIMIT),
highlightedOnly: args.highlightedOnly,
nonSuspiciousOnly: args.nonSuspiciousOnly,
})) as SkillSearchEntry[])
const mergedMatches = mergeUniqueBySkillId(exactMatches, fallbackMatches)
return mergedMatches
.map((entry) => {
const vectorScore = entry.embeddingId ? (scoreById.get(entry.embeddingId) ?? 0) : 0
return {
...entry,
score: scoreSkillResult(
queryTokens,
vectorScore,
entry.skill.displayName,
entry.skill.slug,
entry.skill.stats.downloads,
),
}
})
.filter((entry) => entry.skill)
.sort((a, b) => b.score - a.score || b.skill.stats.downloads - a.skill.stats.downloads)
.slice(0, limit)
},
})
export const hydrateResults = internalQuery({
args: { embeddingIds: v.array(v.id('skillEmbeddings')) },
handler: async (ctx, args): Promise<HydratedEntry[]> => {
const entries: HydratedEntry[] = []
export const getBadgeMapsForSkills = internalQuery({
args: { skillIds: v.array(v.id('skills')) },
handler: async (ctx, args): Promise<Array<[Id<'skills'>, SkillBadgeMap]>> => {
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
return Array.from(badgeMap.entries())
},
})
for (const embeddingId of args.embeddingIds) {
const embedding = await ctx.db.get(embeddingId)
if (!embedding) continue
const skill = await ctx.db.get(embedding.skillId)
if (skill?.softDeletedAt) continue
const version = await ctx.db.get(embedding.versionId)
entries.push({ embeddingId, skill, version })
export const hydrateResults = internalQuery({
args: {
embeddingIds: v.array(v.id('skillEmbeddings')),
nonSuspiciousOnly: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SkillSearchEntry[]> => {
const ownerHandleCache = new Map<Id<'users'>, Promise<string | null>>()
const getOwnerHandle = (ownerUserId: Id<'users'>) => {
const cached = ownerHandleCache.get(ownerUserId)
if (cached) return cached
const handlePromise = ctx.db
.get(ownerUserId)
.then((owner) => owner?.handle ?? owner?._id ?? null)
ownerHandleCache.set(ownerUserId, handlePromise)
return handlePromise
}
return entries
const entries: Array<SkillSearchEntry | null> = await Promise.all(
args.embeddingIds.map(async (embeddingId) => {
const embedding = await ctx.db.get(embeddingId)
if (!embedding) return null
const skill = await ctx.db.get(embedding.skillId)
if (!skill || skill.softDeletedAt) return null
if (args.nonSuspiciousOnly && isSkillSuspicious(skill)) return null
const [version, ownerHandle] = await Promise.all([
ctx.db.get(embedding.versionId),
getOwnerHandle(skill.ownerUserId),
])
const publicSkill = toPublicSkill(skill)
if (!publicSkill) return null
return { embeddingId, skill: publicSkill, version, ownerHandle }
}),
)
return entries.filter((entry): entry is SkillSearchEntry => entry !== null)
},
})
export const lexicalFallbackSkills = internalQuery({
args: {
query: v.string(),
queryTokens: v.array(v.string()),
limit: v.optional(v.number()),
highlightedOnly: v.optional(v.boolean()),
nonSuspiciousOnly: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SkillSearchEntry[]> => {
const limit = Math.min(Math.max(args.limit ?? 200, 10), FALLBACK_SCAN_LIMIT)
const seenSkillIds = new Set<Id<'skills'>>()
const candidateSkills: Doc<'skills'>[] = []
const slugQuery = args.query.trim().toLowerCase()
if (/^[a-z0-9][a-z0-9-]*$/.test(slugQuery)) {
const exactSlugSkill = await ctx.db
.query('skills')
.withIndex('by_slug', (q) => q.eq('slug', slugQuery))
.unique()
if (
exactSlugSkill &&
!exactSlugSkill.softDeletedAt &&
(!args.nonSuspiciousOnly || !isSkillSuspicious(exactSlugSkill))
) {
seenSkillIds.add(exactSlugSkill._id)
candidateSkills.push(exactSlugSkill)
}
}
const recentSkills = await ctx.db
.query('skills')
.withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined))
.order('desc')
.take(FALLBACK_SCAN_LIMIT)
for (const skill of recentSkills) {
if (seenSkillIds.has(skill._id)) continue
if (args.nonSuspiciousOnly && isSkillSuspicious(skill)) continue
seenSkillIds.add(skill._id)
candidateSkills.push(skill)
}
const matched = candidateSkills.filter((skill) =>
matchesExactTokens(args.queryTokens, [skill.displayName, skill.slug, skill.summary]),
)
if (matched.length === 0) return []
const ownerHandleCache = new Map<Id<'users'>, Promise<string | null>>()
const getOwnerHandle = (ownerUserId: Id<'users'>) => {
const cached = ownerHandleCache.get(ownerUserId)
if (cached) return cached
const handlePromise = ctx.db
.get(ownerUserId)
.then((owner) => owner?.handle ?? owner?._id ?? null)
ownerHandleCache.set(ownerUserId, handlePromise)
return handlePromise
}
const entries = await Promise.all(
matched.map(async (skill) => {
const [version, ownerHandle] = await Promise.all([
skill.latestVersionId ? ctx.db.get(skill.latestVersionId) : Promise.resolve(null),
getOwnerHandle(skill.ownerUserId),
])
const publicSkill = toPublicSkill(skill)
if (!publicSkill) return null
return { skill: publicSkill, version, ownerHandle }
}),
)
const validEntries = entries.filter((entry): entry is SkillSearchEntry => entry !== null)
if (validEntries.length === 0) return []
const badgeMap = await getSkillBadgeMaps(
ctx,
validEntries.map((entry) => entry.skill._id),
)
const withBadges = validEntries.map((entry) => ({
...entry,
skill: {
...entry.skill,
badges: badgeMap.get(entry.skill._id) ?? {},
},
}))
const filtered = args.highlightedOnly
? withBadges.filter((entry) => isSkillHighlighted(entry.skill))
: withBadges
return filtered.slice(0, limit)
},
})
type HydratedSoulEntry = {
embeddingId: Id<'soulEmbeddings'>
soul: Doc<'souls'> | null
soul: NonNullable<ReturnType<typeof toPublicSoul>>
version: Doc<'soulVersions'> | null
}
@@ -119,10 +353,17 @@ export const searchSouls: ReturnType<typeof action> = action({
if (!query) return []
const queryTokens = tokenize(query)
if (queryTokens.length === 0) return []
const vector = await generateEmbedding(query)
let vector: number[]
try {
vector = await generateEmbedding(query)
} catch (error) {
console.warn('Search embedding generation failed', error)
return []
}
const limit = args.limit ?? 10
const maxCandidate = Math.min(Math.max(limit * 10, 200), 1000)
let candidateLimit = Math.max(limit * 3, 50)
// Convex vectorSearch max limit is 256; clamp candidate sizes accordingly.
const maxCandidate = Math.min(Math.max(limit * 10, 200), 256)
let candidateLimit = Math.min(Math.max(limit * 3, 50), 256)
let hydrated: HydratedSoulEntry[] = []
let scoreById = new Map<Id<'soulEmbeddings'>, number>()
let exactMatches: HydratedSoulEntry[] = []
@@ -144,9 +385,9 @@ export const searchSouls: ReturnType<typeof action> = action({
exactMatches = hydrated.filter((entry) =>
matchesExactTokens(queryTokens, [
entry.soul?.displayName,
entry.soul?.slug,
entry.soul?.summary,
entry.soul.displayName,
entry.soul.slug,
entry.soul.summary,
]),
)
@@ -180,11 +421,27 @@ export const hydrateSoulResults = internalQuery({
const soul = await ctx.db.get(embedding.soulId)
if (soul?.softDeletedAt) continue
const version = await ctx.db.get(embedding.versionId)
entries.push({ embeddingId, soul, version })
const publicSoul = toPublicSoul(soul)
if (!publicSoul) continue
entries.push({ embeddingId, soul: publicSoul, version })
}
return entries
},
})
export const __test = { getNextCandidateLimit }
export const getSkillBadgeMapsInternal = internalQuery({
args: { skillIds: v.array(v.id('skills')) },
handler: async (ctx, args) => {
const badgeMap = await getSkillBadgeMaps(ctx, args.skillIds)
return Array.from(badgeMap.entries())
},
})
export const __test = {
getNextCandidateLimit,
matchesAllTokens,
getLexicalBoost,
scoreSkillResult,
mergeUniqueBySkillId,
}
+2 -1
View File
@@ -242,7 +242,8 @@ export const ensureSeedUserInternal = internalMutation({
})
async function sha256Hex(bytes: Uint8Array) {
const digest = await crypto.subtle.digest('SHA-256', bytes)
const data = new Uint8Array(bytes)
const digest = await crypto.subtle.digest('SHA-256', data)
return toHex(new Uint8Array(digest))
}
+3 -3
View File
File diff suppressed because one or more lines are too long
+110
View File
@@ -0,0 +1,110 @@
/* @vitest-environment node */
import { describe, expect, it } from 'vitest'
// Test the aggregateEvents function by importing and testing the module logic
// Since aggregateEvents is not exported, we test the behavior indirectly through
// the event processing contract
describe('skill stat events - comment delta handling', () => {
it('aggregates comment and uncomment events into net deltas', () => {
// Simulate the aggregation logic from processSkillStatEventsAction
type EventKind =
| 'download'
| 'star'
| 'unstar'
| 'comment'
| 'uncomment'
| 'install_new'
| 'install_reactivate'
| 'install_deactivate'
| 'install_clear'
const events: { kind: EventKind; occurredAt: number }[] = [
{ kind: 'star', occurredAt: 1000 },
{ kind: 'comment', occurredAt: 2000 },
{ kind: 'comment', occurredAt: 3000 },
{ kind: 'uncomment', occurredAt: 4000 },
{ kind: 'download', occurredAt: 5000 },
]
// Replicate the aggregation logic
const result = {
downloads: 0,
stars: 0,
comments: 0,
installsAllTime: 0,
installsCurrent: 0,
downloadEvents: [] as number[],
installNewEvents: [] as number[],
}
for (const event of events) {
switch (event.kind) {
case 'download':
result.downloads += 1
result.downloadEvents.push(event.occurredAt)
break
case 'star':
result.stars += 1
break
case 'unstar':
result.stars -= 1
break
case 'comment':
result.comments += 1
break
case 'uncomment':
result.comments -= 1
break
case 'install_new':
result.installsAllTime += 1
result.installsCurrent += 1
result.installNewEvents.push(event.occurredAt)
break
case 'install_reactivate':
result.installsCurrent += 1
break
case 'install_deactivate':
result.installsCurrent -= 1
break
}
}
expect(result.stars).toBe(1)
expect(result.comments).toBe(1) // 2 comments - 1 uncomment
expect(result.downloads).toBe(1)
expect(result.downloadEvents).toEqual([5000])
})
it('should include comments in delta check (regression test for dropped comments)', () => {
// This test verifies the fix: the condition guard in applyAggregatedStatsAndUpdateCursor
// must include comments !== 0 so comment-only batches are not skipped
const delta = {
downloads: 0,
stars: 0,
comments: 3,
installsAllTime: 0,
installsCurrent: 0,
}
// The OLD buggy condition (missing comments):
const oldCondition =
delta.downloads !== 0 ||
delta.stars !== 0 ||
delta.installsAllTime !== 0 ||
delta.installsCurrent !== 0
// The FIXED condition (includes comments):
const fixedCondition =
delta.downloads !== 0 ||
delta.stars !== 0 ||
delta.comments !== 0 ||
delta.installsAllTime !== 0 ||
delta.installsCurrent !== 0
// With only comment deltas, the old condition would skip the patch
expect(oldCondition).toBe(false)
// The fixed condition correctly triggers the patch
expect(fixedCondition).toBe(true)
})
})
+591
View File
@@ -0,0 +1,591 @@
/**
* Skill Stat Events - Event-sourced stats processing for skills
*
* Instead of updating skill stats synchronously in the hot path (which can cause
* contention when multiple users download/star/install the same skill), we insert
* lightweight event records and process them in batches via a cron job.
*
* Flow:
* 1. User action (download, star, install) → insertStatEvent() writes to skillStatEvents table
* 2. Cron job runs every 5 minutes → processSkillStatEventsInternal() processes batches
* 3. Events are aggregated per-skill to minimize database operations
* 4. Stats are applied to skill documents and daily stats tables
* 5. Events are marked as processed (kept forever for auditing)
*/
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import type { MutationCtx } from './_generated/server'
import { internalAction, internalMutation, internalQuery } from './_generated/server'
import { applySkillStatDeltas, bumpDailySkillStats } from './lib/skillStats'
/**
* Event types that affect skill stats:
*
* - download: User downloaded skill as zip (+1 downloads)
* - star: User starred the skill (+1 stars)
* - unstar: User removed their star (-1 stars)
* - install_new: First time this user installed this skill (+1 installsAllTime, +1 installsCurrent)
* - install_reactivate: User re-added skill after removing it (+1 installsCurrent only)
* - install_deactivate: User removed skill from all projects (-1 installsCurrent)
* - install_clear: User cleared all telemetry data (custom delta for both allTime and current)
*/
export type StatEventKind =
| 'download'
| 'star'
| 'unstar'
| 'comment'
| 'uncomment'
| 'install_new'
| 'install_reactivate'
| 'install_deactivate'
| 'install_clear'
/**
* Insert a stat event to be processed later by the cron job.
*
* This is called from the hot path (downloads, stars, telemetry) instead of
* directly updating skill stats. It's a single insert with no read-modify-write
* cycle, so it's fast and doesn't contend with other operations on the same skill.
*
* @param ctx - Mutation context
* @param params.skillId - The skill being affected
* @param params.kind - Type of event (download, star, install_new, etc.)
* @param params.occurredAt - When the event happened (defaults to now). Important for
* daily stats bucketing - we want downloads at 11:55 PM Monday
* to count toward Monday's stats even if processed on Tuesday.
* @param params.delta - Only used for install_clear events, specifies exact delta amounts
*/
export async function insertStatEvent(
ctx: MutationCtx,
params: {
skillId: Id<'skills'>
kind: StatEventKind
occurredAt?: number
delta?: { allTime: number; current: number }
},
) {
await ctx.db.insert('skillStatEvents', {
skillId: params.skillId,
kind: params.kind,
delta: params.delta,
occurredAt: params.occurredAt ?? Date.now(),
processedAt: undefined,
})
}
/**
* Aggregated deltas for a single skill after processing multiple events.
*
* When we process a batch of 100 events, many might be for the same skill.
* Instead of updating the skill document once per event, we aggregate all
* events for each skill and apply a single update.
*
* The downloadEvents and installNewEvents arrays store the original timestamps
* so we can update daily stats with the correct day bucket for each event.
*/
type AggregatedDeltas = {
downloads: number
stars: number
comments: number
installsAllTime: number
installsCurrent: number
/** Original timestamps for each download event (for daily stats bucketing) */
downloadEvents: number[]
/** Original timestamps for each new install event (for daily stats bucketing) */
installNewEvents: number[]
}
/**
* Aggregate multiple events for a single skill into net deltas.
*
* Example: If a skill has these events in the batch:
* - download (Mon 11pm)
* - download (Tue 1am)
* - star
* - unstar
* - star
*
* The result would be:
* - downloads: 2
* - stars: 1 (net: +1 -1 +1 = +1)
* - downloadEvents: [<Mon 11pm timestamp>, <Tue 1am timestamp>]
*
* This aggregation reduces the number of database operations from N events
* to 1 skill update + N daily stat updates (which themselves may coalesce
* if multiple events fall on the same day).
*/
function aggregateEvents(events: Doc<'skillStatEvents'>[]): AggregatedDeltas {
const result: AggregatedDeltas = {
downloads: 0,
stars: 0,
comments: 0,
installsAllTime: 0,
installsCurrent: 0,
downloadEvents: [],
installNewEvents: [],
}
for (const event of events) {
switch (event.kind) {
case 'download':
result.downloads += 1
result.downloadEvents.push(event.occurredAt)
break
case 'star':
result.stars += 1
break
case 'unstar':
result.stars -= 1
break
case 'comment':
result.comments += 1
break
case 'uncomment':
result.comments -= 1
break
case 'install_new':
// New user installing for the first time: count toward both lifetime and current
result.installsAllTime += 1
result.installsCurrent += 1
result.installNewEvents.push(event.occurredAt)
break
case 'install_reactivate':
// User re-added skill after removing: only affects current count
result.installsCurrent += 1
break
case 'install_deactivate':
// User removed skill from all projects: only affects current count
result.installsCurrent -= 1
break
case 'install_clear':
// User cleared telemetry: uses custom delta values (typically negative)
if (event.delta) {
result.installsAllTime += event.delta.allTime
result.installsCurrent += event.delta.current
}
break
}
}
return result
}
/**
* Process a batch of unprocessed stat events.
*
* Called by cron every 5 minutes. Processes up to batchSize events (default 100).
* If the batch is full, schedules an immediate follow-up run to drain the queue.
*
* Processing steps:
* 1. Query unprocessed events (processedAt is undefined)
* 2. Group events by skillId to minimize skill document fetches
* 3. For each skill:
* a. Fetch the skill document once
* b. Aggregate all events for this skill into net deltas
* c. Apply deltas to skill stats (downloads, stars, installs)
* d. Update daily stats for trending (using original event timestamps)
* e. Mark all events as processed
* 4. If batch was full, schedule another run immediately
*
* Aggregation levels:
* - Level 1: Batch of 100 events from the queue
* - Level 2: Group by skillId (e.g., 100 events → 30 unique skills)
* - Level 3: Aggregate events per skill (e.g., 5 events → 1 skill update)
* - Level 4: Daily stats may coalesce (e.g., 3 downloads same day → 1 upsert)
*/
export const processSkillStatEventsInternal = internalMutation({
args: { batchSize: v.optional(v.number()) },
handler: async (ctx, args) => {
const batchSize = args.batchSize ?? 100
const now = Date.now()
// Level 1: Fetch a batch of unprocessed events
const events = await ctx.db
.query('skillStatEvents')
.withIndex('by_unprocessed', (q) => q.eq('processedAt', undefined))
.take(batchSize)
if (events.length === 0) {
return { processed: 0 }
}
// Level 2: Group events by skillId to minimize database reads
// Instead of fetching the same skill document multiple times,
// we fetch it once and process all its events together
const eventsBySkill = new Map<Id<'skills'>, Doc<'skillStatEvents'>[]>()
for (const event of events) {
const existing = eventsBySkill.get(event.skillId) ?? []
existing.push(event)
eventsBySkill.set(event.skillId, existing)
}
// Process each skill's events
for (const [skillId, skillEvents] of eventsBySkill) {
const skill = await ctx.db.get(skillId)
// Skill was deleted - just mark events as processed
if (!skill) {
for (const event of skillEvents) {
await ctx.db.patch(event._id, { processedAt: now })
}
continue
}
// Level 3: Aggregate all events for this skill into net deltas
// e.g., 3 downloads + 2 stars - 1 unstar → { downloads: 3, stars: 1 }
const deltas = aggregateEvents(skillEvents)
// Apply aggregated deltas to skill stats (single update per skill)
if (
deltas.downloads !== 0 ||
deltas.stars !== 0 ||
deltas.comments !== 0 ||
deltas.installsAllTime !== 0 ||
deltas.installsCurrent !== 0
) {
const patch = applySkillStatDeltas(skill, {
downloads: deltas.downloads,
stars: deltas.stars,
comments: deltas.comments,
installsAllTime: deltas.installsAllTime,
installsCurrent: deltas.installsCurrent,
})
await ctx.db.patch(skill._id, {
...patch,
updatedAt: now,
})
}
// Update daily stats for trending/leaderboards
// We use the ORIGINAL event timestamp (occurredAt) so that:
// - A download at Mon 11:55 PM counts toward Monday's stats
// - Even if the cron processes it on Tuesday
//
// Level 4: bumpDailySkillStats does its own coalescing - multiple
// events on the same day will update the same daily record
for (const occurredAt of deltas.downloadEvents) {
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, downloads: 1 })
}
for (const occurredAt of deltas.installNewEvents) {
await bumpDailySkillStats(ctx, { skillId, now: occurredAt, installs: 1 })
}
// Mark all events for this skill as processed
for (const event of skillEvents) {
await ctx.db.patch(event._id, { processedAt: now })
}
}
// If we hit the batch limit, there may be more events waiting.
// Schedule an immediate follow-up run to drain the queue.
// This ensures high-volume periods don't create a backlog.
if (events.length === batchSize) {
await ctx.scheduler.runAfter(0, internal.skillStatEvents.processSkillStatEventsInternal, {
batchSize,
})
}
return { processed: events.length }
},
})
// ============================================================================
// Action-based processing (cursor-based, runs outside transaction window)
// ============================================================================
const CURSOR_KEY = 'skill_stat_events'
const EVENT_BATCH_SIZE = 500
const MAX_SKILLS_PER_RUN = 50
/**
* Fetch a batch of events after the given cursor (by _creationTime).
* Returns events sorted by _creationTime ascending.
*/
export const getUnprocessedEventBatch = internalQuery({
args: {
cursorCreationTime: v.optional(v.number()),
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const limit = args.limit ?? EVENT_BATCH_SIZE
const cursor = args.cursorCreationTime
// Query events after the cursor using the built-in creation time index
const events = await ctx.db
.query('skillStatEvents')
.withIndex('by_creation_time', (q) =>
cursor !== undefined ? q.gt('_creationTime', cursor) : q,
)
.take(limit)
return events
},
})
/**
* Get the current cursor position from the cursors table.
*/
export const getStatEventCursor = internalQuery({
args: {},
handler: async (ctx) => {
const cursor = await ctx.db
.query('skillStatUpdateCursors')
.withIndex('by_key', (q) => q.eq('key', CURSOR_KEY))
.unique()
return cursor?.cursorCreationTime
},
})
/**
* Validator for skill deltas passed to the mutation.
*/
const skillDeltaValidator = v.object({
skillId: v.id('skills'),
downloads: v.number(),
stars: v.number(),
comments: v.number(),
installsAllTime: v.number(),
installsCurrent: v.number(),
downloadEvents: v.array(v.number()),
installNewEvents: v.array(v.number()),
})
/**
* Apply aggregated stats to skills and update the cursor.
* This is a single atomic mutation that:
* 1. Updates all affected skills with their aggregated deltas
* 2. Updates daily stats for trending
* 3. Advances the cursor to the new position
*/
export const applyAggregatedStatsAndUpdateCursor = internalMutation({
args: {
skillDeltas: v.array(skillDeltaValidator),
newCursor: v.number(),
},
handler: async (ctx, args) => {
const now = Date.now()
// Process each skill's aggregated deltas
for (const delta of args.skillDeltas) {
const skill = await ctx.db.get(delta.skillId)
// Skill was deleted - skip
if (!skill) {
continue
}
// Apply aggregated deltas to skill stats
if (
delta.downloads !== 0 ||
delta.stars !== 0 ||
delta.comments !== 0 ||
delta.installsAllTime !== 0 ||
delta.installsCurrent !== 0
) {
const patch = applySkillStatDeltas(skill, {
downloads: delta.downloads,
stars: delta.stars,
comments: delta.comments,
installsAllTime: delta.installsAllTime,
installsCurrent: delta.installsCurrent,
})
await ctx.db.patch(skill._id, {
...patch,
updatedAt: now,
})
}
// Update daily stats for trending/leaderboards
for (const occurredAt of delta.downloadEvents) {
await bumpDailySkillStats(ctx, { skillId: delta.skillId, now: occurredAt, downloads: 1 })
}
for (const occurredAt of delta.installNewEvents) {
await bumpDailySkillStats(ctx, { skillId: delta.skillId, now: occurredAt, installs: 1 })
}
}
// Update cursor position (upsert)
const existingCursor = await ctx.db
.query('skillStatUpdateCursors')
.withIndex('by_key', (q) => q.eq('key', CURSOR_KEY))
.unique()
if (existingCursor) {
await ctx.db.patch(existingCursor._id, {
cursorCreationTime: args.newCursor,
updatedAt: now,
})
} else {
await ctx.db.insert('skillStatUpdateCursors', {
key: CURSOR_KEY,
cursorCreationTime: args.newCursor,
updatedAt: now,
})
}
return { skillsUpdated: args.skillDeltas.length }
},
})
/**
* Action that processes skill stat events in batches outside the transaction window.
*
* Algorithm:
* 1. Get current cursor position
* 2. Fetch events in batches of 500, aggregating as we go
* 3. Stop when we have >= 500 unique skills OR run out of events
* 4. Call mutation to apply all deltas and update cursor atomically
* 5. Self-schedule if we stopped due to skill limit (not exhaustion)
*/
export const processSkillStatEventsAction = internalAction({
args: {},
handler: async (ctx) => {
// Get current cursor position (convert null to undefined for consistency)
const cursorResult = await ctx.runQuery(internal.skillStatEvents.getStatEventCursor)
let cursor: number | undefined = cursorResult ?? undefined
console.log(`[STAT-AGG] Starting aggregation, cursor=${cursor ?? 'none'}`)
// Aggregated deltas per skill
const aggregatedBySkill = new Map<
Id<'skills'>,
{
downloads: number
stars: number
comments: number
installsAllTime: number
installsCurrent: number
downloadEvents: number[]
installNewEvents: number[]
}
>()
let maxCreationTime: number | undefined = cursor
let exhausted = false
let totalEventsFetched = 0
// Fetch and aggregate until we have enough skills or run out of events
while (aggregatedBySkill.size < MAX_SKILLS_PER_RUN) {
const events = await ctx.runQuery(internal.skillStatEvents.getUnprocessedEventBatch, {
cursorCreationTime: cursor,
limit: EVENT_BATCH_SIZE,
})
if (events.length === 0) {
exhausted = true
break
}
totalEventsFetched += events.length
const skillsBefore = aggregatedBySkill.size
// Aggregate events into per-skill deltas
for (const event of events) {
let skillDelta = aggregatedBySkill.get(event.skillId)
if (!skillDelta) {
skillDelta = {
downloads: 0,
stars: 0,
comments: 0,
installsAllTime: 0,
installsCurrent: 0,
downloadEvents: [],
installNewEvents: [],
}
aggregatedBySkill.set(event.skillId, skillDelta)
}
// Apply event to aggregated deltas
switch (event.kind) {
case 'download':
skillDelta.downloads += 1
skillDelta.downloadEvents.push(event.occurredAt)
break
case 'star':
skillDelta.stars += 1
break
case 'unstar':
skillDelta.stars -= 1
break
case 'comment':
skillDelta.comments += 1
break
case 'uncomment':
skillDelta.comments -= 1
break
case 'install_new':
skillDelta.installsAllTime += 1
skillDelta.installsCurrent += 1
skillDelta.installNewEvents.push(event.occurredAt)
break
case 'install_reactivate':
skillDelta.installsCurrent += 1
break
case 'install_deactivate':
skillDelta.installsCurrent -= 1
break
case 'install_clear':
if (event.delta) {
skillDelta.installsAllTime += event.delta.allTime
skillDelta.installsCurrent += event.delta.current
}
break
}
// Track highest _creationTime seen
if (maxCreationTime === undefined || event._creationTime > maxCreationTime) {
maxCreationTime = event._creationTime
}
}
// Update cursor for next batch fetch
cursor = events[events.length - 1]._creationTime
console.log(
`[STAT-AGG] Fetched ${events.length} events, ${aggregatedBySkill.size - skillsBefore} new skills (${aggregatedBySkill.size} total)`,
)
// If we got fewer than requested, we've exhausted the events
if (events.length < EVENT_BATCH_SIZE) {
exhausted = true
break
}
}
// If we have nothing to process, we're done
if (aggregatedBySkill.size === 0 || maxCreationTime === undefined) {
console.log('[STAT-AGG] No events to process, done')
return { processed: 0, skillsUpdated: 0, exhausted: true }
}
// Convert map to array for mutation
const skillDeltas = Array.from(aggregatedBySkill.entries()).map(([skillId, delta]) => ({
skillId,
...delta,
}))
console.log(
`[STAT-AGG] Running mutation for ${skillDeltas.length} skills (${totalEventsFetched} total events)`,
)
// Apply all deltas and update cursor atomically
await ctx.runMutation(internal.skillStatEvents.applyAggregatedStatsAndUpdateCursor, {
skillDeltas,
newCursor: maxCreationTime,
})
// Self-schedule if we stopped because of skill limit, not exhaustion
if (!exhausted) {
console.log('[STAT-AGG] More events remaining, self-scheduling')
await ctx.scheduler.runAfter(0, internal.skillStatEvents.processSkillStatEventsAction, {})
} else {
console.log('[STAT-AGG] All events processed, done')
}
return {
skillsUpdated: skillDeltas.length,
exhausted,
}
},
})
+349
View File
@@ -0,0 +1,349 @@
import { describe, expect, it, vi } from 'vitest'
import {
approveSkillByHashInternal,
clearOwnerSuspiciousFlagsInternal,
escalateByVtInternal,
insertVersion,
} from './skills'
type WrappedHandler<TArgs> = {
_handler: (ctx: unknown, args: TArgs) => Promise<unknown>
}
const insertVersionHandler = (insertVersion as unknown as WrappedHandler<Record<string, unknown>>)
._handler
const approveSkillByHashHandler = (
approveSkillByHashInternal as unknown as WrappedHandler<Record<string, unknown>>
)._handler
const escalateByVtHandler = (
escalateByVtInternal as unknown as WrappedHandler<Record<string, unknown>>
)._handler
const clearOwnerSuspiciousFlagsHandler = (
clearOwnerSuspiciousFlagsInternal as unknown as WrappedHandler<Record<string, unknown>>
)._handler
function createPublishArgs(overrides?: Partial<Record<string, unknown>>) {
return {
userId: 'users:owner',
slug: 'spam-skill',
displayName: 'Spam Skill',
version: '1.0.0',
changelog: 'Initial release',
changelogSource: 'user',
tags: ['latest'],
fingerprint: 'f'.repeat(64),
files: [
{
path: 'SKILL.md',
size: 128,
storageId: '_storage:1',
sha256: 'a'.repeat(64),
contentType: 'text/markdown',
},
],
parsed: {
frontmatter: { description: 'test' },
metadata: {},
clawdis: {},
},
embedding: [0.1, 0.2],
...overrides,
}
}
describe('skills anti-spam guards', () => {
it('blocks low-trust users after hourly new-skill cap', async () => {
const now = Date.now()
const ownerSkills = Array.from({ length: 5 }, (_, i) => ({
_id: `skills:${i}`,
createdAt: now - i * 10_000,
}))
const db = {
get: vi.fn(async () => ({
_id: 'users:owner',
_creationTime: now - 2 * 24 * 60 * 60 * 1000,
createdAt: now - 2 * 24 * 60 * 60 * 1000,
deletedAt: undefined,
})),
query: vi.fn((table: string) => {
if (table === 'skills') {
return {
withIndex: (name: string) => {
if (name === 'by_slug') {
return { unique: async () => null }
}
if (name === 'by_owner') {
return {
order: () => ({
take: async () => ownerSkills,
}),
}
}
throw new Error(`unexpected index ${name}`)
},
}
}
throw new Error(`unexpected table ${table}`)
}),
}
await expect(
insertVersionHandler({ db } as never, createPublishArgs() as never),
).rejects.toThrow(/max 5 new skills per hour/i)
})
it('keeps suspicious skills visible for low-trust publishers', async () => {
const patch = vi.fn(async () => {})
const version = { _id: 'skillVersions:1', skillId: 'skills:1' }
const skill = {
_id: 'skills:1',
slug: 'spam-skill',
ownerUserId: 'users:owner',
moderationFlags: undefined,
moderationReason: undefined,
}
const owner = {
_id: 'users:owner',
_creationTime: Date.now() - 2 * 24 * 60 * 60 * 1000,
createdAt: Date.now() - 2 * 24 * 60 * 60 * 1000,
deletedAt: undefined,
}
const db = {
get: vi.fn(async (id: string) => {
if (id === 'skills:1') return skill
if (id === 'users:owner') return owner
return null
}),
query: vi.fn((table: string) => {
if (table === 'skillVersions') {
return {
withIndex: () => ({
unique: async () => version,
}),
}
}
if (table === 'skills') {
return {
withIndex: (name: string) => {
if (name === 'by_owner') {
return {
order: () => ({
take: async () => [],
}),
}
}
throw new Error(`unexpected skills index ${name}`)
},
}
}
throw new Error(`unexpected table ${table}`)
}),
patch,
}
await approveSkillByHashHandler(
{ db, scheduler: { runAfter: vi.fn() } } as never,
{
sha256hash: 'h'.repeat(64),
scanner: 'vt',
status: 'suspicious',
} as never,
)
expect(patch).toHaveBeenCalledWith(
'skills:1',
expect.objectContaining({
moderationStatus: 'active',
moderationReason: 'scanner.vt.suspicious',
moderationFlags: ['flagged.suspicious'],
}),
)
})
it('keeps admin-owned skills non-suspicious for suspicious scanner verdicts', async () => {
const patch = vi.fn(async () => {})
const version = { _id: 'skillVersions:1', skillId: 'skills:1' }
const skill = {
_id: 'skills:1',
slug: 'trusted-skill',
ownerUserId: 'users:owner',
moderationFlags: ['flagged.suspicious'],
moderationReason: 'scanner.vt.suspicious',
}
const owner = {
_id: 'users:owner',
role: 'admin',
_creationTime: Date.now() - 60 * 24 * 60 * 60 * 1000,
createdAt: Date.now() - 60 * 24 * 60 * 60 * 1000,
deletedAt: undefined,
}
const db = {
get: vi.fn(async (id: string) => {
if (id === 'skills:1') return skill
if (id === 'users:owner') return owner
return null
}),
query: vi.fn((table: string) => {
if (table === 'skillVersions') {
return {
withIndex: () => ({
unique: async () => version,
}),
}
}
if (table === 'skills') {
return {
withIndex: (name: string) => {
if (name === 'by_owner') {
return {
order: () => ({
take: async () => [],
}),
}
}
throw new Error(`unexpected skills index ${name}`)
},
}
}
throw new Error(`unexpected table ${table}`)
}),
patch,
}
await approveSkillByHashHandler(
{ db, scheduler: { runAfter: vi.fn() } } as never,
{
sha256hash: 'h'.repeat(64),
scanner: 'llm',
status: 'suspicious',
} as never,
)
expect(patch).toHaveBeenCalledWith(
'skills:1',
expect.objectContaining({
moderationStatus: 'active',
moderationReason: 'scanner.llm.clean',
moderationFlags: undefined,
}),
)
})
it('vt suspicious escalation does not keep suspicious flags for admin owners', async () => {
const patch = vi.fn(async () => {})
const version = { _id: 'skillVersions:1', skillId: 'skills:1' }
const skill = {
_id: 'skills:1',
slug: 'trusted-skill',
ownerUserId: 'users:owner',
moderationFlags: ['flagged.suspicious'],
moderationReason: 'scanner.llm.suspicious',
}
const owner = {
_id: 'users:owner',
role: 'admin',
deletedAt: undefined,
}
const db = {
get: vi.fn(async (id: string) => {
if (id === 'skills:1') return skill
if (id === 'users:owner') return owner
return null
}),
query: vi.fn((table: string) => {
if (table === 'skillVersions') {
return {
withIndex: () => ({
unique: async () => version,
}),
}
}
throw new Error(`unexpected table ${table}`)
}),
patch,
}
await escalateByVtHandler(
{ db, scheduler: { runAfter: vi.fn() } } as never,
{
sha256hash: 'h'.repeat(64),
status: 'suspicious',
} as never,
)
expect(patch).toHaveBeenCalledWith(
'skills:1',
expect.objectContaining({
moderationFlags: undefined,
moderationReason: 'scanner.llm.clean',
}),
)
})
it('bulk-clears suspicious flags/reasons for privileged owner skills', async () => {
const patch = vi.fn(async () => {})
const owner = {
_id: 'users:owner',
role: 'admin',
deletedAt: undefined,
}
const skills = [
{
_id: 'skills:1',
moderationFlags: ['flagged.suspicious'],
moderationReason: 'scanner.vt.suspicious',
moderationStatus: 'hidden',
softDeletedAt: undefined,
},
{
_id: 'skills:2',
moderationFlags: undefined,
moderationReason: 'scanner.llm.clean',
moderationStatus: 'active',
softDeletedAt: undefined,
},
]
const db = {
get: vi.fn(async (id: string) => {
if (id === 'users:owner') return owner
return null
}),
query: vi.fn((table: string) => {
if (table === 'skills') {
return {
withIndex: (name: string) => {
if (name !== 'by_owner') throw new Error(`unexpected skills index ${name}`)
return {
order: () => ({
take: async () => skills,
}),
}
},
}
}
throw new Error(`unexpected table ${table}`)
}),
patch,
}
const result = await clearOwnerSuspiciousFlagsHandler(
{ db } as never,
{ ownerUserId: 'users:owner', limit: 20 } as never,
)
expect(result).toEqual({ inspected: 2, updated: 1 })
expect(patch).toHaveBeenCalledWith(
'skills:1',
expect.objectContaining({
moderationFlags: undefined,
moderationReason: 'scanner.vt.clean',
moderationStatus: 'active',
}),
)
})
})
+2643 -65
View File
File diff suppressed because it is too large Load Diff
+5 -4
View File
@@ -1,7 +1,8 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { assertRole, requireUser } from './lib/access'
import { assertModerator, requireUser } from './lib/access'
import { type PublicUser, toPublicUser } from './lib/public'
export const listBySoul = query({
args: { soulId: v.id('souls'), limit: v.optional(v.number()) },
@@ -13,10 +14,10 @@ export const listBySoul = query({
.order('desc')
.take(limit)
const results: Array<{ comment: Doc<'soulComments'>; user: Doc<'users'> | null }> = []
const results: Array<{ comment: Doc<'soulComments'>; user: PublicUser | null }> = []
for (const comment of comments) {
if (comment.softDeletedAt) continue
const user = await ctx.db.get(comment.userId)
const user = toPublicUser(await ctx.db.get(comment.userId))
results.push({ comment, user })
}
return results
@@ -59,7 +60,7 @@ export const remove = mutation({
const isOwner = comment.userId === user._id
if (!isOwner) {
assertRole(user, ['admin', 'moderator'])
assertModerator(user)
}
await ctx.db.patch(comment._id, {
+5 -3
View File
@@ -1,7 +1,7 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { requireUser } from './lib/access'
import { toPublicSoul } from './lib/public'
export const isStarred = query({
args: { soulId: v.id('souls') },
@@ -59,10 +59,12 @@ export const listByUser = query({
.order('desc')
.take(limit)
const souls: Doc<'souls'>[] = []
const souls: NonNullable<ReturnType<typeof toPublicSoul>>[] = []
for (const star of stars) {
const soul = await ctx.db.get(star.soulId)
if (soul) souls.push(soul)
const publicSoul = toPublicSoul(soul)
if (!publicSoul) continue
souls.push(publicSoul)
}
return souls
},
+36 -12
View File
@@ -2,7 +2,8 @@ import { ConvexError, v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalMutation, internalQuery, mutation, query } from './_generated/server'
import { assertRole, requireUser, requireUserFromAction } from './lib/access'
import { assertModerator, requireUser, requireUserFromAction } from './lib/access'
import { toPublicSoul, toPublicUser } from './lib/public'
import { getFrontmatterValue, hashSkillFiles } from './lib/skills'
import { generateSoulChangelogPreview } from './lib/soulChangelog'
import { fetchText, type PublishResult, publishSoulVersionForUser } from './lib/soulPublish'
@@ -27,9 +28,11 @@ export const getBySlug = query({
const soul = matches[0] ?? null
if (!soul || soul.softDeletedAt) return null
const latestVersion = soul.latestVersionId ? await ctx.db.get(soul.latestVersionId) : null
const owner = await ctx.db.get(soul.ownerUserId)
const owner = toPublicUser(await ctx.db.get(soul.ownerUserId))
const publicSoul = toPublicSoul(soul)
if (!publicSoul) return null
return { soul, latestVersion, owner }
return { soul: publicSoul, latestVersion, owner }
},
})
@@ -59,13 +62,21 @@ export const list = query({
.withIndex('by_owner', (q) => q.eq('ownerUserId', ownerUserId))
.order('desc')
.take(limit * 5)
return entries.filter((soul) => !soul.softDeletedAt).slice(0, limit)
return entries
.filter((soul) => !soul.softDeletedAt)
.slice(0, limit)
.map((soul) => toPublicSoul(soul))
.filter((soul): soul is NonNullable<typeof soul> => Boolean(soul))
}
const entries = await ctx.db
.query('souls')
.order('desc')
.take(limit * 5)
return entries.filter((soul) => !soul.softDeletedAt).slice(0, limit)
return entries
.filter((soul) => !soul.softDeletedAt)
.slice(0, limit)
.map((soul) => toPublicSoul(soul))
.filter((soul): soul is NonNullable<typeof soul> => Boolean(soul))
},
})
@@ -82,12 +93,17 @@ export const listPublicPage = query({
.order('desc')
.paginate({ cursor: args.cursor ?? null, numItems: limit })
const items: Array<{ soul: Doc<'souls'>; latestVersion: Doc<'soulVersions'> | null }> = []
const items: Array<{
soul: NonNullable<ReturnType<typeof toPublicSoul>>
latestVersion: Doc<'soulVersions'> | null
}> = []
for (const soul of page) {
if (soul.softDeletedAt) continue
const latestVersion = soul.latestVersionId ? await ctx.db.get(soul.latestVersionId) : null
items.push({ soul, latestVersion })
const publicSoul = toPublicSoul(soul)
if (!publicSoul) continue
items.push({ soul: publicSoul, latestVersion })
}
return { items, nextCursor: isDone ? null : continueCursor }
@@ -129,6 +145,14 @@ export const getVersionById = query({
handler: async (ctx, args) => ctx.db.get(args.versionId),
})
export const getVersionsByIdsInternal = internalQuery({
args: { versionIds: v.array(v.id('soulVersions')) },
handler: async (ctx, args) => {
const versions = await Promise.all(args.versionIds.map((id) => ctx.db.get(id)))
return versions.filter((v): v is NonNullable<typeof v> => v !== null)
},
})
export const getVersionByIdInternal = internalQuery({
args: { versionId: v.id('soulVersions') },
handler: async (ctx, args) => ctx.db.get(args.versionId),
@@ -309,7 +333,7 @@ export const updateTags = mutation({
const soul = await ctx.db.get(args.soulId)
if (!soul) throw new Error('Soul not found')
if (soul.ownerUserId !== user._id) {
assertRole(user, ['admin', 'moderator'])
assertModerator(user)
}
const nextTags = { ...soul.tags }
@@ -370,14 +394,14 @@ export const insertVersion = internalMutation({
handler: async (ctx, args) => {
const userId = args.userId
const user = await ctx.db.get(userId)
if (!user || user.deletedAt) throw new Error('User not found')
if (!user || user.deletedAt || user.deactivatedAt) throw new Error('User not found')
const soulMatches = await ctx.db
.query('souls')
.withIndex('by_slug', (q) => q.eq('slug', args.slug))
.order('desc')
.take(2)
let soul = soulMatches[0] ?? null
let soul: Doc<'souls'> | null = soulMatches[0] ?? null
if (soul && soul.ownerUserId !== userId) {
throw new Error('Only the owner can publish updates')
@@ -492,7 +516,7 @@ export const setSoulSoftDeletedInternal = internalMutation({
},
handler: async (ctx, args) => {
const user = await ctx.db.get(args.userId)
if (!user || user.deletedAt) throw new Error('User not found')
if (!user || user.deletedAt || user.deactivatedAt) throw new Error('User not found')
const slug = args.slug.trim().toLowerCase()
if (!slug) throw new Error('Slug required')
@@ -506,7 +530,7 @@ export const setSoulSoftDeletedInternal = internalMutation({
if (!soul) throw new Error('Soul not found')
if (soul.ownerUserId !== args.userId) {
assertRole(user, ['admin', 'moderator'])
assertModerator(user)
}
const now = Date.now()
+50 -12
View File
@@ -1,7 +1,8 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { internalMutation, mutation, query } from './_generated/server'
import { requireUser } from './lib/access'
import { toPublicSkill } from './lib/public'
import { insertStatEvent } from './skillStatEvents'
export const isStarred = query({
args: { skillId: v.id('skills') },
@@ -29,10 +30,7 @@ export const toggle = mutation({
if (existing) {
await ctx.db.delete(existing._id)
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, stars: Math.max(0, skill.stats.stars - 1) },
updatedAt: Date.now(),
})
await insertStatEvent(ctx, { skillId: skill._id, kind: 'unstar' })
return { starred: false }
}
@@ -42,10 +40,7 @@ export const toggle = mutation({
createdAt: Date.now(),
})
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, stars: skill.stats.stars + 1 },
updatedAt: Date.now(),
})
await insertStatEvent(ctx, { skillId: skill._id, kind: 'star' })
return { starred: true }
},
@@ -60,11 +55,54 @@ export const listByUser = query({
.withIndex('by_user', (q) => q.eq('userId', args.userId))
.order('desc')
.take(limit)
const skills: Doc<'skills'>[] = []
const skills: NonNullable<ReturnType<typeof toPublicSkill>>[] = []
for (const star of stars) {
const skill = await ctx.db.get(star.skillId)
if (skill) skills.push(skill)
const publicSkill = toPublicSkill(skill)
if (!publicSkill) continue
skills.push(publicSkill)
}
return skills
},
})
export const addStarInternal = internalMutation({
args: { userId: v.id('users'), skillId: v.id('skills') },
handler: async (ctx, args) => {
const skill = await ctx.db.get(args.skillId)
if (!skill) throw new Error('Skill not found')
const existing = await ctx.db
.query('stars')
.withIndex('by_skill_user', (q) => q.eq('skillId', args.skillId).eq('userId', args.userId))
.unique()
if (existing) return { ok: true as const, starred: true, alreadyStarred: true }
await ctx.db.insert('stars', {
skillId: args.skillId,
userId: args.userId,
createdAt: Date.now(),
})
await insertStatEvent(ctx, { skillId: skill._id, kind: 'star' })
return { ok: true as const, starred: true, alreadyStarred: false }
},
})
export const removeStarInternal = internalMutation({
args: { userId: v.id('users'), skillId: v.id('skills') },
handler: async (ctx, args) => {
const skill = await ctx.db.get(args.skillId)
if (!skill) throw new Error('Skill not found')
const existing = await ctx.db
.query('stars')
.withIndex('by_skill_user', (q) => q.eq('skillId', args.skillId).eq('userId', args.userId))
.unique()
if (!existing) return { ok: true as const, unstarred: false, alreadyUnstarred: true }
await ctx.db.delete(existing._id)
await insertStatEvent(ctx, { skillId: skill._id, kind: 'unstar' })
return { ok: true as const, unstarred: true, alreadyUnstarred: false }
},
})
+301
View File
@@ -0,0 +1,301 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction, internalMutation, internalQuery } from './_generated/server'
const DEFAULT_BATCH_SIZE = 200
const MAX_BATCH_SIZE = 1000
const DEFAULT_MAX_BATCHES = 5
const MAX_MAX_BATCHES = 50
const BACKFILL_STATE_KEY = 'default'
export const backfillSkillStatFieldsInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const { page, isDone, continueCursor } = await ctx.db
.query('skills')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
let patched = 0
for (const skill of page) {
const next = buildSkillStatPatch(skill)
if (!next) continue
await ctx.db.patch(skill._id, next)
patched += 1
}
return {
ok: true as const,
scanned: page.length,
patched,
cursor: isDone ? null : continueCursor,
isDone,
}
},
})
type BackfillState = {
cursor: string | null
doneAt?: number
}
type BackfillActionArgs = {
batchSize?: number
maxBatches?: number
resetCursor?: boolean
}
type BackfillStats = {
scanned: number
patched: number
batches: number
}
type BackfillActionResult = {
ok: true
isDone: boolean
cursor: string | null
stats: BackfillStats
}
export const getSkillStatBackfillStateInternal = internalQuery({
args: {},
handler: async (ctx): Promise<BackfillState> => {
const state = await ctx.db
.query('skillStatBackfillState')
.withIndex('by_key', (q) => q.eq('key', BACKFILL_STATE_KEY))
.unique()
return { cursor: state?.cursor ?? null, doneAt: state?.doneAt }
},
})
export const setSkillStatBackfillStateInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
doneAt: v.optional(v.number()),
},
handler: async (ctx, args) => {
const now = Date.now()
const state = await ctx.db
.query('skillStatBackfillState')
.withIndex('by_key', (q) => q.eq('key', BACKFILL_STATE_KEY))
.unique()
if (!state) {
await ctx.db.insert('skillStatBackfillState', {
key: BACKFILL_STATE_KEY,
cursor: args.cursor,
doneAt: args.doneAt,
updatedAt: now,
})
return { ok: true as const }
}
await ctx.db.patch(state._id, {
cursor: args.cursor,
doneAt: args.doneAt,
updatedAt: now,
})
return { ok: true as const }
},
})
async function runSkillStatBackfillInternalHandler(
ctx: ActionCtx,
args: BackfillActionArgs,
): Promise<BackfillActionResult> {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
if (args.resetCursor) {
await ctx.runMutation(internal.statsMaintenance.setSkillStatBackfillStateInternal, {
cursor: undefined,
doneAt: undefined,
})
}
const state = (await ctx.runQuery(
internal.statsMaintenance.getSkillStatBackfillStateInternal,
{},
)) as BackfillState
if (state.doneAt && !args.resetCursor) {
return {
ok: true,
isDone: true,
cursor: null,
stats: { scanned: 0, patched: 0, batches: 0 },
}
}
let cursor: string | null = state.cursor ?? null
const stats: BackfillStats = { scanned: 0, patched: 0, batches: 0 }
for (let i = 0; i < maxBatches; i += 1) {
const result = (await ctx.runMutation(
internal.statsMaintenance.backfillSkillStatFieldsInternal,
{
cursor: cursor ?? undefined,
batchSize,
},
)) as { scanned: number; patched: number; cursor: string | null; isDone: boolean }
stats.scanned += result.scanned
stats.patched += result.patched
stats.batches += 1
cursor = result.cursor
if (result.isDone) {
await ctx.runMutation(internal.statsMaintenance.setSkillStatBackfillStateInternal, {
cursor: undefined,
doneAt: Date.now(),
})
return { ok: true, isDone: true, cursor: null, stats }
}
await ctx.runMutation(internal.statsMaintenance.setSkillStatBackfillStateInternal, {
cursor: cursor ?? undefined,
doneAt: undefined,
})
}
return { ok: true, isDone: false, cursor, stats }
}
export const runSkillStatBackfillInternal: ReturnType<typeof internalAction> = internalAction({
args: {
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
resetCursor: v.optional(v.boolean()),
},
handler: runSkillStatBackfillInternalHandler,
})
function buildSkillStatPatch(skill: Doc<'skills'>) {
const stats = skill.stats
const nextDownloads = stats.downloads
const nextStars = stats.stars
const nextInstallsCurrent = stats.installsCurrent ?? 0
const nextInstallsAllTime = stats.installsAllTime ?? 0
if (
skill.statsDownloads === nextDownloads &&
skill.statsStars === nextStars &&
skill.statsInstallsCurrent === nextInstallsCurrent &&
skill.statsInstallsAllTime === nextInstallsAllTime
) {
return null
}
return {
statsDownloads: nextDownloads,
statsStars: nextStars,
statsInstallsCurrent: nextInstallsCurrent,
statsInstallsAllTime: nextInstallsAllTime,
}
}
/**
* Reconcile skill stats by counting actual records in source-of-truth tables.
*
* This fixes stats that got out of sync due to missed events, cursor issues,
* or bugs in the event processing pipeline. It counts:
* - stars: actual records in the `stars` table for each skill
* - comments: actual records in the `comments` table for each skill
*
* Downloads and installs are event-sourced only (no separate table to count from),
* so they cannot be reconciled this way.
*/
export const reconcileSkillStarCounts = internalMutation({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 50, 1, 200)
const now = Date.now()
const { page, isDone, continueCursor } = await ctx.db
.query('skills')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
let patched = 0
for (const skill of page) {
// Count actual star records for this skill
const starRecords = await ctx.db
.query('stars')
.withIndex('by_skill_user', (q) => q.eq('skillId', skill._id))
.collect()
const actualStars = starRecords.length
// Count actual comment records for this skill
const commentRecords = await ctx.db
.query('comments')
.withIndex('by_skill', (q) => q.eq('skillId', skill._id))
.collect()
const actualComments = commentRecords.filter((c) => !c.softDeletedAt).length
// Check if stats are out of sync
if (skill.stats.stars !== actualStars || skill.stats.comments !== actualComments) {
const updatedStats = {
...skill.stats,
stars: actualStars,
comments: actualComments,
}
await ctx.db.patch(skill._id, {
statsStars: actualStars,
stats: updatedStats,
updatedAt: now,
})
patched += 1
}
}
return {
scanned: page.length,
patched,
cursor: isDone ? null : continueCursor,
isDone,
}
},
})
export const runReconcileSkillStarCountsInternal = internalAction({
args: {
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
},
handler: async (ctx, args) => {
const batchSize = clampInt(args.batchSize ?? 50, 1, 200)
const maxBatches = clampInt(args.maxBatches ?? 10, 1, 50)
let cursor: string | undefined
let totalScanned = 0
let totalPatched = 0
for (let i = 0; i < maxBatches; i++) {
const result = (await ctx.runMutation(internal.statsMaintenance.reconcileSkillStarCounts, {
cursor,
batchSize,
})) as { scanned: number; patched: number; cursor: string | null; isDone: boolean }
totalScanned += result.scanned
totalPatched += result.patched
if (result.isDone) break
cursor = result.cursor ?? undefined
}
return { scanned: totalScanned, patched: totalPatched }
},
})
function clampInt(value: number, min: number, max: number) {
return Math.min(Math.max(value, min), max)
}
+28 -37
View File
@@ -4,6 +4,7 @@ import type { Id } from './_generated/dataModel'
import type { MutationCtx, QueryCtx } from './_generated/server'
import { internalMutation, mutation, query } from './_generated/server'
import { requireUser } from './lib/access'
import { insertStatEvent } from './skillStatEvents'
const TELEMETRY_STALE_MS = 120 * 24 * 60 * 60 * 1000
@@ -87,7 +88,13 @@ export const getMyInstalled = query({
lastSeenAt: number
expiredAt?: number
skills: Array<{
skill: { slug: string; displayName: string; summary?: string; stats: unknown }
skill: {
slug: string
displayName: string
summary?: string
stats: unknown
ownerUserId: Id<'users'>
}
firstSeenAt: number
lastSeenAt: number
lastVersion?: string
@@ -104,7 +111,13 @@ export const getMyInstalled = query({
const filtered = includeRemoved ? installs : installs.filter((entry) => !entry.removedAt)
const skills: Array<{
skill: { slug: string; displayName: string; summary?: string; stats: unknown }
skill: {
slug: string
displayName: string
summary?: string
stats: unknown
ownerUserId: Id<'users'>
}
firstSeenAt: number
lastSeenAt: number
lastVersion?: string
@@ -120,6 +133,7 @@ export const getMyInstalled = query({
displayName: skill.displayName,
summary: skill.summary,
stats: skill.stats,
ownerUserId: skill.ownerUserId,
},
firstSeenAt: entry.firstSeenAt,
lastSeenAt: entry.lastSeenAt,
@@ -157,24 +171,13 @@ async function clearTelemetryForUser(ctx: MutationCtx, params: { userId: Id<'use
await ctx.db.delete(entry._id)
continue
}
const stats = skill.stats as {
downloads: number
installsCurrent?: number
installsAllTime?: number
stars: number
versions: number
comments: number
}
await ctx.db.patch(skill._id, {
stats: {
...stats,
installsCurrent: Math.max(
0,
(stats.installsCurrent ?? 0) - (entry.activeRoots > 0 ? 1 : 0),
),
installsAllTime: Math.max(0, (stats.installsAllTime ?? 0) - 1),
await insertStatEvent(ctx, {
skillId: skill._id,
kind: 'install_clear',
delta: {
allTime: -1,
current: entry.activeRoots > 0 ? -1 : 0,
},
updatedAt: Date.now(),
})
await ctx.db.delete(entry._id)
}
@@ -381,25 +384,13 @@ async function bumpSkillInstallCounts(
ctx: MutationCtx,
params: { skillId: Id<'skills'>; deltaAllTime: number; deltaCurrent: number },
) {
const skill = await ctx.db.get(params.skillId)
if (!skill) return
const stats = skill.stats as {
downloads: number
installsCurrent?: number
installsAllTime?: number
stars: number
versions: number
comments: number
if (params.deltaAllTime === 1 && params.deltaCurrent === 1) {
await insertStatEvent(ctx, { skillId: params.skillId, kind: 'install_new' })
} else if (params.deltaAllTime === 0 && params.deltaCurrent === 1) {
await insertStatEvent(ctx, { skillId: params.skillId, kind: 'install_reactivate' })
} else if (params.deltaAllTime === 0 && params.deltaCurrent === -1) {
await insertStatEvent(ctx, { skillId: params.skillId, kind: 'install_deactivate' })
}
await ctx.db.patch(skill._id, {
stats: {
...stats,
installsAllTime: Math.max(0, (stats.installsAllTime ?? 0) + params.deltaAllTime),
installsCurrent: Math.max(0, (stats.installsCurrent ?? 0) + params.deltaCurrent),
},
updatedAt: Date.now(),
})
}
async function expireStaleRoots(
+21 -3
View File
@@ -1,7 +1,25 @@
{
"extends": "../tsconfig.json",
/* This TypeScript project config describes the environment that
* Convex functions run in and is used to typecheck them.
* You can modify it, but some settings are required to use Convex.
*/
"compilerOptions": {
/* These settings are not required by Convex and can be modified. */
"allowJs": true,
"strict": true,
"moduleResolution": "Bundler",
"skipLibCheck": true
}
"jsx": "react-jsx",
"skipLibCheck": true,
"allowSyntheticDefaultImports": true,
/* These compiler options are required by Convex */
"target": "ESNext",
"lib": ["ES2022", "dom", "dom.iterable"],
"forceConsistentCasingInFileNames": true,
"module": "ESNext",
"isolatedModules": true,
"noEmit": true
},
"include": ["./**/*"],
"exclude": ["./_generated"]
}
+1 -1
View File
@@ -14,7 +14,7 @@ export const generateUploadUrlForUserInternal = internalMutation({
args: { userId: v.id('users') },
handler: async (ctx, args) => {
const user = await ctx.db.get(args.userId)
if (!user || user.deletedAt) throw new Error('User not found')
if (!user || user.deletedAt || user.deactivatedAt) throw new Error('User not found')
return ctx.storage.generateUploadUrl()
},
})
+90
View File
@@ -0,0 +1,90 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
vi.mock('./lib/access', async () => {
const actual = await vi.importActual<typeof import('./lib/access')>('./lib/access')
return { ...actual, requireUser: vi.fn() }
})
const { requireUser } = await import('./lib/access')
const { ensureHandler } = await import('./users')
function makeCtx() {
const patch = vi.fn()
const get = vi.fn()
return { ctx: { db: { patch, get } } as never, patch, get }
}
describe('ensureHandler', () => {
afterEach(() => {
vi.mocked(requireUser).mockReset()
})
it('updates handle and display name when GitHub login changes', async () => {
const { ctx, patch } = makeCtx()
vi.mocked(requireUser).mockResolvedValue({
userId: 'users:1',
user: {
_creationTime: 1,
handle: 'old-handle',
displayName: 'old-handle',
name: 'new-handle',
email: 'old@example.com',
role: 'user',
createdAt: 1,
},
} as never)
await ensureHandler(ctx)
expect(patch).toHaveBeenCalledWith('users:1', {
handle: 'new-handle',
displayName: 'new-handle',
updatedAt: expect.any(Number),
})
})
it('does not override a custom display name when syncing handle', async () => {
const { ctx, patch } = makeCtx()
vi.mocked(requireUser).mockResolvedValue({
userId: 'users:2',
user: {
_creationTime: 1,
handle: 'old-handle',
displayName: 'Custom Name',
name: 'new-handle',
role: 'user',
createdAt: 1,
},
} as never)
await ensureHandler(ctx)
expect(patch).toHaveBeenCalledWith('users:2', {
handle: 'new-handle',
updatedAt: expect.any(Number),
})
})
it('fills display name from existing handle when missing', async () => {
const { ctx, patch } = makeCtx()
vi.mocked(requireUser).mockResolvedValue({
userId: 'users:3',
user: {
_creationTime: 1,
handle: 'steady-handle',
displayName: undefined,
name: undefined,
email: undefined,
role: 'user',
createdAt: 1,
},
} as never)
await ensureHandler(ctx)
expect(patch).toHaveBeenCalledWith('users:3', {
displayName: 'steady-handle',
updatedAt: expect.any(Number),
})
})
})
+423 -37
View File
@@ -1,52 +1,148 @@
import { getAuthUserId } from '@convex-dev/auth/server'
import { v } from 'convex/values'
import { internal } from './_generated/api'
import { mutation, query } from './_generated/server'
import { assertRole, requireUser } from './lib/access'
import type { Doc, Id } from './_generated/dataModel'
import type { MutationCtx } from './_generated/server'
import { internalMutation, internalQuery, mutation, query } from './_generated/server'
import { assertAdmin, assertModerator, requireUser } from './lib/access'
import { toPublicUser } from './lib/public'
import { buildUserSearchResults } from './lib/userSearch'
const DEFAULT_ROLE = 'user'
const ADMIN_HANDLE = 'steipete'
export const getById = query({
args: { userId: v.id('users') },
handler: async (ctx, args) => toPublicUser(await ctx.db.get(args.userId)),
})
export const getByIdInternal = internalQuery({
args: { userId: v.id('users') },
handler: async (ctx, args) => ctx.db.get(args.userId),
})
export const getGitHubProviderAccountIdInternal = internalQuery({
args: { userId: v.id('users') },
handler: async (ctx, args) => {
const account = await ctx.db
.query('authAccounts')
.withIndex('userIdAndProvider', (q) => q.eq('userId', args.userId).eq('provider', 'github'))
.unique()
return account?.providerAccountId ?? null
},
})
export const searchInternal = internalQuery({
args: {
actorUserId: v.id('users'),
query: v.optional(v.string()),
limit: v.optional(v.number()),
},
handler: async (ctx, args) => {
const actor = await ctx.db.get(args.actorUserId)
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error('Unauthorized')
assertAdmin(actor)
const limit = Math.min(Math.max(args.limit ?? 20, 1), 200)
const users = await ctx.db.query('users').order('desc').collect()
const result = buildUserSearchResults(users, args.query)
const items = result.items.slice(0, limit).map((user) => ({
userId: user._id,
handle: user.handle ?? null,
displayName: user.displayName ?? null,
name: user.name ?? null,
role: user.role ?? null,
}))
return { items, total: result.total }
},
})
export const updateGithubMetaInternal = internalMutation({
args: {
userId: v.id('users'),
githubCreatedAt: v.number(),
githubFetchedAt: v.number(),
},
handler: async (ctx, args) => {
await ctx.db.patch(args.userId, {
githubCreatedAt: args.githubCreatedAt,
githubFetchedAt: args.githubFetchedAt,
updatedAt: args.githubFetchedAt,
})
},
})
export const me = query({
args: {},
handler: async (ctx) => {
const userId = await getAuthUserId(ctx)
if (!userId) return null
const user = await ctx.db.get(userId)
if (!user || user.deletedAt) return null
if (!user || user.deletedAt || user.deactivatedAt) return null
return user
},
})
export const ensure = mutation({
args: {},
handler: async (ctx) => {
const { userId, user } = await requireUser(ctx)
const now = Date.now()
const updates: Record<string, unknown> = {}
const handle = user.handle ?? user.name ?? user.email?.split('@')[0]
if (!user.handle && handle) updates.handle = handle
if (!user.displayName) updates.displayName = handle
if (!user.role) {
updates.role = handle === ADMIN_HANDLE ? 'admin' : DEFAULT_ROLE
}
if (!user.createdAt) updates.createdAt = user._creationTime
updates.updatedAt = now
if (Object.keys(updates).length > 0) {
await ctx.db.patch(userId, updates)
}
return ctx.db.get(userId)
},
handler: ensureHandler,
})
function normalizeHandle(handle: string | undefined) {
const normalized = handle?.trim()
return normalized ? normalized : undefined
}
function deriveHandle(args: { existingHandle?: string; githubLogin?: string; email?: string }) {
// Prefer the GitHub login; only fall back to email-derived handle when we don't already have one.
if (args.githubLogin) return args.githubLogin
if (!args.existingHandle && args.email) return args.email.split('@')[0]?.trim() || undefined
return undefined
}
function computeEnsureUpdates(user: Doc<'users'>) {
const updates: Record<string, unknown> = {}
const existingHandle = normalizeHandle(user.handle)
const githubLogin = normalizeHandle(user.name)
const derivedHandle = deriveHandle({
existingHandle,
githubLogin,
email: user.email,
})
const baseHandle = derivedHandle ?? existingHandle
if (derivedHandle && existingHandle !== derivedHandle) {
updates.handle = derivedHandle
}
const displayName = normalizeHandle(user.displayName)
if (!displayName && baseHandle) {
updates.displayName = baseHandle
} else if (derivedHandle && displayName === existingHandle) {
updates.displayName = derivedHandle
}
if (!user.role) {
updates.role = baseHandle === ADMIN_HANDLE ? 'admin' : DEFAULT_ROLE
}
if (!user.createdAt) updates.createdAt = user._creationTime
return updates
}
export async function ensureHandler(ctx: MutationCtx) {
const { userId, user } = await requireUser(ctx)
const updates = computeEnsureUpdates(user)
if (Object.keys(updates).length > 0) {
updates.updatedAt = Date.now()
await ctx.db.patch(userId, updates)
}
return ctx.db.get(userId)
}
export const updateProfile = mutation({
args: {
displayName: v.string(),
@@ -66,31 +162,63 @@ export const deleteAccount = mutation({
args: {},
handler: async (ctx) => {
const { userId } = await requireUser(ctx)
const now = Date.now()
const tokens = await ctx.db
.query('apiTokens')
.withIndex('by_user', (q) => q.eq('userId', userId))
.collect()
for (const token of tokens) {
if (!token.revokedAt) {
await ctx.db.patch(token._id, { revokedAt: now })
}
}
await ctx.db.patch(userId, {
deletedAt: Date.now(),
updatedAt: Date.now(),
deactivatedAt: now,
purgedAt: now,
deletedAt: undefined,
banReason: undefined,
role: 'user',
handle: undefined,
displayName: undefined,
name: undefined,
image: undefined,
email: undefined,
emailVerificationTime: undefined,
phone: undefined,
phoneVerificationTime: undefined,
isAnonymous: undefined,
bio: undefined,
githubCreatedAt: undefined,
githubFetchedAt: undefined,
updatedAt: now,
})
await ctx.runMutation(internal.telemetry.clearUserTelemetryInternal, { userId })
},
})
export const list = query({
args: { limit: v.optional(v.number()) },
args: { limit: v.optional(v.number()), search: v.optional(v.string()) },
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
assertRole(user, ['admin'])
const limit = args.limit ?? 50
return ctx.db.query('users').order('desc').take(limit)
assertAdmin(user)
const limit = Math.min(Math.max(args.limit ?? 50, 1), 200)
const query = args.search?.trim().toLowerCase()
const users = await ctx.db.query('users').order('desc').collect()
const result = buildUserSearchResults(users, query)
return { items: result.items.slice(0, limit), total: result.total }
},
})
export const getByHandle = query({
args: { handle: v.string() },
handler: async (ctx, args) => {
return ctx.db
const user = await ctx.db
.query('users')
.withIndex('handle', (q) => q.eq('handle', args.handle))
.unique()
return toPublicUser(user)
},
})
@@ -101,15 +229,273 @@ export const setRole = mutation({
},
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
assertRole(user, ['admin'])
await ctx.db.patch(args.userId, { role: args.role, updatedAt: Date.now() })
return setRoleWithActor(ctx, user, args.userId, args.role)
},
})
export const setRoleInternal = internalMutation({
args: {
actorUserId: v.id('users'),
targetUserId: v.id('users'),
role: v.union(v.literal('admin'), v.literal('moderator'), v.literal('user')),
},
handler: async (ctx, args) => {
const actor = await ctx.db.get(args.actorUserId)
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error('User not found')
return setRoleWithActor(ctx, actor, args.targetUserId, args.role)
},
})
async function setRoleWithActor(
ctx: MutationCtx,
actor: Doc<'users'>,
targetUserId: Id<'users'>,
role: 'admin' | 'moderator' | 'user',
) {
assertAdmin(actor)
const target = await ctx.db.get(targetUserId)
if (!target) throw new Error('User not found')
const now = Date.now()
await ctx.db.patch(targetUserId, { role, updatedAt: now })
await ctx.db.insert('auditLogs', {
actorUserId: actor._id,
action: 'role.change',
targetType: 'user',
targetId: targetUserId,
metadata: { role },
createdAt: now,
})
return { ok: true as const, role }
}
export const banUser = mutation({
args: { userId: v.id('users'), reason: v.optional(v.string()) },
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
return banUserWithActor(ctx, user, args.userId, args.reason)
},
})
export const banUserInternal = internalMutation({
args: {
actorUserId: v.id('users'),
targetUserId: v.id('users'),
reason: v.optional(v.string()),
},
handler: async (ctx, args) => {
const actor = await ctx.db.get(args.actorUserId)
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error('User not found')
return banUserWithActor(ctx, actor, args.targetUserId, args.reason)
},
})
export const unbanUser = mutation({
args: { userId: v.id('users'), reason: v.optional(v.string()) },
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
return unbanUserWithActor(ctx, user, args.userId, args.reason)
},
})
export const unbanUserInternal = internalMutation({
args: {
actorUserId: v.id('users'),
targetUserId: v.id('users'),
reason: v.optional(v.string()),
},
handler: async (ctx, args) => {
const actor = await ctx.db.get(args.actorUserId)
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error('User not found')
return unbanUserWithActor(ctx, actor, args.targetUserId, args.reason)
},
})
async function banUserWithActor(
ctx: MutationCtx,
actor: Doc<'users'>,
targetUserId: Id<'users'>,
reasonRaw?: string,
) {
assertModerator(actor)
if (targetUserId === actor._id) throw new Error('Cannot ban yourself')
const target = await ctx.db.get(targetUserId)
if (!target) throw new Error('User not found')
if (target.role === 'admin' && actor.role !== 'admin') {
throw new Error('Forbidden')
}
const now = Date.now()
const reason = reasonRaw?.trim()
if (reason && reason.length > 500) {
throw new Error('Reason too long (max 500 chars)')
}
if (target.deletedAt || target.deactivatedAt) {
return { ok: true as const, alreadyBanned: true, deletedSkills: 0 }
}
const skills = await ctx.db
.query('skills')
.withIndex('by_owner', (q) => q.eq('ownerUserId', targetUserId))
.collect()
for (const skill of skills) {
await ctx.scheduler.runAfter(0, internal.skills.hardDeleteInternal, {
skillId: skill._id,
actorUserId: actor._id,
})
}
const tokens = await ctx.db
.query('apiTokens')
.withIndex('by_user', (q) => q.eq('userId', targetUserId))
.collect()
for (const token of tokens) {
await ctx.db.patch(token._id, { revokedAt: now })
}
await ctx.db.patch(targetUserId, {
deletedAt: now,
role: 'user',
updatedAt: now,
banReason: reason || undefined,
})
await ctx.runMutation(internal.telemetry.clearUserTelemetryInternal, { userId: targetUserId })
await ctx.db.insert('auditLogs', {
actorUserId: actor._id,
action: 'user.ban',
targetType: 'user',
targetId: targetUserId,
metadata: { deletedSkills: skills.length, reason: reason || undefined },
createdAt: now,
})
return { ok: true as const, alreadyBanned: false, deletedSkills: skills.length }
}
async function unbanUserWithActor(
ctx: MutationCtx,
actor: Doc<'users'>,
targetUserId: Id<'users'>,
reasonRaw?: string,
) {
assertAdmin(actor)
if (targetUserId === actor._id) throw new Error('Cannot unban yourself')
const target = await ctx.db.get(targetUserId)
if (!target) throw new Error('User not found')
if (target.deactivatedAt) {
throw new Error('Cannot unban a permanently deleted account')
}
if (!target.deletedAt) {
return { ok: true as const, alreadyUnbanned: true }
}
const reason = reasonRaw?.trim()
if (reason && reason.length > 500) {
throw new Error('Reason too long (max 500 chars)')
}
const now = Date.now()
await ctx.db.patch(targetUserId, {
deletedAt: undefined,
banReason: undefined,
role: 'user',
updatedAt: now,
})
await ctx.db.insert('auditLogs', {
actorUserId: actor._id,
action: 'user.unban',
targetType: 'user',
targetId: targetUserId,
metadata: { reason: reason || undefined },
createdAt: now,
})
return { ok: true as const, alreadyUnbanned: false }
}
/**
* Auto-ban a user whose skill was flagged malicious by VT.
* Skips moderators/admins. No actor required this is a system-level action.
*/
export const autobanMalwareAuthorInternal = internalMutation({
args: {
ownerUserId: v.id('users'),
sha256hash: v.string(),
slug: v.string(),
},
handler: async (ctx, args) => {
const target = await ctx.db.get(args.ownerUserId)
if (!target) return { ok: false, reason: 'user_not_found' }
if (target.deletedAt || target.deactivatedAt) return { ok: true, alreadyBanned: true }
// Never auto-ban moderators or admins
if (target.role === 'admin' || target.role === 'moderator') {
console.log(`[autoban] Skipping ${target.handle ?? args.ownerUserId}: role=${target.role}`)
return { ok: false, reason: 'protected_role' }
}
const now = Date.now()
// Soft-delete all their skills
const skills = await ctx.db
.query('skills')
.withIndex('by_owner', (q) => q.eq('ownerUserId', args.ownerUserId))
.collect()
for (const skill of skills) {
if (!skill.softDeletedAt) {
await ctx.db.patch(skill._id, { softDeletedAt: now, updatedAt: now })
}
}
// Revoke all API tokens
const tokens = await ctx.db
.query('apiTokens')
.withIndex('by_user', (q) => q.eq('userId', args.ownerUserId))
.collect()
for (const token of tokens) {
if (!token.revokedAt) {
await ctx.db.patch(token._id, { revokedAt: now })
}
}
// Ban the user
await ctx.db.patch(args.ownerUserId, {
deletedAt: now,
role: 'user',
updatedAt: now,
banReason: 'malware auto-ban',
})
await ctx.runMutation(internal.telemetry.clearUserTelemetryInternal, {
userId: args.ownerUserId,
})
// Audit log — use the target as actor since there's no human actor
await ctx.db.insert('auditLogs', {
actorUserId: user._id,
action: 'role.change',
actorUserId: args.ownerUserId,
action: 'user.autoban.malware',
targetType: 'user',
targetId: args.userId,
metadata: { role: args.role },
createdAt: Date.now(),
targetId: args.ownerUserId,
metadata: {
trigger: 'vt.malicious',
sha256hash: args.sha256hash,
slug: args.slug,
deletedSkills: skills.length,
},
createdAt: now,
})
console.warn(
`[autoban] Banned ${target.handle ?? args.ownerUserId} — malicious skill: ${args.slug}`,
)
return { ok: true, alreadyBanned: false, deletedSkills: skills.length }
},
})
+1262
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -11,7 +11,7 @@ export const sendDiscordWebhook = internalAction({
summary: v.optional(v.string()),
version: v.optional(v.string()),
ownerHandle: v.optional(v.string()),
batch: v.optional(v.string()),
highlighted: v.optional(v.boolean()),
tags: v.optional(v.array(v.string())),
}),
},
@@ -21,7 +21,7 @@ export const sendDiscordWebhook = internalAction({
event: args.event,
slug: args.skill.slug,
version: args.skill.version ?? null,
batch: args.skill.batch ?? null,
highlighted: args.skill.highlighted ?? false,
highlightedOnly: config.highlightedOnly,
}
if (!shouldSendWebhook(args.event, args.skill, config)) {
+2 -1
View File
@@ -22,7 +22,8 @@ Reading order (new contributor):
Feature/ops docs (already present):
- `docs/spec.md`: product + implementation spec (data model + flows).
- `docs/telemetry.md`: what `clawdhub sync` reports; opt-out.
- `docs/security.md`: moderation, reporting, bans, upload gating.
- `docs/telemetry.md`: what `clawhub sync` reports; opt-out.
- `docs/webhook.md`: Discord webhook events/payload.
- `docs/diffing.md`: version-to-version diff UI spec.
- `docs/manual-testing.md`: CLI smoke scripts.
+3 -2
View File
@@ -7,7 +7,7 @@ read_when:
# API v1
Base: `https://clawdhub.com`
Base: `https://clawhub.ai`
OpenAPI: `/api/v1/openapi.json`
@@ -30,7 +30,8 @@ Headers: `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Ret
Public read:
- `GET /api/v1/search?q=...`
- `GET /api/v1/skills?limit=&cursor=`
- `GET /api/v1/skills?limit=&cursor=&sort=`
- `sort`: `updated` (default), `downloads`, `stars` (`rating`), `installsCurrent` (`installs`), `installsAllTime`, `trending`
- `GET /api/v1/skills/{slug}`
- `GET /api/v1/skills/{slug}/versions?limit=&cursor=`
- `GET /api/v1/skills/{slug}/versions/{version}`
+4 -4
View File
@@ -11,8 +11,8 @@ read_when:
- Web app: TanStack Start (React) under `src/`.
- Backend: Convex under `convex/` (DB, storage, actions, HTTP routes).
- CLI: `packages/clawdhub/` (published as `clawdhub`).
- Shared schemas/routes: `packages/schema/` (`clawdhub-schema`).
- CLI: `packages/clawdhub/` (published as `clawhub`, legacy `clawdhub`).
- Shared schemas/routes: `packages/schema/` (`clawhub-schema`).
## Data + storage
@@ -38,8 +38,8 @@ read_when:
- Download zip via `/api/v1/download?slug=...&version=...`.
- Extract into `./skills/<slug>` (default).
- Persist install state:
- `./.clawdhub/lock.json` (per workdir)
- `./skills/<slug>/.clawdhub/origin.json` (per skill folder)
- `./.clawhub/lock.json` (per workdir, legacy `.clawdhub`)
- `./skills/<slug>/.clawhub/origin.json` (per skill folder, legacy `.clawdhub`)
### Update (CLI)
+4 -4
View File
@@ -23,7 +23,7 @@ The CLI uses a long-lived API token (Bearer token) for publish/sync/delete.
### Browser flow (default)
`clawdhub login` does:
`clawhub login` does:
1. Starts a loopback HTTP server on `127.0.0.1` (random port).
2. Opens `<site>/cli/auth?redirect_uri=http://127.0.0.1:<port>/callback&state=...`.
@@ -35,18 +35,18 @@ The CLI uses a long-lived API token (Bearer token) for publish/sync/delete.
Create a token in the web UI (Settings → API tokens) and paste it:
```bash
clawdhub login --token clh_...
clawhub login --token clh_...
```
### Token storage
Default global config path:
- macOS: `~/Library/Application Support/clawdhub/config.json`
- macOS: `~/Library/Application Support/clawhub/config.json`
Override:
- `CLAWDHUB_CONFIG_PATH=/path/to/config.json`
- `CLAWHUB_CONFIG_PATH=/path/to/config.json` (legacy `CLAWDHUB_CONFIG_PATH`)
### Revocation
+80 -15
View File
@@ -7,62 +7,88 @@ read_when:
# CLI
CLI package: `packages/clawdhub/` (bin: `clawdhub`).
CLI package: `packages/clawdhub/` (published as `clawhub`, bin: `clawhub`).
From this repo you can run it via the wrapper script:
```bash
bun clawdhub --help
bun clawhub --help
```
## Global flags
- `--workdir <dir>`: working directory (default: cwd; falls back to Clawdbot workspace if configured)
- `--dir <dir>`: install dir under workdir (default: `skills`)
- `--site <url>`: base URL for browser login (default: `https://clawdhub.com`)
- `--registry <url>`: API base URL (default: discovered, else `https://clawdhub.com`)
- `--site <url>`: base URL for browser login (default: `https://clawhub.ai`)
- `--registry <url>`: API base URL (default: discovered, else `https://clawhub.ai`)
- `--no-input`: disable prompts
Env equivalents:
- `CLAWDHUB_SITE`
- `CLAWDHUB_REGISTRY`
- `CLAWDHUB_WORKDIR`
- `CLAWHUB_SITE` (legacy `CLAWDHUB_SITE`)
- `CLAWHUB_REGISTRY` (legacy `CLAWDHUB_REGISTRY`)
- `CLAWHUB_WORKDIR` (legacy `CLAWDHUB_WORKDIR`)
## Config file
Stores your API token + cached registry URL.
- macOS: `~/Library/Application Support/clawdhub/config.json`
- override: `CLAWDHUB_CONFIG_PATH`
- macOS: `~/Library/Application Support/clawhub/config.json`
- override: `CLAWHUB_CONFIG_PATH` (legacy `CLAWDHUB_CONFIG_PATH`)
## Commands
### `login` / `auth login`
- Default: opens browser to `<site>/cli/auth` and completes via loopback callback.
- Headless: `clawdhub login --token clh_...`
- Headless: `clawhub login --token clh_...`
### `whoami`
- Verifies the stored token via `/api/v1/whoami`.
### `star <slug>` / `unstar <slug>`
- Adds/removes a skill from your highlights.
- Calls `POST /api/v1/stars/<slug>` and `DELETE /api/v1/stars/<slug>`.
- `--yes` skips confirmation.
### `search <query...>`
- Calls `/api/v1/search?q=...`.
### `explore`
- Lists latest updated skills via `/api/v1/skills?limit=...` (sorted by `updatedAt` desc).
- Flags:
- `--limit <n>` (1-200, default: 25)
- `--sort newest|downloads|rating|installs|installsAllTime|trending` (default: newest)
- `--json` (machine-readable output)
- Output: `<slug> v<version> <age> <summary>` (summary truncated to 50 chars).
### `inspect <slug>`
- Fetches skill metadata and version files without installing.
- `--version <version>`: inspect a specific version (default: latest).
- `--tag <tag>`: inspect a tagged version (e.g. `latest`).
- `--versions`: list version history (first page).
- `--limit <n>`: max versions to list (1-200).
- `--files`: list files for the selected version.
- `--file <path>`: fetch raw file content (text files only; 200KB limit).
- `--json`: machine-readable output.
### `install <slug>`
- Resolves latest version via `/api/v1/skills/<slug>`.
- Downloads zip via `/api/v1/download`.
- Extracts into `<workdir>/<dir>/<slug>`.
- Writes:
- `<workdir>/.clawdhub/lock.json`
- `<skill>/.clawdhub/origin.json`
- `<workdir>/.clawhub/lock.json` (legacy `.clawdhub`)
- `<skill>/.clawhub/origin.json` (legacy `.clawdhub`)
### `list`
- Reads `<workdir>/.clawdhub/lock.json`.
- Reads `<workdir>/.clawhub/lock.json` (legacy `.clawdhub`).
### `update [slug]` / `update --all`
@@ -77,6 +103,45 @@ Stores your API token + cached registry URL.
- Publishes via `POST /api/v1/skills` (multipart).
- Requires semver: `--version 1.2.3`.
### `delete <slug>`
- Soft-delete a skill (moderator/admin only).
- Calls `DELETE /api/v1/skills/{slug}`.
- `--yes` skips confirmation.
### `undelete <slug>`
- Restore a hidden skill (moderator/admin only).
- Calls `POST /api/v1/skills/{slug}/undelete`.
- `--yes` skips confirmation.
### `hide <slug>`
- Hide a skill (moderator/admin only).
- Alias for `delete`.
### `unhide <slug>`
- Unhide a skill (moderator/admin only).
- Alias for `undelete`.
### `ban-user <handleOrId>`
- Ban a user and delete owned skills (moderator/admin only).
- Calls `POST /api/v1/users/ban`.
- `--id` treats the argument as a user id instead of a handle.
- `--fuzzy` resolves the handle via fuzzy user search (admin only).
- `--reason` records an optional ban reason.
- `--yes` skips confirmation.
### `set-role <handleOrId> <role>`
- Change a user role (admin only).
- Calls `POST /api/v1/users/role`.
- `--id` treats the argument as a user id instead of a handle.
- `--fuzzy` resolves the handle via fuzzy user search (admin only).
- `--yes` skips confirmation.
### `sync`
- Scans for local skill folders and publishes new/changed ones.
@@ -86,7 +151,7 @@ Stores your API token + cached registry URL.
- `routing.agents.*.workspace/skills` (per-agent)
- `~/.clawdbot/skills` (shared)
- `skills.load.extraDirs` (shared packs)
- Respects `CLAWDBOT_CONFIG_PATH` and `CLAWDBOT_STATE_DIR`.
- Respects `CLAWDBOT_CONFIG_PATH` / `CLAWDBOT_STATE_DIR` and `OPENCLAW_CONFIG_PATH` / `OPENCLAW_STATE_DIR`.
- Flags:
- `--root <dir...>` extra scan roots
- `--all` upload without prompting
@@ -98,5 +163,5 @@ Stores your API token + cached registry URL.
Telemetry:
- Sent during `sync` when logged in, unless `CLAWDHUB_DISABLE_TELEMETRY=1`.
- Sent during `sync` when logged in, unless `CLAWHUB_DISABLE_TELEMETRY=1` (legacy `CLAWDHUB_DISABLE_TELEMETRY=1`).
- Details: `docs/telemetry.md`.
+7 -5
View File
@@ -7,7 +7,7 @@ read_when:
# Deploy
ClawdHub is two deployables:
ClawHub is two deployables:
- Web app (TanStack Start) → typically Vercel.
- Convex backend → Convex deployment (serves `/api/...` routes).
@@ -30,6 +30,7 @@ Ensure Convex env is set (auth + embeddings):
- `OPENAI_API_KEY`
- `SITE_URL` (your web app URL)
- Optional webhook env (see `docs/webhook.md`)
- Optional: `GITHUB_TOKEN` (recommended; raises GitHub account lookup limit used by publish gate)
## 2) Deploy web app (Vercel)
@@ -55,12 +56,13 @@ For self-host:
The CLI can discover the API base from:
- `/.well-known/clawdhub.json`
- `/.well-known/clawhub.json` (preferred)
- `/.well-known/clawdhub.json` (legacy)
If you dont serve that file, users must set:
```bash
export CLAWDHUB_REGISTRY=https://your-site.example
export CLAWHUB_REGISTRY=https://your-site.example
```
## 5) Post-deploy checks
@@ -73,6 +75,6 @@ curl -i "https://<site>/api/v1/skills/gifgrep"
Then:
```bash
clawdhub login --site https://<site>
clawdhub whoami
clawhub login --site https://<site>
clawhub whoami
```
+2 -2
View File
@@ -10,7 +10,7 @@ read_when:
## Goals
- Compare any file between two versions.
- Default compare: `latest` vs `previous` (SemVer precedence).
- UX feels native to ClawdHub (theme + typography + motion).
- UX feels native to ClawHub (theme + typography + motion).
- Inline or side-by-side toggle.
- Public access.
@@ -57,7 +57,7 @@ Optional helper action:
- Feed into Monaco diff editor.
## Monaco theming
- Define `clawdhub-light` / `clawdhub-dark` via `monaco.editor.defineTheme`.
- Define `clawhub-light` / `clawhub-dark` via `monaco.editor.defineTheme`.
- Derive colors from CSS variables on `document.documentElement`:
- `--surface`, `--surface-muted`, `--ink`, `--ink-soft`, `--line`, `--accent`.
- Apply theme on load + when theme changes (`data-theme`).
+110 -8
View File
@@ -7,7 +7,7 @@ read_when:
# HTTP API
Base URL: `https://clawdhub.com` (default).
Base URL: `https://clawhub.ai` (default).
All v1 paths are under `/api/v1/...` and implemented by Convex HTTP routes (`convex/http.ts`).
Legacy `/api/...` and `/api/cli/...` remain for compatibility (see `DEPRECATIONS.md`).
@@ -19,11 +19,17 @@ Enforced per IP + per API key:
- Read: 120/min per IP, 600/min per key
- Write: 30/min per IP, 120/min per key
- Download: 20/min per IP, 120/min per key (`/api/v1/download`)
Headers:
- `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After` (when limited)
IP source:
- Uses `cf-connecting-ip` first, then falls back to `x-real-ip`, `x-forwarded-for`, or `fly-client-ip`.
- Set `TRUST_FORWARDED_IPS=false` to disable forwarded-header fallback.
## Public endpoints (no auth)
### `GET /api/v1/search`
@@ -40,12 +46,21 @@ Response:
{ "results": [{ "score": 0.123, "slug": "gifgrep", "displayName": "GifGrep", "summary": "…", "version": "1.2.3", "updatedAt": 1730000000000 }] }
```
Notes:
- Results are returned in relevance order (embedding similarity + exact slug/name token boosts + popularity prior from downloads).
### `GET /api/v1/skills`
Query params:
- `limit` (optional): integer
- `cursor` (optional): pagination cursor
- `limit` (optional): integer (1200)
- `cursor` (optional): pagination cursor (only for `sort=updated`)
- `sort` (optional): `updated` (default), `downloads`, `stars` (alias: `rating`), `installsCurrent` (alias: `installs`), `installsAllTime`, `trending`
Notes:
- `trending` ranks by installs in the last 7 days (telemetry-based).
Response:
@@ -116,6 +131,7 @@ Notes:
- If neither `version` nor `tag` is provided, the latest version is used.
- Soft-deleted versions return `410`.
- Download stats are counted as unique identities per hour (`userId` when API token is valid, otherwise IP).
## Auth endpoints (Bearer token)
@@ -138,7 +154,92 @@ Publishes a new version.
### `DELETE /api/v1/skills/{slug}` / `POST /api/v1/skills/{slug}/undelete`
Soft-delete / restore a skill (owner/admin only).
Soft-delete / restore a skill (moderator/admin only).
### `POST /api/v1/users/ban`
Ban a user and hard-delete owned skills (moderator/admin only).
Body:
```json
{ "handle": "user_handle", "reason": "optional ban reason" }
```
or
```json
{ "userId": "users_...", "reason": "optional ban reason" }
```
Response:
```json
{ "ok": true, "alreadyBanned": false, "deletedSkills": 3 }
```
### `POST /api/v1/users/role`
Change a user role (admin only).
Body:
```json
{ "handle": "user_handle", "role": "moderator" }
```
or
```json
{ "userId": "users_...", "role": "admin" }
```
Response:
```json
{ "ok": true, "role": "moderator" }
```
### `GET /api/v1/users`
List or search users (admin only).
Query params:
- `q` (optional): search query
- `query` (optional): alias for `q`
- `limit` (optional): max results (default 20, max 200)
Response:
```json
{
"items": [
{
"userId": "users_...",
"handle": "user_handle",
"displayName": "User",
"name": "User",
"role": "moderator"
}
],
"total": 1
}
```
### `POST /api/v1/stars/{slug}` / `DELETE /api/v1/stars/{slug}`
Add/remove a star (highlights). Both endpoints are idempotent.
Responses:
```json
{ "ok": true, "starred": true, "alreadyStarred": false }
```
```json
{ "ok": true, "unstarred": true, "alreadyUnstarred": false }
```
## Legacy CLI endpoints (deprecated)
@@ -153,16 +254,17 @@ Still supported for older CLI versions:
See `DEPRECATIONS.md` for removal plan.
## Registry discovery (`/.well-known/clawdhub.json`)
## Registry discovery (`/.well-known/clawhub.json`)
The CLI can discover registry/auth settings from the site:
- `/.well-known/clawdhub.json` (JSON)
- `/.well-known/clawhub.json` (JSON, preferred)
- `/.well-known/clawdhub.json` (legacy)
Schema:
```json
{ "apiBase": "https://clawdhub.com", "authBase": "https://clawdhub.com", "minCliVersion": "0.0.5" }
{ "apiBase": "https://clawhub.ai", "authBase": "https://clawhub.ai", "minCliVersion": "0.0.5" }
```
If you self-host, serve this file (or set `CLAWDHUB_REGISTRY` explicitly).
If you self-host, serve this file (or set `CLAWHUB_REGISTRY` explicitly; legacy `CLAWDHUB_REGISTRY`).

Some files were not shown because too many files have changed in this diff Show More