Compare commits

...
Author SHA1 Message Date
Val AlexanderandClaude Opus 4.6 fec5db744c fix: clean up slot timers on unmount, fix about-grid specificity
Add useEffect cleanup to clear slot machine timers/intervals when
the home route unmounts mid-animation. Fix about-grid media query
specificity by including .about-panel-categories .about-grid to
override the higher-specificity base rule.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 16:04:53 -05:00
Val AlexanderandClaude Opus 4.6 cb75011244 feat: tune slot machine odds to 1/25 any jackpot, 1/100 Hack jackpot
Replace pure random picks with controlled probability: 4% chance of
any jackpot per spin, with 25% of jackpots being Hack (= 1% overall).
Non-jackpot spins re-roll accidental triple matches to keep odds exact.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 16:02:25 -05:00
Val AlexanderandClaude Opus 4.6 18b23886ab feat: add cooldown, longer celebration, and Hack x3 lobster jackpot
- 18s cooldown after a win, 3s after a loss to prevent spam
- Win celebration extended to 10s for screenshot opportunities
- Hack x3 jackpot triggers aquatic theme: cyan/teal text glow,
  ocean-colored confetti with bubble and claw particles, and the
  lobster logo fades in behind the headline

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 15:54:14 -05:00
copilot-swe-agent[bot]andBunsDev beb50f4b49 Merge remote-tracking branch 'origin/okcode/about-page-refresh' into okcode/about-page-refresh
Co-authored-by: BunsDev <68980965+BunsDev@users.noreply.github.com>
2026-04-18 20:53:19 +00:00
copilot-swe-agent[bot]andBunsDev 623a414b9b Merge remote-tracking branch 'origin/staging' into okcode/about-page-refresh
# Conflicts:
#	src/routes/about.tsx
#	src/styles.css

Co-authored-by: BunsDev <68980965+BunsDev@users.noreply.github.com>
2026-04-18 20:49:49 +00:00
Val AlexanderandClaude Opus 4.6 91c7e44031 feat: slot machine Easter egg on hero label triple-click
Triple-clicking "BUILT BY THE COMMUNITY" triggers a casino-style slot
machine across all 3 headline words. Reels spin and stop sequentially
with a 1/13 jackpot chance. Winning fires a confetti celebration with
golden text glow. Auto-resets after the animation completes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 15:47:09 -05:00
Val AlexanderandClaude Opus 4.6 850fc9c78e fix: use :is(h2, h3) selector for about-rule-card headings
The /souls page reuses about-rule-card with <h2> elements. Using
:is(h2, h3) ensures both heading levels get styled consistently.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 15:40:43 -05:00
Val AlexanderandClaude Opus 4.6 f5ec0882d5 style: widen navbar search bar and polish hero section
Let the search bar span the full width between brand and theme toggle
by removing the oversized right-column minimum and theme-toggle min-width.
Widen the hero search container, subtitle, and tighten vertical padding
for a sleeker feel.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 15:40:08 -05:00
Val AlexanderandClaude Opus 4.6 b2e751c5fc consolidate logo assets: delete SVGs, use only PNGs with transparent bg
- Delete public/logo.svg, public/og.svg, src/logo.svg (dead/unused SVGs)
- Remove logo.svg favicon link from __root.tsx (favicon.ico remains)
- Remove white backgrounds from clawd-logo.png and clawd-mark.png
- Convert white strokes to dark (#1a0808), defringe antialiased edges
- Regenerate logo192.png, logo512.png, favicon.ico from clean sources
- Only canonical logo files are now clawd-logo.png and clawd-mark.png

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 15:13:34 -05:00
Val AlexanderandClaude Opus 4.6 4c8032bb97 fix: remove white backgrounds from all logo assets
- Remove white backgrounds from clawd-logo.png, clawd-mark.png,
  logo192.png, logo512.png — now transparent PNGs
- Convert white strokes to dark (#1a0808) in both PNGs and logo.svg
  so segments separate cleanly on any background
- Defringe antialiased edges to eliminate white halos
- Regenerate favicon.ico from transparent source
- Update manifest.json background_color from #ffffff to #0a0a0a

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 15:00:51 -05:00
Val AlexanderandClaude Opus 4.6 4c566268a9 fix: hide logo, clean up rejection categories layout (#1727)
* fix: hide logo, use ClawHub as home link, and clean up rejection categories grid

Comment out the brand logo image for now, rename "Immediate rejection
categories" to "Rejection Categories", remove the featured card variant,
and switch to an auto-fill grid so cards spread evenly at full width.
Add overflow: visible on the categories panel to prevent hover shadow
clipping.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: impose max page width on home page using --page-max (1536px)

Constrain .home-v2-main to max-width: var(--page-max) and center it
with margin-inline: auto. Extend the home page background color to the
full viewport via .app-shell:has(.home-v2-main) for both light and dark
themes so the background bleeds edge-to-edge beyond the content column.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove extra footer padding and ensure full-width nav/footer for boxed layout

Zero out the outer .site-footer padding and set background to transparent
on home-v2 pages so the app-shell background bleeds through edge-to-edge.
Remove the redundant light-mode footer background override (app-shell
background already covers it). Nav and footer now visually span full
viewport width while .home-v2-main content stays boxed at --page-max.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: reduce carousel card hover effect and increase track padding

The carousel cards were getting clipped by the parent overflow:hidden
container. Reduce the hover transform from translateY(-4px) scale(1.01)
to translateY(-2px) and shrink box-shadow spread across all theme
variants. Increase carousel track top padding from 4px to 12px to
accommodate the upward shift without cutoff.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address review feedback — mobile brand, category grid, hover drama

- Keep brand name visible on mobile (remove display:none for
  .brand-name-responsive at ≤639px) so the home link is always
  discoverable. Add TODO comment on the commented-out logo block.
- Add .about-panel-categories .about-grid to the ≤640px media query
  so the category grid correctly collapses to single-column on mobile.
- Bump carousel card hover to translateY(-3px) with 0 6px 24px shadow
  for a slightly more dramatic lift — still within the 12px top / 48px
  bottom track padding so nothing clips.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove unused footer divider element

The site-footer-divider was already hidden via CSS (display: none) on
home-v2 pages. Remove the element entirely since it serves no purpose.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 14:37:43 -05:00
Val AlexanderandClaude Opus 4.6 2c435a3d7b fix: use h3 for card titles to fix heading hierarchy (WCAG 1.3.1)
Change rejection category card titles from <h2> to <h3> since the parent
section already uses <h2> for "Immediate rejection categories". Updates
the matching CSS selector from .about-rule-card h2 to h3.

Also adds tests for renderWithInlineCode helper covering plain text,
single/multiple code spans, empty input, and code-only strings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 14:20:44 -05:00
Val AlexanderandClaude Opus 4.6 1f990cf66d improve about page rejection categories: add icons, fix grid, polish cards
- Add lucide-react icons to each rejection category card for visual scanning
- Fix unbalanced grid layout by removing featured card sizing, using clean 2/3-col grid
- Fix broken hover transitions (var(--transition-fast) was undefined outside reduced-motion)
- Add lift-on-hover effect and icon glow matching home page card patterns
- Render backtick-wrapped text as styled inline code elements
- Improve description text contrast from ~3.5:1 to ~4.8:1 (WCAG AA)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 14:04:41 -05:00
Val Alexander e54fc1939a fix: normalize nav and footer layout 2026-04-18 13:41:36 -05:00
copilot-swe-agent[bot]andBunsDev 8b87c31a99 Merge remote-tracking branch 'origin/main' into staging
# Conflicts:
#	src/routes/management.tsx
#	src/routes/settings.tsx

Co-authored-by: BunsDev <68980965+BunsDev@users.noreply.github.com>
2026-04-17 08:49:21 +00:00
Momo f7bc8b6349 fix(stats): fix skill stat field sync direction and reconcile logic (#1704)
Merged via squash.

Prepared head SHA: e814278382
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Reviewed-by: @momothemage
2026-04-16 20:04:13 +08:00
5b8f09167a fix(api): align inspect security snapshot with static scan moderation
Include static scan results in the skill version security snapshot so inspect/API responses reflect the same moderation-relevant signal already used elsewhere. Also add regression coverage for suspicious, malicious, and static-only scan combinations.

Co-authored-by: davida-ps <232346510+davida-ps@users.noreply.github.com>
Co-authored-by: Luke <92253590+ImLukeF@users.noreply.github.com>
2026-04-16 18:39:39 +10:00
hugh 17fbd13bc9 fix(cli): use explorer on Windows to preserve auth URL params
On Windows, opening auth URLs via `cmd /c start` can truncate query parameters because `&` is treated as a command separator. Use `explorer` instead so the browser opener gets the full URL without shell parsing, and cover the Windows spawn args in the CLI UI test.

Co-authored-by: hugh <1012760428@qq.com>
2026-04-16 14:00:03 +10:00
aab7dc9ba4 fix(upload): fall back to octet-stream for empty Content-Type
Handle browser uploads that provide an empty MIME type by falling back to `application/octet-stream` before sending the storage request.

Co-authored-by: Arthur Katcher <192321283+arthurkatcher@users.noreply.github.com>
Co-authored-by: Luke <92253590+ImLukeF@users.noreply.github.com>
2026-04-16 12:54:51 +10:00
ImLukeF dde8796790 feat: tag skills needing sensitive credentials 2026-04-14 20:09:24 +10:00
Val Alexander acc6d292de Home v2 styles: layout, theme & navbar tweaks
Add and refine styles for the Home V2 UI: introduce navbar search/home styles, motto and headline variants, section copy/eyebrow rules, discovery and categories layouts, and responsive grid stacking. Adjust hv2 color variables (text-secondary/tertiary) and move category border to the grid element; update spacing/alignment for carousel and section headers. Add light/dark theme overrides to improve navbar, tabs and search contrast and hover states. Misc minor typographic and spacing refinements for a more cohesive Home V2 appearance.
2026-04-14 00:27:04 -05:00
ImLukeF 2236ed7be1 feat: add org profile editing 2026-04-14 14:10:52 +10:00
ImLukeF f6fb7ccfc0 Revert "Reapply "feat: allow moderators to transfer skill publishers (#1663)""
This reverts commit b73758c7c8.
2026-04-14 13:39:49 +10:00
ImLukeF b73758c7c8 Reapply "feat: allow moderators to transfer skill publishers (#1663)"
This reverts commit fbc07c5617.
2026-04-14 13:38:30 +10:00
ImLukeF fbc07c5617 Revert "feat: allow moderators to transfer skill publishers (#1663)"
This reverts commit 80e5aec577.
2026-04-14 13:37:08 +10:00
Luke 80e5aec577 feat: allow moderators to transfer skill publishers (#1663) 2026-04-14 13:36:28 +10:00
Val AlexanderandNova f869b31ad6 fix: remove leftover theme-family UI remnants
- drop mobile theme-family section in header
- remove unused theme-family settings bindings

Co-authored-by: Nova <nova@openknot.ai>
2026-04-13 22:15:06 -05:00
Val AlexanderandNova 9a853f2fcc chore: update lockfile and favicon
- refresh bun.lock after dependency reinstall
- include favicon update

Co-authored-by: Nova <nova@openknot.ai>
2026-04-13 22:13:03 -05:00
Val AlexanderandNova b4a7540157 feat: homepage redesign + unified theme + UI polish
- Redesign homepage with hero, search, featured carousel, categories, proof bar, trending
- Add cream/peach/tan light mode palette with inset-shadow pattern (dark + light)
- Remove Hub theme — single Claw theme only (light/dark mode toggle remains)
- Semi-rounded radius system (Claw × Hub midpoint: 4/7/10px)
- Consistent button radius site-wide (--r-btn: 4px), zero makeshift buttons
- Add VITE_FEATURE_SOULS env flag (default: false) to gate Souls pages
- Hide Souls from nav, footer, and homepage categories
- Remove theme family toggle from Header + Settings
- Widen page max to screen-2xl (1536px)
- Slow featured carousel 15% (40s → 46s)

Co-authored-by: Nova <nova@openknot.ai>
2026-04-13 21:59:23 -05:00
Val Alexander 0ea1127a2b fix: refine header and about responsiveness (#1661) 2026-04-13 13:03:40 -05:00
Val Alexander aeab23a6d6 Fix dark-mode styling for skills filter chips (#1660)
- Add readable dark-surface and active-state colors to filter chips
- Cover the toolbar styling with a jsdom test
2026-04-13 13:01:37 -05:00
Val Alexander 15bc4440cc style: add claw red accents to hub theme 2026-04-13 10:44:51 -05:00
Val Alexander 11a20f5755 refactor: reduce marketplace themes to claw and hub 2026-04-13 10:38:36 -05:00
Val Alexander 05f8674628 refactor: simplify footer browse links and theme switching 2026-04-13 10:19:15 -05:00
Val Alexander d2b2252770 Merge branch 'main' into staging 2026-04-13 09:48:02 -05:00
Val Alexander a2387253ec style: give knot theme distinct component treatments 2026-04-13 04:22:39 -05:00
Val Alexander 411260767b style: make knot theme darker and purple 2026-04-13 04:18:42 -05:00
Val Alexander 731d0ce0c5 feat: restyle dash theme to match clawhub v2 2026-04-13 03:59:25 -05:00
Val Alexander 361f2affde feat: add clawhub theme families and tweakcn import 2026-04-13 03:19:17 -05:00
Val AlexanderClaude Sonnet 4.6ImLukeFSaurabh Jaindependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>MomopqdongJhollyloongYaovi
a17f7bb07e feat: modernize clawhub app store (#1655)
* build(deps-dev): bump vite in the npm_and_yarn group across 1 directory (#1561)

Bumps the npm_and_yarn group with 1 update in the / directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite).


Updates `vite` from 8.0.1 to 8.0.5
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.0.5/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 8.0.5
  dependency-type: direct:development
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: detect generated-source template injection in skill scans (#1597)

* fix: detect exposed resource identifiers in skill scans (#1598)

* fix: restore ci checks after lockfile drift

* refactor: address actionable review cleanup (#1601)

* fix: prevent starring soft-deleted skills and fix star count reconciliation (#1605)

* feat: Add support for Chinese Japanese and Korean(CJK) skills search (#1596)

Merged via squash.

Prepared head SHA: ab58f01be7
Co-authored-by: pq-dong <40668796+pq-dong@users.noreply.github.com>
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Reviewed-by: @momothemage

* docs: document CLI config paths across platforms (#1252)

* docs: document CLI config paths across platforms

* docs: clarify legacy config fallback

---------

Co-authored-by: ImLukeF <92253590+ImLukeF@users.noreply.github.com>

* fix: point plugin metadata help link to OpenClaw docs (#1399)

* fix: point plugin metadata help link to OpenClaw docs

* fix: open plugin metadata docs in a new tab

* fix(cli-auth): ensure fallback token renders before redirect on Windows/Chrome (#1486)

* fix(cli-auth): ensure fallback token renders before redirect on Windows/Chrome

React batches state updates, so setToken() and window.location.assign()
previously raced: the navigation could fire before React re-rendered the
fallback token UI. On Chrome/Windows this means a failed http:// redirect
(ERR_CONNECTION_REFUSED, HTTPS-first interference) would replace the page
with an error screen before the user ever saw the token.

Use flushSync() to render the token synchronously, then attempt
window.location.assign(). If the redirect fails the token and a "Retry
redirect to CLI" link are already painted on screen.

Fixes #1469

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: cover cli auth fallback redirect

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: ImLukeF <92253590+ImLukeF@users.noreply.github.com>

* fix: reduce souls browse overfetch (#1637)

* fix: improve admin user search coverage (#1466)

* fix admin user search coverage

* fix admin user search without full table scan

* feat: include stats in package detail API response

Expose package detail stats through the shared API contract and the app client.

This lands the original package detail stats work and folds in the follow-up cleanup to keep the response shape sourced from the shared schema instead of a hand-maintained app-local type.

Co-authored-by: Saurabh Jain <saurabhjain1592@gmail.com>

* test: cover package detail stats response

* fix: normalize misleading MIME types for text files

* feat: modernize clawhub app store

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Luke <92253590+ImLukeF@users.noreply.github.com>
Co-authored-by: Momo <35096042+momothemage@users.noreply.github.com>
Co-authored-by: pqdong <40668796+pq-dong@users.noreply.github.com>
Co-authored-by: Jholly <xiangjunkong90@gmail.com>
Co-authored-by: loong <46096863+robinspt@users.noreply.github.com>
Co-authored-by: Yaovi <dkpoga@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Saurabh Jain <saurabhjain1592@gmail.com>
2026-04-13 01:36:18 -05:00
ImLukeF 7980788ea6 fix: normalize misleading MIME types for text files 2026-04-13 15:54:42 +10:00
Val Alexander 2d03b827d3 fix: close plugins route loader return 2026-04-13 00:44:04 -05:00
Val Alexanderandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> 835094ea2c Update convex/seedDemo.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-04-13 00:35:16 -05:00
Val Alexandergreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>v0 <v0[bot]@users.noreply.github.com>
7bd7e4c99e Modern utility store (#1646)
* feat: redesign ClawHub marketplace with modern utility store theme

Update styles.css and index.tsx for new modern design

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* style: polish marketplace UI with modern design updates

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* fix: resolve 500 errors in TanStack Router loaders

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* fix: resolve SSR error bubbling in TanStack Router

Ensure loader errors don't escape SSR and hydrate correctly.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* feat: handle errors in Package API calls to prevent SSR and HMR errors

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* feat: update branding to OpenClaw's black, white, and red color scheme

Implement new color scheme across dark, light themes and interactive elements

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* feat: use Vite's native tsconfig paths

Replace deprecated plugin with native option and remove unused import.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* fix: add defensive checks in fetchPluginCatalog

Ensure proper handling of undefined and unexpected API responses.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* fix: add defensive checks to PluginsIndex for SSR errors

Handle undefined loader data in PluginsIndex component.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* refactor: optimize skill detail page layout

Redesign skill detail page to maximize space, remove sidebar, create metadata bar, and add responsive breakpoints.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* fix: add global overflow protection to detail pages

Add overflow prevention for text elements and links.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* feat: highlight parent tabs with activePathPrefixes

Add activePathPrefixes to NavItem and update navigation to highlight parent tabs.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* feat: add user preferences customization section

Add 'usePreferences' hook and new Switch component; enhance settings page with Customization section.

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* feat: optimize layout and create DESIGN.md

Fix orphan cards, enforce equal card heights, add branding accents, improve visual hierarchy, add responsive breakpoints, create design document

Co-authored-by: Val Alexander <68980965+BunsDev@users.noreply.github.com>

* Update src/routes/plugins/index.tsx

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update src/components/layout/Container.tsx

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: v0 <v0[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-04-12 23:23:08 -05:00
Val Alexander 1722a48055 fix: resolve review feedback on layout and auth state (#1652) 2026-04-12 23:21:11 -05:00
Val Alexander 29178898bb fix: resolve review feedback on layout and auth state (#1647) 2026-04-12 22:41:30 -05:00
ImLukeF b16861f422 test: cover package detail stats response 2026-04-12 22:18:01 +10:00
Saurabh Jain 6896e61fa1 feat: include stats in package detail API response
Expose package detail stats through the shared API contract and the app client.

This lands the original package detail stats work and folds in the follow-up cleanup to keep the response shape sourced from the shared schema instead of a hand-maintained app-local type.

Co-authored-by: Saurabh Jain <saurabhjain1592@gmail.com>
2026-04-12 21:59:38 +10:00
Luke cf137aa592 fix: improve admin user search coverage (#1466)
* fix admin user search coverage

* fix admin user search without full table scan
2026-04-12 21:18:19 +10:00
Luke a28d014d4f fix: reduce souls browse overfetch (#1637) 2026-04-12 21:04:12 +10:00
b2038fc931 fix(cli-auth): ensure fallback token renders before redirect on Windows/Chrome (#1486)
* fix(cli-auth): ensure fallback token renders before redirect on Windows/Chrome

React batches state updates, so setToken() and window.location.assign()
previously raced: the navigation could fire before React re-rendered the
fallback token UI. On Chrome/Windows this means a failed http:// redirect
(ERR_CONNECTION_REFUSED, HTTPS-first interference) would replace the page
with an error screen before the user ever saw the token.

Use flushSync() to render the token synchronously, then attempt
window.location.assign(). If the redirect fails the token and a "Retry
redirect to CLI" link are already painted on screen.

Fixes #1469

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: cover cli auth fallback redirect

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: ImLukeF <92253590+ImLukeF@users.noreply.github.com>
2026-04-12 19:58:11 +10:00
loong 4a72b543b2 fix: point plugin metadata help link to OpenClaw docs (#1399)
* fix: point plugin metadata help link to OpenClaw docs

* fix: open plugin metadata docs in a new tab
2026-04-12 18:05:57 +10:00
JhollyandImLukeF 54e99c8cc2 docs: document CLI config paths across platforms (#1252)
* docs: document CLI config paths across platforms

* docs: clarify legacy config fallback

---------

Co-authored-by: ImLukeF <92253590+ImLukeF@users.noreply.github.com>
2026-04-12 18:02:41 +10:00
pqdongandmomothemage 5826001795 feat: Add support for Chinese Japanese and Korean(CJK) skills search (#1596)
Merged via squash.

Prepared head SHA: ab58f01be7
Co-authored-by: pq-dong <40668796+pq-dong@users.noreply.github.com>
Co-authored-by: momothemage <35096042+momothemage@users.noreply.github.com>
Reviewed-by: @momothemage
2026-04-10 12:20:40 +08:00
Momo 0708a43fde fix: prevent starring soft-deleted skills and fix star count reconciliation (#1605) 2026-04-09 22:16:51 +08:00
Nimrod Gutman 9a45c371fc fix(ui): align browse page widths across tabs 2026-04-09 14:19:27 +03:00
Luke 311a123fbe refactor: address actionable review cleanup (#1601) 2026-04-09 20:00:25 +10:00
ImLukeF 59e93862ed fix: restore ci checks after lockfile drift 2026-04-09 19:45:51 +10:00
Luke 8fcd53f899 fix: detect exposed resource identifiers in skill scans (#1598) 2026-04-09 19:32:55 +10:00
Luke ba2c73e180 fix: detect generated-source template injection in skill scans (#1597) 2026-04-09 19:10:39 +10:00
Val Alexander 298cbdd6db fix: denormalize user hover stats 2026-04-08 15:16:15 -05:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> fa87dc3509 build(deps-dev): bump vite in the npm_and_yarn group across 1 directory (#1561)
Bumps the npm_and_yarn group with 1 update in the / directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite).


Updates `vite` from 8.0.1 to 8.0.5
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.0.5/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 8.0.5
  dependency-type: direct:development
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-07 08:21:53 -05:00
Val AlexanderandNova f636b31fca feat: design foundation sweep (#1570) thanks @BunsDev
Co-authored-by: Nova <nova@openknot.ai>
2026-04-07 07:48:20 -05:00
Val Alexander b255b5865f Merge pull request #1567 from openclaw/okcode/conflict-resolution-plan
feat: marketplace UI overhaul with main security fixes
2026-04-07 02:07:33 -05:00
Val AlexanderandClaude Opus 4.6 5003c1bec8 fix: address remaining PR review comments
- Re-throw non-rate-limit errors in plugin loader so route error
  boundary handles real failures instead of showing empty results
- Bump requestRef on query clear to invalidate in-flight searches
  and prevent stale results from repopulating
- Replace Promise.all with Promise.allSettled in unified search so
  one failing provider doesn't blank results from other sources
- Log unexpected errors in unified search catch block

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 02:04:19 -05:00
Val AlexanderandClaude Opus 4.6 a8a6242f87 fix: address PR review — reason guard, one-shot fetch, index scan
- Reject empty reason strings in setSoftDeleted calls + re-add .catch()
  for error feedback (both reported-skills and skill-tools sections)
- Replace useQuery with ConvexHttpClient.query() on /users public
  browse page per CLAUDE.md policy
- Add by_active_handle compound index on users table to avoid full
  table scan in queryUsersForPublicList

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 00:55:41 -05:00
Val AlexanderandClaude Opus 4.6 39c0fa2531 merge: integrate origin/main into feat/marketplace-ui-overhaul
Keep feature branch UI overhaul (custom CSS) while incorporating
security/stability fixes from main:
- setSoftDeleted now requires moderation reason (runtime-critical)
- moderationNotes displayed in skill detail when available
- Rate limit handling for plugin catalog
- Tailwind @theme block for auto-merged component compatibility
- Capability tag passthrough to SecurityScanResults
- ALL_CATEGORY_KEYWORDS export for skills browse model

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-07 00:28:17 -05:00
Val Alexander 9df6fe37c5 Merge pull request #1564 from openclaw/okcodes/fix-lightmode
Add sortable paginated data table components
2026-04-06 20:37:44 -05:00
Val Alexander d5d806516d Merge remote-tracking branch 'origin/main' into okcodes/fix-lightmode
# Conflicts:
#	src/routes/skills/-SkillsToolbar.tsx
2026-04-06 20:35:54 -05:00
Val Alexander 67c74e10ea Hide empty data table pagination
- Make pagination conditional on there being rows and pages
- Tighten props so manual pagination requires a page count
2026-04-06 20:29:09 -05:00
Val Alexander 3501af6bb4 update create-markdown packages to 2.0.1 2026-04-06 20:28:47 -05:00
Vincent Koc 051b1dafcd fix: audit fixes — warm colors, lint, ARIA labels
- Replace ~20 remaining warm hex colors in upload/form styles
  (#ffddc9, #9a3a24, #fff3ec, etc.) with monochrome equivalents
- Fix 2 lint errors: remove unused Link import (search.tsx),
  prefix unused parseDir with underscore (souls/index.tsx)
- Add aria-label to PluginListItem and UserListItem for
  screen reader identification
2026-04-06 22:58:12 +01:00
Vincent Koc 383844cacf chore: various component changes 2026-04-06 22:03:49 +01:00
Vincent Koc b7923edbfd Update .gitignore 2026-04-06 20:41:09 +01:00
Val Alexander 655c914c77 Restore dark-mode styling in skills toolbar
- Resolve merge markers in `-SkillsToolbar.tsx`
- Keep control surfaces readable in dark mode
- Refresh lockfile for dependency bumps
2026-04-06 13:31:33 -05:00
Val Alexander b15eeab93f Improve dark mode contrast in skills toolbar
- Apply dark-themed control surface styles to search, select, and view toggles
- Adjust filter chips and labels for better visibility in dark mode
2026-04-06 13:31:33 -05:00
Val Alexander a39f07427e Stop empty public page scans from reporting more pages
- Clear `hasMore` and `nextCursor` when a scan hits the budget but returns no items
- Prevent the client IntersectionObserver from looping on empty auto-load responses
2026-04-06 13:31:33 -05:00
Val Alexanderandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> 1a42207879 Update src/__tests__/package-detail-route.test.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-04-06 13:31:33 -05:00
Val Alexander 9bd3a63edf Equalize stat widths across listings
- Align skill and soul metric chips to fixed widths
- Rebalance skills table columns for better summary space
- Tighten spacing in soul list rows
2026-04-06 13:31:33 -05:00
Luke be4c51ed1b fix: improve auth flow and webkit compatibility (#1555)
* fix: improve auth flow, skill filters, and webkit compatibility

* test: mock auth actions in settings route

* fix: remove stale skills toolbar props

* fix: address skills filter and webkit review feedback

* test: avoid monaco lazy import in skill detail test

* fix: recompute other skills category filter
2026-04-06 13:31:33 -05:00
Val Alexander 8711de4441 Merge pull request #1549 from openclaw/okcode/equalize-widths
Bump create-markdown to 2.0.1 and stabilize package detail tests
2026-04-06 12:18:57 -05:00
Luke f4db0ee32b fix: improve auth flow and webkit compatibility (#1555)
* fix: improve auth flow, skill filters, and webkit compatibility

* test: mock auth actions in settings route

* fix: remove stale skills toolbar props

* fix: address skills filter and webkit review feedback

* test: avoid monaco lazy import in skill detail test

* fix: recompute other skills category filter
2026-04-06 21:14:47 +10:00
Val AlexanderandClaude Opus 4.6 f1cf715b89 Merge main into okcode/equalize-widths
Resolve conflicts keeping our fixes:
- convex/skills.ts: retain empty-page hasMore guard for tag filter
- package-detail-route.test.tsx: retain widened MarkdownPreview mock type

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 22:38:52 -05:00
Val Alexander 57ed1f62a4 Add TanStack data table pagination and sorting
- Introduce reusable data table, column header, and pagination components
- Add TanStack Table dependency for sortable, paginated table UIs
2026-04-05 21:44:03 -05:00
Val Alexander bde7529fbe Widen the About page container
- Switch the About page from `narrow` to `wide` layout
- Align page width with the rest of the app
2026-04-05 21:44:03 -05:00
Val Alexander 5892718a23 Add capability tag filtering to skills browse
- Thread capability tags through skills search and public listing
- Add toolbar tag picker and preserve tag state in routing
- Cover tag-filtered search and browse query behavior with tests
2026-04-05 21:44:03 -05:00
Val Alexander 922eecdbd9 Stop empty public page scans from reporting more pages
- Clear `hasMore` and `nextCursor` when a scan hits the budget but returns no items
- Prevent the client IntersectionObserver from looping on empty auto-load responses
2026-04-05 20:57:34 -05:00
Val Alexanderandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> e53a4433ae Update src/__tests__/package-detail-route.test.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-04-05 20:36:35 -05:00
Val Alexander 68870a1dcc Equalize stat widths across listings
- Align skill and soul metric chips to fixed widths
- Rebalance skills table columns for better summary space
- Tighten spacing in soul list rows
2026-04-05 20:26:51 -05:00
Val Alexander c6aaf27886 Widen the About page container
- Switch the About page from `narrow` to `wide` layout
- Align page width with the rest of the app
2026-04-05 20:26:51 -05:00
Val Alexander f71db92fc3 Add capability tag filtering to skills browse
- Thread capability tags through skills search and public listing
- Add toolbar tag picker and preserve tag state in routing
- Cover tag-filtered search and browse query behavior with tests
2026-04-05 20:26:51 -05:00
Val Alexander 07a1da4285 Merge pull request #1548 from openclaw/okcode/align-about-page-width
Add capability tag filtering and widen About page
2026-04-05 20:01:20 -05:00
Val Alexander cb9d854e92 Widen the About page container
- Switch the About page from `narrow` to `wide` layout
- Align page width with the rest of the app
2026-04-05 19:54:55 -05:00
Val Alexander 74ec5191e0 Mock MarkdownPreview in package detail route tests
- Add a lightweight MarkdownPreview mock for the package detail route test
- Keep the test focused on route behavior instead of markdown rendering
2026-04-05 19:54:50 -05:00
Val Alexander cf98936105 update create-markdown packages to 2.0.1 2026-04-05 19:54:50 -05:00
Val Alexander dd9c42be28 Merge pull request #1546 from openclaw/okcode/tags-filter-update
Add capability tag filtering to skills browse
2026-04-05 19:35:24 -05:00
Val Alexander e8a4e094e7 Add capability tag filtering to skills browse
- Thread capability tags through skills search and public listing
- Add toolbar tag picker and preserve tag state in routing
- Cover tag-filtered search and browse query behavior with tests
2026-04-05 19:31:54 -05:00
Val Alexander 068dd78a05 Mock MarkdownPreview in package detail route tests
- Add a lightweight MarkdownPreview mock for the package detail route test
- Keep the test focused on route behavior instead of markdown rendering
2026-04-05 18:44:04 -05:00
Val Alexander 2f5e17f56e update create-markdown packages to 2.0.1 2026-04-05 17:50:45 -05:00
Val Alexander 5302bf8598 Merge pull request #1537 from openclaw/okcode/fix-rate-limit-errors
Handle public package API rate limits gracefully
2026-04-05 06:44:32 -05:00
Val Alexander 749c89e77b Move local hook install to an explicit script
- Add `install:local-hooks` for manual hook setup
- Remove the automatic `prepare` hook install
2026-04-05 06:42:28 -05:00
Val Alexander fd08d74071 Handle missing push base SHA in secret scan workflow
- Resolve scan range explicitly for pull_request and push events
- Fall back to the default branch when GitHub provides a zero or empty base SHA
2026-04-05 06:39:31 -05:00
Val Alexander 67c8e188e3 Fix rate-limit retry messaging
- share retry delay formatting for plugin pages
- handle secret scan diffs in push and PR workflows
2026-04-05 06:26:42 -05:00
Val Alexander bad07ff95c Improve plugin layout responsiveness
- tighten container padding on small screens
- make plugin install and filter controls stack cleanly
- reflow detail sections and metadata for mobile
2026-04-05 06:25:04 -05:00
Val Alexander a7547eee51 Handle package API rate limits gracefully
- Surface retryable empty states for plugin catalog and detail pages
- Preserve Retry-After metadata in package API errors
- Add staged-secret scanning and auto-installed git hooks
2026-04-05 06:15:53 -05:00
Val Alexander fddbb35b40 Merge pull request #1536 from openclaw/okcode/modern-ui-ux-refactor
Format codebase with consistent linting and import ordering
2026-04-05 06:02:11 -05:00
Val Alexander ba313e2d0a Refactor app UI around shared design system
- Add shared Radix/Tailwind UI primitives and empty/loading/error states
- Modernize route layouts, forms, tabs, nav, and markdown rendering
- Tighten package/auth API behavior and update tests for the new UX
2026-04-05 05:39:40 -05:00
Val Alexander f6f31ac78d Merge pull request #1527 from openclaw/okcode/modern-ui-ux-refactor
Refactor app UI with shared design system
2026-04-05 05:39:19 -05:00
Val Alexander f32500b1b4 Update tests for refreshed skill UI copy and behavior
- Align assertions with skeleton loading states and tab roles
- Update skills index expectations for new copy and ordering
2026-04-05 05:30:37 -05:00
Val Alexander 5bc6402b5e Add skill capability tag controls to management
- surface capability tags in the management view
- align tests with updated loading and toggle behavior
- relax package API credential assertion for env-dependent URLs
2026-04-05 05:24:00 -05:00
Val Alexander 9721461aca Handle package fetch credentials by origin
- Send cookies only for same-origin package requests
- Avoid CORS failures on cross-origin Convex site URL fetches
2026-04-05 05:11:06 -05:00
Val Alexander 91f80c264a Fix API URL routing for local and SSR requests
- Route local browser API calls to the Convex site URL
- Keep SSR loaders on the Convex site URL and remove the Vite proxy
2026-04-05 05:06:47 -05:00
Val Alexander 712302eb30 Widen skills and plugins layouts
- Expand skills and plugins pages to the wide container
- Rename settings test route file to match routing convention
2026-04-05 04:19:58 -05:00
Val Alexander f044164bae Prompt for hide and restore reasons
- Ask for a reason before soft-deleting or restoring a skill
- Pass the trimmed reason through to `setSoftDeleted` in both management views
2026-04-05 04:09:36 -05:00
Val Alexander 997da8857a Add Vite proxy for Convex API requests
- Proxy `/api` to `VITE_CONVEX_SITE_URL` in dev
- Keep the proxy disabled when the env var is unset
2026-04-05 04:05:38 -05:00
Val Alexander 1a31e07332 Use GitHub sign-in for publish skill empty state
- Replace the sign-in link with an auth action
- Trigger GitHub login directly from the publish flow
2026-04-05 03:51:32 -05:00
Val Alexander cb761a73a9 Wrap management links in button components
- Use `Button asChild` for view/manage actions in management screens
- Import `Button` in the dashboard route
2026-04-05 03:19:41 -05:00
Val Alexander d4ad4ea489 Refactor links to use Button asChild
- Centralize link-styled actions through the Button component
- Surface publish validation errors in the form state as well as toasts
2026-04-05 03:17:18 -05:00
Val Alexander d7c9126c4b Show fallback messages from API error payloads
- Accept unknown error shapes in `ErrorFallback`
- Extract user-facing text from `error.message`, `error.error`, or string values
- Keep a default message when no useful detail is available
2026-04-05 02:51:09 -05:00
Val Alexander 5dc6720d9e Refine markdown previews and tab/table styling
- Auto-link bare URLs in rendered markdown
- Add shared table primitives and markdown table styles
- Update tabs to use an underlined active state
2026-04-05 02:51:09 -05:00
Val Alexander 087f2c75b0 Add rich markdown preview for skill and plugin docs
- replace react-markdown rendering with a shared Shiki-powered preview
- update skill, soul, and plugin detail views to use the new component
- add markdown preview dependencies and adjust tests
2026-04-05 02:51:09 -05:00
Val Alexander 03c42fa947 Reset root error boundary on route changes
- Add a resetKey prop to `ErrorBoundary` so caught errors clear when navigation changes
- Wrap root children in a pathname-aware boundary to recover from route-level failures
2026-04-05 02:51:09 -05:00
Val Alexander 572110b1d5 Add confirmation dialogs for skill ownership actions
- Confirm rename and merge before submitting
- Remove unused form validation schema helpers
- Silence unhandled Promise.all lint in home route
2026-04-05 02:51:09 -05:00
Val Alexanderandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> 812276f1b7 Update src/lib/schemas.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-04-05 02:51:09 -05:00
Val Alexander b9676f674c Refactor app UI with shared design system
- Add reusable Radix-based UI primitives and layout helpers
- Refresh skill, soul, and dashboard pages with new empty/error/loading states
- Update dependencies for the modernized component stack
2026-04-05 02:51:09 -05:00
ImLukeF 32d601aab7 fix: show user-facing moderation override errors 2026-04-05 17:09:50 +10:00
ImLukeF 4aae925e76 fix: keep suspicious package scans from promoting to clean 2026-04-05 13:34:40 +10:00
ImLukeF e41fd677bb fix: use full skill description on detail page 2026-04-05 13:13:33 +10:00
ImLukeF 31520ef02a feat: add manual skill capability tag controls 2026-04-05 12:02:41 +10:00
ImLukeF e829cb1ae6 fix: use multiline moderation note dialog 2026-04-05 11:23:56 +10:00
ImLukeF 24cc59a425 fix: require moderation notes for hide and restore 2026-04-05 11:08:05 +10:00
Vincent Koc 522fa22026 fix: polish pass — tighter spacing, no empty sections, subtle skeletons
- Home page sections only render when data arrives (no skeleton flash)
- Removed SkeletonRows component from home page (unused)
- Skeleton bars simplified: static gray bars, no shimmer animation
- Skeleton row padding matches list item padding
- Hero padding tightened (48px top → 32px)
- Hero subtitle made concise ("20 skill bundles... Browse, install, publish.")
- Removed redundant explainer paragraph
- Browse results count shows em dash while loading (not "Loading...")
- .section class uses spacing tokens
- .section-title uses monospace font at --fs-lg
- .section-subtitle uses --fs-sm
- results-list removed fadeIn animation (subtle state changes only)
2026-04-04 19:26:44 +09:00
ImLukeF 7e76336ae4 fix: restore banned accounts in management 2026-04-04 20:50:55 +11:00
ImLukeF e41d5d6314 fix: restore banned accounts in management 2026-04-04 20:49:24 +11:00
Vincent Koc c62ab8bc96 fix: complete audit cleanup — monochrome purity, touch targets, a11y
Theming (P0):
- Removed all 78 [data-theme="dark"] override selectors (dark is now
  default, these were dead code with conflicting warm colors)
- Replaced 56 instances of rgba(255,107,74,x) warm coral with
  rgba(255,255,255,x) monochrome equivalents
- Replaced hard-coded warm hex colors (#c35640, #ff6b4a, etc.) with
  gray monochrome values
- CSS file reduced from ~6300 to 5909 lines

Touch targets (P1):
- Added min-height: 36px to .btn (was ~24px)
- Added min-height: 36px to .navbar-tab (was ~27px)
- Added min-height: 32px to .sidebar-option and .sidebar-checkbox
- Increased padding on buttons and tabs

Accessibility (P2/P3):
- Comprehensive prefers-reduced-motion: reduce rule — disables all
  animations AND transitions for users who prefer reduced motion
- Covers shimmer, fadeIn, fadeUp, and all CSS transitions
2026-04-04 17:15:28 +09:00
Vincent Koc 6df3a3b9ef feat: monochrome dark TUI aesthetic — complete design pivot
Complete visual redesign to a dark, monochrome, terminal-inspired
aesthetic inspired by Warp, modern TUI tools, and blueprint designs.

Color system:
- Default is now dark (#0a0a0a bg, #e0e0e0 ink, #141414 surface)
- All accent colors removed — monochrome only (white as accent)
- Borders use rgba(255,255,255,0.08) for subtle separation
- Light theme available as optional override via [data-theme="light"]

Typography:
- All fonts now IBM Plex Mono (display, body, code all monospace)
- Brand name is lowercase monospace
- Section titles are uppercase monospace with letter-spacing
- Tags and badges use monospace font

Geometry:
- All border-radius reduced to 1-2px (sharp TUI corners)
- No shadows anywhere (--shadow: none)
- No backdrop-filter blur on navbar
- Cards, buttons, inputs all have sharp edges

Components:
- Buttons: transparent bg with border, monospace text
- Primary buttons: white on black (inverted)
- Tags: border-only, no colored backgrounds
- Cards: dark surface with subtle border
- Brand mark: 24px square instead of 28px circle

Layout:
- Replaced category grid with simple quick links
- Removed all warm color references
- Home section titles are small uppercase labels
- Skill list item names use --ink (no accent color)
2026-04-04 16:08:12 +09:00
Vincent Koc 7452cf6f69 feat: white background, onboarding explainer, filter reset, transitions
- Switch light theme from warm beige (#f8f2ed) to neutral white (#fafafa)
  with neutral gray ink (#1a1a1a) and borders (rgba black)
- Switch dark theme from warm brown to neutral dark (#111111) with
  neutral gray borders (rgba white)
- Replace fake category grid (8 keyword-search cards) with curated
  quick links (Most starred, New this week, Browse plugins, Staff picks)
- Add "What are skills?" explainer paragraph below hero CTAs
- Add fadeIn animation on results list when data arrives
- Add "Clear" button in browse results toolbar when filters are active
- Tighten browse layout gap from 24px to 16px
2026-04-04 15:01:57 +09:00
ImLukeF 8ded548da2 fix: reject invalid capability tag filters 2026-04-04 15:41:26 +11:00
Luke 4ff2d1a0cd feat: add skill capability tags (#1513)
* feat: add skill capability tags

* fix: address capability tag review feedback

* fix: tighten capability tag review heuristics
2026-04-04 15:17:37 +11:00
Vincent Koc b4e8a26eb4 chore: remove accidentally committed skill/agent config files 2026-04-04 00:01:59 +09:00
Vincent Koc 7ff601bcb8 fix: resolve all lint errors
- Remove unused imports (v, getRuntimeEnv) from seedDemo.ts and SkillHeader.tsx
- Replace `as any` casts with proper Id<"publishers"> types in seedDemo.ts
- Prefix unused params with underscore (_clawdis, _osLabels, _nixSystems, _listDoneLoading)
- Remove unused convexSiteUrl variable from SkillHeader
2026-04-04 00:01:43 +09:00
Vincent Koc 70af109cb2 fix: address PR review feedback
- Import internalMutation from convex/functions (not _generated/server)
  to get trigger wrapping per CLAUDE.md rules
- Derive activeCategory from current search query so sidebar category
  selection shows correct visual/ARIA state
- Push moderationStatus filter server-side in repairGlobalStats to
  avoid full table scan
- Reset skillCount/pluginCount to 0 in useUnifiedSearch catch block
  to prevent stale badge values after search errors
2026-04-03 23:41:54 +09:00
Vincent Koc 1504708208 feat: marketplace UI overhaul — HuggingFace/npm-style discovery hub
Complete frontend rebuild of ClawHub into a marketplace-style discovery
hub inspired by HuggingFace and npm. No backend changes.

Navigation:
- Two-row header: brand + search bar + user actions on top, content
  type tabs (Skills, Plugins) with count badges below
- Inline search in navbar navigates to /search
- Mobile: search collapses behind icon, tabs scroll horizontally

Home page:
- Value-prop hero with Browse/Publish CTAs (no duplicate search)
- Trending section (8 skills by downloads)
- Recently updated section (8 skills by update time)
- Staff picks grid (6 highlighted skill cards)
- Browse by category grid (8 categories with Lucide icons)
- Skeleton loading rows while data fetches

Browse pages (Skills + Plugins):
- Full-width search bar above sidebar+results grid
- Left sidebar with sort options, categories, filter checkboxes
  (proper ARIA: fieldset/legend, role=radiogroup, aria-checked)
- List view uses compact SkillListItem rows (owner/name/summary/meta)
- Card view with hover border feedback
- View toggle (List/Cards)
- Better empty states with guidance text

Skill detail page:
- README tab as default (was Files)
- Two-column layout: tabs+comments on left, metadata sidebar on right
- Removed duplicate README from Files tab
- Removed duplicate Download button from header (kept in sidebar)
- Removed SkillInstallCard from header (license info in sidebar)
- Nix/config snippets moved inside two-column layout
- Friendly "No README available" instead of raw Convex errors

Unified search (/search):
- Real search results page (was redirect-only)
- Type tabs: All / Skills / Plugins with counts
- useUnifiedSearch hook fires skill search + plugin catalog in parallel
- Consistent SkillListItem rendering for results

Dashboard:
- Welcome state for new users with empty dashboard
- Simplified header copy

Profiles & Footer:
- Richer user profiles: large avatar, stat row, SkillListItem for
  published/starred skills
- Multi-column footer: Browse / Publish / Community / Platform

Design system:
- Spacing tokens: --space-1 (4px) through --space-8 (64px)
- Typography scale: --fs-xs through --fs-3xl (8 values, was 42)
- Radius tokens: --r-lg/md/sm/xs/pill (renamed from --radius-* to
  avoid Tailwind CSS v4 variable collision)
- Flat buttons (killed gradient, removed hover lift/shadow)
- Complete markdown styles: tables, blockquotes, lists, images, hr,
  heading hierarchy with h1/h2 bottom borders (npm-style)
- Removed decorative elements: body gradient backgrounds, card
  shadows, brand mark animation, category card glow

New components:
- SkillListItem — compact HF-style row
- BrowseSidebar — faceted filter sidebar with ARIA
- SkillMetadataSidebar — detail page right sidebar
- useUnifiedSearch — parallel search hook
- timeAgo — relative time formatter
- categories — static skill category taxonomy

Seed data:
- seedDemo.ts with 20 realistic skills, 5 publishers
- repairHighlightedBadges for skillBadges table
- repairGlobalStats for correct count

Test updates:
- Updated 5 test files for new text, class names, and prop changes
- All 122 test files, 915 tests passing
2026-04-03 22:58:30 +09:00
Onur 4af2bd50a7 Fix reusable package publish workflow (#1505)
* fix: harden reusable package publish workflow

* Fix reusable package publish CLI path

* Resolve reusable workflow source via OIDC

* Harden OIDC env lookup in publish workflow
2026-04-03 14:37:29 +02:00
OnurandOnur 88fe310ff1 feat: add deploy workflow targets (#1499)
Co-authored-by: Onur <onur@solmaz.io>
2026-04-03 09:43:41 +02:00
OnurandOnur 0a31b31f1b docs: remove production approval step from deploy docs (#1497)
Co-authored-by: Onur <onur@solmaz.io>
2026-04-03 09:26:18 +02:00
OnurandOnur 079e390da6 fix: repair auth sign-in typecheck (#1493)
Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 22:22:59 +02:00
Peter Steinberger 15da02c70a fix: improve banned account sign-in errors 2026-04-03 03:51:23 +09:00
Peter Steinberger 8b289618ba fix: recover broken auth state for package and user flows 2026-04-03 03:51:18 +09:00
OnurandOnur 4ae0406948 refactor: rename internal clawdhub package path (#1490)
* refactor: rename internal clawdhub package path

* fix: update workflow paths after clawhub dir rename

* fix: preserve old tag npm release compatibility

---------

Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 17:50:33 +02:00
OnurandOnur 78d0a637fa fix: make trusted publisher environment optional (#1489)
* fix: make trusted publisher environment optional

* fix: avoid env mismatch on unpinned trusted publishes

---------

Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 17:20:40 +02:00
OnurandOnur 65bc5d3335 ops: add guarded npm release workflow for clawhub cli (#1487)
* ops: add guarded npm release workflow for clawhub cli

* ops: limit clawhub cli release flow to stable npm tags

* docs: add clawhub cli release note to package readme

---------

Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 16:45:08 +02:00
OnurandOnur 8592272720 feat: add package trusted publishing via GitHub OIDC (#1461)
* feat: add package trusted publishing via GitHub OIDC

* fix: harden trusted publishing flow

* fix: finish trusted publishing rollout

* fix: harden trusted publish fallback

* ci: run package publish workflow from source

* fix: fall back when GitHub OIDC request fails

* test: fix plugin detail route mock

* fix: keep caller checkout pinned in package publish

* fix: restore auth query types

* ci: pin package publish workflow sources

* fix: tighten trusted package publish flow

* fix: add override reason for token fallback publishes

---------

Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 15:53:21 +02:00
OnurandOnur 15f5769cda docs: document manual production release flow (#1485)
Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 15:24:45 +02:00
OnurandOnur bb2a05e501 ops: make production deploy manual only (#1484)
Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 15:13:36 +02:00
OnurandOnur af29ca7f9d ops: gate production deploys behind environment approval (#1482)
* ops: gate production deploys behind environment approval

* ops: use existing production environment

* fix: retry transient deploy status polling failures

---------

Co-authored-by: Onur <onur@solmaz.io>
2026-04-02 15:05:57 +02:00
Luke 13caea7cfd fix: resolve admin search exact matches via personal publisher handles (#1467)
* fix admin user search coverage

* support personal publisher handle admin search

* fix admin search fallback pagination

* fix package detail route test mock

* fix users test mock typing
2026-04-02 11:25:34 +11:00
Val Alexander 86259eef42 Merge pull request #1455 from openclaw/feature/mobile-nav-spacing
Improve mobile navigation and plugin detail layouts
2026-04-01 09:29:46 -05:00
Val Alexander bf2ecab92e Handle source repo values without URL scheme
- Normalize `verification.sourceRepo` to a full GitHub URL when needed
- Keep the rendered source link text consistent for external links
2026-04-01 09:28:56 -05:00
Val Alexander 70516bd0f5 Improve plugin copy fallback and capability labels
- Add textarea-based clipboard fallback for unsupported contexts
- Show copied and failed states in the copy button
- Add labels and formatting for plugin capability values
2026-04-01 09:24:00 -05:00
Val Alexanderandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> 66fca9a286 Update src/routes/plugins/$name.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-04-01 09:16:29 -05:00
Val Alexander a03fe3bbd0 Handle clipboard copy failures gracefully
- Ignore clipboard write errors when copy is unavailable
- Keep the copy button from throwing in insecure or denied contexts
2026-04-01 09:12:23 -05:00
Val Alexander d3d9298389 Refine skill header layout and surface version
- Rework the hero into a two-column grid with sidebar actions and metadata
- Move version, badges, and security scan content into clearer sections
- Add responsive spacing tweaks for mobile navigation
2026-04-01 09:07:46 -05:00
Val Alexander 2fd0aa01d5 Adjust mobile nav trigger spacing and add Node version pin
- Shrink the mobile nav trigger and add a larger hit area
- Add `.nvmrc` to pin Node 22 for local development
2026-04-01 09:03:13 -05:00
Val Alexander 5aad36dab0 Refine plugin navigation and detail layout
- Rework plugin list filters into a more compact mobile-friendly toolbar
- Expand plugin detail pages with install, capability, compatibility, and verification sections
- Tighten shared toggle and scan result spacing for the new layout
2026-04-01 08:47:56 -05:00
Val Alexander 1bab92313f Tighten mobile navigation spacing and controls
- Reduce navbar padding and gaps on small screens
- Compact toggle group, theme buttons, and user trigger
- Hide extra dashboard summary text on narrow layouts
2026-04-01 08:23:55 -05:00
Val Alexander f77993d614 Make dashboard list responsive on mobile
- Relax desktop grid column sizing
- Collapse dashboard list headers to three columns below 768px
2026-03-31 20:24:02 -05:00
Val Alexander 3a28344c53 Improve mobile wrapping and layout resilience
- Wrap long URLs, token strings, and changelogs to prevent overflow
- Make key controls and dashboard grids shrink more gracefully on small screens
- Relax textarea and file viewer sizing for better mobile usability
2026-03-31 19:42:30 -05:00
Val Alexander ea49143fe0 Merge pull request #1439 from openclaw/okcode/mobile-friendly-review
fix: improve mobile touch targets and layout
2026-03-31 19:02:52 -05:00
Val Alexander bd38fef5c5 Relax ghost button min-height 2026-03-31 18:58:12 -05:00
Val Alexander b7a015523d Make management controls responsive on small screens
- Let management inputs shrink to container width
- Remove mobile-only overrides that duplicated desktop layout rules
2026-03-31 17:30:19 -05:00
Valentina AlexanderandClaude Opus 4.6 d36f98faa1 fix: comprehensive mobile-friendly improvements
Bring all interactive elements to 44px WCAG touch target minimum,
fix diff editor horizontal scroll on mobile, stack skills table on
small phones, add 480px breakpoint for tiny devices, and tighten
spacing across dashboard/management/dialog components.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 16:18:04 -05:00
George Zhang eeb0ecd932 fix: address post-merge vercel workflow/docs issues (#1386)
* fix: address post-merge vercel review issues

* fix: address remaining publish flow review findings

* docs: note explicit plugin compatibility requirements
2026-03-29 10:38:27 -07:00
LukeandJeff 51beceeb20 feat: add ability to delete version tags from skill detail page (#1380)
* feat: add ability to delete version tags from skill detail page

- Add deleteTags mutation to convex/skills.ts (protects 'latest' tag)
- Add delete button (×) on each tag in SkillHeader (visible to owner/moderator only)
- Wire up onTagDelete prop from SkillDetailPage to SkillHeader
- Add .tag-delete CSS styles

Closes: version tags accumulate across publishes with no way to remove them

* fix: address review feedback on deleteTags PR

- Add window.confirm() before deleting a tag (P2: missing confirmation)
- Skip db.patch when no tags are actually removed (P2: unnecessary write)
- Add test suite for deleteTags mutation covering:
  - Tag deletion with latest protection
  - No-op when only latest is targeted
  - No-op for nonexistent tags
  - Permission check for non-owner
  - Moderator access on other user's skill
  - Skill not found error

* fix: repair deleteTags test harness

* fix: satisfy deleteTags test typecheck

---------

Co-authored-by: Jeff <tjefferson518@gmail.com>
2026-03-29 22:10:38 +11:00
George Zhang a8a687eba2 [codex] streamline plugin publish flow (#1373)
* fix: streamline plugin publish flow

* fix: stabilize github package publish e2e

* test: fix clawdhub ci portability

* test: make settings route assertion portable

* test: fix ci typechecks
2026-03-29 02:37:58 -07:00
Val Alexander b9341202c7 Merge pull request #1349 from openclaw/okcode/fix-convex-tokens 2026-03-29 03:15:32 -05:00
George Zhang d701c2a33b Merge pull request #1362 from openclaw/codex/fix-handle-resolution
[codex] unify personal publisher handle resolution
2026-03-28 15:58:25 -07:00
George Zhang 79d17a91ed fix: ignore inactive personal publisher handles 2026-03-28 15:34:31 -07:00
George Zhang 2f15202a68 fix: unify personal publisher handle resolution 2026-03-28 15:14:17 -07:00
Nimrod Gutman 342a2b1ca4 fix(skill-detail): stop owner canonical redirect loop (#1357) 2026-03-28 20:49:41 +03:00
ImLukeF 75915cd2b5 fix: improve Safari compatibility and auth callbacks 2026-03-28 23:59:23 +11:00
Val Alexanderandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> 5395d9d159 Update src/routes/settings.test.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-03-28 07:36:38 -05:00
Val Alexander 87c236e8ce Skip token query until settings auth resolves
- avoid loading tokens before `me` is available
- add a test that verifies the query is skipped while auth is pending
2026-03-28 07:31:06 -05:00
ImLukeF d8db9b99a2 fix: resolve user profiles via personal publisher handles 2026-03-28 22:37:25 +11:00
Luke d8e1f0daa1 fix: align transferred skill publisher ownership (#1344) 2026-03-28 22:22:09 +11:00
ImLukeF 8a350d953c fix: preserve exact skill slug search matches
Fixes #1322.

Credit to @Yyang100 for the original report and repro.
2026-03-28 16:27:26 +11:00
Peter Steinberger 3fb3150ee2 fix: allow package downloads while VT scan is pending 2026-03-28 02:54:10 +00:00
Peter Steinberger 7dbc0fc3bb fix: unblock verified community package scans 2026-03-27 23:44:45 +00:00
Peter Steinberger fc4f8644eb refactor: centralize package scan state resolution 2026-03-27 02:10:01 +00:00
Peter Steinberger dda6d55fbf fix: unblock package vt scan fallback stalls 2026-03-27 02:06:03 +00:00
Vincent Koc e014759b40 Merge pull request #1191 from openclaw/vincentkoc-code/fix-plugin-compatibility 2026-03-24 22:55:12 -07:00
Peter Steinberger 2186c41c48 fix: prioritize recent package vt backfills 2026-03-24 10:19:28 -07:00
Peter Steinberger c30c182478 fix: harden package vt scan scheduling 2026-03-24 10:11:26 -07:00
Peter Steinberger 3080567964 perf: tighten package search candidate reads 2026-03-23 23:23:38 -07:00
Peter Steinberger f304541561 perf: collapse package readme fallback requests 2026-03-23 23:23:36 -07:00
Peter Steinberger 91224ada13 perf: parallelize plugin detail reads 2026-03-23 23:04:36 -07:00
Peter Steinberger d5fbaeef81 perf: unify plugin catalog reads 2026-03-23 23:03:01 -07:00
Peter Steinberger 3706018b72 docs: clarify abuse policy for leaked data and account farming 2026-03-23 19:42:20 -07:00
Peter Steinberger 62e616f635 fix: raise http api rate limits 2026-03-23 19:06:49 -07:00
Peter Steinberger 9013d324c8 fix: forward client ip headers for ssr package api 2026-03-23 19:03:12 -07:00
Peter Steinberger c1363ec8d0 feat: add acceptable usage about page 2026-03-23 17:37:17 -07:00
Peter Steinberger 7e09196f92 feat: backfill static plugin scans 2026-03-23 15:58:02 -07:00
Peter Steinberger 807043b4b0 feat: surface plugin security scans 2026-03-23 15:42:59 -07:00
Vincent Koc 972fe35935 fix(plugins): improve package upload detection UX 2026-03-23 15:04:30 -07:00
Vincent Koc 16ee540f5d fix(vt): recover plugin package scan retries 2026-03-23 14:57:39 -07:00
Vincent Koc f541882d55 feat(plugins): gate publish form behind upload 2026-03-23 14:49:42 -07:00
Vincent Koc 95bc156747 test(e2e): align header smoke nav labels 2026-03-23 14:46:06 -07:00
Vincent Koc bf7422022f fix(auth): unblock safari cli github login 2026-03-23 14:28:42 -07:00
Vincent Koc 230e5b91f8 fix(publishing): rename publish routes and clarify import scope 2026-03-23 13:54:47 -07:00
Vincent Koc 70dcf21e37 fix(shell): streamline publisher inventory navigation 2026-03-23 13:45:42 -07:00
Vincent Koc 932a1fb30c fix(dashboard): improve publisher plugin card layout 2026-03-23 13:34:28 -07:00
Vincent Koc 44fe60b701 fix(plugins): keep public package routes public 2026-03-23 13:28:26 -07:00
Vincent Koc 1c057ca9b9 fix(dashboard): streamline publisher plugin workflow 2026-03-23 13:24:46 -07:00
Vincent Koc 9f793d1336 feat(dashboard): add publisher plugin status panel 2026-03-23 13:16:30 -07:00
Vincent Koc aa9295bea9 fix(plugins): forward browser auth to package api 2026-03-23 13:05:41 -07:00
Vincent Koc 370eea4977 fix(plugins): recover owner package detail after auth 2026-03-23 12:53:35 -07:00
Vincent Koc 59e6819020 fix(publishers): remove leftover conflict marker 2026-03-23 12:37:53 -07:00
Vincent Koc 52b633f4e9 fix(publishers): skip digest scheduling without scheduler 2026-03-23 12:28:22 -07:00
Vincent Koc 6b1e6ca1c9 Merge pull request #1203 from openclaw/vincentkoc-code/fix-personal-publisher-pagination
fix: split publisher digest sync scheduling
2026-03-23 12:21:51 -07:00
Vincent Koc 6c112eccb7 chore: use typed internal digest refs 2026-03-23 12:20:59 -07:00
Vincent Koc f9e9effcdd fix(publishers): split owner digest sync scheduling 2026-03-23 12:14:35 -07:00
Peter Steinberger b5d2d0fefa fix: heal missing personal publishers during publish 2026-03-23 10:43:17 -07:00
Vincent Koc 48d0fc91f3 fix(packages): keep plugin api separate from gateway version 2026-03-23 09:08:49 -07:00
LukeandPeter Steinberger fefb2340a8 feat: redesign plugins page and skills list view (#1166)
* feat: redesign plugins page and skills list view

Redesigned the plugins page with a cleaner toolbar (pill search,
toggle filter buttons) and simplified card layout. Added a proper
table-style list view for skills with skill name, version, summary,
and author avatar columns. Also polished the sort dropdown with a
chevron indicator, added card shadows for better separation, and
tightened up the theme toggle and sign-in button.

* feat: add publisher org ownership

* feat: migrate legacy publisher handles to orgs

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-03-23 03:44:17 -07:00
Peter Steinberger 22287558b9 feat: migrate legacy publisher handles to orgs 2026-03-23 03:39:28 -07:00
Peter Steinberger f6ce8f9e1e feat: add publisher org ownership 2026-03-23 03:27:08 -07:00
Peter Steinberger b5cdee50a9 feat(packages): support shared publisher owners 2026-03-23 02:14:38 -07:00
Vincent Koc b5fdee1c13 feat(plugins): clarify pending verification and add backfill 2026-03-23 02:07:05 -07:00
Vincent Koc 2530beaf51 fix(plugins): hide pending and malicious packages 2026-03-23 02:04:00 -07:00
Vincent Koc c627b202f7 fix(plugins): add async malware scan parity 2026-03-23 02:01:04 -07:00
Vincent Koc 5cbeb54c8e fix(upload): cap published skill and plugin file sizes 2026-03-23 01:57:59 -07:00
Vincent Koc eb8136b7fb fix(ui): keep query when filtering plugins 2026-03-23 01:47:31 -07:00
Peter Steinberger a166c95eb0 fix: avoid skill scans on verified plugin browse 2026-03-23 01:14:40 -07:00
scoootscooob 86dc196e9b fix: tolerate stale auth on public package reads (#1157) 2026-03-23 01:06:32 -07:00
Peter Steinberger 649c14a43f feat: rename Packages to Plugins, filter out skills, add verified badge
- Rename /packages route to /plugins with /packages redirecting
- Nav links now say "Plugins" and point to /plugins
- Plugins page only shows code-plugin and bundle-plugin (no skills)
- "Official only" → "Verified only"; blue checkmark badge for verified publishers
- Compact card footer: "by author · v1.2.3" inline with verified badge
- Remove duplicate Skill/Skill tag bubbles
- Update tests to match new routes and behavior
2026-03-23 01:02:15 -07:00
Peter Steinberger 7cbf0434b0 ci: wait for vercel git deploy status 2026-03-23 00:58:21 -07:00
Peter Steinberger 76944f0e32 chore: prepare 0.9.0 release 2026-03-23 00:45:56 -07:00
Peter Steinberger 6ead1da08b fix: use read rate limits for package files 2026-03-23 00:44:31 -07:00
Val Alexander 22413f4ff8 Revert "chore: update .gitignore and enhance Convex documentation in AGENTS.md and CLAUDE.md"
This reverts commit 3326c235da.
2026-03-22 20:17:58 -05:00
Val Alexander 3326c235da chore: update .gitignore and enhance Convex documentation in AGENTS.md and CLAUDE.md 2026-03-22 20:03:04 -05:00
Peter Steinberger 2b97c13191 feat: reserve official handles and package archives 2026-03-22 12:48:47 -07:00
Peter Steinberger 17977c1248 fix: support nested package upload files 2026-03-22 11:14:58 -07:00
Peter Steinberger aea0127663 fix: add package cleanup mutation 2026-03-22 11:09:31 -07:00
Peter Steinberger 9ea750852f fix: stabilize package catalog search and contract verify 2026-03-22 11:04:20 -07:00
Peter Steinberger 801cc550ab fix: narrow skill package catalog search 2026-03-22 11:03:17 -07:00
Peter Steinberger 522512c9d2 fix: use paginator for skill package catalog queries 2026-03-22 11:00:18 -07:00
Peter Steinberger 1ceb5d2bad fix: tolerate prefixed convex function-spec output 2026-03-22 10:51:04 -07:00
Peter Steinberger c7b3b2a30e ci: restore bun in deploy web job 2026-03-22 10:49:21 -07:00
Peter Steinberger f1e3629a99 ci: deploy web via vercel direct build 2026-03-22 10:47:49 -07:00
Peter Steinberger 931843aeac ci: fail deploy preflight when secrets are missing 2026-03-22 10:43:36 -07:00
Peter Steinberger 1a56fc5427 test: isolate package api env fallback 2026-03-22 10:22:25 -07:00
Peter Steinberger 03a19f6497 fix: normalize skill and soul slugs 2026-03-22 10:20:48 -07:00
Peter Steinberger f9087fc6e0 Merge pull request #1093 from openclaw/plugin
feat: add native package and plugin registry flows
2026-03-22 10:18:11 -07:00
Peter Steinberger 5429c28667 Merge remote-tracking branch 'origin/main' into plugin
# Conflicts:
#	bun.lock
#	package.json
2026-03-22 10:15:03 -07:00
Peter Steinberger c234009b7c fix: expose private packages to owners 2026-03-22 10:13:16 -07:00
Peter Steinberger 962171a590 feat: unify skills into package catalog 2026-03-22 16:04:56 +00:00
Peter Steinberger 852b54a722 fix: preserve package publish outputs and site fallbacks 2026-03-22 15:33:23 +00:00
Peter Steinberger dff83a52bf fix: tighten package auth boundaries 2026-03-22 07:27:25 +00:00
Peter Steinberger ec8601c0eb fix: tighten package fallback and readme errors 2026-03-22 04:09:46 +00:00
Peter Steinberger ee6a5ab037 test: increase package API coverage 2026-03-22 03:56:42 +00:00
Peter Steinberger 4814ce39d6 fix: remove dead skill family from package catalog 2026-03-22 03:54:11 +00:00
Peter Steinberger ccace09242 fix: preserve package latest metadata and rename publish route 2026-03-22 03:34:42 +00:00
Peter Steinberger 8f9c9ffba2 fix: tighten package filter indexes and private UI states 2026-03-22 03:20:29 +00:00
Peter Steinberger 7460c084a5 test: cover package publish route plugin uploads 2026-03-22 02:53:30 +00:00
Peter Steinberger e7ef691013 fix: filter ignored package uploads and block skill publishes 2026-03-22 02:50:05 +00:00
Peter Steinberger da76bf7554 test: ignore generated vercel output in vitest 2026-03-21 19:35:46 -07:00
Peter Steinberger 3959a7ea85 fix: preserve package cursor tails and latest tags 2026-03-22 02:29:59 +00:00
Peter Steinberger 08da7eb477 fix: remove runtime devtools from app shell 2026-03-21 19:27:43 -07:00
Peter Steinberger 958a37fa6b fix: respect runtime env in prod SSR 2026-03-21 19:25:11 -07:00
Peter Steinberger 91d963cf13 fix: repoint packages to the right fallback release 2026-03-22 02:23:09 +00:00
Peter Steinberger 31191b7771 fix: lazy-load devtools on client 2026-03-21 19:21:01 -07:00
Peter Steinberger 371367f01d chore: update dependencies 2026-03-21 19:13:19 -07:00
Peter Steinberger 2ace8f9ab4 fix: freeze code plugin runtime ids 2026-03-22 02:11:59 +00:00
Peter Steinberger 914fa895c5 build: add China mirror redirect entrypoint 2026-03-21 19:08:17 -07:00
Peter Steinberger 93e398e7a5 fix: harden package catalog scans and downloads 2026-03-22 02:06:28 +00:00
Peter Steinberger 043f029f2c fix: resync package latest pointers on release hides 2026-03-22 01:56:54 +00:00
Peter Steinberger de90f80d72 fix: tighten package summary and digest sync 2026-03-22 01:29:34 +00:00
Peter Steinberger cc9de82034 fix: tighten package pagination and version indexing 2026-03-22 01:17:38 +00:00
Peter Steinberger 05924a36e1 fix: keep private package flows on app origin 2026-03-22 00:42:26 +00:00
Peter Steinberger ed9dcc39b7 fix: restore package auth and download metadata 2026-03-22 00:20:36 +00:00
Peter Steinberger 1511aa2082 fix: tighten package latest visibility and filter indexes 2026-03-21 23:09:06 +00:00
Peter Steinberger f88f3ef925 fix: tighten package release serving and uploads 2026-03-21 22:55:23 +00:00
Peter Steinberger adce8ac1b5 fix: preserve skill family filters across package listings 2026-03-21 22:40:39 +00:00
Peter Steinberger ddafee7e0e fix: address final package registry review issues 2026-03-21 17:38:07 +00:00
Peter Steinberger 877f37d052 fix: resolve package publish and search regressions 2026-03-21 17:28:30 +00:00
Peter Steinberger 5210966a19 test: add package registry regression coverage 2026-03-20 18:30:50 +00:00
Peter Steinberger 4ccfb9d3f5 fix: address package registry review regressions 2026-03-20 18:27:52 +00:00
Peter Steinberger fc7d4dadca feat: add native package and plugin registry flows 2026-03-20 18:00:15 +00:00
Peter Steinberger fb86e790b8 docs: add plugin hosting plan 2026-03-20 10:29:47 -07:00
Val Alexander deb592d4ce docs: update repository guidelines and improve formatting across multiple files
- Enhanced AGENTS.md with clearer project structure and development commands.
- Updated CHANGELOG.md to reflect recent fixes and additions.
- Improved formatting in CONTRIBUTING.md for better readability.
- Adjusted package.json and configuration files for consistent command structure.
- Refined README.md and VISION.md for clarity and organization.
- Standardized code formatting in various TypeScript files for consistency.

These changes aim to enhance documentation clarity and maintainability across the repository.
2026-03-18 21:56:01 -05:00
Peter Steinberger 8030c554e8 test: fix ci for download regression coverage 2026-03-17 09:59:58 -07:00
Peter Steinberger 472e1e875d test: harden download counter regression coverage 2026-03-17 09:56:49 -07:00
Peter Steinberger ba9cdde703 fix: internalize download counter mutations 2026-03-17 09:38:11 -07:00
magicseth 8d5a64b599 Merge pull request #965 from sethconvex/fix/github-backup-retry-on-conflict
fix: retry GitHub backup push on fast-forward conflict
2026-03-17 00:07:44 -07:00
Seth RaphaelandClaude Opus 4.6 351dfd48c3 fix: retry GitHub backup push on fast-forward conflict
When a publish-time backup and the cron backup push concurrently, the
second push fails with "not a fast forward" because the branch moved.

Split backupSkillToGitHub into two phases:
1. Create blobs (storage downloads) — done once
2. Fetch ref, build tree, commit, push — retried up to 3x on conflict

Same retry applied to deleteGitHubSkillBackup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 23:16:22 -07:00
magicseth 0e2a7e1bba Merge pull request #956 from sethconvex/feat/listpublic-v4-staged-release
feat: switch /skills and homepage to listPublicPageV4
2026-03-16 18:11:10 -07:00
Seth RaphaelandClaude Opus 4.6 54c9d57a22 fix: add authTables to @convex-dev/auth/server mocks in all test files
The schema import in skills.ts (needed for getPage) transitively pulls
in authTables from @convex-dev/auth/server. All test files that import
from skills.ts need this export in their mock.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 18:07:34 -07:00
Seth RaphaelandClaude Opus 4.6 343836a026 fix: use skillBadges index for highlightedOnly filter in V4
Instead of scanning 500+ rows of the sort index and filtering for
highlighted skills in JS (which fails when highlighted skills are
sparse among 25K+ rows), query the skillBadges table via by_kind_at
index to find highlighted skill IDs directly, then look up their
digests. Also simplifies the non-highlighted path to a single getPage
call since the multi-round loop was only needed for highlighted.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 17:59:58 -07:00
Seth RaphaelandClaude Opus 4.6 ac86be7e14 chore: remove useV4 flag — V4 is now the only code path
Remove the V3 fallback branch and useV4 parameter from
useSkillsBrowseModel since V4 is verified and the only path used.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 17:38:07 -07:00
Seth RaphaelandClaude Opus 4.6 6ef8843f1c feat: switch /skills and homepage to V4, remove test routes
- /skills browse page now uses listPublicPageV4 via useV4 flag
- Homepage popular skills section uses listPublicPageV4
- Remove /skillsv4 and /test-v4 temporary test routes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 17:33:24 -07:00
magicseth 33a7cec1af Merge pull request #955 from sethconvex/feat/listpublic-v4-staged-release
feat: V4 staged release — /skillsv4 test route
2026-03-16 17:30:54 -07:00
Seth RaphaelandClaude Opus 4.6 81e734b1fc fix: guard against hasMore=true with nextCursor=null in V4 path
When V4 returns hasMore=true but nextCursor=null, the next load-more
call would pass cursor=null, triggering the replace branch instead of
append. Treat this edge case as 'done' to prevent silent list reset.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 17:30:12 -07:00
Seth RaphaelandClaude Opus 4.6 b3dd6c1219 feat: V4 staged release — fix getPage schema, add /skillsv4 test route
Root cause of V4 returning empty in production: `getPage()` ignores the
`indexFields` property at runtime and always calls `getIndexFields(table,
index, schema)`. Without `schema`, it threw "schema is required" silently.

Fixes:
- Pass `schema` instead of `indexFields` to `getPage()`
- Use `absoluteMaxRows` instead of `targetMaxRows` (ignored when
  `endIndexKey` is provided)
- Remove unused `DIGEST_INDEX_FIELDS` constant

Staged release:
- Add `/skillsv4` route (same UI as `/skills` but using V4 backend)
- Add `/test-v4` debug page for raw V4 API testing
- Add `useV4` flag to `useSkillsBrowseModel` hook
- Keep `/skills` on V3 until V4 is verified in production

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 17:23:47 -07:00
magicseth ecf71e8664 Merge pull request #953 from sethconvex/feat/listpublic-v4-backend-only
feat: add listPublicPageV4 backend query (frontend stays on V3)
2026-03-16 16:58:45 -07:00
Seth RaphaelandClaude Opus 4.6 1d170cf634 fix: revert homepage to V3 — V4 failing in production
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:57:47 -07:00
magicseth c19917e9a4 Merge pull request #950 from sethconvex/feat/listpublic-v4-deterministic-cursors
feat: listPublicPageV4 with cacheable deterministic cursors
2026-03-16 16:55:55 -07:00
Seth RaphaelandClaude Opus 4.6 9922c22291 fix: use indexFields instead of schema import to avoid auth dep in tests
- Replace `import schema from './schema'` with inline DIGEST_INDEX_FIELDS
  lookup, avoiding @convex-dev/auth/server transitive dependency in tests
- Gut V1 test to match gutted handler (single stub verification)
- Fix load-more test to use V4 response shape

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:51:16 -07:00
Seth RaphaelandClaude Opus 4.6 f6c0735238 fix: update tests for V4 response shape and gutted V2
- Replace V2 test suite with single stub verification (no DB reads)
- Update skills-index tests: paginationOpts → cursor/numItems,
  isDone/continueCursor → hasMore/nextCursor
- Update default convexHttpMock to return V4 shape

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:47:57 -07:00
Seth RaphaelandClaude Opus 4.6 1d2a822f2a fix: lint — remove dead code and fix variable shadowing
- Remove sortToIndex and getTrendingEntries (only used by gutted V1)
- Remove unused leaderboard imports
- Rename shadowed 'v' parameter to 'val' in encode/decodeIndexKey

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:45:03 -07:00
Seth RaphaelandClaude Opus 4.6 92e640a46b fix: address codex review — cursor safety, pagination stall, stale doc
- Use tagged object encoding for undefined index key values instead of
  plain string sentinel to avoid collisions
- Add try/catch in decodeIndexKey, treat malformed cursors as first page
- When highlightedOnly filters out all fetched rows, advance nextCursor
  to last fetched position instead of returning null (prevents restart loop)
- Update V3 JSDoc to reflect its current role

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:39:57 -07:00
Seth RaphaelandClaude Opus 4.6 618cb3141c fix: bound getPage with equality prefix to exclude soft-deleted rows
getPage walks the entire index unless bounded. Without constraining
startIndexKey/endIndexKey to the equality prefix ([undefined] for base,
[undefined, false] for nonsuspicious), desc order returns soft-deleted
items first, producing empty pages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:36:45 -07:00
Seth RaphaelandClaude Opus 4.6 49d0e0246a feat: add listPublicPageV4 with cacheable deterministic cursors
Use convex-helpers getPage() instead of .paginate() so that page cursors
are derived from actual index field values. Two users requesting the same
page now produce identical query args, enabling shared query caching.

- Add listPublicPageV4 with IndexKey-based cursor encoding
- Gut listPublicPage (V1) and listPublicPageV2 to return empty results
- Keep listPublicPageV3 intact for any remaining subscribers
- Switch frontend browse model and homepage to V4

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 16:24:01 -07:00
magicseth 0a4e542ba9 Merge pull request #949 from sethconvex/chore/listpublic-v3-stale-tab-check
chore: add listPublicPageV3 to identify stale-tab websocket traffic
2026-03-16 15:53:49 -07:00
Seth RaphaelandClaude Opus 4.6 0721c57fae chore: add listPublicPageV3 to distinguish new clients from stale tabs
Duplicate of listPublicPageV2 as a separate Convex function. Frontend
switched to V3 so any remaining V2 calls in the dashboard are from
stale browser tabs with old bundles.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 15:52:02 -07:00
magicseth 57cec34d53 Merge pull request #948 from sethconvex/fix/listpublic-v2-test-typecheck
fix: typecheck error in listPublicPageV2 test
2026-03-16 15:13:23 -07:00
Seth RaphaelandClaude Opus 4.6 c20f836d71 fix: typecheck error in listPublicPageV2 test
Cast page item to Record to access latestVersion property that isn't
on the narrow return type.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 15:12:20 -07:00
magicseth 390f52ade0 Merge pull request #947 from sethconvex/fix/listpublic-v2-test-update
test: update listPublicPageV2 tests for digest-only behavior
2026-03-16 15:07:12 -07:00
Seth RaphaelandClaude Opus 4.6 1f30223b54 test: update listPublicPageV2 tests to match digest-only behavior
Remove fallback expectations — pre-backfill rows without owner fields
are now skipped, and missing latestVersionSummary returns null instead
of fetching from skillVersions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 15:06:33 -07:00
magicseth b7d273daea Merge pull request #945 from sethconvex/perf/listpublic-v2-digest-only
perf: listPublicPageV2 reads only from digest, no extra table joins
2026-03-16 14:37:06 -07:00
Seth RaphaelandClaude Opus 4.6 8226e418f4 refactor: use filteredMap to avoid redundant digestToHydratableSkill call
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:22:51 -07:00
Seth RaphaelandClaude Opus 4.6 48849dec89 perf: build listPublicPageV2 response directly from digest, no extra table reads
listPublicPageV2 was calling buildPublicSkillEntries which fell back to
ctx.db.get() for owners and versions, adding skills/users/skillVersions
to the query's read set. Now that digest rows have owner fields and
latestVersionSummary backfilled, we can construct the full response from
skillSearchDigest alone — writes to other tables no longer bust the cache.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:09:38 -07:00
Peter Steinberger 864b24fe03 test: add prod http smoke coverage 2026-03-15 22:22:22 -07:00
Peter Steinberger 26e6a435f2 test: add ssr and og regression coverage 2026-03-15 20:49:18 -07:00
Peter Steinberger f8104fd759 fix: package og assets for server rendering 2026-03-15 20:36:12 -07:00
Peter Steinberger ef2cd7a1f4 fix: restore ssr-compatible web stack 2026-03-15 20:30:29 -07:00
Peter Steinberger ee2d06e622 feat: server render public skill pages 2026-03-15 20:12:53 -07:00
Nimrod Gutmanandgeoffrey 69e68e696e fix(web): default compare diff to inline on narrow screens (#898)
* fix: isolate diff editor rendering and styling

* fix: scrollbar style

* fix(web): default compare diff to inline on narrow screens

---------

Co-authored-by: geoffrey <1377499035@qq.com>
2026-03-15 18:53:57 +02:00
magicseth 64b32d88c0 Merge pull request #872 from sethconvex/perf/search-skip-users-and-compound-indexes
perf: skip users table in search, use compound indexes in listPublicPageV2
2026-03-14 09:37:19 -07:00
Seth RaphaelandClaude Opus 4.6 a0d0ec0e1e test: update assertions for compound index and digest owner changes
- search.test: expect users lookup NOT called when digest has owner data
- listPublicPageV2.test: expect by_nonsuspicious_* indexes when nonSuspiciousOnly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 09:34:13 -07:00
Seth RaphaelandClaude Opus 4.6 da578cfcad fix: handle null-owner digest fallback and safe compound index migration
- P2: When digestToOwnerInfo returns { owner: null } (deactivated/deleted
  user), fall back to live users table lookup instead of dropping the skill
  from results. Applies to hydrateResults, lexicalFallbackSkills, and
  buildPublicSkillEntries.

- P1: If compound index returns zero results on the first page (isSuspicious
  not yet backfilled), fall back to base index with JS filtering so the
  homepage isn't empty during migration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 09:28:02 -07:00
Seth RaphaelandClaude Opus 4.6 14e4ab59cb docs: add CLAUDE.md with Convex performance rules
Encodes the patterns learned from bandwidth optimization so AI coding
assistants get them right from the start — digest owner fields over
users table reads, compound indexes over JS filtering, one-shot fetches
for public pages, change detection in triggers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 09:14:29 -07:00
Seth RaphaelandClaude Opus 4.6 7c48828d69 perf: skip users table in search, use compound indexes in listPublicPageV2
Three bandwidth fixes:

1. search.hydrateResults / lexicalFallbackSkills: use digestToOwnerInfo()
   to resolve owner data from the digest instead of ctx.db.get(ownerUserId)
   on the users table. Eliminates users from the read set for ~99% of
   search calls (8+ GB in 72h).

2. skills.listPublicPageV2: use compound by_nonsuspicious_* indexes when
   nonSuspiciousOnly is true, filtering isSuspicious at the DB level
   instead of scanning and discarding in JS (30+ GB in 72h).

3. maintenance.backfillDigestIsSuspicious: targeted backfill that sets
   isSuspicious on digest rows where it's undefined, using the digest's
   own moderationFlags/moderationReason. Must run before compound indexes
   take effect.

Deploy sequence:
  1. Deploy functions
  2. npx convex run maintenance:backfillDigestIsSuspicious --prod
  3. Compound indexes work immediately for backfilled rows

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 09:13:40 -07:00
Nimrod GutmanandJonathan Deamer 674ca01a3b fix(cli): validate forced install version before rm (#864)
* test(cli): add tests for install --force rm ordering

Add three tests to verify that --force install does not delete the local
skill directory before pre-download checks have passed:

- rm not called when skill is malware-blocked
- rm not called when API fetch fails (skill not found)
- rm called before download when all checks pass (happy path)

* fix(cli): move rm after checks in install --force

Previously, install --force deleted the local skill directory before
fetching metadata or running moderation checks. If any check failed
(skill deleted, malware-blocked, not found), the local copy was lost.

This is inconsistent with cmdUpdate, which already checks before
deleting. Move rm to after all checks pass, just before download.

This does not alter the meaning of --force; it narrows the window in
which data is removed before the command has confirmed the replacement
is viable.

* fix(cli): validate forced install version before rm

---------

Co-authored-by: Jonathan Deamer <202770+jonathandeamer@users.noreply.github.com>
2026-03-14 13:33:33 +02:00
Wangnovandwangnov f0b6335966 fix: default skills search to relevance (#802)
* fix: default skills search to relevance

* test: cover explicit search sort guard

---------

Co-authored-by: wangnov <1694546283@qq.com>
2026-03-14 13:10:13 +02:00
Vincent KocandNimrod Gutman 165f132613 fix: guard direct version file reads (#797)
* fix: guard soul version file reads

* test: cover version file access guards

* fix(convex): tighten version file access guards

---------

Co-authored-by: Nimrod Gutman <nimrod.gutman@gmail.com>
2026-03-14 12:46:16 +02:00
Vincent KocandNimrod Gutman da0448923b fix: sanitize public soul version queries (#796)
* fix: sanitize public soul version queries

* test: cover public soul version sanitization

* fix(api): preserve soul file downloads after sanitization

---------

Co-authored-by: Nimrod Gutman <nimrod.gutman@gmail.com>
2026-03-14 12:14:27 +02:00
Vincent Koc cca8b4421b fix: sanitize public skill version queries 2026-03-14 11:42:56 +02:00
magicseth 07df1bcec3 Merge pull request #844 from sethconvex/perf/oneshot-browse-page
perf: replace reactive browse/home queries with one-shot fetches
2026-03-13 22:09:34 -07:00
Seth RaphaelandClaude Opus 4.6 0c5dc63bf5 fix: mock convexHttp in skills-route-default-sort test for CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 22:08:02 -07:00
Seth RaphaelandClaude Opus 4.6 a9699684ce merge: resolve conflict with origin/main in index.tsx
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 22:06:16 -07:00
Seth RaphaelandClaude Opus 4.6 1bc3ba98ed fix: allow retry on load-more fetch failure instead of hiding control
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 20:58:18 -07:00
Seth RaphaelandClaude Opus 4.6 08284d8f3d test: update tests for one-shot fetch and preResolved owner short-circuit
- Replace usePaginatedQuery mocks with convexHttp.query mocks in
  browse page tests
- Update load-more test to use convexHttp instead of loadMorePaginated
- Update backend tests to reflect new behavior: getOwnerInfo skips
  db.get when digest has pre-resolved owner data

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 20:53:30 -07:00
Seth RaphaelandClaude Opus 4.6 1848fcb819 fix: add error handling and unmount cleanup to home page fetches
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 20:48:21 -07:00
Seth RaphaelandClaude Opus 4.6 6224c05e82 perf: replace reactive browse/home queries with one-shot HTTP fetches
listPublicPageV2 is the #1 DB bandwidth consumer (31GB+ spikes) because
usePaginatedQuery creates reactive subscriptions. Any write to
skillSearchDigest invalidates all active subscribers simultaneously —
a thundering herd. This replaces reactive subscriptions with one-shot
ConvexHttpClient.query() calls on both the /skills browse page and the
home page, eliminating the reactive read set entirely.

Also short-circuits getOwnerInfo() to return pre-resolved owner data
from the digest before hitting ctx.db.get(ownerUserId), removing the
users table from the reactive read set for listPublicPageV2.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 20:28:23 -07:00
magicseth 68fc915444 Merge pull request #843 from sethconvex/perf/home-page-oneshot-queries
perf: replace reactive subscriptions with one-shot fetches on home page
2026-03-13 17:44:47 -07:00
Seth RaphaelandClaude Opus 4.6 7cd68cc3b5 fix: add .catch() handlers to one-shot home page queries
Prevents unhandled promise rejections on transient network/backend
failures. Empty state is an acceptable fallback for the home page.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:43:10 -07:00
Seth RaphaelandClaude Opus 4.6 e37ccd95d1 perf: replace reactive subscriptions with one-shot fetches on home page
The home page used useQuery for highlighted and popular skills, creating
live reactive subscriptions that re-executed on every skillSearchDigest
write (crons, triggers). Since the home page doesn't need live updates,
switch to one-shot convex.query() fetches on mount to eliminate unnecessary
reactive invalidation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:33:52 -07:00
magicseth 871e89e0b7 Merge pull request #841 from sethconvex/feat/backfill-digest-version-summary
feat: backfill latestVersionSummary into skillSearchDigest rows
2026-03-13 16:33:43 -07:00
Seth RaphaelandClaude Opus 4.6 78077028fe feat: add backfillDigestVersionSummary to patch latestVersionSummary into digest rows
Existing skillSearchDigest rows created before the latestVersionSummary
denormalization lack the field, causing listPublicPageV2 to fall back to
reading full skillVersions docs (~6KB each, 14MB per call).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 16:20:22 -07:00
magicseth 5d0f5155cf Merge pull request #840 from sethconvex/fix/stable-pagination-global-stats
fix: use stable _creationTime pagination in countPublicDigestPageInternal
2026-03-13 15:48:44 -07:00
Seth RaphaelandClaude Opus 4.6 946ca4febd fix: use stable _creationTime pagination in countPublicDigestPageInternal
The by_active_updated index orders by updatedAt which is mutable —
rows can shift during a paginated scan causing double-counting or
skipping. Default _creationTime ordering is immutable and stable.
isPublicSkillDoc already filters softDeletedAt in JS.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 15:48:07 -07:00
magicseth 593e05f72e Merge pull request #837 from sethconvex/perf/aggregate-global-stats
perf: split global stats recount into batched action
2026-03-13 15:39:17 -07:00
Seth RaphaelandClaude Opus 4.6 faaedb17ea perf: skip digest write when no fields changed to prevent reactive thundering herd
The skills trigger unconditionally wrote to skillSearchDigest on every skill
mutation, even when only stat fields were updated with identical values.
This caused every cron that patches skills (stat sync, backfill) to invalidate
all active listPublicPageV2 subscriptions, triggering massive re-execution
storms (55 GB bandwidth spikes).

Now upsertSkillSearchDigest compares new fields against the existing row and
skips the write when nothing changed. This prevents unnecessary reactive
invalidation while still keeping the digest current for real changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 15:24:07 -07:00
Peter Steinberger 0ab23862a9 feat: harden skill moderation and canonicalization 2026-03-13 21:35:39 +00:00
Seth RaphaelandClaude Opus 4.6 7b03a44a9c docs: clarify deprecated mutation is a manual fallback only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 13:37:59 -07:00
Seth RaphaelandClaude Opus 4.6 50715d6467 perf: split global stats recount into batched action to avoid bytes-read limit
The daily updateGlobalStatsInternal cron read all ~19K skillSearchDigest docs
(17.8 MB) in a single mutation, exceeding the Convex bytes-read limit.

Switch to an action-based approach that pages through the table in ~1000-doc
queries (each ~900 KB), then writes the result in a separate mutation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 13:37:59 -07:00
magicseth 0bb513c19a Merge pull request #836 from sethconvex/perf/denormalize-owner-into-digest
feat: add start/stop/status controls to digest owner backfill
2026-03-13 13:18:55 -07:00
Peter Steinberger 536c252310 fix: hide banned-owner skills from public surfaces 2026-03-13 20:07:06 +00:00
Seth RaphaelandClaude Opus 4.6 d1b2ca0a29 feat: add start/stop/status controls to digest owner backfill
Add delayMs param, stop flag via skillStatBackfillState, and status
query so backfill speed can be adjusted without redeploying.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 12:47:16 -07:00
magicseth 9730ba65bc Merge pull request #831 from sethconvex/perf/denormalize-owner-into-digest
perf: denormalize owner fields into skillSearchDigest
2026-03-13 12:25:20 -07:00
DangerouslyShipandClaude Opus 4.6 247580c456 fix: preserve owner profile data for handle-less visible users
digestToOwnerInfo now checks for profile data (name/displayName/image)
in addition to handle when deciding whether to return an owner object.
Handle-less visible users get their full profile; deactivated users
(no handle AND no profile data) correctly get owner: null.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 11:53:39 -07:00
DangerouslyShipandClaude Opus 4.6 114cca2049 fix: use empty string sentinel for handle-less owners in digest
Write ownerHandle: '' (not undefined) for visible users without a
handle and for deactivated users, so digestToOwnerInfo can distinguish
"not backfilled" (undefined → fallback to DB) from "backfilled but
no handle" ('' → use userId fallback, skip DB read).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 11:08:53 -07:00
DangerouslyShipandClaude Opus 4.6 928a65a29c fix: gate digest owner fields on deletedAt/deactivatedAt
Prevent baking deactivated/deleted user info into the digest.
The trigger and backfill now write undefined for owner fields
when the owner is not visible, matching the live query path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 10:52:24 -07:00
DangerouslyShipandClaude Opus 4.6 1a88a06948 perf: denormalize owner fields into skillSearchDigest to eliminate users reads
listPublicPageV2 (419 GB) and search.hydrateResults (1.76 TB) both read
full users docs for every unique owner. Denormalize ownerHandle, ownerName,
ownerDisplayName, ownerImage into the digest so query paths skip ctx.db.get
entirely. One extra read per skill mutation (rare) vs eliminating reads on
every query (very frequent).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 10:41:38 -07:00
Peter Steinberger 44638b73d4 chore(release): 0.8.0 2026-03-13 13:33:03 +00:00
Peter Steinberger db15896a6b ci: pin setup-bun to Node 24 action commit 2026-03-13 13:22:20 +00:00
Peter Steinberger bf160445dd ci: opt GitHub actions into Node 24 2026-03-13 13:19:08 +00:00
Peter Steinberger d2956bc64b test: fix lint and coverage compatibility after upgrades 2026-03-13 13:11:05 +00:00
Peter Steinberger 2486159e96 build(deps): update workspace dependencies and workflows 2026-03-13 13:11:05 +00:00
Nimrod Gutman b461dcb2bd fix(convex): avoid trending leaderboard read limit (#821)
* fix(convex): avoid trending leaderboard read limit

* test(convex): exercise trending cold start path

* docs: update convex query guidance

* fix(convex): preserve shim return contract
2026-03-13 14:57:03 +02:00
Peter Steinberger 1f474b68ce docs(changelog): reconstruct 0.7.0 release notes 2026-03-13 12:39:20 +00:00
Peter Steinberger e6ec1ec060 fix(ci): harden release verification 2026-03-13 12:39:20 +00:00
Peter Steinberger 1b038d55a2 fix(clawhub): inline packaged license exports 2026-03-13 12:39:04 +00:00
Nimrod Gutman f4fd8fe6f1 fix(convex): fix leaderboard test typecheck 2026-03-13 14:25:51 +02:00
e9f731b57f feat(api): add scan security verification endpoint and non-suspicious filters (#820)
* feat(api): add scan verification endpoint and non-suspicious filters

* fix(api): dedupe bool query parsing and backfill trending non-suspicious

* fix(api): preserve llm dimension warnings in security snapshot

* fix(api): clarify scan result semantics

* fix(api): clarify scan version context

* docs(api): clarify filtered pagination behavior

* fix(api): restore safe non-suspicious behavior

---------

Co-authored-by: VAC <vac@vacs-mac-mini.localdomain>
Co-authored-by: VACInc <3279061+VACInc@users.noreply.github.com>
2026-03-13 14:17:35 +02:00
Nimrod Gutman e2fb0355df fix(convex): align public skills test typing 2026-03-13 10:47:15 +02:00
Nimrod Gutman 4057431f63 fix(api): allow legacy cli publish payloads (#815) 2026-03-13 10:41:29 +02:00
Vincent Koc 0dcfa0be81 Merge pull request #793 from neeravmakwana/fix/public-skill-owner-leak
fix: sanitize public skill owners
2026-03-12 20:36:54 -04:00
Neerav Makwana 3348e87cb7 fix: sanitize public skill owners
Use the public user serializer for `skills.getBySlug` so Convex query responses no longer expose private account metadata like email addresses from public endpoints.

Made-with: Cursor
2026-03-12 20:19:13 -04:00
DangerouslyShipandClaude Opus 4.6 b32a499774 test: add coverage for fallback when latestVersionSummary is absent
Verifies that old digest rows without latestVersionSummary correctly
fall back to ctx.db.get(latestVersionId) for version data.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 08:28:44 -07:00
DangerouslyShipandClaude Opus 4.6 576588b70d perf: denormalize latestVersionSummary into skillSearchDigest
Eliminates ~9MB of skillVersions reads per listPublicPageV2 call by
copying latestVersionSummary from skills into the digest via the
existing trigger. Old rows without the field fall back to fetching
the full version doc.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 08:14:35 -07:00
d725a381d7 fix: allow ownership healing when previous owner is deleted/banned (#689)
* fix: allow ownership healing when previous owner is deleted/banned

The GitHub identity check in `publishOrUpdateSkillInternal` and
`checkSlugAvailability` was unreachable when the skill owner's account
was deleted or deactivated. This created a permanent deadlock: the
original owner could not sign in, and no one (not even the same GitHub
user with a new Convex Auth record) could reclaim the slug.

Move the `canHealSkillOwnershipByGitHubProviderAccountId` check before
the deleted/deactivated early-exit so ownership healing still works for
duplicate Convex Auth user records where the old record was later banned.

When healing is not possible (different GitHub identity or missing
auth records), show a message directing the user to contact
security@openclaw.ai instead of a generic "Slug is already taken".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: drop unused skills sort index map

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Nimrod Gutman <nimrod.gutman@gmail.com>
2026-03-12 11:53:52 +02:00
Nimrod Gutman 83f5e07f9d chore(convex): upgrade convex to 1.32.0 (#766) 2026-03-12 11:51:10 +02:00
a9556ee3f0 fix: surface auth errors from OAuth callback to the UI (#688)
* fix: surface auth errors from OAuth callback to the UI

When `afterUserCreatedOrUpdated` throws a `ConvexError` (e.g. banned or
deleted account), the error was silently discarded — the user was
redirected back to the sign-in page with no feedback.

Parse the error from the OAuth callback URL hash fragment in the
`ConvexAuthProvider` `replaceURL` callback and expose it via a
lightweight `useAuthError` hook (backed by `useSyncExternalStore`).
Display the error next to the sign-in button in both Header and the
CLI auth page, and add `.catch()` to `signIn()` calls to avoid
unhandled promise rejections.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(auth): handle oauth callback sign-in errors

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Nimrod Gutman <nimrod.gutman@gmail.com>
2026-03-12 11:33:47 +02:00
magicseth 041d8b4b92 Merge pull request #749 from sethconvex/fix/revert-nonsuspicious-index-query
fix: disable nonsuspicious index queries until backfill
2026-03-11 18:13:27 -07:00
DangerouslyShipandClaude Opus 4.6 2058a53c1d fix: disable nonsuspicious index queries until isSuspicious is backfilled
Most skillSearchDigest rows have isSuspicious: undefined (not false),
so eq('isSuspicious', false) returns zero results. Revert to regular
indexes with JS filtering until the field is backfilled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 18:12:12 -07:00
magicseth 123fecf04a Merge pull request #748 from sethconvex/fix/digest-nonsuspicious-indexes
fix: add nonsuspicious indexes to skillSearchDigest
2026-03-11 18:04:19 -07:00
DangerouslyShipandClaude Opus 4.6 b321025921 fix: use nonsuspicious indexes on skillSearchDigest for filtered pagination
Add by_nonsuspicious_* indexes to skillSearchDigest (matching the
existing ones on the skills table) so nonSuspiciousOnly filtering
happens at the index level instead of in JS. This eliminates empty
filtered pages without needing a multi-paginate loop.

TODO: once deployed and stable, remove the duplicate by_nonsuspicious_*
indexes from the skills table (no longer queried by listPublicPageV2).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 18:01:27 -07:00
magicseth c1115b1491 Merge pull request #746 from sethconvex/fix/listpublicpagev2-multi-paginate
fix: remove multi-paginate loop in listPublicPageV2
2026-03-11 17:51:19 -07:00
DangerouslyShipandClaude Opus 4.6 cd6403fec8 fix: remove multi-paginate loop in listPublicPageV2
Convex only allows a single .paginate() call per query function.
The while loop that skipped empty filtered pages violated this
constraint, causing "ran multiple paginated queries" errors on prod.

Remove the loop — clients will handle empty filtered pages by
requesting the next page via continueCursor.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 17:46:09 -07:00
magicseth b7528760b5 Merge pull request #743 from sethconvex/fix/trending-leaderboard-doc-limit
fix: split trending leaderboard rebuild to avoid 32K doc read limit
2026-03-11 15:55:38 -07:00
DangerouslyShipandClaude Opus 4.6 d47c774f8c fix: use Id<'skills'> cast instead of unsafe as any
Addresses review feedback from Greptile.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 15:42:21 -07:00
DangerouslyShipandClaude Opus 4.6 8bf9414387 fix: split trending leaderboard rebuild to avoid 32K document read limit
The rebuildTrendingLeaderboardInternal mutation queries ~31,500
skillDailyStats docs (7 days × ~4,500/day) in a single transaction,
hitting Convex's 32K document read limit on prod (71 errors/72h).

Split into action → query → mutation pattern so each day's query runs
in its own transaction with its own 32K budget:
- getDailyStats (internalQuery): reads one day's stats
- writeTrendingLeaderboard (internalMutation): writes leaderboard + prunes
- rebuildTrendingLeaderboardAction (internalAction): orchestrates the above

The old single-mutation path is kept as a fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 15:31:33 -07:00
magicseth 0c6c71d167 Merge pull request #741 from sethconvex/perf/lexical-fallback-digest
perf: switch listPublicPageV2 and countPublicSkills to skillSearchDigest
2026-03-11 14:33:21 -07:00
Shakker e3b80a848c fix: narrow moderation external override 2026-03-11 21:24:52 +00:00
Shakker 9b33abc0ea fix: harden moderation state reconciliation 2026-03-11 21:24:52 +00:00
Linfang Wang d68facae8e fix: reduce false-positive skill moderation flags and enable recovery
Problem:
Skills using legitimate API integrations (process.env + fetch) were
permanently flagged as malicious due to CREDENTIAL_HARVEST being
classified as a malicious-level reason code. Once flagged, skills could
not recover to normal status even after clean VT and OpenClaw scans,
because:
1. syncModerationReasons used a partial-update path that only patched
   moderationReason without reconciling moderationFlags, moderationStatus,
   or moderationVerdict.
2. Static scan "you are now a/an" regex over-matched common skill
   preambles, adding spurious INJECTION_INSTRUCTIONS flags.
3. No mechanism existed for external scanner results (VT/LLM) to
   override static suspicious findings when both independently
   confirmed the skill as safe.

Solution:
- Downgrade CREDENTIAL_HARVEST from malicious.env_harvesting to
  suspicious.env_credential_access — env+network is suspicious, not
  malicious, for API integration skills (moderationReasonCodes.ts).
- Remove "you are now a/an" regex from markdown scanning to stop
  false INJECTION_INSTRUCTIONS flags (moderationEngine.ts).
- Add external scanner override in buildModerationSnapshot: when both
  VT and LLM report clean/benign, demote suspicious.* static codes
  from verdict calculation while preserving malicious.* codes and
  keeping all findings in evidence for transparency (moderationEngine.ts).
- Route syncModerationReasons through approveSkillByHashInternal for
  rows with sha256hash, ensuring full moderation state reconciliation.
  For legacy no-hash rows: malicious → escalateSkillByIdInternal
  (immediate hide); clean/suspicious → updateSkillModerationReasonInternal
  (partial fix, matches pre-existing behavior) (vt.ts, skills.ts).
- Add escalateSkillByIdInternal mutation for atomic emergency
  escalation by skillId (sets moderationReason, moderationFlags,
  moderationStatus, hiddenAt, isSuspicious) (skills.ts).
- Ensure approveSkillByHashInternal explicitly hides malicious skills
  by setting moderationStatus to 'hidden' (skills.ts).
- Bump MODERATION_ENGINE_VERSION to v2.1.0.

Frontend:
- Add StaticAnalysisDetail component to display static scan findings
  with severity-aware styling (SkillSecurityScanResults.tsx).
- getStaticGuidance now accepts vtStatus/llmStatus and shows "Confirmed
  safe by external scanners" (benign/green) when both are clean, instead
  of always showing yellow "Patterns worth reviewing" for critical
  severity findings.
- Render SecurityScanResults and disclaimer when only static findings
  are present (SkillHeader.tsx).

Testing:
- 7 new unit tests in moderationEngine.test.ts covering:
  - CREDENTIAL_HARVEST downgrade (suspicious, not malicious)
  - "you are now" no longer flagged in markdown
  - "ignore previous instructions" still flagged
  - buildModerationSnapshot: VT+LLM clean demotes suspicious codes
  - buildModerationSnapshot: malicious codes preserved despite clean VT+LLM
  - Single-scanner-clean does not demote suspicious codes
  - VT suspicious + LLM clean does not demote suspicious codes
- All existing tests pass with engine version bump to v2.1.0.

Follow-up needed (not in this commit):
- One-time backfill for already-misflagged skills (cursor-based,
  re-run approveSkillByHashInternal on isSuspicious=true + clean VT).

Made-with: Cursor
2026-03-11 21:24:52 +00:00
DangerouslyShipandClaude Opus 4.6 73dfb7b2ba perf: switch listPublicPageV2 and countPublicSkills to skillSearchDigest
Both functions were hitting Bytes Read Limit errors scanning the full
skills table (~1.9KB/doc × 9K docs ≈ 17MB). Switch to the lightweight
skillSearchDigest table (~800 bytes/row) which carries all fields
needed by toPublicSkill/isPublicSkillDoc/isSkillSuspicious.

- Add 5 sort indexes to skillSearchDigest matching the ones used by
  SORT_INDEXES (by_active_created, by_active_name, by_active_stats_*)
- listPublicPageV2: query skillSearchDigest, map via digestToHydratableSkill
- countPublicSkillsForGlobalStats: query skillSearchDigest
- Widen buildPublicSkillEntries/filterPublicSkillPage to HydratableSkill[]
- Guard latestVersionSummary access (digest rows don't carry it)
- Update test mocks to expect skillSearchDigest table

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 13:46:36 -07:00
Nimrod Gutman e689a33a09 fix: resolve typescript errors blocking deploy 2026-03-11 21:21:39 +02:00
DangerouslyShipandClaude Opus 4.6 d68bcc43dd perf: use skillSearchDigest for lexical fallback scan
Switches the 500-row lexicalFallbackSkills scan from full skill docs
(~3-5KB each) to lightweight digest rows (~800 bytes each), reducing
DB read bandwidth by ~75%.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 12:15:04 -07:00
magicseth 9861da02d5 Merge pull request #735 from sethconvex/feat/skill-search-digest
perf: add skillSearchDigest table to reduce search hydration bandwidth
2026-03-11 12:12:57 -07:00
DangerouslyShip 94c805d0f5 Merge remote-tracking branch 'origin/main' into feat/skill-search-digest
# Conflicts:
#	convex/skills.ts
2026-03-11 12:02:39 -07:00
Nimrod Gutman 487ecb3890 Merge pull request #682 from openclaw/feat/moderation-override-audit-tools
feat(moderation): add manual override audit tools
2026-03-11 20:46:53 +02:00
DangerouslyShipandClaude Opus 4.6 869c45b5c0 perf: track all attempted embedding IDs to avoid redundant hydration
Build the seen-ID set from vector results rather than only successful
hydrations, so soft-deleted and suspicious embeddings aren't re-hydrated
on each candidate-limit expansion loop iteration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 11:34:51 -07:00
DangerouslyShipandClaude Opus 4.6 5433c66200 refactor: adopt convex-helpers Triggers for automatic digest sync
Replace ~28 manual syncSkillSearchDigest/upsertSkillSearchDigest calls
across 4 files with a single Triggers handler in convex/functions.ts
that fires automatically on every skills table write. This eliminates
the risk of new mutations silently breaking digest consistency.

- Add convex-helpers as direct dependency
- Create convex/functions.ts wrapping mutation/internalMutation with triggers
- Update all 39 convex modules to import from ./functions
- Remove syncSkillSearchDigest from lib (no longer needed)
- Add normalizeId mock to test db objects for trigger wrapper compat

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 11:07:52 -07:00
DangerouslyShipandClaude Opus 4.6 c74419d834 fix: add missing maintenance.ts sync hooks, type-safe digest hydration
- Add syncSkillSearchDigest calls to 6 maintenance mutations that patch
  digest-relevant fields without syncing (applyEmptySkillCleanup,
  applySkillBadgeBackfillPatch, upsertSkillBadgeRecord,
  backfillDenormalizedBadges, backfillIsSuspicious, applySkillBackfillPatch)
- Replace unsafe `as unknown as Doc<'skills'>` cast with typed
  HydratableSkill interface and digestToHydratableSkill mapper — compiler
  now catches field drift between digest and skill doc
- DRY up extractDigestFields/digestToHydratableSkill with shared
  SHARED_KEYS array and pick() helper

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 10:16:43 -07:00
DangerouslyShipandClaude Opus 4.6 e2c48d893c fix: clean up orphaned digest rows on hard-delete and fix reclaim test mock
When a skill is hard-deleted, syncSkillSearchDigest now removes the
corresponding digest row instead of silently no-oping. Also adds
skillSearchDigest table handling to reclaim test mock.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 10:09:22 -07:00
DangerouslyShipandClaude Opus 4.6 b360de5291 refactor: extract shared validators between skills and skillSearchDigest tables
DRY up duplicated validator definitions (forkOf, badges, stats, moderationStatus)
into shared constants reused by both tables to prevent schema drift.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 10:02:53 -07:00
DangerouslyShipandClaude Opus 4.6 ed2ecab0a2 fix: add missing digest sync hooks and address PR review feedback
- Remove redundant digest write for new skills (Greptile review)
- Add sync to hardDeleteSkillStep init/canonical/forks phases (Codex review)
- Add sync to patchStructuredModerationFromVersion (LLM analysis path)
- Add sync to report mutation (auto-hide path)
- Add sync to applyBanToOwnedSkillsBatchInternal (bulk ban)
- Add sync to restoreOwnedSkillsForUnbanBatchInternal (bulk unban)
- Add sync to transferSkillOwnershipAndEmbeddings (slug reclaim)
- Add sync to setSkillSoftDeletedInternal (internal soft-delete)
- Make digest test fixture derive from makeSkillDoc to avoid fragility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 09:58:44 -07:00
DangerouslyShipandClaude Opus 4.6 872045681b perf: add skillSearchDigest table to reduce search hydration bandwidth
hydrateResults reads full skill docs (~3-5KB each) but only needs ~800
bytes for toPublicSkill/isPublicSkillDoc/isSkillSuspicious. Add a
lightweight skillSearchDigest projection table that is kept in sync by
all skill mutation paths.

Also fix the searchSkills while loop to incrementally hydrate only new
embedding IDs on each expansion instead of re-hydrating all candidates
from scratch (475 → 250 reads per search).

Expected impact: ~7x bandwidth reduction for hydrateResults
(495 GB → ~70 GB at current traffic).

Post-deploy: npx convex run maintenance:backfillSkillSearchDigestInternal --prod

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 09:34:55 -07:00
Nimrod Gutman 2528c1c35a test(skills): align public list pagination assertions 2026-03-11 11:03:59 +02:00
Nimrod Gutman e93f9411f3 fix(moderation): tighten override safety guards 2026-03-11 10:49:09 +02:00
Nimrod Gutman 4049a3b58a feat(moderation): add manual override audit tools 2026-03-11 10:32:18 +02:00
magicseth 6318a74adf Merge pull request #709 from sethconvex/fix/pre-existing-type-errors
fix: resolve type errors and update OG image branding
2026-03-10 23:42:05 -07:00
DangerouslyShipandClaude Opus 4.6 e7101f155e fix: shrink OG subtitle to fit within card bounds
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 23:38:20 -07:00
DangerouslyShipandClaude Opus 4.6 0f1c7536ba fix: regenerate og.png with correct ClawHub branding
The static OG image still said "ClawdHub" and "clawdhub.com" — regenerated
from the already-correct og.svg source.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 23:36:25 -07:00
DangerouslyShipandClaude Opus 4.6 dc89ab643e fix: resolve pre-existing type errors in test files
- Add missing sha256 field to file mocks in moderation.test.ts
- Accept softDeletedAt param in makeSkillDoc in search.test.ts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 23:24:16 -07:00
magicseth 0a400437ff Merge pull request #697 from sethconvex/fix/nonsuspicious-index-path
fix: use nonsuspicious indexes in listPublicPageV2

Should reduce bandwidth significantly
2026-03-10 23:20:50 -07:00
DangerouslyShipandClaude Opus 4.6 023a01f411 fix: use nonsuspicious index for combined highlightedOnly + nonSuspiciousOnly
When both filters are active, use the nonsuspicious index for isSuspicious
and apply highlightedOnly as a JS filter on top, instead of scanning the
full table with both filters in JS.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 21:32:46 -07:00
DangerouslyShipandClaude Opus 4.6 2c42bf9900 fix: remove backfill fallback — isSuspicious backfill is complete
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 12:26:35 -07:00
DangerouslyShipandClaude Opus 4.6 523b65e443 fix: use nonsuspicious indexes in listPublicPageV2 to avoid full table scans
Restore the NONSUSPICIOUS_SORT_INDEXES map and index branching logic that
was lost during the PR #572 merge. When nonSuspiciousOnly is set, queries
now use by_nonsuspicious_* indexes with isSuspicious=false in the predicate
instead of scanning the full table and filtering in JS — eliminating
bytesReadLimit errors under load.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 12:12:44 -07:00
Peter Steinberger e07198ad41 test: cover key site workflows in playwright 2026-03-09 05:18:18 +00:00
Peter Steinberger be6761526a test: strengthen playwright smoke error assertions 2026-03-09 05:18:15 +00:00
Peter Steinberger 72b6c5ede6 fix: fail deploy workflow clearly without secrets 2026-03-09 05:10:27 +00:00
Peter Steinberger 8dddcea5c4 fix: unblock deploy workflow smoke gate 2026-03-09 05:08:55 +00:00
Peter Steinberger 0e8c00a8eb fix: stabilize deployment drift query subscription 2026-03-09 04:27:25 +00:00
Peter Steinberger 0aa702fa70 fix: tolerate missing deployment info query 2026-03-09 04:23:38 +00:00
Ayaan Zaidi 4d72506b1c fix: isolate deployment drift banner failures 2026-03-09 09:52:04 +05:30
Peter Steinberger 2be9b67e74 ci: harden deploy pipeline against web/backend drift 2026-03-08 21:58:37 +00:00
Peter Steinberger c617ef124a test: fix timeout mock typing 2026-03-08 03:36:51 +00:00
Peter Steinberger 114e480388 fix: expose structured moderation API (#334) (thanks @ArthurzKV) 2026-03-08 03:18:57 +00:00
Peter Steinberger e31a8e9d32 fix: add structured moderation snapshots (#333) (thanks @ArthurzKV) 2026-03-08 03:13:13 +00:00
Peter Steinberger 460ad3c13d feat: add skill transfer API and CLI 2026-03-07 22:51:49 +00:00
Peter Steinberger 2687d671a0 feat: enforce MIT-0 skill licensing 2026-03-07 22:46:28 +00:00
Peter Steinberger deb216e3b4 docs: fix explore flag list formatting (#601) (thanks @gandli) 2026-03-07 21:15:53 +00:00
gandli e122569d2c docs(cli): fix indentation of --limit flag in explore command
The --limit flag under the 'explore' command's Flags section was
missing the proper two-space indentation, making it inconsistent
with other flag lists in the document.
2026-03-07 21:15:53 +00:00
Peter Steinberger 65649bc032 docs: clarify local dev setup workflow (#584) (thanks @jack-piplabs) 2026-03-07 21:14:45 +00:00
Jack Chan cf59b41790 docs: fix local dev setup instructions in CONTRIBUTING.md
- Add Node.js v18/20/22/24 prerequisite (Convex backend rejects v25+)
- Remove duplicate CONVEX_SITE_URL from .env.local example
- Reorder steps so Convex backend starts before auth/JWT setup
- Add "Set backend environment variables" section (bunx convex env set)
- Clarify that AUTH_GITHUB_ID/SECRET and SITE_URL must be set on the
  Convex backend, not just in .env.local
- Make frontend port explicit (bun run dev -- --port 3000)
- Add updateGlobalStatsInternal step after seeding
2026-03-07 21:14:45 +00:00
Peter Steinberger 09054bb053 fix: add soft-delete search regression coverage (#552) (thanks @MunemHashmi) 2026-03-07 21:12:55 +00:00
Munem Hashmi ea0b14dca5 test(search): add soft-delete filtering tests for vector and lexical paths (#29)
Verify that soft-deleted skills are excluded from both vector search
hydration and lexical fallback exact-slug matching.
2026-03-07 21:12:55 +00:00
Peter Steinberger 4a80758357 fix: update manifest branding changelog (#569) (thanks @Glucksberg) 2026-03-07 18:44:11 +00:00
Glucksberg efd0f50d56 fix: update manifest.json with correct app name
Updates manifest.json from TanStack defaults to ClawHub branding.
This fixes the app name shown when installing as PWA.
2026-03-07 18:44:11 +00:00
Peter Steinberger a45c5b91f3 fix: stabilize browse pagination during safety backfill (#572) (thanks @sethconvex) 2026-03-07 18:42:53 +00:00
Seth RaphaelandClaude Opus 4.6 fc22bfb2a1 fix: keep pagination cursor on same index path during backfill fallback
The nonsuspicious index fallback now fires on any page (not just the
first) and reuses the client's cursor via stale-cursor recovery. This
prevents pagination from breaking when a SORT_INDEXES cursor is sent
back to the NONSUSPICIOUS_SORT_INDEXES path on page 2+.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:42:53 +00:00
Seth RaphaelandClaude Opus 4.6 2152879cd7 fix: address PR review comments for bandwidth reduction
- Fix P1: remove !result.isDone guard from listPublicPageV2 backfill
  fallback so it fires when the nonsuspicious index is empty (isDone=true)
- Fix updateTags to update latestVersionSummary when repointing latest tag
- Parallelize leaderboard daily queries with Promise.all
- Over-fetch stale-reason candidates (2x limit) before VT filtering
- Reconcile existing latestVersionSummary in backfill instead of skipping
- Add _creationTime approximation comment
- Rebuild schema dist to include author field on ClawdisSkillMetadata

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:42:53 +00:00
Seth RaphaelandClaude Opus 4.6 f8d57f7c70 fix(schema): add compile-time guard for ClawdisSkillMetadata drift
The explicit ClawdisSkillMetadata interface (needed because ArkType's
[inferred] doesn't resolve all fields) now has a keyof-based type guard
that triggers a compile error if the interface keys drift from the schema.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:42:53 +00:00
Seth RaphaelandClaude Opus 4.6 a3fe5cbc43 fix: resolve all 26 pre-existing typecheck errors
- Replace ArkType `[inferred]` type alias for ClawdisSkillMetadata with
  an explicit interface so TS can see envVars/dependencies/author/links
- Extract listBySkillHandler from comments.ts so tests can call it
  directly without accessing private _handler property
- Rebuild packages/schema dist output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:42:53 +00:00
Seth RaphaelandClaude Opus 4.6 a90608d240 docs: use convex CLI for insights instead of MCP
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:42:53 +00:00
Seth RaphaelandClaude Opus 4.6 8d938e3b44 docs: tell agents to check Convex insights before writing queries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:42:53 +00:00
Seth RaphaelandClaude Opus 4.6 7b7e2b3dc4 perf: reduce DB bandwidth ~1.5 TB/day with indexes, denormalization, and query rewrites
Phase 1: Add `by_moderation` compound index and rewrite 8 cron query
functions to use `.withIndex()` instead of `.filter().collect()` full
table scans (~6 GB/day saved).

Phase 2: Denormalize `isSuspicious` onto skills table with 6 compound
indexes so `listPublicPageV2` can filter at the index level instead of
paginating the entire table (~1 TB/day saved). Includes backfill
mutation and write-path updates across all moderation mutations.

Phase 3: Add `latestVersionSummary` denormalization to avoid reading
full ~6.4 KB `skillVersions` docs on list pages (~500 GB/day saved).

Phase 4: Split trending leaderboard query to one day at a time to stay
under 32K doc limit. Reduce global stats recount from hourly to daily
since delta tracking handles real-time accuracy (~400 MB/day saved).

Phase 5: Add "Convex Query & Bandwidth Rules" section to AGENTS.md.

Backfill commands (run after deploy):
  bunx convex run maintenance:backfillIsSuspiciousInternal
  bunx convex run maintenance:backfillLatestVersionSummaryInternal

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:42:53 +00:00
Peter Steinberger 8f2c86a878 fix: relax moderation false positives for auth skills (#273) (thanks @superlowburn) 2026-03-07 18:37:44 +00:00
SteveandClaude Opus 4.6 06a528c5d9 fix: resolve biome lint errors in moderation test
- Fix import ordering (alphabetical: describe, expect, test)
- Add biome-ignore comments for test mock `as any` casts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:37:44 +00:00
SteveandClaude Opus 4.6 bb1636f255 fix: reduce false positives in suspicious pattern detection for OAuth skills
Fixes #209 by removing overly broad regex patterns that flag legitimate
authentication and payment integration skills.

## Problem

Skills like openbotauth (OAuth identity verification) were being flagged
as suspicious because they mention "token", "api key", or "password" in
their description or metadata. The regex scanner was too aggressive,
catching legitimate auth flows alongside actual threats.

## Solution

Removed three overly broad patterns:
- `suspicious.secrets` - flagged ANY mention of token/api key/password
- `suspicious.crypto` - flagged ANY mention of wallet/seed phrase/crypto

These are common in legitimate skills:
- OAuth skills mention "token" for authentication flows
- API integrations mention "api key" for service credentials
- Database skills mention "password" for connections
- Crypto wallet skills mention "seed phrase" for key management

The LLM evaluator already handles credential proportionality analysis
(section 4 of security prompt). The regex scan should only catch
ACTUAL malicious patterns, not keywords that appear in legitimate contexts.

## What Still Gets Flagged

Kept patterns that catch real threats:
- `suspicious.keyword` - malware, stealer, phishing, keylogger
- `suspicious.webhook` - discord/slack webhooks (data exfiltration)
- `suspicious.script` - curl | bash (arbitrary code execution)
- `suspicious.url_shortener` - bit.ly etc (URL obfuscation)

## Testing

- Added 18 comprehensive tests for pattern detection
- Verified OAuth skills (openbotauth, trello) are NOT flagged
- Verified malicious patterns ARE still flagged
- All 418 existing tests pass

## Security Impact

This does NOT weaken security:
- LLM evaluator still analyzes credential proportionality
- Actual malicious patterns (webhooks, curl|bash, etc) still caught
- Only removes false positives on legitimate auth keywords

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 18:37:44 +00:00
Neerav Makwana 619489a93b fix: debounce URL navigation in skills search to reduce input lag
onQueryChange called navigate() on every keystroke to sync the query
to the URL. Each navigate triggers history.replaceState, TanStack
Router re-evaluation, and useSearch() invalidation, causing multiple
re-renders per keystroke.

Keep setQuery() immediate so the controlled input stays responsive,
but debounce the navigate() call at 220 ms (matching the existing
search-action debounce). Cancel the pending timer when search.q
changes externally (browser back/forward) to prevent the debounced
navigate from overwriting the external URL change.

Made-with: Cursor
2026-03-07 18:35:10 +00:00
Munem Hashmi 0f0086591c fix(ui): persist folder upload input across hydration and re-renders (#58)
Replace the useEffect + useRef approach for setting webkitdirectory/
directory attributes with a ref callback that sets the attributes
every time the input element is mounted. This ensures folder selection
mode persists after page refresh, where React hydration could strip
the non-standard attributes.

Also removes the @ts-expect-error JSX props since the attributes are
now set imperatively via the ref callback.
2026-03-07 18:33:03 +00:00
Peter Steinberger 531dcc8d26 fix: avoid auth crash in slug availability preflight 2026-03-07 18:29:01 +00:00
Tristan Manchester b1c710e1ea fix: add dedicated slug availability preflight 2026-03-07 15:38:01 +00:00
Tristan Manchester a4a9fc62bc fix: address review comments for slug-collision error handling 2026-03-07 15:38:01 +00:00
Tristan Manchester e58fbc8d1f fix: surface slug-collision publish errors and block conflicts preflight 2026-03-07 15:38:01 +00:00
Peter Steinberger b4b4f266a8 fix: align VT engine fallback verdict mapping (#591) (thanks @Shuai-DaiDai) 2026-03-07 15:33:30 +00:00
帅小呆1号 f9d35cc5e6 fix(vt): sync scan status from AV engines when Code Insight unavailable
When VirusTotal returns scan results with AV engine stats but no Code Insight
AI analysis, the skill status was stuck on 'Pending'. This fix adds fallback
logic to check last_analysis_stats (malicious/suspicious/harmless/undetected)
to determine scan status.

Functions updated:
- pollPendingScans: Check AV engines before requesting rescan
- backfillPendingScans: Check AV engines before marking as no results
- rescanActiveSkills: Check AV engines before keeping as pending
- backfillActiveSkillsVTCache: Check AV engines before skipping

Fixes #33435
2026-03-07 15:33:30 +00:00
Peter Steinberger 1892f72a13 test: lock multipart upload timeout behavior (#550) (thanks @MunemHashmi) 2026-03-07 15:31:20 +00:00
Munem Hashmi fdbc184e0a fix(cli): improve publish timeout handling and error messages (#533)
- Increase upload timeout from 15s to 120s for multipart form uploads
  (apiRequestForm and curl-based form upload). Regular API requests
  remain at 15s.
- Improve timeout error message from bare "Timeout" to
  "Request timed out after Ns" so users know what happened.
- Normalize non-Error throws (e.g. DOMException from AbortController
  across runtimes) into proper Error instances, preventing the
  misleading "Non-error was thrown" message from p-retry.
- Preserve the original error as `cause` on the wrapped Error.
2026-03-07 15:31:20 +00:00
Peter Steinberger 18cbfc6788 test: cover auth token forwarding for search/explore (#608) (thanks @artdaal) 2026-03-07 15:29:37 +00:00
Артемов Даниил Алексеевич c821b0ffc4 fix: pass auth token in search and explore commands
cmdSearch and cmdExplore were not calling getOptionalAuthToken()
and did not pass the token to apiRequest, unlike install/update/uninstall.
This caused 'missing API token' errors on registries that require auth
(e.g. private Hermit instances).
2026-03-07 15:29:37 +00:00
Peter Steinberger 9833a5038d docs: note top-level frontmatter metadata parsing fix (#548) (thanks @MunemHashmi) 2026-03-07 15:28:12 +00:00
Munem Hashmi 40a89e02d5 fix: extract requires.env and homepage from top-level frontmatter (#522)
parseFrontmatterLevelDeclarations did not handle the requires block
(env, bins, anyBins, config) or primaryEnv when declared at the
top level of SKILL.md frontmatter without a metadata.openclaw wrapper.
This caused the security scanner to always show "Required env vars: none"
for skills using that format, triggering false-positive suspicious flags.

Also extends the evalCtx.homepage fallback chain to check
clawdis.homepage and clawdis.links.homepage so skills declaring
homepage inside the metadata block are picked up by the scanner.
2026-03-07 15:28:12 +00:00
Timothy JordanandClaude Opus 4.6 bc06dbffd0 chore: add Vercel attribution in footer (#557)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 14:06:35 +00:00
Agent f5fa23e0c1 chore: add convex attribution in footer 2026-02-27 23:10:26 +01:00
Vincent Koc e8c3947b21 Merge pull request #547 from openclaw/fix/secret-scan-trufflehog-ref
fix(ci): restore secret scan action reference
2026-02-27 10:39:18 -08:00
Vincent Koc ef26ee0d1f fix(ci): pin trufflehog to published v3.93.6 tag 2026-02-27 10:38:31 -08:00
Vincent Koc 3d006ec663 fix(ci): use resolvable trufflehog action ref 2026-02-27 09:47:17 -08:00
Peter Steinberger 52590e84dd chore: commit local pending changes 2026-02-26 13:03:25 +01:00
Peter Steinberger e3a1c95851 docs(agents): reject skill-in-source PRs; require CLI publish 2026-02-26 13:03:25 +01:00
Mahsum AktaşandPeter Steinberger db4540743f feat(registry): support env vars, dependencies, author, and links in skill manifest (#360)
* feat(registry): support env vars, dependencies, author, and links in skill manifest

Closes #350

Add structured declarations for environment variables, package
dependencies, author identity, and project links to the skill
registry manifest. These fields can be declared in the clawdis
metadata block or as top-level frontmatter keys.

Changes:
- schema: add EnvVarDeclaration, DependencyDeclaration, SkillLinks
  types to ClawdisSkillMetadata
- parser: extract envVars, dependencies, author, links from both
  clawdis block and top-level frontmatter (fallback for skills
  without a clawdis block)
- UI: render env vars with required/optional badges and descriptions,
  dependencies with type/version/links, and project links in the
  skill detail page install card
- security: update evaluator prompt to recognize envVars alongside
  requires.env and primaryEnv
- tests: 7 new test cases covering all declaration formats

* fix(ui): handle unspecified env required state and stable keys

* docs(changelog): credit metadata manifest expansion (#360) (thanks @mahsumaktas)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-26 12:02:25 +00:00
David Abutbulgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>Peter Steinberger
0cb0963c2b feat(api): expose security evaluation results (#362)
* feat(api): expose security evaluation results

- Add security field to skill version API responses
- Map llmAnalysis database field to public API format
- Display security info in CLI inspect command
- Enable security tools like clawsec-clawhub-checker to access internal security checks

Security field includes:
- status: clean|suspicious|malicious|pending|error
- hasWarnings: boolean
- checkedAt: timestamp
- model: evaluation model name

Backward compatible: optional field, no breaking changes.

* fix: ensure hasWarnings is always boolean

- Add ?? false to coerce undefined to false when dimensions is undefined
- Fixes Greptile comment: hasWarnings can be undefined instead of boolean
- Ensures SecurityStatusSchema validation passes on client side

* Update convex/httpApiV1/skillsV1.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(api-cli): harden security inspect output + tests (#362) (thanks @abutbul)

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-26 12:01:23 +00:00
David AronchickandPeter Steinberger beae065794 fix(cli): handle missing browser opener gracefully (#163)
* fix(cli): handle missing browser opener gracefully

On headless Linux servers without xdg-open, 'clawhub login' crashes with
ENOENT error. This change catches the error and prints the URL for manual
copy-paste instead of crashing.

Fixes crash on:
- VPS/cloud servers
- Docker containers
- CI environments
- WSL without browser integration

* fix(cli): test browser-opener fallback messaging (#163) (thanks @aronchick)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-26 11:59:24 +00:00
46c5637dae Improve error handling in GitHub import (#512)
* Improve error handling in GitHub import

- Add detailed error messages for storage failures
- Wrap publishVersionForUser in try/catch with helpful context
- Include file size and path in storage error messages
- Guide users to check skill format and slug availability

* fix(github-import): improve failure messaging + coverage (#512) (thanks @vassiliylakhonin)

---------

Co-authored-by: Vassiliy Lakhonin <vassiliy.lakhonin@example.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-26 11:30:01 +00:00
Tristan Manchester 2217a327e7 fix(upload): ignore macOS junk files during publish (#526) 2026-02-26 11:28:27 +00:00
45d8f0d217 feat: surface platform/architecture labels on skill cards and API (#499)
* feat: surface platform/architecture labels on skill cards and API

Expose existing `os` and `nix.systems` metadata from skill frontmatter
through the HTTP API and render as compact tags on browse/search views.

- Widen `PublicSkillListVersion` and `SkillListEntry` types to include
  `os` and `nix.systems` fields (data already flows through, types were
  artificially narrow)
- Add `metadata: { os, systems }` to `/api/v1/skills/{slug}` and
  `/api/v1/skills` list responses
- Add `formatSystemsList` and `getPlatformLabels` helpers to map nix
  system strings to human-readable labels (e.g. aarch64-darwin → macOS ARM64)
- Add `platformLabels` prop to `SkillCard`, render as `.tag .tag-compact`
- Show platform labels in both card grid and list views
- Update HTTP API docs with new `metadata` field

Coded by Claude Opus 4.6 (Claude Code)
Reviewed and tested by Jason (@asyncjason)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove redundant optional chaining on clawdis

Address greptile-apps review comment — clawdis is already confirmed
truthy by the ternary condition, so `?.` is unnecessary.

Coded by Claude Opus 4.6 (Claude Code)
Reviewed and tested by Jason (@asyncjason)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: include version data in listPublicPageV2 for platform labels

The browse listing passed includeVersion: false, causing latestVersion
to always be null and platform/arch labels to never render.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Jason Separovic <jason@wilma.dog>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 11:27:20 +00:00
Nicolas GreniéandClaude Opus 4.6 add5d83014 docs: add CONTRIBUTING.md and refresh README header (#400)
* docs: add CONTRIBUTING.md and refresh README header

Add a comprehensive CONTRIBUTING.md covering local Convex setup,
env var configuration, GitHub OAuth, JWT keys, database seeding,
CLI development, PR guidelines, and AI-generated code policy.

Refresh the README with a centered logo, quick links row, and
clickable doc references. Condense the Local dev section to link
to CONTRIBUTING.md for full setup details.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add #clawhub discord channel

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 11:25:51 +00:00
Abdul B.andCursor 9751199231 fix: prevent filtered skills pagination flicker (#372)
* fix: prevent filtered skills pagination flicker

Skip fully filtered-out pages in public skills pagination so highlighted/non-suspicious filtering doesn't return empty pages with more cursor state, which caused repeated loading-more flicker.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: rely on inferred Convex paginate result type

Remove the custom runPaginate annotation so TypeScript infers the exact Convex paginate result shape and preserves stronger type-safety.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-26 11:25:15 +00:00
Luke 883221f8ec fix(cli): clarify owner delete permissions in command text (#417) 2026-02-26 11:23:53 +00:00
Peter Steinberger 3e45d67e0d fix: delete and hide comments from banned users 2026-02-26 05:52:09 +01:00
Peter Steinberger df346aeea9 feat: require 14-day GitHub age for publish and comments 2026-02-26 05:35:44 +01:00
Peter Steinberger 4317369480 fix: stabilize comment moderation tests without env keys 2026-02-26 02:18:00 +01:00
Peter Steinberger e04d16bdae feat: add ai comment scam backfill and auto-ban flow 2026-02-26 02:16:31 +01:00
Peter Steinberger cb66d8d6f3 feat: add abuse-resistant comment reporting 2026-02-26 01:28:22 +01:00
Peter Steinberger 14a2fa80f6 test: lock 5xx retry behavior in HTTP client (#457) (thanks @YonghaoZhao722) 2026-02-25 13:03:16 +00:00
Peter Steinberger ee788b7af3 test: pin Retry-After relative-delay behavior (#421) (thanks @apoorvdarshan) 2026-02-25 12:19:10 +00:00
Apoorv Darshan 3956ca7e55 fix: use relative delay for Retry-After header on 429 responses
Retry-After was set to an absolute Unix epoch timestamp (e.g. 1771404540),
which violates RFC 9110 §10.2.3. Clients treating it as delay-seconds
would wait ~56 years. Now emits the actual seconds until reset.

Closes #407
2026-02-25 12:19:10 +00:00
Peter Steinberger bc37ec7156 fix: complete registry-url migration with test coverage (#486) (thanks @Liknox) 2026-02-25 12:17:30 +00:00
Nazar Koval f07408eb81 test: url formatter 2026-02-25 12:17:30 +00:00
Nazar Koval cdf5baef7f ref: url entity utilization 2026-02-25 12:17:30 +00:00
Nazar Koval 65b154f36c feat: url formatter entity 2026-02-25 12:17:30 +00:00
Peter Steinberger 6ea7a0792d fix: finalize proxy env support + changelog credits (#363) (thanks @kerrypotter) 2026-02-25 12:14:00 +00:00
Jarvis ed961e459f fix: use EnvHttpProxyAgent for proper proxy support
Address review feedback:
- Use undici's EnvHttpProxyAgent instead of ProxyAgent. This properly
  handles HTTPS_PROXY vs HTTP_PROXY per-scheme, respects NO_PROXY,
  and uses connect.timeout instead of requestTls.
- Update docs to mention NO_PROXY support.
2026-02-25 12:14:00 +00:00
Jarvis 8b5f242f73 fix: respect HTTP_PROXY/HTTPS_PROXY environment variables
The CLI creates a custom undici Agent via setGlobalDispatcher() which
overrides any proxy configuration. Since Node.js native fetch (backed
by undici) does not automatically respect HTTP_PROXY/HTTPS_PROXY env
vars, the CLI fails with 'fetch failed' on systems that require a
proxy for outbound connections.

Import ProxyAgent from undici and use it when any of the standard proxy
environment variables (HTTPS_PROXY, HTTP_PROXY, https_proxy, http_proxy)
is set. When no proxy variable is present, behavior is unchanged.

Also adds proxy documentation to cli.md and a troubleshooting entry.
2026-02-25 12:14:00 +00:00
Phineas1500 311bf1a88a fix: guard de-escalation on clean status and fix alreadyFlagged blocking clean verdicts
Address review feedback:
- Guard rescan de-escalation with `status === 'clean'` so pending/unknown
  verdicts don't accidentally clear the suspicious flag
- Fix approveSkillByHashInternal where `alreadyFlagged` in the condition
  `(isSuspicious || alreadyFlagged) && !bypassSuspicious` prevented clean
  verdicts from reaching the isClean branch that properly checks whether
  a different scanner set the flag
2026-02-25 12:09:41 +00:00
Phineas1500 8343f0bb23 fix: clear stale suspicious flag when VT verdict improves to clean
The daily VT rescan updated vtAnalysis on the version but only called
escalateByVtInternal for suspicious/malicious verdicts. When a verdict
improved from suspicious to clean, the version's vtAnalysis was updated
(website shows "Benign") but the skill's moderationFlags kept the stale
"flagged.suspicious" entry (CLI warns "suspicious"). Now the rescan
calls approveSkillByHashInternal to clear the flag on de-escalation.
2026-02-25 12:09:41 +00:00
Peter Steinberger 3b73a09d36 fix: keep denormalized badge reads consistent (#441) (thanks @sethconvex) 2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 412249d2d1 fix: use destructuring instead of undefined assignment in removeSkillBadge
Avoids leaving an explicit undefined key in the badges object which
could fail Convex validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 480125d859 feat: add backfill for denormalized skill badges, clean up fallback
- Add backfillDenormalizedBadgesInternal: syncs skillBadges table →
  skill.badges field so listing/search reads are correct
- Simplify hydrateResults fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 0ab1d1e051 fix: restore fallback in hydrateResults for un-backfilled embeddings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 85374fa44b perf: eliminate redundant badge reads and add embedding lookup table
- Remove badge table queries from listing and search paths (~200 queries
  per page load eliminated). Use denormalized skill.badges field instead.
- Sync skill.badges when badges are mutated (upsertSkillBadge/removeSkillBadge).
- Add embeddingSkillMap lookup table (~100 bytes/doc) so search hydration
  can skip reading full skillEmbeddings docs (~12KB each with vector).
- Remove dead badge query exports from search module.
- Reduce lexical fallback scan limit from 1200 to 500.
- Add backfill mutation for embeddingSkillMap with graceful fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 c319e46c8b perf: cap badge query to 10 records per skill
Skills should never have more than a handful of badge records.
Using .take(10) instead of .collect() avoids unbounded reads.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
Seth RaphaelandClaude Opus 4.6 f17087d1f4 perf: reduce db bandwidth by splitting stat processing into two frequencies
The 5-minute stat event processor was patching skill documents on every run,
which invalidated listPublicPageV2 reactive queries for ALL subscribers —
causing a thundering herd responsible for ~17 TB (59%) of the 28.65 TB
monthly db bandwidth.

Split into two paths:
- Daily stats (15-min cron): writes to skillDailyStats only, no skill doc patches
- Skill doc sync (6-hour cron): patches skill documents with accumulated deltas

Also skip reading version docs in listPublicPageV2 and search hydration
(version data is only needed on detail pages, not listings).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:00:34 +00:00
LK da4469e1e0 fix(rate-limit): skip ip consumption for authenticated requests 2026-02-25 02:56:20 +00:00
LK 42a4648475 refactor(http): parse rate-limit headers once per error 2026-02-25 02:56:20 +00:00
Lukeandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> edc8ec274b Update docs/http-api.md
fix header precedence in docs to match code

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-02-25 02:56:20 +00:00
LK 15b1a05fee fix(rate-limit): make limiter auth-aware for shared proxy ips 2026-02-25 02:56:20 +00:00
LK 1e216c03c6 fix(rate-limit): standardize retry-after and improve 429 ux 2026-02-25 02:56:20 +00:00
ɐʞsǝs 0a0b2e6cb1 pin the th version for better stability / security 2026-02-25 02:50:17 +00:00
ɐʞsǝsandgreptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> ac6770acff Update .github/workflows/secret-scan.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-02-25 02:50:17 +00:00
ɐʞsǝs bfc87e5932 refined message to give better guidance 2026-02-25 02:50:17 +00:00
ɐʞsǝs 3e1bd19a45 Add secret scanning workflow using TruffleHog
This ensures that a given PR will immediately error on vulnerable, live credentials found by trufflehog
2026-02-25 02:50:17 +00:00
Peter Steinberger d71b747d1c fix: harden VT fallback activation rules (#300) (thanks @superlowburn) 2026-02-25 02:48:57 +00:00
Steve 4d211fcf73 fix: check moderation reason before activating skills
- Prevent activating skills with quality.low moderation reason
- Add skill lookup and moderationReason check in 3 locations where skills are activated
- This ensures quality gate quarantine is not bypassed when VT scan is unavailable or stale

Resolves review comments on #300
2026-02-25 02:48:57 +00:00
SteveandClaude Opus 4.6 bc5ab8f3e1 fix: activate skills when VT scan is unavailable or stales out
Published skills stay permanently hidden in search when VirusTotal
cannot produce a verdict. Three code paths leave moderationStatus as
'hidden' with no recovery:

1. VT_API_KEY not configured — scan skipped, skill stays hidden
2. VT hash not found after 10 poll attempts — marked stale, stays hidden
3. VT hash found but no Code Insight after 10 attempts — same

Fix: call setSkillModerationStatusActiveInternal in all three paths so
the skill becomes searchable. If VT later returns a malicious verdict,
approveSkillByHashInternal will correctly re-hide and flag it.

Closes #139

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 02:48:57 +00:00
Peter Steinberger bb528ea4b9 fix: dedupe OpenAI response parsing (#502) (thanks @ianalloway) 2026-02-25 02:46:35 +00:00
Nimrod Gutman 464a04c1e5 fix(vt): prevent pending scan starvation and retry unresolved results (#468)
* fix(vt): prevent pending scan starvation and retry unresolved results

* fix(vt): remove exhaustive pending-scan clamp
2026-02-23 21:26:47 -06:00
Peter Steinberger f4f8e7276f docs: clarify skill delete/undelete permissions 2026-02-18 17:24:09 +01:00
Peter Steinberger 3ccf2e05f5 fix: make users reclaim transfer root slug ownership in place 2026-02-18 17:10:08 +01:00
Shadow 0ee2872f5b fix: prune deleted skill backups 2026-02-17 11:02:57 -06:00
Peter Steinberger 5112d1b215 docs: cross-link README and vision 2026-02-17 17:35:28 +01:00
Peter Steinberger 275a170f15 docs: clarify MCP support policy in vision 2026-02-17 17:33:54 +01:00
Peter Steinberger 17aa24baf9 docs: refine vision priorities and security stance 2026-02-17 17:31:24 +01:00
Peter Steinberger f01476757a fix(ui): remove upvote-style metric and clarify installs icon 2026-02-17 15:08:47 +01:00
Peter Steinberger fe011d00fd test(e2e): update search menu route expectation 2026-02-17 15:08:42 +01:00
Peter Steinberger 30ae099825 fix: avoid paginated fallback in skills count query (#76) 2026-02-17 00:32:22 +01:00
Peter Steinberger 812641342d fix: harden /skills count when globalStats is missing (#76) 2026-02-17 00:25:46 +01:00
Peter Steinberger 88848c224c fix: keep public skill counts consistent after moderation/visibility changes (#76) (thanks @rknoche6) 2026-02-17 00:25:46 +01:00
c107adabac display total skills count on /skills page (#76)
Co-authored-by: rknoche <richard.knoche@holidaycheck.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-17 00:25:46 +01:00
Peter Steinberger 38c4a673da Revert "display total skills count on /skills page (#76)" (#359)
This reverts commit 89933951f5.
2026-02-16 17:47:19 +01:00
89933951f5 display total skills count on /skills page (#76)
Co-authored-by: rknoche <richard.knoche@holidaycheck.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-16 16:06:45 +01:00
Peter Steinberger e3523093b1 chore: bump clawhub to 0.7.0 2026-02-16 06:00:49 +01:00
Peter Steinberger 1faf3ee5ed test: boost cli http and user branch coverage 2026-02-16 05:58:20 +01:00
Peter Steinberger 286c76a05f test: add even more user auth and ranking regressions 2026-02-16 05:44:12 +01:00
Peter Steinberger 5745b5a096 test: add more users search and auth regressions 2026-02-16 05:40:30 +01:00
Peter Steinberger a060ae3b15 test: expand users list and search coverage 2026-02-16 05:33:44 +01:00
Peter Steinberger 77982c5d8e test: harden users search coverage for malformed data 2026-02-16 05:31:44 +01:00
Peter Steinberger 4cb84df36a fix: bound users list scans for management 2026-02-16 05:19:05 +01:00
Peter Steinberger d82c8f66c2 test: add regressions for pagination edge cases 2026-02-16 05:09:05 +01:00
Peter Steinberger 43fd834d23 refactor: simplify pagination + compact stat formatting 2026-02-16 05:07:18 +01:00
Peter Steinberger 9b2fc48a55 fix: harden listPublicPageV2 cursor recovery 2026-02-16 05:03:57 +01:00
Peter Steinberger a4dad5dc9d refactor: split skills page model and centralize stat rendering 2026-02-16 04:46:13 +01:00
Peter Steinberger 84830a268a feat: compact-format skill and soul stats 2026-02-16 04:35:01 +01:00
Peter Steinberger 37ef3eb7c5 test: add listPublicPageV2 regression coverage 2026-02-16 04:32:15 +01:00
Peter Steinberger 7f987fcc26 fix: prevent skills pagination dead-end and flicker (#339) (thanks @Marvae) 2026-02-16 04:24:24 +01:00
Hongwei Ma a0ea45c9a6 test: add coverage for search empty state 2026-02-16 04:24:24 +01:00
Hongwei Ma 1f5a782ecd fix: show empty state immediately for search results 2026-02-16 04:24:24 +01:00
Hongwei Ma c300d4b447 test: add edge case coverage for LoadingMore with empty results 2026-02-16 04:24:24 +01:00
Hongwei Ma c3a6cd7356 fix: keep load more visible during pagination
Prevents flicker when loading more pages by showing the load more
area during LoadingMore status instead of hiding it completely.
2026-02-16 04:24:24 +01:00
Hongwei Ma b75e25c4d6 fix: prevent loading state flicker on skills page
- Show 'Loading skills…' instead of 'No skills match' when pagination is not exhausted
- Hide 'Scroll to load more' when results are empty
- Add tests for both cases
2026-02-16 04:24:24 +01:00
Peter Steinberger 54383665d8 refactor: split skill detail page and optimize lazy loading 2026-02-16 03:52:24 +01:00
+22 697cc1a08f feat: add skill file viewer (#44)
* fix: return proper HTTP status codes for delete/undelete errors

The delete and undelete handlers for skills and souls were catching all
errors and returning 401 Unauthorized, even for errors like:
- 'Skill not found' (should be 404)
- 'Forbidden' (should be 403)
- Other validation errors (should be 400)

This change updates the error handling to return appropriate status codes:
- 401 Unauthorized: authentication failures
- 403 Forbidden: authorization failures (not owner/admin/moderator)
- 404 Not Found: skill/soul/user not found
- 400 Bad Request: other errors with descriptive message

Fixes #34

* fix(cli): use proper Error objects in abort timeouts

When AbortController.abort() receives a string instead of an Error,
the string itself is thrown. pRetry then wraps it in a confusing
message: 'Non-error was thrown: Timeout'

Changed all 3 occurrences in http.ts:
- apiRequest (line 57)
- apiRequestForm (line 106)
- downloadZip (line 141)

Now timeouts will surface as proper Error objects with clear messages.

* test: add e2e test for delete error handling

Verifies that deleting a non-existent skill returns a proper 'not found'
error instead of a generic 'Unauthorized' message.

* fix: use Error for timeout abort in e2e helper

* feat: add skill file viewer

* fix: prevent file viewer state updates after unmount

* feat: add ban reasons to moderation

* chore: release 0.6.0

* docs: reset changelog for next release

* feat: add LLM security evaluation at publish time

Add OpenClaw LLM-based security evaluator that runs alongside VirusTotal
when skills are published. Reads SKILL.md prose, metadata, install specs,
and file manifest, then assesses coherence across 5 dimensions to catch
social engineering vectors that VT/regex miss (e.g. instruction-only skills
with no code files).

- convex/lib/securityPrompt.ts: system prompt, message assembly, response
  parsing, injection pattern detection
- convex/llmEval.ts: evaluateWithLlm action, evaluateBySlug convenience
  action, backfillLlmEval for existing skills
- convex/schema.ts: llmAnalysis field on skillVersions
- convex/skills.ts: updateVersionLlmAnalysisInternal mutation,
  getActiveSkillBatchForLlmBackfillInternal query, defense-in-depth
  multi-scanner flag merging in approveSkillByHashInternal
- convex/lib/skillPublish.ts: schedule LLM eval alongside VT scan
- SkillDetailPage.tsx: OpenClaw row, LlmAnalysisDetail expandable
  component with 5 dimension rows, guidance panel, findings section
- styles.css: analysis detail styles from mockup

* fix: collapse OpenClaw analysis by default, fix row spacing, switch to gpt-5-mini

* fix: add retry with backoff for OpenAI rate limits, fix JSON mode requirement

* fix: increase max_output_tokens for reasoning model, fix backfill error retry

* feat: recognize metadata.openclaw as valid frontmatter namespace

* fix: eval assembler falls back to metadata.openclaw for requirements

* feat: evaluator reads all file contents, not just SKILL.md

Reads all files from storage and includes their full source in the eval
prompt so the LLM can detect malicious code hidden behind clean READMEs.
Injection detection now scans all content. Per-file cap 10K chars, total
cap 50K chars.

* feat: add skill metadata docs, suspicious appeal banner for owners

- Document full frontmatter metadata reference in docs/skill-format.md
- Add metadata section + quick example to README
- Show appeal message on suspicious skills (owner-only) linking to GitHub issues
- Accept metadata.openclaw alias in README docs
- Re-evaluate all skills with full file content reading (backfill in progress)

* fix: trailing comma tolerance in JSON metadata, tone down persistence flags

- Strip trailing commas in frontmatter JSON before parsing (silent failure fix)
- Stop flagging disable-model-invocation default as a concern (it's the normal default)
- Stop flagging skills configuring themselves as privilege escalation
- Add MITRE ATLAS AML.T0051 context for when autonomous invocation actually matters
- Show actual defaults in assembled eval message instead of "not set"

* chore: fix lint issues (#213)

* perf: lazy-load diff viewer (Monaco) (#212)

* chore: fix review comments

* fix: VT scan sync race condition + LLM-first moderation model

VT no longer overwrites LLM moderation verdicts. LLM is the primary
moderation authority; VT only escalates (hides + flags) for malicious/
suspicious content via new escalateByVtInternal mutation. Stale VT polls
write vtAnalysis marker instead of overwriting moderationReason. Query
pools expanded to include LLM-evaluated skills awaiting VT results.
Ban message now references malicious skills and security@openclaw.ai.

* fix: handle GitHub API rate limits in account age check (#246)

* fix: handle GitHub API rate limits in account age check

The GitHub account lookup uses unauthenticated requests (60 req/hr
per IP). Since this runs server-side in Convex, all users share the
same IP and quickly exhaust the rate limit, causing "GitHub account
lookup failed" errors during skill publish.

- Detect 403/429 responses and surface a clear rate-limit message
- Support optional GITHUB_TOKEN env var for authenticated requests
  (5,000 req/hr)

Fixes #155

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: stabilize GitHub account gate tests and docs

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* docs: thank @superlowburn for PR #246

* fix: prioritize relevant skills in search

* fix: add lexical fallback for skill search recall

* test: add search fallback coverage

* test: fix search test handler typing

* fix(http): remove allowH2 from undici Agent — causes fetch failed on Node.js 22+ (#245)

* Remove allowH2 option from global dispatcher

fix/remove-allowH2-undici-node22-compat

* fix(http): remove allowH2 from e2e dispatcher

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* docs: add 0.6.1 unreleased changelog from post-0.6.0 commits

* fix: allow soft-deleted users to re-authenticate

Fixes Issue #32 where users who soft-deleted their accounts were unable to sign back in because the re-auth logic was only triggering when an existingUserId was passed by the auth provider, which doesn't happen during a standard fresh login flow.

* test: update auth tests for direct deletedAt check

* fix: restore existingUserId check for type safety

* fix: update tests to include required existingUserId parameter

* fix: resolve final lint error in auth tests

* fix: ensure reactivation only matches soft-deleted user (prevents bypass)

* fix: allow re-auth when existingUserId is null

* fix: use valid crons.interval and set to 1 minute

* test: add missing coverage for fresh-login reactivation and identity mismatch guard

* fix: scope reauth fix; keep banned users blocked (#177) (thanks @tanujbhaud)

* fix: include comment deltas in action-based stat processing & add stats reconciliation (#194)

Bug 1: applyAggregatedStatsAndUpdateCursor was missing 'comments' in both
the guard condition and the applySkillStatDeltas call. This caused comment
count deltas to be silently dropped during cron-based event processing,
while stars/downloads/installs were processed correctly.

Bug 2: No reconciliation mechanism existed. If events were missed due to
cursor issues or processing errors, skill stats (stars, comments) would
remain stale with no way to recover. Added reconcileSkillStarCounts
maintenance mutation that counts actual records in the stars and comments
tables and patches any out-of-sync skill stats.

Fixes #193

Co-authored-by: Limitless2023 <limitless@users.noreply.github.com>

* fix: prevent horizontal overflow from long code blocks in skill pages (#183)

* Fix: Prevent horizontal overflow from long code blocks in skill pages

- Add max-width: 100% to .file-list-body and .file-row
- Prevents page-wide overflow when skills contain long code examples
- Markdown pre blocks already have overflow-x: auto, but parent containers were expanding infinitely
- Fixes issue where skills with 400+ char lines (e.g. browser automation commands) cause horizontal scrolling

Affected: Skills with long inline code in markdown (browser act commands, etc.)

* fix: add max-width to .file-list container to prevent overflow

- Also ensures .file-list-body constraint is inherited properly
- Prevents long code blocks from expanding file list container

* fix: add max-width to all markdown containers and pre tags

- Add max-width: 100% to .markdown, .tab-body, .markdown pre
- Ensures code blocks are constrained and show horizontal scrollbar
- Prevents content from expanding parent containers beyond viewport

* fix: add overflow-x to parent containers for horizontal scroll

Adds overflow-x: auto to .skill-detail-stack, .tab-card, and .tab-body
to ensure long code blocks are scrollable within the content area
instead of causing page-wide horizontal overflow.

Fixes horizontal overflow issue on skill pages with long code examples
(e.g., browser automation commands with 400+ character lines).

Tested on zepto skill page - page now stays within viewport (1200px)
and code blocks are accessible via horizontal scrollbar in tab area.

* docs: note code-block overflow fix in changelog (#183) (thanks @bewithgaurav)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* chore(release): 0.6.1

* fix: prevent infinite loading loop on skills page  (#90)

* fix: prevent infinite loading loop on skills pageAdd isLoadingMore guard to IntersectionObserver useEffect to preventcontinuous WebSocket queries when user is idle at bottom of page.The observer now won't set up while a request is in progress, breakingthe infinite loop cycle.Fixes: Related to #89

* fix: prevent repeated skills auto-load requests (#90) (thanks @xcqtnr)

* fix: resolve PR merge conflicts and keep observer regression test (#90) (thanks @xcqtnr)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* fix(cli): secure config file permissions (#164)

* fix(cli): secure config file permissions and reduce duplication

Security:
- Config files now created with 0600 permissions (owner read/write only)
- Config directories created with 0700 permissions
- Protects API tokens from other users on shared systems

Maintainability:
- Extract resolveConfigPath() helper to reduce code duplication
- Same legacy fallback logic (clawhub -> clawdhub) now in one place

* fix(cli): tolerate unsupported chmod errors for config

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* fix: make /search host-aware in SSR (#257)

* fix: make /search mode-aware

Notes:\n- Medium: /search now depends on getSiteMode() during beforeLoad. On server-side routing, if VITE_SITE_MODE isn’t set and VITE_SOULHUB_SITE_URL is set (as in .env.local), getSiteMode() will resolve to souls and redirect /search to / even on the ClawdHub deployment. This is a regression risk vs the old always-/skills redirect. Confirm deployment envs guarantee correct mode. src/routes/search.tsx:9-31

* fix: make /search host-aware in SSR

* chore: fix lint and route tree for /search route

---------

Co-authored-by: Sash Zats <sash@zats.io>

* fix(vt): explicit return types and missing undici dependency (#255)

* fix(vt): explicit return types and missing undici dependency

Refactor action handlers in convex/vt.ts to use explicit return types, resolving circular type inference (TS7022). Also add undici to devDependencies for E2E tests.

* fix: add root undici devDependency for e2e (#255) (thanks @tanujbhaud)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* Fix initial skill sorting (#92)

* fix: initial skill sorting

* chore: update unit test

* fix: use correct indexes for skill sorting

* chore: cleanup

* fix: land skill sorting update (#92) (thanks @bpk9)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* fix: harden download rate limiting and dedupe (#43) (thanks @regenrek)

- add download-specific rate limit tier\n- add per-IP/day dedupe + daily pruning\n- keep moderation gating + deterministic zips\n- add optional forwarded-IP trust via TRUST_FORWARDED_IPS

* fix: harden skill listing and rate limiting under load

* fix: replace skill report prompt with modal

* fix: add skill publish anti-spam caps and quarantine

* docs: add git local-branch cleanup fallback

* fix: enforce quality gate and trust-tier spam checks

* fix: prevent autobanned users from self-reactivating

* test: expand reauth ban regression coverage

* feat: add empty-skill cleanup backfill with ban nominations

* fix: make empty-skill cleanup resumable

* feat: add non-suspicious skills filter toggle

* style: polish selected states in skills toolbar

* feat: default skills sort to downloads

* fix: enforce downloads as canonical default skills sort

* fix: force canonical downloads sort in skills browse mode

* fix: bypass suspicious flags for privileged owners and polish comment delete UI

* fix: add privileged-owner suspicious flag reconciler

* fix: force auth redirects and registry to canonical clawhub host

* feat: auto-generate missing skill summaries

* fix: make skill summary backfill resumable

* feat: add self-scheduling skill summary backfill job

* perf: short-circuit empty skill summary generation

* style: polish upload page layout and actions

* feat: show popular non-suspicious skills on homepage

* fix: normalize legacy skill stats to prevent homepage crash

* fix: render homepage popular cards from nested skill entries

* style: refine global UI theme, borders, and spacing

* fix: resolve search timeout and improve skills page UI alignment (#53)

* fix: resolve search timeout and improve skills page UI alignment

- Added a 10s timeout to OpenAI embedding requests to prevent hanging searches.
- Fixed a TypeScript error in search.ts regarding entry hydration.
- Restructured skills page layout and CSS to ensure consistent alignment between the search toolbar and skill cards.

* fix: resolve search timeout and improve skills page UI alignment

- Added a 10s timeout to OpenAI embedding requests to prevent hanging searches.
- Fixed a TypeScript error in search.ts regarding entry hydration.
- Restructured skills page layout and CSS to ensure consistent alignment between the search toolbar and skill cards.

* style: format skills index layout block

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* docs: thank @GhadiSaab for #53

* style: shift UI palette to cool blue tones

* style: remove remaining warm accent literals

* style: darken hero primary CTA in dark mode

* fix: show stars in popular skill cards

* fix: simplify skills CTA label

* fix: dedupe download metrics hourly by user-or-ip identity (#278)

* style: restore brown palette and dark-mode CTA tone

* fix(comments): stop updating skills.updatedAt on comment add/remove (#55)

* fix(comments): stop updating skills.updatedAt on comment add/remove

Comments are not content changes, so they shouldn't invalidate skill
list queries that depend on updatedAt. This reduces query invalidation
when users add or remove comments.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(comments): add updatedAt invalidation regression coverage

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* refactor(comments): extract handlers and harden mutation tests

* feat: make account deletion irreversible and migrate lint to oxlint

* chore: add oxfmt config

* fix(cli): throw Error for all timeout aborts (#283)

* fix(cli): throw Error on timeout aborts

Users have seen an elevated number of:\n  clawdhub search image\n  ✖ Non-error was thrown: "Timeout". You should only throw errors.\n\nInvestigation shows we were aborting with a string instead of an Error. Switching to controller.abort(new Error('Timeout')) makes retries/formatting treat it as a real error and clears the message.\n\nExample after change:\n  clawdhub search image\n  table-image v1.0.0  Table Image  (0.332)\n  nano-banana-pro v1.0.1  Nano Banana Pro  (0.319)\n  vap-media v1.0.1  AI media generation API - Flux2pro, Veo3.1, Suno Ai  (0.281)\n  clawdbot-meshyai-skill v0.1.0  Meshy AI  (0.276)\n  venice-ai-media v1.0.0  Venice AI Media  (0.274)\n  daily-recap v1.0.2  Daily Recap  (0.260)\n  openai-image-gen v1.0.1  Openai Image Gen  (0.260)\n  bible-votd v1.0.1  Bible Verse of the Day  (0.248)\n  orf v1.0.1  ORF  (0.224)\n  smalltalk v1.0.1  Smalltalk  (0.161)

* fix(http): wrap fetch calls in try-finally to prevent timer leaks

Addresses Vercel review comment: clearTimeout was not called on error paths when fetch throws an exception.

* fix(cli): unify timeout abort handling

---------

Co-authored-by: Sash Zats <sash@zats.io>

* refactor(cli): centralize HTTP status errors and timeout tests (#286)

* fix: keep new skill versions pending until VT verdict

* style: remove residual blue accents and warm base palette

* fix: add retry logic for OpenAI embedding API failures (#272)

* fix: add retry logic for OpenAI embedding API failures

Fixes #149

When importing or uploading skills, the OpenAI embedding API call could
fail with transient errors (rate limits, timeouts, network issues),
causing the entire import to fail with a generic "Server Error".

This adds retry logic with exponential backoff (1s, 2s, 4s delays):
- Retries on 429 (rate limit) and 5xx server errors
- Retries on network/fetch errors
- Logs warnings for debugging
- Max 3 retries before failing with clear error message

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: correct retry count and broaden network error catch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address retry loop off-by-one, broaden error catch, preserve original error

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: harden embeddings retry semantics

* style: format embeddings retry changes

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* fix: sync handle on user ensure

* fix: sync handle on user ensure (#293) (thanks @christianhpoe)

* feat: improve moderation/admin UX + language-aware quality gate

- API: owner-visible responses for hidden/soft-deleted skills\n- Admin: add unban user mutations + docs\n- Quality: Intl.Segmenter tokenization + CJK signal to reduce false rejects\n- Jobs: skill-stat-events interval 15m -> 5m\n- Tests: add coverage for owner-visible states + non-Latin docs\n- Changelog: add Unreleased entry

* refactor: simplify user ensure updates

* fix(cors): complete CORS + tokenized CLI reads (#296)

* fix(cors): add Access-Control-Allow-Origin headers to API and downloads

* fix: add CORS to error/raw paths & add CLI install auth

* fix: add OPTIONS handler for CORS preflight

* fix(cors): complete CORS + tokenized CLI reads

* test(cli): fix config mock typing

---------

Co-authored-by: Grenghis-Khan <63885013+Grenghis-Khan@users.noreply.github.com>

* refactor: centralize CORS + CLI auth token (#297)

* refactor(convex): centralize CORS headers

* refactor(cli): centralize auth token lookup

* fix(skills): keep global sorting across pagination (#98)

* fix: initial skill sorting

* chore: update unit test

* fix: use correct indexes for skill sorting

* chore: cleanup

* fix(skills): preserve server order for paginated sorting

* chore(lint): apply biome formatting fixes

* chore(convex): bump tsconfig lib to ES2022

* fix(skills): add deterministic tie-breaker for search sorting

* fix(skills): stable sorting across pagination (#98) (thanks @CodeBBakGoSu)

---------

Co-authored-by: Brian Kasper <bkasperr@gmail.com>
Co-authored-by: knox-glorang <knox@glorang.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* chore: drop convex-helpers (#302)

* perf: batch tag resolution to reduce action→query round-trips

- Add getVersionsByIds batch query to skills.ts and souls.ts
- Replace per-item tag resolution with batch resolution in httpApiV1.ts
- Reduces N action→query round-trips to 1 for list endpoints

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add null guard and short-circuit for empty tags

- Short-circuit when no version IDs to resolve
- Add null coalescing for runQuery response
- Fixes potential crash when tags are empty or query returns null

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: batch resolve tags in v1 API (#112) (thanks @mkrokosz)

* fix: handle duplicate Convex Auth user records in publish ownership check (#180)

* fix: handle duplicate user records in publish ownership check

* fix: heal publish ownership via GitHub auth identity

---------

Co-authored-by: Emmet Brown <emmet@Emmets-Mac-mini.local>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* fix: gate publish by immutable GitHub account ID

* refactor: simplify GitHub age gate cache

* fix(api): centralize v1 soft-delete error mapping

* chore(cli): align http client with main

* test(api): cover v1 soft-delete error mapping

* test(api): reposition soft-delete mapping test

* fix: default to CF-only client IP parsing

* docs: changelog credit + v1 delete status codes

* fix(cli): clarify logout only affects local config (#166)

* fix(cli): clarify logout only affects local config

Users may assume 'clawhub logout' revokes their token everywhere.
In reality, the token remains valid on the server until explicitly
revoked in the web UI. This could be a security concern on shared
machines.

Update the message to set correct expectations.

* fix(cli): clarify logout revocation scope (#166) (thanks @aronchick)

* chore: sync changelog for merge (#166) (thanks @aronchick)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>

* feat: anti-squatting protection, backup restore, and ban flow improvements (#298)

* feat: anti-squatting protection, backup restore, and ban flow improvements

- Add `reservedSlugs` table with 90-day cooldown to prevent slug squatting
  after skill deletion. Hard-delete finalize phase reserves slugs for the
  original owner; `insertVersion` blocks non-owners during cooldown.

- Change ban flow from hard-delete to soft-delete: `banUserWithActor` now
  sets `moderationReason: 'user.banned'` and syncs embedding visibility.
  `unbanUserWithActor` restores all ban-hidden skills and releases slug
  reservations automatically.

- Align `autobanMalwareAuthorInternal` with the same soft-delete + embedding
  visibility pattern so unban recovery works uniformly.

- Add admin `reclaimSlug` / `reclaimSlugInternal` mutations for reclaiming
  squatted slugs, with audit logging.

- Add GitHub backup restore system (`githubRestore.ts`,
  `githubRestoreMutations.ts`, `githubRestoreHelpers.ts`) that reads from
  the `clawdbot/skills` backup repo and re-creates skill records. Squatter
  eviction runs synchronously in the same transaction as restore to avoid
  async race conditions.

- Add `POST /api/v1/users/restore` and `POST /api/v1/users/reclaim` admin
  HTTP endpoints for bulk operations.

- Add `trustedPublisher` flag on users; trusted publishers bypass the
  `pending.scan` auto-hide for new skill publishes.

- Add `setTrustedPublisher` / `setTrustedPublisherInternal` admin mutations.

Addresses: slug squatting prevention, skill backup/restore, ban recovery,
and trusted publisher workflow improvements.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: harden restore/reclaim + ban flow (#298) (thanks @autogame-17)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* refactor: post-#298 cleanup (#313)

* refactor: consolidate slug + embedding helpers

* refactor: batch ban/unban skill updates

* refactor: report batched ban/unban scheduling

* fix: unblock package typecheck

* refactor: split httpApiV1 + consolidate moderation batches (#315)

* refactor: dedupe v1 file response + unify embedding patches (#316)

* Devin/1771112524 skill metadata update (#312)

* fix: sync GitHub profile on login to handle username renames (#303)

When a user renames their GitHub account, the stored username becomes stale
and causes 'GitHub account lookup failed' errors during skill publishing.

This fix:
- Adds syncGitHubProfile function that fetches current profile using the
  immutable GitHub numeric ID
- Adds syncGitHubProfileInternal mutation to update user's name, handle,
  displayName, and image when they change
- Schedules the sync as a background action on every login via
  afterUserCreatedOrUpdated callback

The sync is best-effort (silently fails if GitHub API unavailable) since
it's not on the critical path. It only updates fields if the username
has actually changed.

Fixes #303

Co-Authored-By: Ian Alloway <adapter_burners.1y@icloud.com>

* fix: allow updating skill summary/description on subsequent publishes (#301)

Previously, the skill summary was only extracted from metadata.description
in the SKILL.md frontmatter. This change also checks for a direct
'description' field in the frontmatter, ensuring that users can update
their skill description by modifying either location.

The fix prioritizes the new description from the current publish over
the existing skill summary, allowing updates to be reflected correctly.

Fixes #301

Co-Authored-By: Ian Alloway <adapter_burners.1y@icloud.com>

* fix: throttle GitHub profile sync

* feat: show skill owner avatars

* fix: avoid nested owner links

* refactor: centralize profile sync + owner lookup

* docs: changelog for #312 (thanks @ianalloway)

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* style: polish markdown code blocks

* feat: show skill owner avatars on home + lists

* feat: sync GitHub profile name

* feat: improve skill card meta layout

* fix: make ghost buttons look like buttons

* fix: match skill hero cta widths

* fix: prefer $HOME over os.homedir() for path resolution (#299)

* fix: prefer $HOME over os.homedir() for path resolution

os.homedir() reads from /etc/passwd which can return a stale path
after a Linux user rename (usermod -l). Prefer the $HOME environment
variable which reflects the current session.

Closes #82

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: normalize resolveHome output

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* UI: allow copying security scan summary text (#322)

* fix(ui): prevent analysis toggle when selecting summary (#324)

* feat: add uninstall command for skills (#241)

* feat: add uninstall command for skills

Implements `clawhub uninstall <slug>` to properly remove installed skills.

Changes:
- Added cmdUninstall function in skills.ts
- Validates skill is installed before removal
- Removes skill directory and lockfile entry
- Supports --yes flag to skip confirmation prompt
- Added comprehensive test coverage

Closes #221

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: require --yes in non-interactive mode and update lockfile before rm

Address review feedback:
- Fail with "Pass --yes (no input)" when running non-interactively
  without --yes flag, matching delete/star/unstar/moderation commands
- Update lockfile before removing directory to avoid inconsistent state
  if rm succeeds but writeLockfile fails

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: harden skill uninstall flow (#241) (thanks @superlowburn)

* docs: document uninstall CLI command (#241) (thanks @superlowburn)

* test: fix cmdUninstall mock typing (#241) (thanks @superlowburn)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

* feat: add skill file viewer

* fix: prevent file viewer state updates after unmount

* fix: lazy-load skill file viewer (#44) (thanks @regenrek)

---------

Co-authored-by: Sergiy Dybskiy <s@serg.tech>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
Co-authored-by: theonejvo <theonejvo@users.noreply.github.com>
Co-authored-by: Vignesh <vigneshnatarajan92@gmail.com>
Co-authored-by: Steve <superlowburn@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: DColl <david.coll.78@gmail.com>
Co-authored-by: Tanuj Bhaud <tanujbhaud@gmail.com>
Co-authored-by: Limitless <127183162+Limitless2023@users.noreply.github.com>
Co-authored-by: Limitless2023 <limitless@users.noreply.github.com>
Co-authored-by: Gaurav Sharma <sharmag@microsoft.com>
Co-authored-by: xcqtnr <xcqtnr0.0@gmail.com>
Co-authored-by: David Aronchick <aronchick@gmail.com>
Co-authored-by: Sash Zats <sash@zats.io>
Co-authored-by: Tanuj Bhaud <128238320+tanujbhaud@users.noreply.github.com>
Co-authored-by: Brian Kasper <brian@bkasper.com>
Co-authored-by: ghadi saab <ghadisaab21@gmail.com>
Co-authored-by: sethconvex <seth@convex.dev>
Co-authored-by: ChristianHPoe <chpoensgen@me.com>
Co-authored-by: Grenghis-Khan <63885013+Grenghis-Khan@users.noreply.github.com>
Co-authored-by: CodeBBakGoSu <127713112+CodeBBakGoSu@users.noreply.github.com>
Co-authored-by: Brian Kasper <bkasperr@gmail.com>
Co-authored-by: knox-glorang <knox@glorang.com>
Co-authored-by: Matthew Krokosz <mattkrokosz@gmail.com>
Co-authored-by: emmet-bot <emmet@universaleverything.io>
Co-authored-by: Emmet Brown <emmet@Emmets-Mac-mini.local>
Co-authored-by: autogame-17 <166480271+autogame-17@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ian Alloway <adapter_burners.1y@icloud.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: CleanApp <165804662+borisolver@users.noreply.github.com>
2026-02-16 03:29:30 +01:00
652beef9c1 feat: add uninstall command for skills (#241)
* feat: add uninstall command for skills

Implements `clawhub uninstall <slug>` to properly remove installed skills.

Changes:
- Added cmdUninstall function in skills.ts
- Validates skill is installed before removal
- Removes skill directory and lockfile entry
- Supports --yes flag to skip confirmation prompt
- Added comprehensive test coverage

Closes #221

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: require --yes in non-interactive mode and update lockfile before rm

Address review feedback:
- Fail with "Pass --yes (no input)" when running non-interactively
  without --yes flag, matching delete/star/unstar/moderation commands
- Update lockfile before removing directory to avoid inconsistent state
  if rm succeeds but writeLockfile fails

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: harden skill uninstall flow (#241) (thanks @superlowburn)

* docs: document uninstall CLI command (#241) (thanks @superlowburn)

* test: fix cmdUninstall mock typing (#241) (thanks @superlowburn)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-15 16:03:06 +01:00
Peter Steinberger 146df7b166 fix(ui): prevent analysis toggle when selecting summary (#324) 2026-02-15 14:42:03 +01:00
CleanApp 8f23eb5ee8 UI: allow copying security scan summary text (#322) 2026-02-15 14:39:13 +01:00
30b263c27c fix: prefer $HOME over os.homedir() for path resolution (#299)
* fix: prefer $HOME over os.homedir() for path resolution

os.homedir() reads from /etc/passwd which can return a stale path
after a Linux user rename (usermod -l). Prefer the $HOME environment
variable which reflects the current session.

Closes #82

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: normalize resolveHome output

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-15 14:26:00 +01:00
Peter Steinberger 10b704278a fix: match skill hero cta widths 2026-02-15 05:28:59 +01:00
Peter Steinberger a289f9cbd9 fix: make ghost buttons look like buttons 2026-02-15 05:27:55 +01:00
Peter Steinberger 97d68a1be5 feat: improve skill card meta layout 2026-02-15 05:26:08 +01:00
Peter Steinberger 57e0d39cdc feat: sync GitHub profile name 2026-02-15 05:26:03 +01:00
Peter Steinberger 1c033868e7 feat: show skill owner avatars on home + lists 2026-02-15 05:06:23 +01:00
Peter Steinberger 4532366009 style: polish markdown code blocks 2026-02-15 05:00:18 +01:00
Ian AllowayDevin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>Peter Steinberger
8e9fa44fc2 Devin/1771112524 skill metadata update (#312)
* fix: sync GitHub profile on login to handle username renames (#303)

When a user renames their GitHub account, the stored username becomes stale
and causes 'GitHub account lookup failed' errors during skill publishing.

This fix:
- Adds syncGitHubProfile function that fetches current profile using the
  immutable GitHub numeric ID
- Adds syncGitHubProfileInternal mutation to update user's name, handle,
  displayName, and image when they change
- Schedules the sync as a background action on every login via
  afterUserCreatedOrUpdated callback

The sync is best-effort (silently fails if GitHub API unavailable) since
it's not on the critical path. It only updates fields if the username
has actually changed.

Fixes #303

Co-Authored-By: Ian Alloway <adapter_burners.1y@icloud.com>

* fix: allow updating skill summary/description on subsequent publishes (#301)

Previously, the skill summary was only extracted from metadata.description
in the SKILL.md frontmatter. This change also checks for a direct
'description' field in the frontmatter, ensuring that users can update
their skill description by modifying either location.

The fix prioritizes the new description from the current publish over
the existing skill summary, allowing updates to be reflected correctly.

Fixes #301

Co-Authored-By: Ian Alloway <adapter_burners.1y@icloud.com>

* fix: throttle GitHub profile sync

* feat: show skill owner avatars

* fix: avoid nested owner links

* refactor: centralize profile sync + owner lookup

* docs: changelog for #312 (thanks @ianalloway)

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-15 04:55:49 +01:00
Peter Steinberger 11a66ea148 refactor: dedupe v1 file response + unify embedding patches (#316) 2026-02-15 03:43:30 +01:00
Peter Steinberger f94e20d4c3 refactor: split httpApiV1 + consolidate moderation batches (#315) 2026-02-15 03:29:53 +01:00
Peter Steinberger 71c74f61e2 refactor: post-#298 cleanup (#313)
* refactor: consolidate slug + embedding helpers

* refactor: batch ban/unban skill updates

* refactor: report batched ban/unban scheduling

* fix: unblock package typecheck
2026-02-15 02:18:54 +01:00
e2592684ed feat: anti-squatting protection, backup restore, and ban flow improvements (#298)
* feat: anti-squatting protection, backup restore, and ban flow improvements

- Add `reservedSlugs` table with 90-day cooldown to prevent slug squatting
  after skill deletion. Hard-delete finalize phase reserves slugs for the
  original owner; `insertVersion` blocks non-owners during cooldown.

- Change ban flow from hard-delete to soft-delete: `banUserWithActor` now
  sets `moderationReason: 'user.banned'` and syncs embedding visibility.
  `unbanUserWithActor` restores all ban-hidden skills and releases slug
  reservations automatically.

- Align `autobanMalwareAuthorInternal` with the same soft-delete + embedding
  visibility pattern so unban recovery works uniformly.

- Add admin `reclaimSlug` / `reclaimSlugInternal` mutations for reclaiming
  squatted slugs, with audit logging.

- Add GitHub backup restore system (`githubRestore.ts`,
  `githubRestoreMutations.ts`, `githubRestoreHelpers.ts`) that reads from
  the `clawdbot/skills` backup repo and re-creates skill records. Squatter
  eviction runs synchronously in the same transaction as restore to avoid
  async race conditions.

- Add `POST /api/v1/users/restore` and `POST /api/v1/users/reclaim` admin
  HTTP endpoints for bulk operations.

- Add `trustedPublisher` flag on users; trusted publishers bypass the
  `pending.scan` auto-hide for new skill publishes.

- Add `setTrustedPublisher` / `setTrustedPublisherInternal` admin mutations.

Addresses: slug squatting prevention, skill backup/restore, ban recovery,
and trusted publisher workflow improvements.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: harden restore/reclaim + ban flow (#298) (thanks @autogame-17)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-15 01:16:46 +01:00
David AronchickandPeter Steinberger 79c9381201 fix(cli): clarify logout only affects local config (#166)
* fix(cli): clarify logout only affects local config

Users may assume 'clawhub logout' revokes their token everywhere.
In reality, the token remains valid on the server until explicitly
revoked in the web UI. This could be a security concern on shared
machines.

Update the message to set correct expectations.

* fix(cli): clarify logout revocation scope (#166) (thanks @aronchick)

* chore: sync changelog for merge (#166) (thanks @aronchick)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 22:56:10 +01:00
Peter Steinberger 6a5712fdb6 Merge pull request #309 from openclaw/chore/merge-all
docs: changelog credit + v1 delete status codes
2026-02-14 22:21:56 +01:00
Peter Steinberger a85faf76ac docs: changelog credit + v1 delete status codes 2026-02-14 22:21:37 +01:00
Peter Steinberger c0a04210e9 fix: default to CF-only client IP parsing 2026-02-14 22:20:58 +01:00
Peter Steinberger 5adb334cb2 Merge pull request #35 from sergical/fix/delete-error-handling
fix: return proper HTTP status codes for delete/undelete errors
2026-02-14 22:11:04 +01:00
Peter Steinberger 182ec8741f test(api): reposition soft-delete mapping test 2026-02-14 22:08:05 +01:00
Peter Steinberger bafd17b00a test(api): cover v1 soft-delete error mapping 2026-02-14 22:06:02 +01:00
Peter Steinberger 802ee58054 chore(cli): align http client with main 2026-02-14 22:04:02 +01:00
Peter Steinberger 0e83ba00b9 fix(api): centralize v1 soft-delete error mapping 2026-02-14 21:59:14 +01:00
Peter Steinberger 3326a5c838 refactor: simplify GitHub age gate cache 2026-02-14 20:53:05 +01:00
Matt Krokosz f05dd556db fix: gate publish by immutable GitHub account ID 2026-02-14 20:25:15 +01:00
964893a622 fix: handle duplicate Convex Auth user records in publish ownership check (#180)
* fix: handle duplicate user records in publish ownership check

* fix: heal publish ownership via GitHub auth identity

---------

Co-authored-by: Emmet Brown <emmet@Emmets-Mac-mini.local>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 19:39:04 +01:00
Peter Steinberger 9a804b951f refactor: batch resolve tags in v1 API (#112) (thanks @mkrokosz) 2026-02-14 19:01:02 +01:00
Matthew KrokoszandClaude Opus 4.5 d699087786 fix: add null guard and short-circuit for empty tags
- Short-circuit when no version IDs to resolve
- Add null coalescing for runQuery response
- Fixes potential crash when tags are empty or query returns null

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-14 19:01:02 +01:00
Matthew KrokoszandClaude Opus 4.5 26b42727fe perf: batch tag resolution to reduce action→query round-trips
- Add getVersionsByIds batch query to skills.ts and souls.ts
- Replace per-item tag resolution with batch resolution in httpApiV1.ts
- Reduces N action→query round-trips to 1 for list endpoints

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-14 19:01:02 +01:00
Peter Steinberger 8756b78a4e chore: drop convex-helpers (#302) 2026-02-14 17:54:24 +01:00
e9c771d55d fix(skills): keep global sorting across pagination (#98)
* fix: initial skill sorting

* chore: update unit test

* fix: use correct indexes for skill sorting

* chore: cleanup

* fix(skills): preserve server order for paginated sorting

* chore(lint): apply biome formatting fixes

* chore(convex): bump tsconfig lib to ES2022

* fix(skills): add deterministic tie-breaker for search sorting

* fix(skills): stable sorting across pagination (#98) (thanks @CodeBBakGoSu)

---------

Co-authored-by: Brian Kasper <bkasperr@gmail.com>
Co-authored-by: knox-glorang <knox@glorang.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 17:44:07 +01:00
Peter Steinberger a58f0166fa refactor: centralize CORS + CLI auth token (#297)
* refactor(convex): centralize CORS headers

* refactor(cli): centralize auth token lookup
2026-02-14 15:31:03 +01:00
Peter SteinbergerandGrenghis-Khan 4328d4d700 fix(cors): complete CORS + tokenized CLI reads (#296)
* fix(cors): add Access-Control-Allow-Origin headers to API and downloads

* fix: add CORS to error/raw paths & add CLI install auth

* fix: add OPTIONS handler for CORS preflight

* fix(cors): complete CORS + tokenized CLI reads

* test(cli): fix config mock typing

---------

Co-authored-by: Grenghis-Khan <63885013+Grenghis-Khan@users.noreply.github.com>
2026-02-14 14:45:15 +01:00
Peter Steinberger 28ee2618c1 refactor: simplify user ensure updates 2026-02-14 13:56:15 +01:00
Peter Steinberger a4b850ec33 feat: improve moderation/admin UX + language-aware quality gate
- API: owner-visible responses for hidden/soft-deleted skills\n- Admin: add unban user mutations + docs\n- Quality: Intl.Segmenter tokenization + CJK signal to reduce false rejects\n- Jobs: skill-stat-events interval 15m -> 5m\n- Tests: add coverage for owner-visible states + non-Latin docs\n- Changelog: add Unreleased entry
2026-02-14 13:54:03 +01:00
Peter Steinberger 7e0b21f7c8 fix: sync handle on user ensure (#293) (thanks @christianhpoe) 2026-02-14 13:48:56 +01:00
ChristianHPoe 71c6705ab1 fix: sync handle on user ensure 2026-02-14 13:48:56 +01:00
a57769771f fix: add retry logic for OpenAI embedding API failures (#272)
* fix: add retry logic for OpenAI embedding API failures

Fixes #149

When importing or uploading skills, the OpenAI embedding API call could
fail with transient errors (rate limits, timeouts, network issues),
causing the entire import to fail with a generic "Server Error".

This adds retry logic with exponential backoff (1s, 2s, 4s delays):
- Retries on 429 (rate limit) and 5xx server errors
- Retries on network/fetch errors
- Logs warnings for debugging
- Max 3 retries before failing with clear error message

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: correct retry count and broaden network error catch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address retry loop off-by-one, broaden error catch, preserve original error

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: harden embeddings retry semantics

* style: format embeddings retry changes

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 04:54:18 +01:00
Peter Steinberger 6a2c131a8a style: remove residual blue accents and warm base palette 2026-02-14 04:40:50 +01:00
Peter Steinberger 67ac157545 fix: keep new skill versions pending until VT verdict 2026-02-14 02:53:02 +01:00
Peter Steinberger ef36cfd698 refactor(cli): centralize HTTP status errors and timeout tests (#286) 2026-02-14 02:39:42 +01:00
Peter SteinbergerandSash Zats e0637ad6aa fix(cli): throw Error for all timeout aborts (#283)
* fix(cli): throw Error on timeout aborts

Users have seen an elevated number of:\n  clawdhub search image\n  ✖ Non-error was thrown: "Timeout". You should only throw errors.\n\nInvestigation shows we were aborting with a string instead of an Error. Switching to controller.abort(new Error('Timeout')) makes retries/formatting treat it as a real error and clears the message.\n\nExample after change:\n  clawdhub search image\n  table-image v1.0.0  Table Image  (0.332)\n  nano-banana-pro v1.0.1  Nano Banana Pro  (0.319)\n  vap-media v1.0.1  AI media generation API - Flux2pro, Veo3.1, Suno Ai  (0.281)\n  clawdbot-meshyai-skill v0.1.0  Meshy AI  (0.276)\n  venice-ai-media v1.0.0  Venice AI Media  (0.274)\n  daily-recap v1.0.2  Daily Recap  (0.260)\n  openai-image-gen v1.0.1  Openai Image Gen  (0.260)\n  bible-votd v1.0.1  Bible Verse of the Day  (0.248)\n  orf v1.0.1  ORF  (0.224)\n  smalltalk v1.0.1  Smalltalk  (0.161)

* fix(http): wrap fetch calls in try-finally to prevent timer leaks

Addresses Vercel review comment: clearTimeout was not called on error paths when fetch throws an exception.

* fix(cli): unify timeout abort handling

---------

Co-authored-by: Sash Zats <sash@zats.io>
2026-02-14 02:27:34 +01:00
Peter Steinberger 09a21a07ff chore: add oxfmt config 2026-02-14 02:15:12 +01:00
Peter Steinberger 65a14dcef3 feat: make account deletion irreversible and migrate lint to oxlint 2026-02-14 02:15:01 +01:00
Peter Steinberger 97c12b2327 refactor(comments): extract handlers and harden mutation tests 2026-02-14 01:56:34 +01:00
a290c81a75 fix(comments): stop updating skills.updatedAt on comment add/remove (#55)
* fix(comments): stop updating skills.updatedAt on comment add/remove

Comments are not content changes, so they shouldn't invalidate skill
list queries that depend on updatedAt. This reduces query invalidation
when users add or remove comments.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(comments): add updatedAt invalidation regression coverage

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 01:53:40 +01:00
Peter Steinberger cc5d5cfee5 style: restore brown palette and dark-mode CTA tone 2026-02-14 01:18:29 +01:00
Peter Steinberger 03cd710abc fix: dedupe download metrics hourly by user-or-ip identity (#278) 2026-02-14 01:13:52 +01:00
Peter Steinberger 287f639fbc fix: simplify skills CTA label 2026-02-14 00:56:52 +01:00
Peter Steinberger aecf66981c fix: show stars in popular skill cards 2026-02-14 00:55:24 +01:00
Peter Steinberger db8090f287 style: darken hero primary CTA in dark mode 2026-02-14 00:50:52 +01:00
Peter Steinberger ae8614fa98 style: remove remaining warm accent literals 2026-02-14 00:38:21 +01:00
Peter Steinberger e7f78ea5a3 style: shift UI palette to cool blue tones 2026-02-14 00:31:55 +01:00
Peter Steinberger b9355f7a0c docs: thank @GhadiSaab for #53 2026-02-14 00:16:24 +01:00
ghadi saabandPeter Steinberger 9530676f8a fix: resolve search timeout and improve skills page UI alignment (#53)
* fix: resolve search timeout and improve skills page UI alignment

- Added a 10s timeout to OpenAI embedding requests to prevent hanging searches.
- Fixed a TypeScript error in search.ts regarding entry hydration.
- Restructured skills page layout and CSS to ensure consistent alignment between the search toolbar and skill cards.

* fix: resolve search timeout and improve skills page UI alignment

- Added a 10s timeout to OpenAI embedding requests to prevent hanging searches.
- Fixed a TypeScript error in search.ts regarding entry hydration.
- Restructured skills page layout and CSS to ensure consistent alignment between the search toolbar and skill cards.

* style: format skills index layout block

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-14 00:15:47 +01:00
Peter Steinberger ef23520d22 style: refine global UI theme, borders, and spacing 2026-02-13 23:50:24 +01:00
Peter Steinberger e67a6e6400 fix: render homepage popular cards from nested skill entries 2026-02-13 22:52:21 +01:00
Peter Steinberger e6871b86e1 fix: normalize legacy skill stats to prevent homepage crash 2026-02-13 22:45:34 +01:00
Peter Steinberger 75937e8b53 feat: show popular non-suspicious skills on homepage 2026-02-13 21:38:56 +01:00
Peter Steinberger d2919791d1 style: polish upload page layout and actions 2026-02-13 21:35:25 +01:00
Peter Steinberger 9019cd8462 perf: short-circuit empty skill summary generation 2026-02-13 21:21:09 +01:00
Peter Steinberger 5e58bd459e feat: add self-scheduling skill summary backfill job 2026-02-13 21:10:51 +01:00
Peter Steinberger 3badf0668f fix: make skill summary backfill resumable 2026-02-13 20:43:46 +01:00
Peter Steinberger c719297d70 feat: auto-generate missing skill summaries 2026-02-13 20:35:03 +01:00
Peter Steinberger e19cd23be2 fix: force auth redirects and registry to canonical clawhub host 2026-02-13 20:22:08 +01:00
Peter Steinberger 9266fb7c20 fix: add privileged-owner suspicious flag reconciler 2026-02-13 20:01:39 +01:00
Peter Steinberger 99645d2c27 fix: bypass suspicious flags for privileged owners and polish comment delete UI 2026-02-13 19:56:53 +01:00
Peter Steinberger a1ad7fac85 fix: force canonical downloads sort in skills browse mode 2026-02-13 19:33:26 +01:00
Peter Steinberger ebd2f12cc4 fix: enforce downloads as canonical default skills sort 2026-02-13 19:27:27 +01:00
Peter Steinberger e2ee7b164c feat: default skills sort to downloads 2026-02-13 19:15:19 +01:00
Peter Steinberger 36ed062739 style: polish selected states in skills toolbar 2026-02-13 19:12:48 +01:00
Peter Steinberger d12d6e3926 feat: add non-suspicious skills filter toggle 2026-02-13 19:05:46 +01:00
Peter Steinberger 1851a9c01f fix: make empty-skill cleanup resumable 2026-02-13 17:54:47 +01:00
Peter Steinberger bbeb0be343 feat: add empty-skill cleanup backfill with ban nominations 2026-02-13 17:48:58 +01:00
Peter Steinberger 9c22fb7e54 test: expand reauth ban regression coverage 2026-02-13 17:33:58 +01:00
Peter Steinberger ef2403179b fix: prevent autobanned users from self-reactivating 2026-02-13 17:28:00 +01:00
Peter Steinberger df178d4bfc fix: enforce quality gate and trust-tier spam checks 2026-02-13 17:16:53 +01:00
Peter Steinberger 318cdd33c5 docs: add git local-branch cleanup fallback 2026-02-13 17:03:38 +01:00
Peter Steinberger 9087b037dd fix: add skill publish anti-spam caps and quarantine 2026-02-13 16:58:18 +01:00
Peter Steinberger 6991569a1c fix: replace skill report prompt with modal 2026-02-13 16:47:16 +01:00
Peter Steinberger 32bc600be4 fix: harden skill listing and rate limiting under load 2026-02-13 16:39:57 +01:00
Kevin Kern a52a37d08c fix: harden download rate limiting and dedupe (#43) (thanks @regenrek)
- add download-specific rate limit tier\n- add per-IP/day dedupe + daily pruning\n- keep moderation gating + deterministic zips\n- add optional forwarded-IP trust via TRUST_FORWARDED_IPS
2026-02-13 16:22:05 +01:00
Brian KasperandPeter Steinberger dd58dd0815 Fix initial skill sorting (#92)
* fix: initial skill sorting

* chore: update unit test

* fix: use correct indexes for skill sorting

* chore: cleanup

* fix: land skill sorting update (#92) (thanks @bpk9)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 16:01:37 +01:00
Tanuj BhaudandPeter Steinberger 37a35c955a fix(vt): explicit return types and missing undici dependency (#255)
* fix(vt): explicit return types and missing undici dependency

Refactor action handlers in convex/vt.ts to use explicit return types, resolving circular type inference (TS7022). Also add undici to devDependencies for E2E tests.

* fix: add root undici devDependency for e2e (#255) (thanks @tanujbhaud)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 15:25:03 +01:00
Peter SteinbergerandSash Zats ddddb431c2 fix: make /search host-aware in SSR (#257)
* fix: make /search mode-aware

Notes:\n- Medium: /search now depends on getSiteMode() during beforeLoad. On server-side routing, if VITE_SITE_MODE isn’t set and VITE_SOULHUB_SITE_URL is set (as in .env.local), getSiteMode() will resolve to souls and redirect /search to / even on the ClawdHub deployment. This is a regression risk vs the old always-/skills redirect. Confirm deployment envs guarantee correct mode. src/routes/search.tsx:9-31

* fix: make /search host-aware in SSR

* chore: fix lint and route tree for /search route

---------

Co-authored-by: Sash Zats <sash@zats.io>
2026-02-13 14:26:31 +01:00
David AronchickandPeter Steinberger 78c27579a3 fix(cli): secure config file permissions (#164)
* fix(cli): secure config file permissions and reduce duplication

Security:
- Config files now created with 0600 permissions (owner read/write only)
- Config directories created with 0700 permissions
- Protects API tokens from other users on shared systems

Maintainability:
- Extract resolveConfigPath() helper to reduce code duplication
- Same legacy fallback logic (clawhub -> clawdhub) now in one place

* fix(cli): tolerate unsupported chmod errors for config

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 13:57:03 +01:00
xcqtnrandPeter Steinberger 9aebc35d86 fix: prevent infinite loading loop on skills page (#90)
* fix: prevent infinite loading loop on skills pageAdd isLoadingMore guard to IntersectionObserver useEffect to preventcontinuous WebSocket queries when user is idle at bottom of page.The observer now won't set up while a request is in progress, breakingthe infinite loop cycle.Fixes: Related to #89

* fix: prevent repeated skills auto-load requests (#90) (thanks @xcqtnr)

* fix: resolve PR merge conflicts and keep observer regression test (#90) (thanks @xcqtnr)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 05:44:36 +01:00
Peter Steinberger fc63f47ffa chore(release): 0.6.1 2026-02-13 05:14:16 +01:00
Gaurav SharmaandPeter Steinberger 0b83ea6ff3 fix: prevent horizontal overflow from long code blocks in skill pages (#183)
* Fix: Prevent horizontal overflow from long code blocks in skill pages

- Add max-width: 100% to .file-list-body and .file-row
- Prevents page-wide overflow when skills contain long code examples
- Markdown pre blocks already have overflow-x: auto, but parent containers were expanding infinitely
- Fixes issue where skills with 400+ char lines (e.g. browser automation commands) cause horizontal scrolling

Affected: Skills with long inline code in markdown (browser act commands, etc.)

* fix: add max-width to .file-list container to prevent overflow

- Also ensures .file-list-body constraint is inherited properly
- Prevents long code blocks from expanding file list container

* fix: add max-width to all markdown containers and pre tags

- Add max-width: 100% to .markdown, .tab-body, .markdown pre
- Ensures code blocks are constrained and show horizontal scrollbar
- Prevents content from expanding parent containers beyond viewport

* fix: add overflow-x to parent containers for horizontal scroll

Adds overflow-x: auto to .skill-detail-stack, .tab-card, and .tab-body
to ensure long code blocks are scrollable within the content area
instead of causing page-wide horizontal overflow.

Fixes horizontal overflow issue on skill pages with long code examples
(e.g., browser automation commands with 400+ character lines).

Tested on zepto skill page - page now stays within viewport (1200px)
and code blocks are accessible via horizontal scrollbar in tab area.

* docs: note code-block overflow fix in changelog (#183) (thanks @bewithgaurav)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 05:09:31 +01:00
LimitlessandLimitless2023 191b5763ec fix: include comment deltas in action-based stat processing & add stats reconciliation (#194)
Bug 1: applyAggregatedStatsAndUpdateCursor was missing 'comments' in both
the guard condition and the applySkillStatDeltas call. This caused comment
count deltas to be silently dropped during cron-based event processing,
while stars/downloads/installs were processed correctly.

Bug 2: No reconciliation mechanism existed. If events were missed due to
cursor issues or processing errors, skill stats (stars, comments) would
remain stale with no way to recover. Added reconcileSkillStarCounts
maintenance mutation that counts actual records in the stars and comments
tables and patches any out-of-sync skill stats.

Fixes #193

Co-authored-by: Limitless2023 <limitless@users.noreply.github.com>
2026-02-13 04:55:16 +01:00
Peter Steinberger 5397a8e5e0 fix: scope reauth fix; keep banned users blocked (#177) (thanks @tanujbhaud) 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 7949402888 test: add missing coverage for fresh-login reactivation and identity mismatch guard 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 2e3920d41a fix: use valid crons.interval and set to 1 minute 2026-02-13 04:34:46 +01:00
Tanuj Bhaud dd4fc823f6 fix: allow re-auth when existingUserId is null 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 135b9ea9b0 fix: ensure reactivation only matches soft-deleted user (prevents bypass) 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 17a106cefe fix: resolve final lint error in auth tests 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 496da99392 fix: update tests to include required existingUserId parameter 2026-02-13 04:34:46 +01:00
Tanuj Bhaud 107486adfb fix: restore existingUserId check for type safety 2026-02-13 04:34:46 +01:00
Tanuj Bhaud ef9e7f0e57 test: update auth tests for direct deletedAt check 2026-02-13 04:34:46 +01:00
Tanuj Bhaud ecc8ad3833 fix: allow soft-deleted users to re-authenticate
Fixes Issue #32 where users who soft-deleted their accounts were unable to sign back in because the re-auth logic was only triggering when an existingUserId was passed by the auth provider, which doesn't happen during a standard fresh login flow.
2026-02-13 04:34:46 +01:00
Peter Steinberger 93c2b23b72 docs: add 0.6.1 unreleased changelog from post-0.6.0 commits 2026-02-13 04:14:16 +01:00
DCollandPeter Steinberger 2e492a5b87 fix(http): remove allowH2 from undici Agent — causes fetch failed on Node.js 22+ (#245)
* Remove allowH2 option from global dispatcher

fix/remove-allowH2-undici-node22-compat

* fix(http): remove allowH2 from e2e dispatcher

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 03:02:57 +01:00
Peter Steinberger 91c87d1322 test: fix search test handler typing 2026-02-13 02:48:16 +01:00
Peter Steinberger 91b8f160f8 test: add search fallback coverage 2026-02-13 02:44:53 +01:00
Peter Steinberger 32f3ce45e9 fix: add lexical fallback for skill search recall 2026-02-13 02:22:32 +01:00
Peter Steinberger 19bfe48a67 fix: prioritize relevant skills in search 2026-02-13 02:15:19 +01:00
Peter Steinberger 19951fccf7 docs: thank @superlowburn for PR #246 2026-02-13 01:22:38 +01:00
7dcada9122 fix: handle GitHub API rate limits in account age check (#246)
* fix: handle GitHub API rate limits in account age check

The GitHub account lookup uses unauthenticated requests (60 req/hr
per IP). Since this runs server-side in Convex, all users share the
same IP and quickly exhaust the rate limit, causing "GitHub account
lookup failed" errors during skill publish.

- Detect 403/429 responses and surface a clear rate-limit message
- Support optional GITHUB_TOKEN env var for authenticated requests
  (5,000 req/hr)

Fixes #155

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: stabilize GitHub account gate tests and docs

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-13 01:21:30 +01:00
theonejvo dab307cb6d fix: VT scan sync race condition + LLM-first moderation model
VT no longer overwrites LLM moderation verdicts. LLM is the primary
moderation authority; VT only escalates (hides + flags) for malicious/
suspicious content via new escalateByVtInternal mutation. Stale VT polls
write vtAnalysis marker instead of overwriting moderationReason. Query
pools expanded to include LLM-evaluated skills awaiting VT results.
Ban message now references malicious skills and security@openclaw.ai.
2026-02-13 01:05:52 +11:00
vignesh07 e96eb4781c chore: fix review comments 2026-02-11 10:38:26 -08:00
Vignesh f64b098fcc perf: lazy-load diff viewer (Monaco) (#212) 2026-02-11 12:31:07 -06:00
Vignesh 243432e04e chore: fix lint issues (#213) 2026-02-11 12:31:01 -06:00
theonejvo 9f7b9b92cd fix: trailing comma tolerance in JSON metadata, tone down persistence flags
- Strip trailing commas in frontmatter JSON before parsing (silent failure fix)
- Stop flagging disable-model-invocation default as a concern (it's the normal default)
- Stop flagging skills configuring themselves as privilege escalation
- Add MITRE ATLAS AML.T0051 context for when autonomous invocation actually matters
- Show actual defaults in assembled eval message instead of "not set"
2026-02-11 18:56:19 +11:00
theonejvo 3402f0e735 feat: add skill metadata docs, suspicious appeal banner for owners
- Document full frontmatter metadata reference in docs/skill-format.md
- Add metadata section + quick example to README
- Show appeal message on suspicious skills (owner-only) linking to GitHub issues
- Accept metadata.openclaw alias in README docs
- Re-evaluate all skills with full file content reading (backfill in progress)
2026-02-11 18:36:45 +11:00
theonejvo 3e39651074 feat: evaluator reads all file contents, not just SKILL.md
Reads all files from storage and includes their full source in the eval
prompt so the LLM can detect malicious code hidden behind clean READMEs.
Injection detection now scans all content. Per-file cap 10K chars, total
cap 50K chars.
2026-02-11 17:23:19 +11:00
theonejvo 741301848f fix: eval assembler falls back to metadata.openclaw for requirements 2026-02-11 14:39:02 +11:00
theonejvo b593dedba1 feat: recognize metadata.openclaw as valid frontmatter namespace 2026-02-11 14:29:11 +11:00
theonejvo e104b8030e fix: increase max_output_tokens for reasoning model, fix backfill error retry 2026-02-11 04:48:08 +11:00
theonejvo e74e879e12 fix: add retry with backoff for OpenAI rate limits, fix JSON mode requirement 2026-02-11 02:48:22 +11:00
theonejvo 1f1d93ded9 fix: collapse OpenClaw analysis by default, fix row spacing, switch to gpt-5-mini 2026-02-11 02:33:17 +11:00
theonejvo 9c31462f15 feat: add LLM security evaluation at publish time
Add OpenClaw LLM-based security evaluator that runs alongside VirusTotal
when skills are published. Reads SKILL.md prose, metadata, install specs,
and file manifest, then assesses coherence across 5 dimensions to catch
social engineering vectors that VT/regex miss (e.g. instruction-only skills
with no code files).

- convex/lib/securityPrompt.ts: system prompt, message assembly, response
  parsing, injection pattern detection
- convex/llmEval.ts: evaluateWithLlm action, evaluateBySlug convenience
  action, backfillLlmEval for existing skills
- convex/schema.ts: llmAnalysis field on skillVersions
- convex/skills.ts: updateVersionLlmAnalysisInternal mutation,
  getActiveSkillBatchForLlmBackfillInternal query, defense-in-depth
  multi-scanner flag merging in approveSkillByHashInternal
- convex/lib/skillPublish.ts: schedule LLM eval alongside VT scan
- SkillDetailPage.tsx: OpenClaw row, LlmAnalysisDetail expandable
  component with 5 dimension rows, guidance panel, findings section
- styles.css: analysis detail styles from mockup
2026-02-11 02:19:03 +11:00
Peter Steinberger f53be5c8d2 docs: reset changelog for next release 2026-02-10 13:22:09 +01:00
Peter Steinberger 03164fe8b1 chore: release 0.6.0 2026-02-10 13:20:16 +01:00
Peter Steinberger 81b53f0b20 feat: add ban reasons to moderation 2026-02-10 13:11:47 +01:00
theonejvo 75474075b1 fix(vt): self-sustaining VT scanning + working stats
- getStatsInternal: derive VT stats from moderationReason instead of
  N+1 version lookups that hit the 16MB byte limit
- UI: read cached vtAnalysis from version docs instead of hitting the
  live VT API on every page view
- Backfill: add vt-cache-backfill cron (30min) with self-scheduling to
  drain the backlog of skills missing cached vtAnalysis
- Daily rescan: cursor-based batching (100/batch) with self-scheduling
  instead of loading all skills in one shot
2026-02-10 20:39:27 +11:00
theonejvo 5946a08267 fix(convex): reduce write conflicts across hot paths
- downloads:increment: remove unnecessary db.get that added skill doc
  to read set, causing conflicts with the stat processing cron
- users:ensure: only patch when there are real field changes, skip
  unconditional updatedAt bump that forced a write on every call
- comments: route stats through event sourcing (insertStatEvent) instead
  of synchronous read-modify-write on the skill doc
- rateLimits: split into query-first check + conditional mutation so
  denied requests are conflict-free reads
- skillStatEvents: reduce MAX_SKILLS_PER_RUN from 500 to 50 to shrink
  the write set and lower conflict probability with concurrent mutations

Co-Authored-By: theonejvo <theonejvo@users.noreply.github.com>
2026-02-10 17:58:43 +11:00
theonejvo b997f5e749 fix(security): block downloads for unscanned/moderated skills
The download endpoint now checks moderation status before serving zips:

- Pending scan (423): "This skill is pending a security scan by VirusTotal. Please try again in a few minutes."
- Malicious (403): "Blocked: this skill has been flagged as malicious by VirusTotal and cannot be downloaded."
- Removed (410): "This skill has been removed by a moderator."
- Hidden (403): "This skill is currently unavailable."

Closes the supply chain gap where a newly published version could be
downloaded before VT scanning completed.
2026-02-08 23:49:23 +11:00
theonejvo 27f300bda5 chore: trigger CI 2026-02-08 16:55:41 +11:00
theonejvo 557f11985d chore: fix lint warnings (unused vars, formatting) 2026-02-08 16:49:43 +11:00
theonejvo 40f9ba5697 chore(clawhub): bump version to 0.5.1 2026-02-08 16:43:45 +11:00
theonejvo 26a353efc7 feat(cli): enforce VT Code Insight moderation on install/update
CLI now checks moderation status before installing skills:

**Suspicious skills** - Shows warning and requires confirmation:
```
⚠️  Warning: "skill-name" is flagged as suspicious by VirusTotal Code Insight.
   This skill may contain risky patterns (crypto keys, external APIs, eval, etc.)
   Review the skill code before use.

? Install anyway? (y/N)
```
Non-interactive mode requires --force flag.

**Malicious skills** - Blocked entirely:
```
✖ Blocked: skill-name is flagged as malicious
Error: This skill has been flagged as malware and cannot be installed.
```

Changes:
- API now returns `moderation` field with `isSuspicious` and `isMalwareBlocked`
- CLI schema updated to expect moderation field
- cmdInstall and cmdUpdate enforce moderation checks

Thanks to @zackkorman for raising this issue.
2026-02-08 16:36:55 +11:00
theonejvo 990d3d730d feat: VT backfill infrastructure and 99.7% scan coverage
- Add getQuickStatsInternal for fast dashboard stats
- Fix getStatsInternal to include null moderationStatus skills
- Add syncModerationReasons to sync vtAnalysis → moderationReason
- Add requestReanalysisForPending to push stuck skills to VT
- Add backfillActiveSkillsVTCache improvements for efficiency
- Add fixNullModerationStatus for legacy skill cleanup
- Add getPendingVTSkillsInternal for monitoring
- Fix getActiveSkillsMissingVTCacheInternal to avoid read limits

Backfilled 5,000+ skills to 99.7% VT Code Insight coverage:
- Clean: 3,537 (70.5%)
- Suspicious: 1,336 (26.6%)
- Malicious: 123 (2.5%)
- Pending: 17 (0.3%)
2026-02-08 15:37:11 +11:00
Peter Steinberger 75f7a93fe8 fix: restore soft-deleted users on reauth (#106) (thanks @mkrokosz) 2026-02-06 16:58:19 -08:00
Matthew KrokoszandClaude Opus 4.5 bc51ab4b5f fix: Convert userId to string for targetId comparison
The auditLogs.targetId field is v.string() in the schema, so explicitly
convert the Id<'users'> to string to ensure type-safe comparison.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-06 16:58:19 -08:00
Matthew KrokoszandClaude Opus 4.5 82fce7f34f fix: Restore soft-deleted users on re-authentication (with ban check)
Users who deleted their account were unable to sign in again with the
same GitHub account. The OAuth flow would complete but the user would
remain logged out due to the `deletedAt` field being set.

This fix adds a `createOrUpdateUser` callback that:
1. Detects soft-deleted users during OAuth
2. Checks audit logs to determine if user was BANNED vs SELF-DELETED
3. If banned → throws error "This account has been suspended"
4. If self-deleted → clears `deletedAt` to restore account

Security: Both `deleteAccount` and `banUser` set the same `deletedAt`
field. This fix ensures banned users cannot restore their accounts.

Performance: The callback runs on every sign-in, but the audit log
query ONLY executes for soft-deleted users (rare edge case). Normal
active users just hit a single `if` check - no extra queries. When
the audit log query does run, it uses the `by_target` index for
efficient lookup.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-06 16:58:19 -08:00
Peter Steinberger 5f4fa02b42 fix: update footer branding to OpenClaw (#122) (thanks @jontsai) 2026-02-06 16:52:01 -08:00
Jonathan Tsai cfef87059b fix: update footer branding from Clawdbot to OpenClaw
- Change 'A Clawdbot project' to 'An OpenClaw project'
- Update link from clawd.bot to openclaw.ai
2026-02-06 16:52:01 -08:00
Peter Steinberger b948b216c6 fix: backfill empty handles in ensure (#158) (thanks @adlai88) 2026-02-06 16:24:28 -08:00
adlai88 f156b909e7 fix: handle empty-string handle in ensure() fallback
`??` (nullish coalescing) treats `""` as a valid value, so users
with `handle=""` never get a fallback derived from name/email.
Change to `||` so empty strings fall through to the next candidate.

This affects any user whose handle field is an empty string rather
than null/undefined — e.g. early accounts or migration artifacts.
2026-02-06 16:24:28 -08:00
nikniknikbbb 0b46210f61 Update README.md (#140) 2026-02-06 16:12:04 -08:00
theonejvo 03eee4b6de fix: self-healing VT scan queue
- Batch size 100 (was 10) - process more per run
- Skip skills checked in last 60 min - avoid hammering same hashes
- After 10 failed checks, mark as pending.scan.stale - drops from queue
- Track scanLastCheckedAt and scanCheckCount on skills
- Add TODO for webhook/notification setup
2026-02-07 00:20:49 +11:00
theonejvo 09ffaba89b fix: randomize VT scan queue + add health monitoring
- Fetch 5x batch size and shuffle to avoid queue head-blocking
- Add getScanQueueHealthInternal to monitor queue status
- Log warnings when queue is unhealthy (>50 pending or >24h stale)
- Return health stats from pollPendingScans
2026-02-07 00:06:03 +11:00
theonejvo b39f203b12 fix: use yellow styling for suspicious code insight blocks
- Malicious verdicts keep red border/background
- Suspicious verdicts now use yellow/amber to match badge color
2026-02-06 20:40:41 +11:00
theonejvo 4002bb615d chore: fix lint errors 2026-02-06 16:39:27 +11:00
theonejvo 583e391d83 chore: add friendly wait message for pending scans 2026-02-06 16:37:24 +11:00
theonejvo 55ae6e1eb7 feat: add VT rescan trigger and backfill function
- Add requestRescan() to trigger Code Insight via /analyse endpoint
- Update pollPendingScans to request rescan when no Code Insight
- Add backfillPendingScans for one-time backlog clearing
2026-02-06 16:36:49 +11:00
theonejvo b4fe685a46 chore: fix lint formatting 2026-02-06 16:01:53 +11:00
theonejvo a93de9cf92 feat: add cron job to poll VT for pending scan results
- Add pollPendingScans action to vt.ts that checks VT for Code Insight verdicts
- Add getPendingScanSkillsInternal query to get skills awaiting scan
- Add vt-pending-scans cron job running every 5 minutes
- Updates skill moderation status when VT analysis is complete
2026-02-06 15:51:24 +11:00
theonejvo 74bf5946e1 chore: fix lint formatting 2026-02-06 15:11:43 +11:00
theonejvo 04c99aafc2 chore: remove debug console.log 2026-02-06 15:09:08 +11:00
theonejvo ee81325cb3 feat: VT Code Insight visibility matrix for malicious/suspicious skills
- Malicious skills: visible for transparency, downloads blocked via moderationFlags
- Suspicious skills: visible with warning banner, downloads allowed
- Neither appears in search/listings (not indexed)
- Add isSuspicious flag to moderation info
- Update approveSkillByHashInternal to set moderationFlags properly
- Add warning banner CSS variant for suspicious skills
2026-02-06 15:08:52 +11:00
theonejvo bdc6348b1a fix: show verdict labels (Benign/Suspicious/Malicious) instead of engine stats 2026-02-06 12:52:54 +11:00
theonejvo 68abaa3642 debug: add logging to getBySlug query 2026-02-05 20:36:43 +11:00
theonejvo 0c3acda26e fix: remove unused isModerated variable 2026-02-05 20:27:33 +11:00
theonejvo 650090d298 fix: allow owners to see all moderated skills, not just pending.scan 2026-02-05 20:26:29 +11:00
theonejvo 70220f9487 merge: resolve conflicts with main 2026-02-05 20:12:55 +11:00
theonejvo aad1fbe2c4 fix: use computed badges in pending skill response 2026-02-05 20:08:26 +11:00
theonejvo 2d7914859c feat: show Code Insight analysis for malicious skills
- Add `source` field to VT results to indicate code_insight vs engines
- Display Code Insight analysis text when AI detects malicious patterns
- Only show "X/Y engines" when traditional AV detection triggers
- Add styled analysis block with red accent for malicious verdicts
2026-02-05 19:58:31 +11:00
ba6a99a65f fix: show pending skill page to owners instead of "Skill not found" (#136)
* fix: show pending skill page to owners instead of "Skill not found"

When a skill owner uploads a skill that's pending VirusTotal scan,
they now see their skill page with a pending banner instead of
"Skill not found". The banner explains the scan is in progress.

Changes:
- Modified getBySlug query to return skill data for owners even when
  moderationStatus is 'hidden' with reason 'pending.scan'
- Added pendingReview flag to query response
- Added pending banner component to SkillDetailPage
- Added CSS for pending banner using existing ClawHub gold theme

* fix: show pending skills on owner's dashboard

Extended the list query to include pending skills when the requester
is viewing their own dashboard. Added "Scanning" badge with gold theme
to indicate skills pending VirusTotal review.

* fix: show all moderation states to owners with appropriate UI

- Owners see their blocked/removed skills with explanatory banners
- Red banner for malware-blocked and removed skills
- Gold banner for pending scan
- Download button hidden for blocked/removed skills
- Added security disclaimer: "Like a lobster shell, security has layers"
- Fixed badges bug (use computed badges, not stale skill.badges)

* feat: make malware-blocked skills publicly visible

Blocked skills are now visible to everyone via direct URL:
- Shows red banner with "security issue detected"
- Displays VT scan results
- No download button
- Still hidden from listings/search

Sends a strong transparency signal about security enforcement.

* fix: allow owners to view pending scan skills (#136)

* fix: make deterministic zip date timezone-safe

* chore: update convex api types

* fix: update changelog for pending scan visibility (#136) (thanks @orlyjamie)

---------

Co-authored-by: theonejvo <theonejvo@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-02-04 23:26:49 -08:00
Peter Steinberger 4ac63c2373 fix: update changelog for pending scan visibility (#136) (thanks @orlyjamie) 2026-02-04 23:25:41 -08:00
Peter Steinberger 8d0f32443a chore: update convex api types 2026-02-04 23:23:13 -08:00
Peter Steinberger 9e75090a39 fix: make deterministic zip date timezone-safe 2026-02-04 23:19:58 -08:00
Peter Steinberger 5cff71104c fix: allow owners to view pending scan skills (#136) 2026-02-04 23:19:53 -08:00
theonejvo 5ef8526874 feat: make malware-blocked skills publicly visible
Blocked skills are now visible to everyone via direct URL:
- Shows red banner with "security issue detected"
- Displays VT scan results
- No download button
- Still hidden from listings/search

Sends a strong transparency signal about security enforcement.
2026-02-05 17:55:29 +11:00
theonejvo 46faafc413 fix: show all moderation states to owners with appropriate UI
- Owners see their blocked/removed skills with explanatory banners
- Red banner for malware-blocked and removed skills
- Gold banner for pending scan
- Download button hidden for blocked/removed skills
- Added security disclaimer: "Like a lobster shell, security has layers"
- Fixed badges bug (use computed badges, not stale skill.badges)
2026-02-05 17:47:04 +11:00
theonejvo 0710b99ac4 fix: show pending skills on owner's dashboard
Extended the list query to include pending skills when the requester
is viewing their own dashboard. Added "Scanning" badge with gold theme
to indicate skills pending VirusTotal review.
2026-02-05 14:51:13 +11:00
theonejvo 9ebe1b6da8 fix: show pending skill page to owners instead of "Skill not found"
When a skill owner uploads a skill that's pending VirusTotal scan,
they now see their skill page with a pending banner instead of
"Skill not found". The banner explains the scan is in progress.

Changes:
- Modified getBySlug query to return skill data for owners even when
  moderationStatus is 'hidden' with reason 'pending.scan'
- Added pendingReview flag to query response
- Added pending banner component to SkillDetailPage
- Added CSS for pending banner using existing ClawHub gold theme
2026-02-05 14:41:00 +11:00
Shakker 98dd49c651 style: fix badges test import formatting
Format import statement to single line per Biome rules.
2026-02-05 03:04:26 +00:00
Shakker a115bb0d65 docs: update changelog for coverage improvements
Add entries for new tests and coverage configuration changes.
2026-02-05 03:04:26 +00:00
Shakker f1625cd5ff chore: add skillZip.ts to coverage tracking
Include convex/lib/skillZip.ts in coverage reports to track
the deterministic ZIP building utility added in PR #130.
2026-02-05 03:04:26 +00:00
Shakker 010ca1677f test: add 4+ errors truncation test for ark schema
Test the formatArkErrors truncation logic when there are more than
3 validation errors, ensuring the "+N more" message is displayed.
2026-02-05 03:04:26 +00:00
Shakker 62d62f976c test: add expandDroppedItems tests for uploadFiles
Add jsdom tests for the expandDroppedItems function:
- Handles null/empty DataTransferItemList
- Collects files via getAsFile fallback
- Collects files via webkitGetAsEntry for file entries
- Recursively collects files from directory entries
- Skips non-file/non-directory entries

Improves branch coverage for uploadFiles.ts.
2026-02-05 03:04:26 +00:00
Shakker 24ad575a4e test: add skillZip module tests
Add tests for the deterministic ZIP building utility from PR #130:
- buildSkillMeta function
- buildDeterministicZip with various scenarios
- Verifies deterministic output and _meta.json inclusion

Achieves 100% coverage for skillZip.ts.
2026-02-05 03:04:26 +00:00
Shakker 1cd6a22284 test: add badges module tests
Add comprehensive tests for the badges utility functions:
- isSkillHighlighted, isSkillOfficial, isSkillDeprecated
- getSkillBadges with all badge combinations

Improves branch coverage from 50% to 100% for badges.ts.
2026-02-05 03:04:26 +00:00
Peter Steinberger 343e065292 fix: stabilize VT scans and UI fetches (#130) (thanks @aleph8) 2026-02-04 17:34:35 -08:00
Alejandro García Peláez 2849a864e0 VirusTotal Integration on ClawHub (#130)
* feat: implementation of dynamic VirusTotal integration and deterministic ZIPs

* fix: do not show security scan results if hash is missing

* ui: show 'Loading...' instead of 'Pending' while fetching VT results

* security: restrict auto-approval to explicit benign verdicts only

* fix: prioritize AI verdict in results and refine stats fallback
2026-02-04 17:33:48 -08:00
Peter Steinberger ae0338e469 feat: add fuzzy user search for moderation CLI 2026-02-04 03:44:54 -08:00
Peter Steinberger ca1ef737cb fix: resolve typecheck errors in api and config 2026-02-04 03:32:09 -08:00
Peter Steinberger 589e46353b fix: management user search and totals 2026-02-04 03:30:07 -08:00
Peter Steinberger 7edbc03494 fix: skill list pagination and footer branding 2026-02-04 03:18:07 -08:00
Peter Steinberger f359071d96 feat(moderation): add set-role 2026-02-02 04:40:53 -08:00
Peter Steinberger 57bae9859e chore(release): 0.5.0 2026-02-02 04:25:11 -08:00
Peter Steinberger 96e9ffdcdc fix(convex): batch hard delete skills 2026-02-02 04:15:22 -08:00
Peter Steinberger eb4601141e fix(cli): honor registry from auth login 2026-02-02 03:25:14 -08:00
Peter Steinberger 39686b3b8d feat(management): add report and user filters 2026-02-02 03:02:52 -08:00
Peter Steinberger a24d3e9809 feat(cli): add inspect and moderation tools 2026-02-02 02:55:56 -08:00
Peter Steinberger 405c74a4ef feat: require report reasons 2026-02-02 00:57:45 -08:00
Peter Steinberger ee828046b8 chore: update dependencies 2026-02-02 00:31:54 -08:00
Peter Steinberger 789082bc00 chore: suppress empty chunk warnings 2026-02-02 00:27:01 -08:00
Peter Steinberger 3e4c2450cd chore: suppress nitro build warnings 2026-02-02 00:25:28 -08:00
Peter Steinberger ea2f51d2ba chore: quiet build warnings 2026-02-02 00:22:04 -08:00
Peter Steinberger 7b2bdbd08f feat: harden moderation and upload safety 2026-02-02 00:17:34 -08:00
Peter Steinberger f654dc9325 fix: allow legacy skill fields in schema 2026-01-31 12:45:35 +01:00
Peter Steinberger d78c105570 feat: add admin user ban 2026-01-31 11:44:31 +01:00
Peter Steinberger a32498ea7d fix: use ClawHub branding for registry 2026-01-31 11:31:46 +01:00
Shadow 5d6ee7adf3 trigger new deploy 2026-01-30 13:05:02 -06:00
Peter Steinberger ffa25f47da chore: rebrand user-facing to OpenClaw 2026-01-30 07:02:35 +01:00
Peter Steinberger f866dc05d1 style: format moderation flags 2026-01-30 05:27:19 +01:00
Peter Steinberger 69436fd79b fix: allow moltbot parsed data 2026-01-30 05:26:09 +01:00
Peter Steinberger f185ca6f55 feat: release 0.4.0 2026-01-30 05:23:12 +01:00
Peter Steinberger f3fc8d62b6 Revert "chore: rename molthub branding"
This reverts commit 8d68b55333.
2026-01-30 05:23:12 +01:00
Shakker 71f94a774f Merge pull request #70 from moltbot/remove-clawd-authenticator-tool
chore(moderation): block ClawdAuthenticatorTool (suspected malware)
2026-01-29 17:18:09 +00:00
vignesh07 07349e6107 chore(moderation): block ClawdAuthenticatorTool listing 2026-01-29 09:14:41 -08:00
Vignesh 5ef00e8c6a chore(security): harden file endpoints CSP + XFO + svg detection (#67) 2026-01-28 23:21:16 -06:00
Jamieson O'Reillyandtheonejvo c5e5e657dd fix: add CSP headers and Content-Disposition to prevent SVG XSS (#61)
Co-authored-by: theonejvo <theonejvo@users.noreply.github.com>
2026-01-28 22:21:39 -06:00
Jamie Turner 69e1e5c507 A few fixes for search. (#64) 2026-01-28 22:21:26 -06:00
Josh Palmer 481f1b9188 Merge pull request #66 from moltbot/fix/ci-lint
fix: restore lint compliance
2026-01-28 21:27:21 +01:00
Josh Palmer d4f5832554 🤖 chore: merge main into fix/ci-lint
- resolve conflicts in search/skill publish files
- apply Biome formatting updates from main

Tests: bun run lint:biome; bun run lint:oxlint
2026-01-28 21:23:50 +01:00
Josh Palmer ca3275ba92 🤖 fix: restore lint compliance
- apply Biome formatting and import ordering across linted files
- fix management useEffect dependencies flagged by Biome

Tests: bun run lint:biome; bun run lint:oxlint
2026-01-28 21:18:19 +01:00
Shadow d577add8a0 fix: unblock convex deploy typecheck 2026-01-28 13:15:18 -06:00
Shadow 90bd065f7e fix: correct public skill entries build 2026-01-28 13:12:41 -06:00
Shadow 219f05b160 fix: sanitize public skill and soul data 2026-01-28 13:03:19 -06:00
Shadow c8091ee8a8 chore: remove unauthenticated badge backfill 2026-01-28 08:33:14 -06:00
Shadow 63f367e1d6 chore: add unauthenticated badge backfill 2026-01-28 08:31:04 -06:00
Shadow e701d7b713 feat: add skill badges table 2026-01-28 01:11:46 -06:00
Shadow 79bd1f1d6c fix: query highlighted skills by batch index 2026-01-28 00:34:33 -06:00
Shadow ac51cc0236 fix: rely on highlighted badge 2026-01-27 23:27:36 -06:00
Shadow b9c23dc00e feat: add reports dashboard for moderation 2026-01-27 22:27:20 -06:00
Shadow f4e96995bc fix: allow clawdis parsed metadata 2026-01-27 21:39:23 -06:00
Shadow 33165db598 feat: unify skill routes with owner slugs 2026-01-27 21:28:46 -06:00
Shadow 4c10e4847b feat: add moderation management and backfill 2026-01-27 21:11:37 -06:00
Shadow 8d68b55333 chore: rename molthub branding 2026-01-27 18:25:02 -06:00
Jamie Turner 251de1f540 Performance optimizations for /skills page 2026-01-27 15:18:21 -06:00
Shadow 123f60fa93 Revert "Performance optimizations for /skills page" temporarily until we can deploy
This reverts commit 643faf71c8.
2026-01-27 12:20:15 -06:00
Jamie Turner 643faf71c8 Performance optimizations for /skills page 2026-01-27 12:09:57 -06:00
Sergiy Dybskiy eb9a67f2af fix: use Error for timeout abort in e2e helper 2026-01-26 12:41:29 +00:00
Sergiy Dybskiy 1ae0498595 test: add e2e test for delete error handling
Verifies that deleting a non-existent skill returns a proper 'not found'
error instead of a generic 'Unauthorized' message.
2026-01-26 12:40:39 +00:00
Sergiy Dybskiy f1a5254755 fix(cli): use proper Error objects in abort timeouts
When AbortController.abort() receives a string instead of an Error,
the string itself is thrown. pRetry then wraps it in a confusing
message: 'Non-error was thrown: Timeout'

Changed all 3 occurrences in http.ts:
- apiRequest (line 57)
- apiRequestForm (line 106)
- downloadZip (line 141)

Now timeouts will surface as proper Error objects with clear messages.
2026-01-25 21:39:54 +00:00
Sergiy Dybskiy de2542e391 fix: return proper HTTP status codes for delete/undelete errors
The delete and undelete handlers for skills and souls were catching all
errors and returning 401 Unauthorized, even for errors like:
- 'Skill not found' (should be 404)
- 'Forbidden' (should be 403)
- Other validation errors (should be 400)

This change updates the error handling to return appropriate status codes:
- 401 Unauthorized: authentication failures
- 403 Forbidden: authorization failures (not owner/admin/moderator)
- 404 Not Found: skill/soul/user not found
- 400 Bad Request: other errors with descriptive message

Fixes #34
2026-01-25 21:12:02 +00:00
Aaron NgandPeter Steinberger a2c46fbb5d Search Fixes (#30)
* more search fixes

* update tests

* comments

* fix: tune search filters and limits (#30) (thanks @aaronn)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-01-25 00:08:29 +00:00
Peter Steinberger f51e0a087d test: fix lockfile mock version 2026-01-24 22:56:39 +00:00
emilianoandPeter Steinberger d9108b0948 feat: show published skills on user profile (#20)
* fix: resolve typecheck and lint errors

* fix: stabilize publish paths and token types

* feat: show published skills on user profile

* fix: document profile published skills (#20) (thanks @njoylab)

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-01-24 22:55:18 +00:00
Peter Steinberger d7a017e1c3 fix: add update lookup test (#22) (thanks @daveonkels) 2026-01-24 22:30:09 +00:00
Dave OnkelsandClaude Opus 4.5 fffdf82540 fix: use path instead of url for skill metadata API call (#22)
The `cmdUpdate` function was passing a relative path to `apiRequest`
using the `url` property, but `url` expects a full URL. When `url` is
provided, it's used as-is without combining with the registry base URL.

This caused "Failed to parse URL from /api/v1/skills/<slug>" errors
when updating skills that don't have a local fingerprint match.

Changed to use `path` property which correctly combines with the
registry base URL via `new URL(args.path, registry)`.

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 22:28:54 +00:00
Ahmed Fuad MireClaude Opus 4.5vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>Ahmed
a16e624766 fix: relax search token matching to require at least one match (#27)
* fix: relax search token matching to require at least one match

The search was requiring ALL query tokens to exist in the skill's
displayName, slug, or summary. This was too strict and caused valid
results to be filtered out. For example, searching "HTTP API client"
would fail to match skills about "HTTP API" that didn't mention "client".

Changed from `.every()` to `.some()` so at least one token must match,
allowing the vector similarity to determine relevance for the rest.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: update matchesExactTokens to require prefix matching for query tokens

* more inclusive token check

* Update convex/lib/searchText.ts

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>

---------

Co-authored-by: Ahmed <ahmed.mire@kaluza.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
2026-01-24 21:23:03 +00:00
Peter Steinberger decce1d35c fix: skip missing skills in search hydration (#28) (thanks @aaronn) 2026-01-24 21:11:46 +00:00
Aaron Ng 468832af3f fix search (#28) 2026-01-24 21:11:06 +00:00
625 changed files with 114602 additions and 20685 deletions
+13 -5
View File
@@ -11,11 +11,11 @@ jobs:
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@v2
- uses: oven-sh/setup-bun@e3914758a49697077f7bcd190d36582a61667aad
with:
bun-version: 1.3.6
bun-version: 1.3.10
- name: Install
run: bun install --frozen-lockfile
@@ -27,14 +27,22 @@ jobs:
- name: Test
run: bun run test
env:
VITE_CONVEX_URL: https://example.invalid
- name: Coverage
run: bun run coverage
env:
VITE_CONVEX_URL: https://example.invalid
- name: Typecheck packages
- name: ClawHub CLI Verify
run: bun run --cwd packages/clawhub verify
- name: Typecheck
run: |
bunx tsc --noEmit
bunx tsc -p packages/schema/tsconfig.json --noEmit
bunx tsc -p packages/clawdhub/tsconfig.json --noEmit
bunx tsc -p packages/clawhub/tsconfig.json --noEmit
- name: Build
run: bun run build
@@ -0,0 +1,314 @@
name: ClawHub CLI NPM Release
on:
workflow_dispatch:
inputs:
tag:
description: Release tag to publish, for example v0.10.0
required: true
type: string
preflight_only:
description: Run validation/build only and skip the gated publish job
required: true
default: false
type: boolean
preflight_run_id:
description: Existing successful preflight workflow run id to promote without rebuilding
required: false
type: string
concurrency:
group: clawhub-cli-npm-release-${{ inputs.tag }}
cancel-in-progress: false
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NODE_VERSION: "24.x"
BUN_VERSION: "1.3.10"
jobs:
preflight_clawhub_cli_npm:
if: ${{ inputs.preflight_only }}
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Forbid preflight artifact promotion on validation-only runs
if: ${{ inputs.preflight_run_id != '' }}
run: |
echo "preflight_run_id is only valid for real publish runs."
exit 1
- name: Checkout
uses: actions/checkout@v6
with:
ref: refs/tags/${{ inputs.tag }}
fetch-depth: 0
- name: Setup Node
uses: actions/setup-node@v6
with:
node-version: ${{ env.NODE_VERSION }}
registry-url: https://registry.npmjs.org
- name: Setup Bun
uses: oven-sh/setup-bun@e3914758a49697077f7bcd190d36582a61667aad
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Resolve CLI package directory
run: |
set -euo pipefail
if [[ -d "packages/clawhub" ]]; then
echo "PACKAGE_DIR=packages/clawhub" >> "$GITHUB_ENV"
elif [[ -d "packages/clawdhub" ]]; then
echo "PACKAGE_DIR=packages/clawdhub" >> "$GITHUB_ENV"
else
echo "Unable to find clawhub CLI package directory." >&2
exit 1
fi
- name: Ensure version is not already published
env:
PREFLIGHT_ONLY: ${{ inputs.preflight_only }}
run: |
set -euo pipefail
PACKAGE_VERSION="$(node --input-type=module <<'EOF'
import { readFileSync } from "node:fs";
const pkg = JSON.parse(readFileSync(`./${process.env.PACKAGE_DIR}/package.json`, "utf8"));
process.stdout.write(String(pkg.version ?? "").trim());
EOF
)"
if npm view "clawhub@${PACKAGE_VERSION}" version >/dev/null 2>&1; then
if [[ "${PREFLIGHT_ONLY}" == "true" ]]; then
echo "clawhub@${PACKAGE_VERSION} is already published on npm; continuing because preflight_only=true."
exit 0
fi
echo "clawhub@${PACKAGE_VERSION} is already published on npm."
exit 1
fi
echo "Publishing clawhub@${PACKAGE_VERSION}"
- name: Validate release tag and package metadata
env:
RELEASE_TAG: ${{ inputs.tag }}
RELEASE_MAIN_REF: origin/main
run: |
set -euo pipefail
RELEASE_SHA="$(git rev-parse HEAD)"
export RELEASE_SHA
git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main
node scripts/clawhub-cli-npm-release-check.mjs
- name: Verify CLI package
run: bun run --cwd "$PACKAGE_DIR" verify
- name: Pack prepared npm tarball
id: packed_tarball
env:
RELEASE_TAG: ${{ inputs.tag }}
run: |
set -euo pipefail
pushd "$PACKAGE_DIR" >/dev/null
PACK_JSON="$(npm pack --json --ignore-scripts)"
echo "$PACK_JSON"
PACK_PATH="$(printf '%s\n' "$PACK_JSON" | node --input-type=module -e 'const chunks=[]; process.stdin.on("data", (chunk) => chunks.push(chunk)); process.stdin.on("end", () => { const parsed = JSON.parse(Buffer.concat(chunks).toString("utf8")); const first = Array.isArray(parsed) ? parsed[0] : null; if (!first || typeof first.filename !== "string" || !first.filename) process.exit(1); process.stdout.write(first.filename); });')"
popd >/dev/null
if [[ -z "${PACK_PATH}" || ! -f "${PACKAGE_DIR}/${PACK_PATH}" ]]; then
echo "npm pack did not produce a tarball file." >&2
exit 1
fi
RELEASE_SHA="$(git rev-parse HEAD)"
PACKAGE_VERSION="$(node --input-type=module <<'EOF'
import { readFileSync } from "node:fs";
const pkg = JSON.parse(readFileSync(`./${process.env.PACKAGE_DIR}/package.json`, "utf8"));
process.stdout.write(String(pkg.version ?? "").trim());
EOF
)"
ARTIFACT_DIR="$RUNNER_TEMP/clawhub-cli-npm-preflight"
rm -rf "$ARTIFACT_DIR"
mkdir -p "$ARTIFACT_DIR"
cp "${PACKAGE_DIR}/${PACK_PATH}" "$ARTIFACT_DIR/"
printf '%s\n' "$RELEASE_TAG" > "$ARTIFACT_DIR/release-tag.txt"
printf '%s\n' "$RELEASE_SHA" > "$ARTIFACT_DIR/release-sha.txt"
printf '%s\n' "$PACKAGE_VERSION" > "$ARTIFACT_DIR/package-version.txt"
echo "dir=$ARTIFACT_DIR" >> "$GITHUB_OUTPUT"
- name: Upload prepared npm publish bundle
uses: actions/upload-artifact@v7
with:
name: clawhub-cli-npm-preflight-${{ inputs.tag }}
path: ${{ steps.packed_tarball.outputs.dir }}
if-no-files-found: error
validate_publish_request:
if: ${{ !inputs.preflight_only }}
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Require main workflow ref for publish
env:
WORKFLOW_REF: ${{ github.ref }}
run: |
set -euo pipefail
if [[ "${WORKFLOW_REF}" != "refs/heads/main" ]]; then
echo "Real publish runs must be dispatched from main. Use preflight_only=true for branch validation."
exit 1
fi
- name: Require preflight artifact promotion on real publish
env:
PREFLIGHT_RUN_ID: ${{ inputs.preflight_run_id }}
run: |
set -euo pipefail
if [[ -z "${PREFLIGHT_RUN_ID}" ]]; then
echo "Real publish requires preflight_run_id from a successful npm preflight run." >&2
exit 1
fi
publish_clawhub_cli_npm:
needs: [validate_publish_request]
if: ${{ !inputs.preflight_only }}
runs-on: ubuntu-latest
environment: npm-release
permissions:
actions: read
contents: read
id-token: write
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: refs/tags/${{ inputs.tag }}
fetch-depth: 0
- name: Setup Node
uses: actions/setup-node@v6
with:
node-version: ${{ env.NODE_VERSION }}
registry-url: https://registry.npmjs.org
- name: Resolve CLI package directory
run: |
set -euo pipefail
if [[ -d "packages/clawhub" ]]; then
echo "PACKAGE_DIR=packages/clawhub" >> "$GITHUB_ENV"
elif [[ -d "packages/clawdhub" ]]; then
echo "PACKAGE_DIR=packages/clawdhub" >> "$GITHUB_ENV"
else
echo "Unable to find clawhub CLI package directory." >&2
exit 1
fi
- name: Ensure version is not already published
run: |
set -euo pipefail
PACKAGE_VERSION="$(node --input-type=module <<'EOF'
import { readFileSync } from "node:fs";
const pkg = JSON.parse(readFileSync(`./${process.env.PACKAGE_DIR}/package.json`, "utf8"));
process.stdout.write(String(pkg.version ?? "").trim());
EOF
)"
if npm view "clawhub@${PACKAGE_VERSION}" version >/dev/null 2>&1; then
echo "clawhub@${PACKAGE_VERSION} is already published on npm."
exit 1
fi
echo "Publishing clawhub@${PACKAGE_VERSION}"
- name: Verify preflight run metadata
env:
GH_TOKEN: ${{ github.token }}
PREFLIGHT_RUN_ID: ${{ inputs.preflight_run_id }}
run: |
set -euo pipefail
RUN_JSON="$(gh run view "$PREFLIGHT_RUN_ID" --repo "$GITHUB_REPOSITORY" --json workflowName,headBranch,event,conclusion,url)"
printf '%s' "$RUN_JSON" | node --input-type=module -e 'const chunks=[]; process.stdin.on("data", (chunk) => chunks.push(chunk)); process.stdin.on("end", () => { const run = JSON.parse(Buffer.concat(chunks).toString("utf8")); const checks = [["workflowName", "ClawHub CLI NPM Release"], ["headBranch", "main"], ["event", "workflow_dispatch"], ["conclusion", "success"]]; for (const [key, expected] of checks) { if (run[key] !== expected) { console.error(`Referenced npm preflight run ${process.env.PREFLIGHT_RUN_ID} must have ${key}=${expected}, got ${run[key] ?? "<missing>"}.`); process.exit(1); } } console.log(`Using npm preflight run ${process.env.PREFLIGHT_RUN_ID}: ${run.url}`); });'
- name: Download prepared npm tarball
uses: actions/download-artifact@v8
with:
name: clawhub-cli-npm-preflight-${{ inputs.tag }}
path: preflight-tarball
repository: ${{ github.repository }}
run-id: ${{ inputs.preflight_run_id }}
github-token: ${{ github.token }}
- name: Validate release tag and package metadata
env:
RELEASE_TAG: ${{ inputs.tag }}
RELEASE_MAIN_REF: origin/main
run: |
set -euo pipefail
RELEASE_SHA="$(git rev-parse HEAD)"
export RELEASE_SHA
git fetch --no-tags origin +refs/heads/main:refs/remotes/origin/main
node scripts/clawhub-cli-npm-release-check.mjs
- name: Verify prepared tarball provenance
env:
RELEASE_TAG: ${{ inputs.tag }}
run: |
set -euo pipefail
EXPECTED_RELEASE_SHA="$(git rev-parse HEAD)"
EXPECTED_PACKAGE_VERSION="$(node --input-type=module <<'EOF'
import { readFileSync } from "node:fs";
const pkg = JSON.parse(readFileSync(`./${process.env.PACKAGE_DIR}/package.json`, "utf8"));
process.stdout.write(String(pkg.version ?? "").trim());
EOF
)"
TAG_FILE="preflight-tarball/release-tag.txt"
SHA_FILE="preflight-tarball/release-sha.txt"
VERSION_FILE="preflight-tarball/package-version.txt"
if [[ ! -f "$TAG_FILE" || ! -f "$SHA_FILE" || ! -f "$VERSION_FILE" ]]; then
echo "Prepared preflight metadata is missing." >&2
ls -la preflight-tarball >&2 || true
exit 1
fi
ARTIFACT_RELEASE_TAG="$(tr -d '\r\n' < "$TAG_FILE")"
ARTIFACT_RELEASE_SHA="$(tr -d '\r\n' < "$SHA_FILE")"
ARTIFACT_PACKAGE_VERSION="$(tr -d '\r\n' < "$VERSION_FILE")"
if [[ "$ARTIFACT_RELEASE_TAG" != "$RELEASE_TAG" ]]; then
echo "Prepared preflight tag mismatch: expected $RELEASE_TAG, got $ARTIFACT_RELEASE_TAG" >&2
exit 1
fi
if [[ "$ARTIFACT_RELEASE_SHA" != "$EXPECTED_RELEASE_SHA" ]]; then
echo "Prepared preflight SHA mismatch: expected $EXPECTED_RELEASE_SHA, got $ARTIFACT_RELEASE_SHA" >&2
exit 1
fi
if [[ "$ARTIFACT_PACKAGE_VERSION" != "$EXPECTED_PACKAGE_VERSION" ]]; then
echo "Prepared preflight package version mismatch: expected $EXPECTED_PACKAGE_VERSION, got $ARTIFACT_PACKAGE_VERSION" >&2
exit 1
fi
- name: Resolve publish tarball
id: publish_tarball
run: |
set -euo pipefail
TARBALL_PATH="$(find preflight-tarball -type f -name '*.tgz' -print | sort | tail -n 1)"
if [[ -z "$TARBALL_PATH" ]]; then
echo "Prepared preflight tarball not found." >&2
ls -la preflight-tarball >&2 || true
exit 1
fi
echo "path=$TARBALL_PATH" >> "$GITHUB_OUTPUT"
- name: Publish
run: |
set -euo pipefail
publish_target="${{ steps.publish_tarball.outputs.path }}"
if [[ -n "${publish_target}" ]]; then
publish_target="./${publish_target}"
fi
bash scripts/clawhub-cli-npm-publish.sh --publish "${publish_target}"
+176
View File
@@ -0,0 +1,176 @@
name: Deploy
on:
workflow_dispatch:
inputs:
target:
description: "What to deploy"
required: true
default: full
type: choice
options:
- full
- backend
- frontend
concurrency:
group: deploy-production
cancel-in-progress: true
jobs:
validate-deploy-request:
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
deploy_backend: ${{ steps.mode.outputs.deploy_backend }}
deploy_frontend: ${{ steps.mode.outputs.deploy_frontend }}
run_smoke: ${{ steps.mode.outputs.run_smoke }}
target: ${{ steps.mode.outputs.target }}
steps:
- name: Require main ref for production deploy
run: |
set -euo pipefail
if [[ "${GITHUB_REF}" != "refs/heads/main" ]]; then
echo "Production deploys must run from main."
exit 1
fi
- name: Resolve deploy mode
id: mode
run: |
set -euo pipefail
target="${{ inputs.target }}"
case "$target" in
full)
echo "deploy_backend=true" >> "$GITHUB_OUTPUT"
echo "deploy_frontend=true" >> "$GITHUB_OUTPUT"
echo "run_smoke=true" >> "$GITHUB_OUTPUT"
;;
backend)
echo "deploy_backend=true" >> "$GITHUB_OUTPUT"
echo "deploy_frontend=false" >> "$GITHUB_OUTPUT"
echo "run_smoke=true" >> "$GITHUB_OUTPUT"
;;
frontend)
echo "deploy_backend=false" >> "$GITHUB_OUTPUT"
echo "deploy_frontend=true" >> "$GITHUB_OUTPUT"
echo "run_smoke=true" >> "$GITHUB_OUTPUT"
;;
*)
echo "Unsupported deploy target: $target" >&2
exit 1
;;
esac
echo "target=$target" >> "$GITHUB_OUTPUT"
deploy-production:
runs-on: ubuntu-latest
timeout-minutes: 45
needs: validate-deploy-request
environment:
name: Production
url: https://clawhub.ai
env:
CONVEX_DEPLOY_KEY: ${{ secrets.CONVEX_DEPLOY_KEY }}
PLAYWRIGHT_AUTH_STORAGE_STATE_JSON: ${{ secrets.PLAYWRIGHT_AUTH_STORAGE_STATE_JSON }}
PLAYWRIGHT_BASE_URL: https://clawhub.ai
steps:
- name: Check deploy configuration
run: |
set -euo pipefail
missing=()
if [[ "${{ needs.validate-deploy-request.outputs.deploy_backend }}" == "true" && -z "$CONVEX_DEPLOY_KEY" ]]; then
missing+=("CONVEX_DEPLOY_KEY")
fi
if (( ${#missing[@]} > 0 )); then
echo "::error::Missing required production environment secrets: ${missing[*]}"
exit 1
fi
echo "Deploy target: ${{ needs.validate-deploy-request.outputs.target }}"
if [[ -z "$PLAYWRIGHT_AUTH_STORAGE_STATE_JSON" ]]; then
echo "PLAYWRIGHT_AUTH_STORAGE_STATE_JSON not set; authenticated smoke will be skipped."
fi
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@e3914758a49697077f7bcd190d36582a61667aad
with:
bun-version: 1.3.10
- name: Install
run: bun install --frozen-lockfile
- name: Stamp Convex build SHA
if: needs.validate-deploy-request.outputs.deploy_backend == 'true'
run: bunx convex env set APP_BUILD_SHA "${GITHUB_SHA}" --prod
- name: Stamp Convex deploy time
if: needs.validate-deploy-request.outputs.deploy_backend == 'true'
run: bunx convex env set APP_DEPLOYED_AT "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" --prod
- name: Deploy Convex
if: needs.validate-deploy-request.outputs.deploy_backend == 'true'
run: bun run convex:deploy
- name: Verify Convex contract
if: needs.validate-deploy-request.outputs.deploy_backend == 'true'
run: bun run verify:convex-contract -- --prod
- name: Wait for Vercel production deployment
if: needs.validate-deploy-request.outputs.deploy_frontend == 'true'
env:
GH_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_SHA: ${{ github.sha }}
VERCEL_STATUS_CONTEXT: Vercel clawhub
run: |
set -euo pipefail
for attempt in {1..90}; do
if ! state="$(gh api "repos/$GITHUB_REPOSITORY/commits/$GITHUB_SHA/status" \
--jq '.statuses[] | select(.context == env.VERCEL_STATUS_CONTEXT) | .state' \
2>/dev/null | head -n1)"; then
echo "GitHub status check failed for $GITHUB_SHA; retrying..."
sleep 10
continue
fi
case "$state" in
success)
echo "Vercel production deployment ready for $GITHUB_SHA"
exit 0
;;
failure|error)
echo "::error::Vercel production deployment failed for $GITHUB_SHA"
exit 1
;;
pending)
echo "Vercel deployment pending for $GITHUB_SHA; waiting..."
;;
*)
echo "Vercel status for $GITHUB_SHA not published yet; waiting..."
;;
esac
sleep 10
done
echo "::error::Timed out waiting for Vercel production deployment for $GITHUB_SHA"
exit 1
- name: Install Playwright browser
if: needs.validate-deploy-request.outputs.run_smoke == 'true'
run: bunx playwright install --with-deps chromium
- name: Write authenticated storage state
if: needs.validate-deploy-request.outputs.run_smoke == 'true' && env.PLAYWRIGHT_AUTH_STORAGE_STATE_JSON != ''
run: |
echo "$PLAYWRIGHT_AUTH_STORAGE_STATE_JSON" > "$RUNNER_TEMP/playwright-auth.json"
echo "PLAYWRIGHT_AUTH_STORAGE_STATE=$RUNNER_TEMP/playwright-auth.json" >> "$GITHUB_ENV"
- name: Smoke test production
if: needs.validate-deploy-request.outputs.run_smoke == 'true'
run: bunx playwright test e2e/menu-smoke.pw.test.ts e2e/upload-auth-smoke.pw.test.ts
+341
View File
@@ -0,0 +1,341 @@
name: Package Publish
on:
workflow_call:
inputs:
source:
description: Package source to publish. Usually owner/repo, owner/repo@ref, or a GitHub URL.
required: false
type: string
default: ""
ref:
description: Optional ref to append to the source when source is not already pinned.
required: false
type: string
dry_run:
description: Preview only. When true, no publish mutation is performed.
required: false
type: boolean
default: true
json:
description: Emit structured JSON output.
required: false
type: boolean
default: true
registry:
description: ClawHub registry URL.
required: false
type: string
default: https://clawhub.ai
site:
description: ClawHub site URL.
required: false
type: string
default: https://clawhub.ai
owner:
description: Optional owner handle override for org/shared publishing.
required: false
type: string
version:
description: Optional package version override.
required: false
type: string
tags:
description: Optional comma-separated tags override.
required: false
type: string
default: latest
source_repo:
description: Optional source repo override for local-folder publishes.
required: false
type: string
source_commit:
description: Optional source commit override for local-folder publishes.
required: false
type: string
source_ref:
description: Optional source ref override for local-folder publishes.
required: false
type: string
clawhub_version:
description: Legacy npm CLI version input. Kept for compatibility; the workflow now runs the checked-out source.
required: false
type: string
default: latest
secrets:
clawhub_token:
required: false
outputs:
publish_json:
description: Structured JSON output from clawhub package publish.
value: ${{ jobs.publish.outputs.publish_json }}
release_id:
description: Published release id when dry_run is false.
value: ${{ jobs.publish.outputs.release_id }}
jobs:
publish:
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
id-token: write
outputs:
publish_json: ${{ steps.capture.outputs.publish_json }}
release_id: ${{ steps.capture.outputs.release_id }}
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.sha }}
- uses: oven-sh/setup-bun@e3914758a49697077f7bcd190d36582a61667aad
with:
bun-version: 1.3.10
- name: Resolve ClawHub workflow source
id: clawhub_source
run: |
python3 - <<'PY'
import base64
import json
import os
from pathlib import Path
from urllib.request import Request, urlopen
request_token = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_TOKEN", "").strip()
request_url = os.environ.get("ACTIONS_ID_TOKEN_REQUEST_URL", "").strip()
if not request_token or not request_url:
raise SystemExit("GitHub OIDC token request env vars are missing; id-token: write is required.")
audience = "clawhub-workflow-source"
joiner = "&" if "?" in request_url else "?"
token_url = f"{request_url}{joiner}audience={audience}"
request = Request(
token_url,
headers={"Authorization": f"Bearer {request_token}"},
)
with urlopen(request) as response:
payload = json.load(response)
token = str(payload.get("value", "")).strip()
if not token:
raise SystemExit("GitHub OIDC token response did not include a token value.")
try:
encoded_payload = token.split(".")[1]
except IndexError as exc:
raise SystemExit("GitHub OIDC token was not a valid JWT.") from exc
padding = "=" * (-len(encoded_payload) % 4)
claims = json.loads(
base64.urlsafe_b64decode(encoded_payload + padding).decode("utf-8")
)
workflow_ref = str(claims.get("job_workflow_ref", "")).strip()
workflow_sha = str(claims.get("job_workflow_sha", "")).strip()
repo, marker, _ = workflow_ref.partition("/.github/workflows/")
if not marker or not repo or not workflow_sha:
raise SystemExit(
"Unable to resolve reusable workflow source from GitHub OIDC claims: "
f"job_workflow_ref={workflow_ref!r} job_workflow_sha={workflow_sha!r}"
)
output_path = Path(os.environ["GITHUB_OUTPUT"])
with output_path.open("a", encoding="utf-8") as fh:
fh.write(f"repository={repo}\n")
fh.write(f"ref={workflow_sha}\n")
PY
- uses: actions/checkout@v6
with:
repository: ${{ steps.clawhub_source.outputs.repository }}
ref: ${{ steps.clawhub_source.outputs.ref }}
path: clawhub-source
- name: Install ClawHub CLI dependencies
working-directory: clawhub-source
run: bun install --frozen-lockfile
- name: Validate publish mode inputs
env:
DRY_RUN: ${{ inputs.dry_run }}
JSON_MODE: ${{ inputs.json }}
CLAWHUB_TOKEN: ${{ secrets.clawhub_token }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
run: |
if [[ "$JSON_MODE" != "true" ]]; then
echo "::warning::This reusable workflow always emits JSON output; forcing --json for downstream parsing."
fi
if [[ "$DRY_RUN" == "true" ]]; then
exit 0
fi
if [[ -n "$CLAWHUB_TOKEN" ]]; then
exit 0
fi
if [[ "$GITHUB_EVENT_NAME" == "workflow_dispatch" && -n "${ACTIONS_ID_TOKEN_REQUEST_URL:-}" && -n "${ACTIONS_ID_TOKEN_REQUEST_TOKEN:-}" ]]; then
echo "No ClawHub token provided; publish will rely on GitHub OIDC trusted publishing."
exit 0
fi
echo "::error::Real publishes need secrets.clawhub_token, or GitHub OIDC on workflow_dispatch runs (permissions.id-token=write)."
exit 1
- name: Write ClawHub config
env:
CLAWHUB_TOKEN: ${{ secrets.clawhub_token }}
CLAWHUB_REGISTRY: ${{ inputs.registry }}
run: |
if [[ -z "$CLAWHUB_TOKEN" ]]; then
echo "No ClawHub token provided, skipping config file creation."
exit 0
fi
python3 - <<'PY'
import json
import os
from pathlib import Path
path = Path(os.environ["RUNNER_TEMP"]) / "clawhub-config.json"
path.write_text(
json.dumps(
{
"registry": os.environ["CLAWHUB_REGISTRY"],
"token": os.environ["CLAWHUB_TOKEN"],
},
indent=2,
)
+ "\n",
encoding="utf-8",
)
print(path)
PY
echo "CLAWHUB_CONFIG_PATH=$RUNNER_TEMP/clawhub-config.json" >> "$GITHUB_ENV"
- name: Resolve publish command
env:
INPUT_SOURCE: ${{ inputs.source }}
INPUT_REF: ${{ inputs.ref }}
INPUT_DRY_RUN: ${{ inputs.dry_run }}
INPUT_OWNER: ${{ inputs.owner }}
INPUT_VERSION: ${{ inputs.version }}
INPUT_TAGS: ${{ inputs.tags }}
INPUT_SOURCE_REPO: ${{ inputs.source_repo }}
INPUT_SOURCE_COMMIT: ${{ inputs.source_commit }}
INPUT_SOURCE_REF: ${{ inputs.source_ref }}
INPUT_SITE: ${{ inputs.site }}
INPUT_REGISTRY: ${{ inputs.registry }}
CLAWHUB_TOKEN: ${{ secrets.clawhub_token }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_REF: ${{ github.ref }}
GITHUB_SHA: ${{ github.sha }}
run: |
python3 - <<'PY'
import json
import os
import shlex
from pathlib import Path
source = os.environ["INPUT_SOURCE"].strip()
if not source:
source = os.environ["GITHUB_REPOSITORY"]
source_is_current_repo = source == os.environ["GITHUB_REPOSITORY"]
ref = os.environ["INPUT_REF"].strip()
if not ref and source_is_current_repo:
ref = os.environ["GITHUB_SHA"].strip()
is_local_source = source.startswith(".") or source.startswith("/") or Path(source).exists()
if ref and "@" not in source and not source.startswith("http") and not is_local_source:
source = f"{source}@{ref}"
cli_entry = (
Path(os.environ["GITHUB_WORKSPACE"])
/ "clawhub-source"
/ "packages"
/ "clawhub"
/ "src"
/ "cli.ts"
)
if not cli_entry.exists():
raise SystemExit(f"Missing ClawHub CLI entrypoint at {cli_entry}")
cmd = [
"bun",
str(cli_entry),
"package",
"publish",
source,
"--site",
os.environ["INPUT_SITE"],
"--registry",
os.environ["INPUT_REGISTRY"],
]
if os.environ["INPUT_DRY_RUN"] == "true":
cmd.append("--dry-run")
cmd.append("--json")
owner = os.environ["INPUT_OWNER"].strip()
version = os.environ["INPUT_VERSION"].strip()
tags = os.environ["INPUT_TAGS"].strip()
if owner:
cmd += ["--owner", owner]
if version:
cmd += ["--version", version]
if tags:
cmd += ["--tags", tags]
source_repo = os.environ["INPUT_SOURCE_REPO"].strip()
source_commit = os.environ["INPUT_SOURCE_COMMIT"].strip()
source_ref = os.environ["INPUT_SOURCE_REF"].strip()
if source_repo:
cmd += ["--source-repo", source_repo]
if source_commit:
cmd += ["--source-commit", source_commit]
if source_ref:
cmd += ["--source-ref", source_ref]
elif source_is_current_repo:
github_ref = os.environ["GITHUB_REF"].strip()
if github_ref:
cmd += ["--source-ref", github_ref]
if os.environ["INPUT_DRY_RUN"] != "true" and os.environ["CLAWHUB_TOKEN"].strip():
cmd += [
"--manual-override-reason",
f"GitHub Actions {os.environ['GITHUB_EVENT_NAME'].strip()} publish via CLAWHUB_TOKEN",
]
path = Path(os.environ["RUNNER_TEMP"]) / "clawhub-package-publish-command.sh"
shell_line = " ".join(shlex.quote(part) for part in cmd)
path.write_text("#!/usr/bin/env bash\nset -euo pipefail\n" + shell_line + "\n", encoding="utf-8")
path.chmod(0o755)
print(shell_line)
PY
- name: Run package publish
run: |
set -euo pipefail
"$RUNNER_TEMP/clawhub-package-publish-command.sh" | tee "$RUNNER_TEMP/package-publish.json"
- name: Capture workflow outputs
id: capture
run: |
python3 - <<'PY'
import json
import os
from pathlib import Path
output_path = Path(os.environ["RUNNER_TEMP"]) / "package-publish.json"
raw = output_path.read_text(encoding="utf-8").strip()
parsed = json.loads(raw)
github_output = Path(os.environ["GITHUB_OUTPUT"])
with github_output.open("a", encoding="utf-8") as fh:
fh.write("publish_json<<__CLAWHUB_JSON__\n")
fh.write(json.dumps(parsed, indent=2))
fh.write("\n__CLAWHUB_JSON__\n")
release_id = str(parsed.get("releaseId", "") or "")
fh.write(f"release_id={release_id}\n")
PY
- name: Upload publish JSON artifact
uses: actions/upload-artifact@v4
with:
name: clawhub-package-publish-json
path: ${{ runner.temp }}/package-publish.json
if-no-files-found: error
+64
View File
@@ -0,0 +1,64 @@
name: "Security Gate: Secret Scanning"
on:
push:
branches: ["**"]
pull_request:
branches: [main, master]
jobs:
trufflehog:
name: Scan for Verified Secrets
runs-on: ubuntu-latest
permissions:
contents: read # Required to scan the code in the PR
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
fetch-depth: 0 # necessary to support the scoping requirements below
- name: Resolve scan range
id: scan_range
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
PUSH_BASE_SHA: ${{ github.event.before }}
PUSH_HEAD_SHA: ${{ github.sha }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
run: |
set -euo pipefail
zero_sha="0000000000000000000000000000000000000000"
if [[ "$EVENT_NAME" == "pull_request" ]]; then
base="$PR_BASE_SHA"
head="$PR_HEAD_SHA"
else
base="$PUSH_BASE_SHA"
head="$PUSH_HEAD_SHA"
if [[ -z "$base" || "$base" == "$zero_sha" ]]; then
base="origin/$DEFAULT_BRANCH"
fi
fi
echo "base=$base" >> "$GITHUB_OUTPUT"
echo "head=$head" >> "$GITHUB_OUTPUT"
- name: TruffleHog OSS
id: trufflehog
# Use a concrete released ref that resolves in upstream action registry.
# v3 (major tag) is not published by trufflesecurity/trufflehog.
uses: trufflesecurity/trufflehog@v3.93.8
with:
path: ./
base: ${{ steps.scan_range.outputs.base }}
head: ${{ steps.scan_range.outputs.head }}
extra_args: --only-verified --debug
- name: Notify on Failure
if: steps.trufflehog.outcome == 'failure'
run: |
echo "::error::Verified secrets found! This PR contains live credentials that must be rotated immediately."
echo "::notice::If these secrets are already in the commit history, they cannot be removed via a simple removal commit/push. A repository owner can contact GitHub Support to purge the cached data: https://support.github.com/contact/private-information"
exit 1
+4
View File
@@ -24,3 +24,7 @@ coverage
playwright-report
test-results
.playwright
convex/_generated/
skills-lock.json
*/skills/*
skills/*
+1
View File
@@ -0,0 +1 @@
22
+20
View File
@@ -0,0 +1,20 @@
{
"$schema": "./node_modules/oxfmt/configuration_schema.json",
"experimentalSortImports": {
"newlinesBetween": false,
},
"experimentalSortPackageJson": {
"sortScripts": true,
},
"ignorePatterns": [
".output/",
".tanstack/",
"convex/_generated/",
"coverage/",
"dist/",
"node_modules/",
"public/",
"src/routeTree.gen.ts",
"test-results/",
],
}
+35 -1
View File
@@ -1,3 +1,37 @@
{
"ignorePatterns": ["node_modules", "dist", "coverage", "convex/_generated", ".tanstack", "public"]
"$schema": "./node_modules/oxlint/configuration_schema.json",
"plugins": ["unicorn", "typescript", "oxc"],
"categories": {
"correctness": "error",
"perf": "error",
"suspicious": "error"
},
"rules": {
"curly": "off",
"eslint-plugin-unicorn/prefer-array-find": "off",
"eslint-plugin-unicorn/no-array-sort": "off",
"eslint/no-await-in-loop": "off",
"eslint/no-new": "off",
"oxc/no-accumulating-spread": "off",
"oxc/no-async-endpoint-handlers": "off",
"oxc/no-map-spread": "off",
"typescript/no-explicit-any": "error",
"typescript/no-extraneous-class": "off",
"typescript/no-unnecessary-boolean-literal-compare": "off",
"typescript/no-unnecessary-type-assertion": "off",
"typescript/no-unsafe-type-assertion": "off",
"unicorn/consistent-function-scoping": "off",
"unicorn/require-post-message-target-origin": "off"
},
"ignorePatterns": [
".output/",
".tanstack/",
"convex/_generated/",
"coverage/",
"dist/",
"node_modules/",
"public/",
"src/routeTree.gen.ts",
"test-results/"
]
}
+54 -1
View File
@@ -1,6 +1,7 @@
# Repository Guidelines
## Project Structure & Module Organization
- `src/` — TanStack Start app code (routes, components, styles).
- `convex/` — Convex backend (schema, queries/mutations/actions, HTTP routes).
- `convex/_generated/` — generated Convex API/types; committed for builds.
@@ -8,6 +9,7 @@
- `public/` — static assets.
## Build, Test, and Development Commands
- `bun run dev` — local app server at `http://localhost:3000`.
- `bun run build` — production build (Vite + Nitro).
- `bun run preview` — preview built app.
@@ -18,27 +20,78 @@
- `bun run coverage` — coverage run; keep global >= 80%.
## Coding Style & Naming Conventions
- TypeScript strict; ESM.
- Indentation: 2 spaces, single quotes (Biome).
- Lint/format: Biome + oxlint (type-aware).
- Convex function names: verb-first (`getBySlug`, `publishVersion`).
## Testing Guidelines
- Framework: Vitest 4 + jsdom.
- Tests live in `src/**` and `convex/lib/**`.
- Coverage threshold: 80% global (lines/functions/branches/statements).
- Example: `convex/lib/skills.test.ts`.
## Commit & Pull Request Guidelines
- Commit messages: Conventional Commits (`feat:`, `fix:`, `chore:`, `docs:`…).
- Keep changes scoped; avoid repo-wide search/replace.
- PRs: include summary + test commands run. Add screenshots for UI changes.
- Before merging any PR, verify TypeScript cleanly with `bunx tsc -p packages/schema/tsconfig.json --noEmit` and `bunx tsc -p packages/clawhub/tsconfig.json --noEmit`; if Convex code changed, also run the repo typecheck path used by deploy so `bunx convex deploy` will not fail on `tsc`.
- GitHub comments: for multiline `gh` comments/close messages, use `--body-file`, `--input`, or stdin/heredoc with real newlines; never pass literal `\\n` in shell strings.
- Reject PRs that add skills into source code/repo content directly (for example under `skills/` or seed-only additions intended as published skills). Skills must be uploaded/published via CLI.
## Production Release
- Production deploys are manual-only. Merging to `main` does **not** deploy.
- To release production, start the GitHub Actions `Deploy` workflow from `main`:
`gh workflow run deploy.yml --repo openclaw/clawhub --ref main`
- The workflow supports `full`, `backend`, and `frontend` targets.
- `frontend` currently means: wait for the Vercel production deploy for the selected `main` SHA, then run production smoke checks. It does not call `vercel deploy` directly yet.
- The workflow uses the GitHub `Production` environment for deploy secrets, but it does not require a separate approval step.
- Prod deploy secrets live on the `Production` environment, not as ordinary repo secrets. Required: `CONVEX_DEPLOY_KEY`. Optional: `PLAYWRIGHT_AUTH_STORAGE_STATE_JSON`.
- CLI npm releases are also manual-only and tag-based. Stable tags only: `vX.Y.Z`. Start `ClawHub CLI NPM Release` from `main`, first with `preflight_only=true`, then rerun it with the same tag and the successful `preflight_run_id`.
- Real CLI publishes wait at the GitHub `npm-release` environment and use npm trusted publishing. Required npm trusted publisher settings: repository `openclaw/clawhub`, workflow `clawhub-cli-npm-release.yml`, environment `npm-release`.
## Git Notes
- If `git branch -d/-D <branch>` is policy-blocked, delete the local ref directly: `git update-ref -d refs/heads/<branch>`.
## URL Quick Reference
- Canonical site: `https://clawhub.ai` (prefer this over legacy domains).
- Skill page URL format: `https://clawhub.ai/<owner>/<slug>` (owner handle preferred; falls back to owner id).
- Skill API detail URL: `https://clawhub.ai/api/v1/skills/<slug>`.
- Skill file URL: `https://clawhub.ai/api/v1/skills/<slug>/file?path=SKILL.md`.
- For “full URL?” requests, return the canonical page URL first, then API URL if useful.
## Configuration & Security
- Local env: `.env.local` (never commit secrets).
- Convex env holds JWT keys; Vercel only needs `VITE_CONVEX_URL` + `VITE_CONVEX_SITE_URL`.
- OAuth: GitHub OAuth App credentials required for login.
## Convex Ops (Gotchas)
- New Convex functions must be pushed before `convex run`: use `bunx convex dev --once` (dev) or `bunx convex deploy --prod` (prod).
- New Convex functions must be pushed before `convex run`: use `bunx convex dev --once` (dev) or `bunx convex deploy` (prod).
- For non-interactive prod deploys, use `bunx convex deploy -y` to skip confirmation.
- If `bunx convex run --env-file .env.local ...` returns `401 MissingAccessToken` despite `bunx convex login`, workaround: omit `--env-file` and use `--deployment-name <name>` / `--prod`.
## Convex Query & Bandwidth Rules
- **Always use `.withIndex()` instead of `.filter()` for fields that can be indexed.** `.filter()` causes full table scans — every doc is read and billed. Even a single `.filter()` on a 16K-row table reads ~16 MB per call.
- **Convex reads entire documents** — no field projections. If you only need a few fields from large docs (~6 KB+), denormalize a lightweight summary onto the parent doc or use a lookup table (see `embeddingSkillMap`, `skill.latestVersionSummary`, `skill.badges` for examples).
- **Denormalization pattern**: persist computed fields so they can be indexed. Every mutation that updates source fields must also update the denormalized field. Always write a cursor-based backfill for new fields (see `backfillIsSuspiciousInternal`, `backfillLatestVersionSummaryInternal`, `backfillDenormalizedBadgesInternal` for examples).
- **Cron jobs must never scan entire tables.** Use indexed queries with equality filters. Use cursor-based pagination for large datasets. Prefer incremental/delta tracking over full recounts.
- **32K document limit per query.** Split `.collect()` calls by a partition field (e.g., one day at a time instead of a 7-day range). See `rebuildTrendingLeaderboardAction` in `convex/leaderboards.ts` for an example.
- **Common mistakes**: `.filter().collect()` without an index; `ctx.db.get()` on large docs in a loop for list views; while loops that paginate the whole table to find filtered results.
- **Before writing or reviewing Convex queries, check deployment health.** Run `bunx convex insights` to check for OCC conflicts, `bytesReadLimit`, and `documentsReadLimit` errors. Run `bunx convex logs --failure` to see individual error messages and stack traces. This helps identify which functions are causing bandwidth issues so you can prioritize fixes.
<!-- convex-ai-start -->
This project uses [Convex](https://convex.dev) as its backend.
When working on Convex code, **always read `convex/_generated/ai/guidelines.md` first** for important guidelines on how to correctly use Convex APIs and patterns. The file contains rules that override what you may have learned about Convex from training data.
Convex agent skills for common tasks can be installed by running `npx convex ai-files install`.
<!-- convex-ai-end -->
+302 -6
View File
@@ -1,8 +1,286 @@
# Changelog
## Unreleased
### Changed
- Search: add CJK tokenization support (Chinese/Japanese/Korean) with Intl.Segmenter plus fallback behavior to improve skill query matching (#1596) (thanks @pq-dong).
### Fixes
- Stats maintenance: keep skill stat migration fields synchronized by treating top-level stat fields as canonical during backfill/reconcile fallback reads (#1704) (thanks @momothemage).
## 0.10.0 - 2026-04-05
### Added
- Design system: introduce a shared UI component library (`src/components/ui/`) built on Radix UI primitives — Button, Card, Badge, Tabs, Dialog, Input, Textarea, Label, Select, Avatar, Separator, Tooltip, ScrollArea, Sheet, Skeleton, and Table — following the shadcn/ui pattern with `cn()` + Tailwind utilities.
- Design system: `Button` supports `asChild` via Radix Slot for polymorphic rendering (e.g., wrapping `<Link>` without extra DOM).
- Layout: add `Container` component with `narrow` / `default` / `wide` size presets and `Breadcrumb` component for hierarchical navigation.
- Loading: add skeleton loading states (`SkillCardSkeleton`, `SkillDetailSkeleton`, `DashboardSkeleton`) replacing text-based "Loading..." indicators with animated placeholders.
- Errors: add `ErrorBoundary` with `resetKey` prop that auto-resets on route changes, wired into the root layout.
- Errors: surface fallback messages from Convex API error payloads in mutation/action error toasts.
- UX: add `EmptyState` component with icon, headline, description, and optional CTA action used across dashboard, stars, profile, and publish pages.
- UX: add confirmation dialogs for destructive skill ownership actions (transfer, abandon).
- Markdown: add `MarkdownPreview` component with `react-markdown`, `remark-gfm`, and `react-syntax-highlighter` for rich rendering of skill/plugin READMEs with syntax-highlighted code blocks, GFM tables, and task lists.
- Markdown: render tables with the new `Table` UI primitive for consistent styling across skill docs.
- Navigation: replace DropdownMenu-based mobile nav with a slide-out `Sheet` panel.
- Validation: add Zod schemas (`src/lib/schemas.ts`) for publish-skill, settings, report, and org forms.
- Management: restore capability-tags UI (crypto, requires-wallet, can-make-purchases, etc.) that was silently removed during the initial refactor.
- Management: add `.catch()` error handling with toast feedback on `setSoftDeleted` calls; prompt for hide/restore reasons.
### Changed
- CSS: migrate from a monolithic 5,161-line `styles.css` to Tailwind utilities on components, pruning CSS to ~1,000 lines (81% reduction). Dark mode now uses Tailwind `dark:` variants via a `@variant dark` directive bridging existing CSS custom properties.
- Tailwind: add `@theme` block mapping all CSS design tokens (`--bg`, `--surface`, `--ink`, `--accent`, `--line`, `--radius-*`, etc.) into first-class Tailwind utilities.
- Pages: modernize all route pages (home, skills browse, skill detail, dashboard, settings, publish-skill, publish-plugin, import, about, CLI auth, stars, souls, user profile, org profile, management, plugins browse, plugin detail) from CSS class selectors to Tailwind + UI primitives.
- Skills browse: widen container to `wide` (1400px) for better use of screen space on desktop; same for plugins browse.
- Skills browse: replace text-based filter toggles with pill chips and modernize toolbar layout.
- Skill detail: migrate tab controls from CSS-styled buttons to Radix `Tabs` primitive with proper `role="tab"` accessibility.
- Skill detail: replace inline CSS class-based install card with `SkillInstallCard` using Card + Button primitives.
- Header/Footer: migrate from CSS classes to Tailwind utilities with responsive Sheet-based mobile navigation.
- Dashboard: replace CSS table layout with `Table` UI primitive; add metric cards and skeleton loading.
- Settings: modernize form inputs with `Input`/`Textarea`/`Label` primitives and structured layout.
- Publish: use `Dialog` primitive for modals; inline validation indicators; modernized file list display.
### Fixed
- Auth: `EmptyState` "Sign in" button on publish page now triggers GitHub OAuth via `useAuthActions` instead of linking to non-existent `/signin` route.
- API: fix plugins page dev-mode `{"error":"Only HTML requests are supported here"}` by routing SSR and localhost API fetches directly to the Convex site URL instead of through TanStack Start's request pipeline.
- API: fix CORS error when `credentials: "include"` conflicts with `Access-Control-Allow-Origin: *` by making credentials conditional on same-origin requests.
- API: fix SSR `packageApiUrl` to always use `VITE_CONVEX_SITE_URL` directly, avoiding `getRequestUrl()` failures when SSR request context is unavailable.
- Management: restore `setSoftDeleted` reason parameter for hide/restore actions.
- Tests: rename `settings.test.tsx` to `-settings.test.tsx` to exclude from TanStack Router's file-based route discovery.
- Tests: add `@convex-dev/auth/react` mock for `useAuthActions` in upload route tests.
- Tests: update skill detail tests for Radix tab roles (`role="tab"` instead of `role="button"`), skeleton loading classes (`animate-pulse`), and capability tag data.
- Tests: update skills index tests for refreshed UI copy (placeholder text, empty state wording, loading indicator patterns).
- Tests: update SkillDiffCard tests for Tailwind active-tab class (`shadow-sm` replacing `.is-active`).
- Tests: update packages publish route tests for Tailwind border classes.
- Tests: update packageApi tests for conditional credentials and SSR URL resolution.
## 0.9.0 - 2026-03-23
### Added
- Packages/Plugins: add a first-class OpenClaw package registry across the web app, CLI, and HTTP API. ClawHub now supports package browse/search/detail/version/file/download flows plus `clawhub package explore`, `clawhub package inspect`, and `clawhub package publish` for `skill`, `code-plugin`, and `bundle-plugin` packages. (#1093)
- Packages/Install: package downloads now ship install-ready archives with a `package/` root, support nested files like `dist/index.js`, and work directly with OpenClaw plugin install flows.
- Skills/Web: server-render public skill pages and OG assets for faster first loads, cleaner sharing previews, and better cache behavior.
### Changed
- Browse/Search: rebuild public browse/search around denormalized digests, one-shot HTTP fetches, and deterministic cursors so the homepage and `/skills` are faster, more cacheable, and less likely to hit stale-tab or pagination dead ends.
- Search: default skill search to relevance, keep load-more retryable after fetch failures, and tighten package/skill catalog query paths to reduce inconsistent results under load.
### Fixed
- Packages/Auth: authenticated owners can now list, search, inspect, download, and read files from their own private packages instead of private packages being direct-URL-only. (#1093)
- Packages/API: stabilize package latest-version pointers, cursor pagination, publish outputs, fallback release resolution, and app-origin auth handling so package publish/search/install flows stay reliable.
- Visibility/API: prevent skills owned by deleted/banned users from showing up in public detail pages, browse/search results, or version API routes.
- Skills/API: sanitize public skill and soul version/file reads so hidden or invalid version data does not leak through direct API access.
- Skills/Web: keep Monaco compare layout toggles reliable while defaulting narrow screens to inline mode (#828) (thanks @geoffrey-xiao).
## 0.8.0 - 2026-03-13
### Added
- Skills/Web: show skill owner avatar + handle on skill cards, lists, and detail pages (#312) (thanks @ianalloway).
- Skills/Web: add file viewer for skill version files on detail page (#44) (thanks @regenrek).
- CLI: add `uninstall` command for skills (#241) (thanks @superlowburn).
- Skills/API/CLI: add ownership transfer workflow with request/list/accept/reject/cancel flows.
- Skills/Web/API: surface platform/architecture labels and security evaluation results in v1 + inspect views (#499, #362).
- API: add structured skill moderation responses plus `GET /api/v1/skills/{slug}/moderation` with redacted public evidence and full owner/staff detail (#334) (thanks @ArthurzKV).
- Moderation: persist structured moderation snapshots (static scan + VT/LLM merged verdict, reason codes, and evidence) on skills and versions (#333) (thanks @ArthurzKV).
- API: add scan security verification endpoint and non-suspicious filters (#820).
- Users: add `trustedPublisher` flag and admin mutations to bypass pending-scan auto-hide for trusted publishers (#298) (thanks @autogame-17).
- Moderation: add comment reporting with per-user active report caps, unique reporter/target enforcement, and auto-hide on the 4th unique report.
- Moderation: add AI-driven comment scam backfill (`commentModeration:*`) with persisted verdict/confidence/explainer metadata and strict auto-ban for `certain_scam` + `high` confidence.
- Admin: add manual unban for banned users (clears `deletedAt` + `banReason`, audit log entry). Revoked API tokens stay revoked.
- Admin: bulk restore skills from GitHub backup; reclaim squatted slugs via v1 endpoints + internal tooling (#298) (thanks @autogame-17).
- Moderation/Admin: add manual override audit tools for suspicious-skill review.
- CI/Security: add TruffleHog pull-request scanning for verified leaked credentials (#505) (thanks @akses0).
### Changed
- Skills: make published skill licensing explicit and fixed to MIT-0; require publish consent, surface no-attribution messaging in web/CLI/API, and remove per-skill license metadata.
- Skill metadata: support env vars, dependency declarations, author, and links in parsed manifest metadata + install UI (#360) (thanks @mahsumaktas).
- Rate limiting: apply authenticated quotas by user bucket (vs shared IP), emit delay-based reset headers, and improve CLI 429 guidance/retries (#412) (thanks @lc0rp).
- Skills: reserve deleted slugs for prior owners (90-day cooldown) to prevent squatting; add admin reclaim flow (#298) (thanks @autogame-17).
- Moderation: ban flow soft-deletes owned skills (reversible) and removes them from vector search (#298) (thanks @autogame-17).
- Security/docs: document comment reporting/auto-hide behavior alongside existing skill reporting rules.
- Security/moderation: add bounded explainable auto-ban reasons for scam comments and protect moderator/admin accounts from automated bans.
- Moderation: banning users now also soft-deletes their authored comments (skill + soul), including legacy cleanup on re-ban.
- Quality gate: language-aware word counting (`Intl.Segmenter`) and new `cjkChars` signal to reduce false rejects for non-Latin docs.
- Jobs: run skill stat event processing every 5 minutes (was 15).
- Deploy: add frontend/backend drift detection plus hardened production smoke/deploy checks.
- API performance: batch resolve skill/soul tags in v1 list/get endpoints (fewer action->query round-trips) (#112) (thanks @mkrokosz).
- LLM helpers: centralize OpenAI Responses text extraction for changelog/summary/eval flows (#502) (thanks @ianalloway).
- Search/listing performance: cut embedding hydration and badge read bandwidth via `embeddingSkillMap` + denormalized skill badges; shift stat-doc sync to low-frequency cron (#441) (thanks @sethconvex).
- Search/listing performance: move public browse/search hydration onto `skillSearchDigest`, add non-suspicious index paths, and split trending rebuilds to stay under Convex document limits.
### Fixed
- API: accept legacy CLI publish payloads during the v1 migration (#815).
- Auth/UI: surface OAuth callback failures in the web UI instead of swallowing them (#688).
- Skills: allow ownership healing when the previous owner was deleted/banned, and sanitize owner data in public payloads (#689, #793).
- CLI: validate explicit `install --force --version` targets before removing an existing local skill, preventing data loss when the requested version does not exist (#825) (thanks @jonathandeamer).
- Skills/Web: debounce search URL updates on `/skills` to keep typing responsive, and cancel stale pending navigations on external query changes (#587) (thanks @neeravmakwana).
- Upload: keep folder-picking enabled after page refresh by reapplying `webkitdirectory`/`directory` on the file input ref (#551) (thanks @MunemHashmi).
- CLI publish: use a longer multipart upload timeout and normalize abort rejections into proper Errors (#550) (thanks @MunemHashmi).
- CLI: forward optional auth tokens for `search` and `explore` against authenticated registries (#608) (thanks @artdaal).
- CLI: respect `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` env vars for outbound registry requests, with troubleshooting docs (#363) (thanks @kerrypotter).
- CLI: preserve registry base paths when composing API URLs for search/inspect/moderation commands (#486) (thanks @Liknox).
- CLI: show manual URL guidance when automatic browser opening is unavailable; add regression tests for opener errors (#163) (thanks @aronchick).
- API/CLI: expose skill security status in version inspect output, with schema wiring and CLI regression coverage (#362) (thanks @abutbul).
- Moderation: remove over-broad keyword flags for common auth/payment/crypto terms so legitimate skills stop tripping regex prefilters (#273) (thanks @superlowburn).
- Skills hard-delete: delete `commentReports` rows during moderation cleanup to avoid orphaned report records.
- Comments: hide entries authored by deleted/deactivated users in `comments:listBySkill`.
- Admin API: `POST /api/v1/users/reclaim` now performs non-destructive root-slug owner transfer
(preserves existing skill versions/stats/metadata) and clears active slug reservations.
- VirusTotal: use shared AV-engine fallback verdict mapping for pending/backfill flows and keep undetected-only results pending (#591) (thanks @Shuai-DaiDai).
- Skills/listing: keep non-suspicious browse pagination on one cursor family during `isSuspicious` backfill, and re-sync stale `latestVersionSummary` metadata fields (#572) (thanks @sethconvex).
- PWA: update `manifest.json` branding so installed apps show the correct ClawHub name (#569) (thanks @Glucksberg).
- Search/tests: cover soft-deleted skill filtering in vector hydration and lexical exact-slug fallback (#552) (thanks @MunemHashmi).
- Docs/dev: fix local setup instructions for Node support, Convex env vars, frontend port, and post-seed stats refresh (#584) (thanks @jack-piplabs).
- Docs/CLI: fix `explore` flag list indentation so `--limit` renders correctly in the command reference (#601) (thanks @gandli).
- Skill metadata: parse top-level `requires.*`, `primaryEnv`, and homepage fallbacks for security review accuracy (#548) (thanks @MunemHashmi).
- Users: sync handle on ensure when GitHub login changes (#293) (thanks @christianhpoe).
- Users/Auth: throttle GitHub profile sync on login; also sync avatar when it changes (#312) (thanks @ianalloway).
- Upload gate: fetch GitHub account age by immutable account ID (prevents username swaps) (#116) (thanks @mkrokosz).
- VT fallback: activate only VT-pending hidden skills when scans are unavailable/stale; keep quality/scanner-blocked skills hidden (#300) (thanks @superlowburn).
- API: return proper status codes for delete/undelete errors (#35) (thanks @sergical).
- API: for owners, return clearer status/messages for hidden/soft-deleted skills instead of a generic 404.
- Web: allow copying OpenClaw scan summary text (thanks @borisolver, #322).
- HTTP/CORS: add preflight handler + include CORS headers on API/download errors; CLI: include auth token for owner-visible installs/updates (#146) (thanks @Grenghis-Khan).
- CLI: clarify `logout` only removes the local token; token remains valid until revoked in the web UI (#166) (thanks @aronchick).
- CLI: validate skill slugs used for filesystem operations (prevents path traversal) (#241) (thanks @superlowburn).
- Skills: keep global sorting across pagination on `/skills` (thanks @CodeBBakGoSu, #98).
- Skills: allow updating skill description/summary from frontmatter on subsequent publishes (#312) (thanks @ianalloway).
- Skills/Web: prevent filtered pagination dead-ends and loading-state flicker on `/skills`; move highlighted browse filtering into server list query (#339) (thanks @Marvae).
- Web: align `/skills` total count with public visibility and format header count (thanks @rknoche6, #76).
- Skills/Web: centralize public visibility checks and keep `globalStats` skill counts in sync incrementally; remove duplicate `/skills` default-sort fallback and share browse test mocks (thanks @rknoche6, #76).
- Moderation: clear stale `flagged.suspicious` flags when VirusTotal rescans improve to clean verdicts (#418) (thanks @Phineas1500).
- API tests: lock `Retry-After` behavior to relative-delay semantics for v1 search 429s (#421) (thanks @apoorvdarshan).
- CLI tests: assert 5xx HTTP responses still perform retry attempts before surfacing final error (#457) (thanks @YonghaoZhao722).
- GitHub import: improve storage/publish failure errors with actionable context; add regression tests for error formatting (#512) (thanks @vassiliylakhonin).
## 0.7.0 - 2026-02-16
Reconstructed from the `clawhub@0.7.0` npm publish timestamp (`2026-02-16T05:02:25Z`) and the repo version bump commit (`e352309`).
### Added
- Skills/Web: show owner avatars/handles across cards, lists, and detail pages (#312) (thanks @ianalloway).
- Skills/Web: add version file viewer on skill detail pages (#44) (thanks @regenrek).
- CLI: add `uninstall` for installed skills (#241) (thanks @superlowburn).
- Skills/Web: add non-suspicious browse filter, downloads-first browse defaults, and popular non-suspicious homepage sections.
- Web: compact-format skill and soul stats, plus split page models for skills/detail rendering.
- Skills: auto-generate missing summaries and add a resumable/self-scheduling summary backfill job.
- Moderation/Admin: add anti-spam publish caps, trust-tier quality checks, empty-skill cleanup tooling, and stronger moderator UX.
### Changed
- HTTP/CLI: centralize CORS handling and allow tokenized owner-visible reads through the CLI (#296, #297).
- API performance: batch resolve tags in v1 list/get flows to cut action-to-query round-trips (#112) (thanks @mkrokosz).
- Quality gate: add language-aware word counting and tighten spam/quarantine handling around publish flows.
### Fixed
- Skills/Web: fix initial sort wiring, keep global ordering across pagination, prevent pagination dead-ends/flicker, and harden cursor recovery (#92, #98, #339).
- CLI: normalize abort/timeout errors, secure config-file permissions, clarify logout semantics, and prefer `$HOME` for path resolution (#164, #166, #283, #286, #299).
- API: return correct delete/undelete status codes and clearer soft-delete/owner-visible error responses (#35) (thanks @sergical).
- Upload/Auth: gate publish ownership by immutable GitHub account ID and handle duplicate auth-user records safely.
- Downloads/Search: harden download dedupe/rate limiting, improve SSR host awareness, and fix homepage/search regressions under legacy data.
## 0.6.1 - 2026-02-13
### Added
- Security: add LLM-based security evaluation during skill publish.
- Parsing: recognize `metadata.openclaw` frontmatter and evaluate all skill files for requirements.
### Changed
- Performance: lazy-load Monaco diff viewer on demand (thanks @alexjcm, #212).
- Search: improve recall/ranking with lexical fallback and relevance prioritization.
- Moderation UX: collapse OpenClaw analysis by default; update spacing and default reasoning model.
### Fixed
- Skills: fix initial `/skills` sort wiring so first page respects selected sort/direction (thanks @bpk9, #92).
- Search/UI: add embedding request timeout and align `/skills` toolbar + list width (thanks @GhadiSaab, #53).
- Upload gate: handle GitHub API rate limits and optional authenticated lookup token (thanks @superlowburn, #246).
- HTTP: remove `allowH2` from Undici agent to prevent `fetch failed` on Node.js 22+ (#245).
- Tests: add root `undici` dev dependency for Node E2E imports (thanks @tanujbhaud, #255).
- Downloads: add download rate limiting + per-IP/day dedupe + scheduled dedupe pruning; preserve moderation gating and deterministic zips (thanks @regenrek, #43).
- VirusTotal: fix scan sync race conditions and retry behavior in scan/backfill paths.
- Metadata: tolerate trailing commas in JSON metadata.
- Auth: allow soft-deleted users to re-authenticate on fresh login, while keeping banned users blocked (thanks @tanujbhaud, #177).
- Web: prevent horizontal overflow from long code blocks in skill pages (thanks @bewithgaurav, #183).
## 0.6.0 - 2026-02-10
### Added
- CLI/API: add `set-role` to change user roles (admin only).
- Security: quarantine skill publishes with VirusTotal scans + UI (thanks @aleph8, #130).
- Testing: add tests for badges, skillZip, uploadFiles expandDroppedItems, and ark schema error truncation.
- Moderation: add ban reasons to API/CLI and show in management UI.
### Changed
- Coverage: track `convex/lib/skillZip.ts` in coverage reports.
### Fixed
- Web: show pending-scan skills to owners without 404 (thanks @orlyjamie, #136).
- Users: backfill empty handles from name/email in ensure (thanks @adlai88, #158).
- Web: update footer branding to OpenClaw (thanks @jontsai, #122).
- Auth: restore soft-deleted users on reauth, block banned users (thanks @mkrokosz, #106).
## 0.5.0 - 2026-02-02
### Added
- Admin: ban users and delete owned skills from management console.
- Moderation: auto-hide skills after 4 unique reports; per-user report cap; moderators can ban users.
- Uploads: require GitHub accounts to be at least 7 days old for skill + soul publish/import.
- CLI: add `inspect` to fetch skill metadata/files without installing.
- CLI: add moderation commands for hide/unhide/delete and ban users.
- Management: add filters for reported skills and users.
### Changed
- Deps: update dependencies to latest available versions.
- Reporting: require reasons, show them in management console, warn about abuse bans.
### Fixed
- Bans: batch hard-delete cleanup to avoid Convex read limits on large skills.
## 0.4.0 - 2026-01-30
### Added
- Web: show published skills on user profiles (thanks @njoylab, #20).
- CLI: include ClawHub + Moltbot fallback skill roots for sync scans.
- CLI: support OpenClaw configuration files (`OPENCLAW_CONFIG_PATH` / `OPENCLAW_STATE_DIR`).
### Changed
- Brand: rebrand to ClawHub and publish CLI as `clawhub` (legacy `clawdhub` supported).
- Domain: default site/registry now `https://clawhub.ai`; `.well-known/clawhub.json` preferred.
- Theme: persist theme under `clawhub-theme` (legacy key still read).
### Fixed
- Registry: drop missing skills during search hydration (thanks @aaronn, #28).
- CLI: use path-based skill metadata lookup for updates (thanks @daveonkels, #22).
- Search: keep highlighted-only filtering and clamp vector candidates to Convex limits (thanks @aaronn, #30).
## 0.3.0 - 2026-01-19
### Added
- CLI: add `explore` command for latest updates, with limit clamping + tests/docs (thanks @jdrhyne, #14).
- CLI: `explore --json` output + new sorts (`installs`, `installsAllTime`, `trending`) and limit up to 200.
- API: `/api/v1/skills` supports installs + trending sorts (7-day installs).
@@ -10,39 +288,44 @@
- Registry: trending leaderboard + daily stats backfill for installs-based sorts.
### Fixed
- Web: keep search mode navigation and state in sync (thanks @NACC96, #12).
## 0.2.0 - 2026-01-13
### Added
- Web: dynamic OG image cards for skills (name, description, version).
- CLI: auto-scan Clawdbot skill roots (per-agent workspaces, shared skills, extraDirs).
- Web: import skills from public GitHub URLs (auto-detect `SKILL.md`, smart file selection, provenance).
- Web/API: SoulHub (SOUL.md registry) with v1 endpoints and first-run auto-seed.
### Fixed
- Web: stabilize skill OG image generation on server runtimes.
- Web: prevent skill OG text overflow outside the card.
- Registry: make SoulHub auto-seed idempotent and non-user-owned.
- Registry: keep GitHub backup state + publish backups intact (thanks @joshp123, #1).
- CLI/Registry: restore fork lineage on sync + clamp bulk list queries (thanks @joshp123, #1).
- CLI: default workdir falls back to Clawdbot workspace (override with `--workdir` / `CLAWDHUB_WORKDIR`).
- CLI: default workdir falls back to Clawdbot workspace (override with `--workdir` / `CLAWHUB_WORKDIR`).
## 0.0.6 - 2026-01-07
### Added
- API: v1 public REST endpoints with rate limits, raw file fetch, and OpenAPI spec.
- Docs: `docs/api.md` and `DEPRECATIONS.md` for the v1 cutover plan.
### Changed
- CLI: publish now uses single multipart `POST /api/v1/skills`.
- Registry: legacy `/api/*` + `/api/cli/*` marked for deprecation (kept for now).
## 0.0.5 - 2026-01-06
### Added
- Telemetry: track installs via `clawdhub sync` (logged-in only), per root, with 120-day staleness.
- Telemetry: track installs via `clawhub sync` (logged-in only), per root, with 120-day staleness.
- Skills: show current + all-time installs; sort by installs.
- Profile: private "Installed" tab with JSON export + delete telemetry controls.
- Docs: add `docs/telemetry.md` (what we track + how to opt out).
@@ -50,30 +333,36 @@
- Web: dashboard for managing your published skills (thanks @dbhurley!).
### Changed
- CLI: telemetry opt-out via `CLAWDHUB_DISABLE_TELEMETRY=1`.
- CLI: telemetry opt-out via `CLAWHUB_DISABLE_TELEMETRY=1`.
- Web: move theme picker into mobile menu.
### Fixed
- Web: handle shorthand hex colors in diff theme (thanks @dbhurley!).
## 0.0.5 - 2026-01-06
### Added
- Maintenance: admin backfill to re-parse `SKILL.md` and repair stored summaries/parsed metadata.
### Fixed
- CLI sync: ignore plural `skills.md` docs files when scanning for skills.
- Registry: parse YAML frontmatter (incl multiline `description`) and accept YAML `metadata` objects.
## 0.0.4 - 2026-01-05
### Added
- Web: `/skills` list view with sorting (newest/downloads/stars/name) + quick filter.
- Web: admin/moderator highlight toggle on skill detail.
- Web: canonical skill URLs as `/<owner>/<slug>` (legacy `/skills/<slug>` redirects).
- Web: upload auto-generates a changelog via OpenAI when left blank (marked as auto-generated).
### Fixed
- Web: skill detail shows a loading state instead of flashing "Skill not found".
- Web: user profile shows avatar + loading state (no "User not found" flash).
- Web: improved mobile responsiveness (nav menu, skill detail layout, install command overflow).
@@ -82,31 +371,37 @@
- CLI: ignore legacy `auth.clawdhub.com` registry and prefer site discovery.
### Changed
- Web: homepage search now expands into full search mode with live results + highlighted toggle.
- CLI: sync no longer prompts for changelog; registry auto-generates when blank.
## 0.0.3 - 2026-01-04
### Added
- CLI sync: concurrency flag to limit registry checks.
- Home: install command switcher (npm/pnpm/bun).
### Changed
- CLI sync: default `--concurrency` is now 4 (was 8).
- CLI sync: replace boxed notes with plain output for long lists.
### Fixed
- CLI sync: wrap note output to avoid terminal overflow; cap list lengths.
- CLI sync: label fallback scans as fallback locations.
- CLI package: bundle schema internally (no external `clawdhub-schema` publish).
- Repo: mark `clawdhub-schema` as private to prevent publishing.
- CLI package: bundle schema internally (no external `clawhub-schema` publish).
- Repo: mark `clawhub-schema` as private to prevent publishing.
## 0.0.2 - 2026-01-04
### Added
- CLI: delete/undelete commands for soft-deleted skills (owner/admin).
### Fixed
- CLI sync: dedupe duplicate slugs across scan roots; skip duplicates to avoid double-publish errors.
- CLI sync: show parsing progress while hashing local skills.
- CLI sync: prompt only actionable skills; preselect all by default; list synced separately; condensed synced summary when nothing to sync.
@@ -119,6 +414,7 @@
## 0.0.1 - 2026-01-04
### Features
- CLI auth: login/logout/whoami; browser loopback auth; token storage; site/registry discovery; config overrides.
- CLI workflow: search, install, update (single/all), list, publish, sync (scan workdir + legacy roots), dry-run, version bumping, tags.
- Registry/API: skills + versions with semver; tags (latest + custom); changelog per version; SKILL.md frontmatter parsing; text-only validation; zip download; hash resolve; stats (downloads/stars/versions/comments).
+55
View File
@@ -0,0 +1,55 @@
# ClawHub — Project Rules
## Convex Performance Rules
- For public listing/browse pages, use `ConvexHttpClient.query()` (one-shot fetch),
not `useQuery`/`usePaginatedQuery` (reactive subscription). Reserve reactive
queries for data the user needs to see update in real time.
- Denormalize hot read paths into a single lightweight "digest" table. Every
`ctx.db.get()` join adds a table to the reactive invalidation scope.
- When a `skillSearchDigest` row is available, use `digestToOwnerInfo(digest)`
to resolve owner data. NEVER call `ctx.db.get(ownerUserId)` when digest
owner fields (`ownerHandle`, `ownerName`, `ownerDisplayName`, `ownerImage`)
are already present. Reading from `users` adds the entire table to the
reactive read set and wastes bandwidth.
- Use `convex-helpers` Triggers to sync denormalized tables automatically.
Always add change detection — skip the write if no fields actually changed.
- Use compound indexes instead of JS filtering. If you're filtering docs after
the query, you're scanning documents you'll throw away.
- For search results scored by computed values (vector + lexical + popularity),
fetch all results once and paginate client-side. Don't re-run the full search
pipeline on "load more."
- Backfills on reactively-subscribed tables need `delayMs` between batches.
- Mutations that read >8 MB should use the Action → Query → Mutation pattern
to split reads across transactions.
## Convex Conventions
- All mutations import from `convex/functions.ts` (not `convex/_generated/server`)
to get trigger wrapping. Type imports still come from `convex/_generated/server`.
- NEVER use `--typecheck=disable` on `npx convex deploy`.
- Use `npx convex dev --once` to push functions once (not long-running watcher).
## Production Release
- Production deploys are manual-only. Merging to `main` does **not** deploy.
- Start the GitHub Actions `Deploy` workflow from `main` with `gh workflow run deploy.yml --repo openclaw/clawhub --ref main`.
- The workflow supports `full`, `backend`, and `frontend` targets.
- `frontend` currently waits for the Vercel production deploy on the selected `main` SHA and then runs smoke checks. It does not trigger Vercel directly yet.
- The workflow uses the `Production` environment for deploy secrets, but it does not wait for a separate approval.
- Required prod secret: `CONVEX_DEPLOY_KEY` on the `Production` environment. Optional smoke secret: `PLAYWRIGHT_AUTH_STORAGE_STATE_JSON`.
- CLI npm releases are manual-only and tag-based through `ClawHub CLI NPM Release`. Stable tags only: `vX.Y.Z`. Run a `preflight_only=true` pass first, then rerun with the same tag plus `preflight_run_id` for the real publish.
- Real CLI publishes wait at `npm-release` and rely on npm trusted publishing for `openclaw/clawhub` + `clawhub-cli-npm-release.yml` + `npm-release`.
## Testing
- Tests use `._handler` to call mutation handlers directly with mock `db` objects.
- Mock `db` objects MUST include `normalizeId: vi.fn()` for trigger wrapper compatibility.
<!-- convex-ai-start -->
This project uses [Convex](https://convex.dev) as its backend.
When working on Convex code, **always read `convex/_generated/ai/guidelines.md` first** for important guidelines on how to correctly use Convex APIs and patterns. The file contains rules that override what you may have learned about Convex from training data.
Convex agent skills for common tasks can be installed by running `npx convex ai-files install`.
<!-- convex-ai-end -->
+192
View File
@@ -0,0 +1,192 @@
# Contributing to ClawHub
Welcome! ClawHub is the public skill registry for [OpenClaw](https://github.com/openclaw/openclaw). We appreciate bug fixes, documentation improvements, and feature contributions.
- **Questions?** Ask in [#clawhub on Discord](https://discord.gg/clawd).
- **Bug fixes** — PRs are welcome.
- **New features or architectural changes** — please start with a Discord conversation in #clawhub first so we can align on scope.
## Local Development Setup
### Prerequisites
- [Bun](https://bun.sh/) (Convex CLI runs via `bunx`, no global install needed)
- [Node.js](https://nodejs.org/) v18, 20, 22, or 24 (required by the local Convex backend; v25+ is not yet supported)
### Install and configure
```bash
bun install
cp .env.local.example .env.local
```
Edit `.env.local` with the following values for **local Convex**:
```bash
# Frontend
VITE_CONVEX_URL=http://127.0.0.1:3210
VITE_CONVEX_SITE_URL=http://127.0.0.1:3210
SITE_URL=http://localhost:3000
# Deployment used by `bunx convex dev`
CONVEX_DEPLOYMENT=anonymous:anonymous-clawhub
```
### GitHub OAuth App (for login)
1. Go to [github.com/settings/developers](https://github.com/settings/developers) and create a new OAuth App.
2. Set **Homepage URL** to `http://localhost:3000`.
3. Set **Authorization callback URL** to `http://127.0.0.1:3210/api/auth/callback/github`.
4. Copy the Client ID and generate a Client Secret.
### Run the Convex backend
Start the local Convex backend first — other setup steps depend on it:
```bash
bunx convex dev --typecheck=disable
```
### Set backend environment variables
The Convex backend has its own env var store separate from `.env.local`. With the backend running, open a new terminal and set the required variables:
```bash
bunx convex env set AUTH_GITHUB_ID <your-client-id>
bunx convex env set AUTH_GITHUB_SECRET <your-client-secret>
bunx convex env set SITE_URL http://localhost:3000
```
### JWT keys (for Convex Auth)
With the backend still running, generate the signing keys:
```bash
bunx @convex-dev/auth
```
This sets `JWT_PRIVATE_KEY` and `JWKS` on the Convex backend and outputs values you can also save to `.env.local` for reference.
### Run the frontend
```bash
bun run dev -- --port 3000
```
Change the port if 3000 is already in use, and update `SITE_URL` in both `.env.local` and the Convex backend (`bunx convex env set SITE_URL ...`) to match.
### Seed the database
Populate sample data so the UI isn't empty:
```bash
# 3 sample skills (padel, gohome, xuezh)
bunx convex run --no-push devSeed:seedNixSkills
# 50 extra skills for pagination testing (optional)
bunx convex run --no-push devSeedExtra:seedExtraSkillsInternal
# Refresh the cached skills count (required after seeding)
bunx convex run --no-push statsMaintenance:updateGlobalStatsInternal
```
To reset and re-seed:
```bash
bunx convex run --no-push devSeed:seedNixSkills '{"reset": true}'
```
### Optional environment variables
These features degrade gracefully without their keys:
| Variable | Purpose |
| ------------------------------------------------------------------------- | --------------------------------------------------------- |
| `OPENAI_API_KEY` | Embeddings and vector search (falls back to zero vectors) |
| `VT_API_KEY` | VirusTotal malware scanning |
| `DISCORD_WEBHOOK_URL` | Discord notifications |
| `GITHUB_APP_ID` / `GITHUB_APP_PRIVATE_KEY` / `GITHUB_APP_INSTALLATION_ID` | GitHub backup sync |
## CLI Development
The CLI source lives in [`packages/clawhub/`](packages/clawhub/). Both `clawhub` and `clawdhub` are registered as bin aliases.
To test the CLI against your local instance:
```bash
CLAWHUB_REGISTRY=http://127.0.0.1:3210 CLAWHUB_SITE=http://localhost:3000 clawhub search "padel"
```
Use the package-local verification contract when working on the CLI:
```bash
bun run --cwd packages/clawhub test
bun run --cwd packages/clawhub verify:build
bun run --cwd packages/clawhub test:artifact
bun run --cwd packages/clawhub verify
```
`bun test packages/clawhub/` is not the supported workflow. Source tests and built-artifact smoke tests are intentionally split.
Manual smoke tests are documented in [`docs/manual-testing.md`](docs/manual-testing.md).
## Skill & Soul Publishing
- Skill format reference: [`docs/skill-format.md`](docs/skill-format.md)
- Soul format reference: [`docs/soul-format.md`](docs/soul-format.md)
- End-to-end walkthrough (search, install, publish, sync): [`docs/quickstart.md`](docs/quickstart.md)
Quick publish:
```bash
clawhub publish <path-to-skill-directory>
```
## Before Submitting a PR
```bash
bun run lint # oxlint
bun run test # Vitest (80% coverage threshold)
bun run build # Vite + Nitro
bun run --cwd packages/clawhub verify
```
These are the same checks that run in CI (`.github/workflows/ci.yml`).
**PR guidelines:**
- Keep PRs focused — one concern per PR.
- Use [Conventional Commits](https://www.conventionalcommits.org/): `feat:`, `fix:`, `chore:`, `docs:`, etc.
- Include test commands and screenshots for UI changes.
- Write a clear description of what changed and why.
## AI-Generated Code
AI-assisted contributions are welcome. When submitting AI-generated or AI-assisted code:
- Note it in the PR description.
- Describe the level of testing you applied.
- Include prompts if useful for reviewers.
- Confirm that you understand and can maintain the code.
## Security Reporting
Report vulnerabilities to **security@openclaw.ai** with:
- Severity assessment
- Technical reproduction steps
- Suggested remediation
See [`docs/security.md`](docs/security.md) for moderation and upload gating details.
## Reading Order for New Contributors
1. This file (local setup)
2. [`docs/quickstart.md`](docs/quickstart.md) — end-to-end workflows
3. [`docs/architecture.md`](docs/architecture.md) — system design
4. [`docs/skill-format.md`](docs/skill-format.md) — skill structure
5. [`docs/cli.md`](docs/cli.md) — CLI reference
6. [`docs/http-api.md`](docs/http-api.md) — HTTP endpoints
7. [`docs/auth.md`](docs/auth.md) — authentication
8. [`docs/deploy.md`](docs/deploy.md) — deployment
9. [`docs/troubleshooting.md`](docs/troubleshooting.md) — common issues
+356
View File
@@ -0,0 +1,356 @@
# ClawHub Design System
This document outlines the design rules, patterns, and guidelines for the ClawHub platform to ensure consistency, accessibility, and maintainability across all components.
---
## Color System
### Brand Palette (OpenClaw)
ClawHub uses a strict **3-5 color palette** based on the OpenClaw brand:
| Token | Light Mode | Dark Mode | Usage |
|-------|------------|-----------|-------|
| `--accent` | `#dc2626` | `#dc2626` | Primary actions, interactive elements, emphasis |
| `--accent-deep` | `#b91c1c` | `#ef4444` | Hover states, secondary emphasis |
| `--ink` | `#0a0a0a` | `#fafafa` | Primary text |
| `--ink-soft` | `#525252` | `#a1a1a1` | Secondary text, descriptions |
| `--surface` | `#ffffff` | `#121212` | Card backgrounds, elevated surfaces |
| `--bg` | `#fafafa` | `#0a0a0a` | Page background |
### Rules
1. **Never exceed 5 colors** without explicit design approval
2. **Never use purple/violet prominently** unless explicitly requested
3. **Always override text color** when changing background color to ensure contrast
4. **Use semantic tokens** (`--accent`, `--ink`, `--surface`) instead of raw colors
---
## Typography
### Font Stack
```css
--font-sans: 'Geist', system-ui, sans-serif;
--font-mono: 'Geist Mono', monospace;
--font-display: 'Geist', system-ui, sans-serif;
```
### Scale
| Token | Size | Usage |
|-------|------|-------|
| `--fs-xs` | 0.75rem (12px) | Labels, badges, metadata |
| `--fs-sm` | 0.875rem (14px) | Body text, descriptions |
| `--fs-base` | 1rem (16px) | Default body text |
| `--fs-md` | 1.125rem (18px) | Subheadings |
| `--fs-lg` | 1.25rem (20px) | Section titles |
| `--fs-xl` | 1.5rem (24px) | Page headings |
### Rules
1. **Maximum 2 font families** per page
2. **Line height 1.4-1.6** for body text (use `leading-relaxed`)
3. **Never use decorative fonts** for body text
4. **Minimum font size: 14px** for readability
5. Use `text-balance` or `text-pretty` for titles
---
## Layout
### Method Priority
Use this hierarchy for layout decisions:
1. **Flexbox** - Default for most layouts
2. **CSS Grid** - Only for complex 2D layouts (cards, galleries)
3. **Never use floats** or absolute positioning unless absolutely necessary
### Spacing Scale
```css
--space-1: 0.25rem /* 4px */
--space-2: 0.5rem /* 8px */
--space-3: 0.75rem /* 12px */
--space-4: 1rem /* 16px */
--space-5: 1.5rem /* 24px */
--space-6: 2rem /* 32px */
```
### Grid Patterns
#### Auto-fit Grid (Recommended for Cards)
```css
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
```
- Automatically adjusts columns based on container width
- Prevents orphan items on partial rows
- Maintains consistent card widths
#### Fixed Grid (When exact columns needed)
```css
/* 3-column at desktop, 2 at tablet, 1 at mobile */
grid-template-columns: repeat(3, minmax(0, 1fr));
@media (max-width: 860px) {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
@media (max-width: 520px) {
grid-template-columns: 1fr;
}
```
### Container Widths
| Size | Max Width | Usage |
|------|-----------|-------|
| Default | `--page-max` (1200px) | Standard pages |
| Narrow | `--page-narrow` (720px) | Reading content, forms |
| Wide | Full width | Dashboards, data tables |
---
## Components
### Cards
```css
.card {
padding: var(--space-4);
border: 1px solid var(--line);
border-radius: var(--r-md);
background: var(--surface);
}
```
**Rules:**
- Always use `display: flex; flex-direction: column;` for consistent height
- Add `flex: 1` to content area for equal-height cards in grids
- Include hover state with `border-color` and subtle `box-shadow`
### Buttons
| Variant | Usage |
|---------|-------|
| `primary` | Main actions (Submit, Save, Download) |
| `secondary` | Alternative actions |
| `ghost` | Tertiary actions, navigation |
| `destructive` | Delete, remove, dangerous actions |
**Rules:**
- Always include visible focus state
- Minimum touch target: 44x44px on mobile
- Include `aria-label` when icon-only
### Form Controls
- Labels above inputs (not inline)
- Error states use `--status-error-fg`
- Focus rings use `--accent` with 0.2 opacity
- Minimum input height: 40px
---
## Responsive Breakpoints
```css
/* Mobile first - base styles for mobile */
@media (min-width: 520px) {
/* Small tablets, large phones */
}
@media (min-width: 640px) {
/* Tablets */
}
@media (min-width: 860px) {
/* Small desktops, landscape tablets */
}
@media (min-width: 1024px) {
/* Desktops */
}
@media (min-width: 1280px) {
/* Large desktops */
}
```
### Rules
1. **Mobile-first approach** - Base styles target mobile
2. **Progressive enhancement** - Add complexity as viewport increases
3. **Test intermediate breakpoints** - Avoid jarring layout jumps
4. **Never hide critical content** on mobile
---
## Accessibility
### Color Contrast
- Normal text: Minimum 4.5:1 ratio
- Large text (18px+): Minimum 3:1 ratio
- Interactive elements: Minimum 3:1 ratio
### Focus States
```css
:focus-visible {
outline: 2px solid var(--accent);
outline-offset: 2px;
border-radius: 2px;
}
```
### Screen Readers
- Use `sr-only` class for visually hidden but accessible text
- Always include `alt` text for images (empty `alt=""` for decorative)
- Use semantic HTML elements (`main`, `nav`, `article`, `section`)
- Proper heading hierarchy (h1 > h2 > h3, no skipping)
### Motion
```css
/* Respect user preference */
@media (prefers-reduced-motion: reduce) {
* {
animation-duration: 0.01ms !important;
transition-duration: 0.01ms !important;
}
}
```
---
## Animation
### Timing
```css
--transition-fast: 150ms;
--transition-base: 200ms;
--transition-slow: 300ms;
```
### Easing
- Use `ease` or `ease-out` for most transitions
- Use `ease-in-out` for enter/exit animations
- Never use `linear` except for continuous animations
### Rules
1. **Subtle by default** - Avoid flashy animations
2. **Purpose-driven** - Animation should provide feedback
3. **Respect preferences** - Support `prefers-reduced-motion`
4. **Performance** - Use `transform` and `opacity` only
---
## Icons
### Usage
- Use Lucide icons consistently
- Standard sizes: 14px, 16px, 20px, 24px
- Include `aria-hidden="true"` for decorative icons
- Never use emojis as icons
### Placement
- Left of labels in buttons and navigation
- Right of labels for external links or dropdowns
- Centered when used alone with `aria-label`
---
## Dark Mode
### Implementation
```css
[data-theme="dark"] {
/* Dark mode overrides */
}
```
### Rules
1. Never use pure white (`#ffffff`) on dark backgrounds
2. Reduce shadow intensity in dark mode
3. Adjust image brightness if needed
4. Test contrast ratios in both modes
---
## Performance
### CSS
1. Use CSS custom properties for theming
2. Avoid deeply nested selectors (max 3 levels)
3. Use `will-change` sparingly
4. Prefer `transform` over `top/left` for animations
### Images
1. Always specify `width` and `height` attributes
2. Use `loading="lazy"` for below-fold images
3. Use appropriate formats (WebP with fallbacks)
4. Include placeholder or skeleton states
---
## Code Style
### CSS Class Naming
```css
/* Component */
.component-name { }
/* Component modifier */
.component-name.variant { }
/* Component child */
.component-name-child { }
/* State */
.component-name.is-active { }
.component-name[data-state="open"] { }
```
### File Organization
```
src/
components/
ui/ # Primitive components (Button, Input, Card)
layout/ # Layout components (Container, Header)
styles.css # Global styles and design tokens
lib/
theme.ts # Theme utilities
preferences.ts # User preference management
```
---
## Checklist
Before shipping any UI changes, verify:
- [ ] Color contrast meets WCAG AA standards
- [ ] Focus states are visible
- [ ] Layout works at all breakpoints
- [ ] Animations respect `prefers-reduced-motion`
- [ ] Text is readable at default browser zoom
- [ ] Interactive elements have 44px minimum touch target
- [ ] Semantic HTML is used appropriately
- [ ] Dark mode has been tested
+111 -31
View File
@@ -1,33 +1,48 @@
# ClawdHub
<p align="center">
<img src="public/clawd-logo.png" alt="ClawHub" width="120">
</p>
<h1 align="center">ClawHub</h1>
<p align="center">
<a href="https://github.com/clawdbot/clawdhub/actions/workflows/ci.yml?branch=main"><img src="https://img.shields.io/github/actions/workflow/status/clawdbot/clawdhub/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a>
<a href="https://github.com/openclaw/clawhub/actions/workflows/ci.yml?branch=main"><img src="https://img.shields.io/github/actions/workflow/status/openclaw/clawhub/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a>
<a href="https://discord.gg/clawd"><img src="https://img.shields.io/discord/1456350064065904867?label=Discord&logo=discord&logoColor=white&color=5865F2&style=for-the-badge" alt="Discord"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
</p>
ClawdHub is the **public skill registry for Clawdbot**: publish, version, and search text-based agent skills (a `SKILL.md` plus supporting files).
Its designed for fast browsing + a CLI-friendly API, with moderation hooks and vector search.
ClawHub is the **public skill registry for Clawdbot**: publish, version, and search text-based agent skills (a `SKILL.md` plus supporting files).
It's designed for fast browsing + a CLI-friendly API, with moderation hooks and vector search.
It also now exposes a native **OpenClaw package catalog** for code plugins and bundle plugins.
onlycrabs.ai is the **SOUL.md registry**: publish and share system lore the same way you publish skills.
Live: `https://clawdhub.com`
onlycrabs.ai: `https://onlycrabs.ai`
<p align="center">
<a href="https://clawhub.ai">ClawHub</a> ·
<a href="https://onlycrabs.ai">onlycrabs.ai</a> ·
<a href="VISION.md">Vision</a> ·
<a href="docs/README.md">Docs</a> ·
<a href="CONTRIBUTING.md">Contributing</a> ·
<a href="https://discord.gg/clawd">Discord</a>
</p>
## What you can do
## What you can do with it
- Browse skills + render their `SKILL.md`.
- Publish new skill versions with changelogs + tags (including `latest`).
- Rename an owned skill without breaking old links or installs.
- Merge duplicate owned skills into one canonical slug.
- Browse souls + render their `SOUL.md`.
- Publish new soul versions with changelogs + tags.
- Search via embeddings (vector index) instead of brittle keywords.
- Star + comment; admins/mods can curate and approve skills.
- Browse OpenClaw packages with family/trust/capability metadata.
- Publish native code plugins and bundle plugins through `/packages` APIs and CLI flows.
## onlycrabs.ai (SOUL.md registry)
- Entry point is host-based: `onlycrabs.ai`.
- On the onlycrabs.ai host, the home page and nav default to souls.
- On ClawdHub, souls live under `/souls`.
- On ClawHub, souls live under `/souls`.
- Soul bundles only accept `SOUL.md` for now (no extra files).
## How it works (high level)
@@ -35,50 +50,71 @@ onlycrabs.ai: `https://onlycrabs.ai`
- Web app: TanStack Start (React, Vite/Nitro).
- Backend: Convex (DB + file storage + HTTP actions) + Convex Auth (GitHub OAuth).
- Search: OpenAI embeddings (`text-embedding-3-small`) + Convex vector search.
- API schema + routes: `packages/schema` (`clawdhub-schema`).
- API schema + routes: `packages/schema` (`clawhub-schema`).
## CLI
Common CLI flows:
- Auth: `clawhub login`, `clawhub whoami`
- Discover: `clawhub search ...`, `clawhub explore`
- Browse unified catalog (skills + plugins): `clawhub package explore`, `clawhub package inspect <name>`
- Manage local installs: `clawhub install <slug>`, `clawhub uninstall <slug>`, `clawhub list`, `clawhub update --all`
- Inspect without installing: `clawhub inspect <slug>`
- Publish/sync skills: `clawhub skill publish <path>`, `clawhub sync`
- Publish plugins: `clawhub package publish <source>`
- Canonicalize owned skills: `clawhub skill rename <slug> <new-slug>`, `clawhub skill merge <source> <target>`
Docs: [`docs/quickstart.md`](docs/quickstart.md), [`docs/cli.md`](docs/cli.md).
### Removal permissions
- `clawhub uninstall <slug>` only removes a local install on your machine.
- Uploaded registry skills use soft-delete/restore (`clawhub delete <slug>` / `clawhub undelete <slug>` or API equivalents).
- Soft-delete/restore is allowed for the skill owner, moderators, and admins.
- Hard delete is admin-only (management tools / ban flows).
- Owner rename keeps the old slug as a redirect alias.
- Owner merge hides the source listing and redirects the old slug to the canonical target.
## Telemetry
ClawdHub tracks minimal **install telemetry** (to compute install counts) when you run `clawdhub sync` while logged in.
ClawHub tracks minimal **install telemetry** (to compute install counts) when you run `clawhub sync` while logged in.
Disable via:
```bash
export CLAWDHUB_DISABLE_TELEMETRY=1
export CLAWHUB_DISABLE_TELEMETRY=1
```
Details: `docs/telemetry.md`.
Details: [`docs/telemetry.md`](docs/telemetry.md).
## Repo layout
- `src/` — TanStack Start app (routes, components, styles).
- `convex/` — schema + queries/mutations/actions + HTTP API routes.
- `packages/schema/` — shared API types/routes for the CLI and app.
- `docs/spec.md` — product + implementation spec (good first read).
- [`docs/`](docs/README.md) — project documentation (architecture, CLI, auth, deployment, and more).
- [`docs/spec.md`](docs/spec.md) — product + implementation spec (good first read).
## Local dev
Prereqs: Bun + Convex CLI.
Prereqs: [Bun](https://bun.sh/) (Convex runs via `bunx`, no global install needed).
```bash
bun install
cp .env.local.example .env.local
# edit .env.local — see CONTRIBUTING.md for local Convex values
# terminal A: web app
# terminal A: local Convex backend
bunx convex dev
# terminal B: web app (port 3000)
bun run dev
# terminal B: Convex dev deployment
bunx convex dev
# seed sample data
bunx convex run --no-push devSeed:seedNixSkills
```
## Auth (GitHub OAuth) setup
Create a GitHub OAuth App, set `AUTH_GITHUB_ID` / `AUTH_GITHUB_SECRET`, then:
```bash
bunx auth --deployment-name <deployment> --web-server-url http://localhost:3000
```
This writes `JWT_PRIVATE_KEY` + `JWKS` to the deployment and prints values for your local `.env.local`.
For full setup instructions (env vars, GitHub OAuth, JWT keys, database seeding), see [CONTRIBUTING.md](CONTRIBUTING.md).
## Environment
@@ -95,7 +131,7 @@ This writes `JWT_PRIVATE_KEY` + `JWKS` to the deployment and prints values for y
## Nix plugins (nixmode skills)
ClawdHub can store a nix-clawdbot plugin pointer in SKILL frontmatter so the registry knows which
ClawHub can store a nix-clawdbot plugin pointer in SKILL frontmatter so the registry knows which
Nix package bundle to install. A nix plugin is different from a regular skill pack: it bundles the
skill pack, the CLI binary, and its config flags/requirements together.
@@ -105,7 +141,17 @@ Add this to `SKILL.md`:
---
name: peekaboo
description: Capture and automate macOS UI with the Peekaboo CLI.
metadata: {"clawdbot":{"nix":{"plugin":"github:clawdbot/nix-steipete-tools?dir=tools/peekaboo","systems":["aarch64-darwin"]}}}
metadata:
{
"clawdbot":
{
"nix":
{
"plugin": "github:clawdbot/nix-steipete-tools?dir=tools/peekaboo",
"systems": ["aarch64-darwin"],
},
},
}
---
```
@@ -123,7 +169,18 @@ You can also declare config requirements + an example snippet:
---
name: padel
description: Check padel court availability and manage bookings via Playtomic.
metadata: {"clawdbot":{"config":{"requiredEnv":["PADEL_AUTH_FILE"],"stateDirs":[".config/padel"],"example":"config = { env = { PADEL_AUTH_FILE = \\\"/run/agenix/padel-auth\\\"; }; };"}}}
metadata:
{
"clawdbot":
{
"config":
{
"requiredEnv": ["PADEL_AUTH_FILE"],
"stateDirs": [".config/padel"],
"example": "config = { env = { PADEL_AUTH_FILE = \\\"/run/agenix/padel-auth\\\"; }; };",
},
},
}
---
```
@@ -133,11 +190,34 @@ To show CLI help (recommended for nix plugins), include the `cli --help` output:
---
name: padel
description: Check padel court availability and manage bookings via Playtomic.
metadata: {"clawdbot":{"cliHelp":"padel --help\\nUsage: padel [command]\\n"}}
metadata: { "clawdbot": { "cliHelp": "padel --help\\nUsage: padel [command]\\n" } }
---
```
`metadata.clawdbot` is preferred, but `metadata.clawdis` is accepted as an alias for compatibility.
`metadata.clawdbot` is preferred, but `metadata.clawdis` and `metadata.openclaw` are accepted as aliases.
## Skill metadata
Skills declare their runtime requirements (env vars, binaries, install specs) in the `SKILL.md` frontmatter. ClawHub's security analysis checks these declarations against actual skill behavior.
Full reference: [`docs/skill-format.md`](docs/skill-format.md#frontmatter-metadata)
Quick example:
```yaml
---
name: my-skill
description: Does a thing with an API.
metadata:
openclaw:
requires:
env:
- MY_API_KEY
bins:
- curl
primaryEnv: MY_API_KEY
---
```
## Scripts
+98
View File
@@ -0,0 +1,98 @@
## OpenClaw Vision
OpenClaw is the AI that actually does things.
It runs on your devices, in your channels, with your rules.
This document explains the current state and direction of the project.
We are still early, so iteration is fast.
Project overview and developer docs: [`README.md`](README.md)
OpenClaw started as my personal playground to learn AI and build something genuinely useful:
an assistant that can run real tasks on my computer.
It evolved through several names and shells: Warelay -> Clawdbot -> Moltbot -> OpenClaw.
The goal? A personal assistant that's easy to use, supports a wide range of platforms, and respects your privacy and security.
The current focus is:
Priority:
- Security and safe defaults
- Bug fixes and stability
- Setup reliability and first-run UX
Next priorities:
- Supporting all major model providers
- Improving support for major messaging channels (and adding a few high-demand ones)
- Performance and test infrastructure
- Better computer-use and agent harness capabilities
- Ergonomics across CLI and web frontend
- Companion apps on macOS, iOS, Android, Windows, and Linux
## Security
Security in OpenClaw is a deliberate tradeoff: strong defaults without killing capability.
The goal is to stay powerful for real work while making risky paths explicit and operator-controlled.
Canonical security policy and reporting:
- https://github.com/openclaw/openclaw/blob/main/SECURITY.md
We prioritize secure defaults, but we also expose clear knobs for trusted high-power workflows.
## Plugins & Memory
OpenClaw has an extensive plugin API.
Core stays lean; optional capability should usually ship as plugins.
Preferred plugin path is npm package distribution plus local extension loading for development.
If you build a plugin, please host and maintain it in your own repository.
The bar for adding optional plugins to core is intentionally high.
Memory is a special plugin slot where only one memory plugin can be active at a time.
Today we ship multiple memory options; over time we plan to converge on one recommended default path.
### Skills
We still ship some bundled skills for baseline UX.
New skills should be published to ClawHub first (`clawhub.ai`), not added to core by default.
Core skill additions should be rare and require a strong product or security reason.
### MCP Support
OpenClaw supports MCP through `mcporter`: https://github.com/steipete/mcporter
This keeps MCP integration flexible and decoupled from core runtime:
- add or change MCP servers without restarting the gateway
- keep core tool/context surface lean
- reduce MCP churn impact on core stability and security
For now, we prefer this bridge model over building first-class MCP runtime into core.
If there is an MCP server or feature `mcporter` does not support yet, please open an issue there.
### Setup
OpenClaw is currently terminal-first by design.
This keeps setup explicit: users see docs, auth, permissions, and security posture up front.
Long term, we want easier onboarding flows as hardening matures.
We do not want convenience wrappers that hide critical security decisions from users.
### Why TypeScript?
OpenClaw is primarily an orchestration system: prompts, tools, protocols, and integrations.
TypeScript was chosen to keep OpenClaw hackable by default.
It is widely known, fast to iterate in, and easy to read, modify, and extend.
## What We Will Not Merge (For Now)
- New core skills when they can live on ClawHub
- Commercial service integrations that do not clearly fit the model-provider category
- Wrapper channels around already supported channels without a clear capability or security gap
- First-class MCP runtime in core when `mcporter` already provides the integration path
- Heavy orchestration layers that duplicate existing agent and tool infrastructure
This list is a roadmap guardrail, not a law of physics.
Strong user demand and strong technical rationale can change it.
-41
View File
@@ -1,41 +0,0 @@
{
"$schema": "https://biomejs.dev/schemas/2.3.11/schema.json",
"files": {
"includes": [
"**",
"!**/.cta.json",
"!**/.vscode",
"!**/node_modules",
"!**/dist",
"!**/.output",
"!**/coverage",
"!**/convex/_generated",
"!**/test-results",
"!**/src/routeTree.gen.ts",
"!**/.tanstack",
"!**/public",
"!**/.devenv",
"!**/.devenv"
]
},
"assist": { "actions": { "source": { "organizeImports": "on" } } },
"formatter": {
"enabled": true,
"indentStyle": "space",
"indentWidth": 2,
"lineWidth": 100
},
"linter": {
"enabled": true,
"rules": {
"recommended": true
}
},
"javascript": {
"formatter": {
"quoteStyle": "single",
"semicolons": "asNeeded",
"trailingCommas": "all"
}
}
}
+627 -429
View File
File diff suppressed because it is too large Load Diff
+6 -3
View File
@@ -3,9 +3,10 @@ import { existsSync } from 'node:fs'
import { stat } from 'node:fs/promises'
import { fileURLToPath } from 'node:url'
const distCliUrl = new URL('./packages/clawdhub/dist/cli.js', import.meta.url)
const packageRootPath = fileURLToPath(new URL('./packages/clawhub/', import.meta.url))
const distCliUrl = new URL('./packages/clawhub/dist/cli.js', import.meta.url)
const distCliPath = fileURLToPath(distCliUrl)
const srcRootPath = fileURLToPath(new URL('./packages/clawdhub/src/', import.meta.url))
const srcRootPath = fileURLToPath(new URL('./packages/clawhub/src/', import.meta.url))
const shouldBuild = await (async () => {
if (!existsSync(distCliPath)) return true
@@ -19,7 +20,8 @@ const shouldBuild = await (async () => {
})()
if (shouldBuild) {
const proc = Bun.spawn(['bunx', 'tsc', '-p', 'packages/clawdhub/tsconfig.json'], {
const proc = Bun.spawn(['bun', 'run', 'build'], {
cwd: packageRootPath,
stdin: 'inherit',
stdout: 'inherit',
stderr: 'inherit',
@@ -34,6 +36,7 @@ async function getLatestMtime(root: string) {
let latest = 0
const glob = new Bun.Glob('**/*.ts')
for await (const rel of glob.scan({ cwd: root, onlyFiles: true })) {
if (rel.endsWith('.test.ts')) continue
const path = `${root}${root.endsWith('/') ? '' : '/'}${rel}`
try {
const entry = await stat(path)
Executable
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bun
import { existsSync } from 'node:fs'
import { stat } from 'node:fs/promises'
import { fileURLToPath } from 'node:url'
const packageRootPath = fileURLToPath(new URL('./packages/clawhub/', import.meta.url))
const distCliUrl = new URL('./packages/clawhub/dist/cli.js', import.meta.url)
const distCliPath = fileURLToPath(distCliUrl)
const srcRootPath = fileURLToPath(new URL('./packages/clawhub/src/', import.meta.url))
const shouldBuild = await (async () => {
if (!existsSync(distCliPath)) return true
try {
const dist = await stat(distCliPath)
const latestSrcMtime = await getLatestMtime(srcRootPath)
return latestSrcMtime > dist.mtimeMs
} catch {
return true
}
})()
if (shouldBuild) {
const proc = Bun.spawn(['bun', 'run', 'build'], {
cwd: packageRootPath,
stdin: 'inherit',
stdout: 'inherit',
stderr: 'inherit',
})
const code = await proc.exited
if (code !== 0) process.exit(code)
}
await import(distCliUrl.href)
async function getLatestMtime(root: string) {
let latest = 0
const glob = new Bun.Glob('**/*.ts')
for await (const rel of glob.scan({ cwd: root, onlyFiles: true })) {
if (rel.endsWith('.test.ts')) continue
const path = `${root}${root.endsWith('/') ? '' : '/'}${rel}`
try {
const entry = await stat(path)
latest = Math.max(latest, entry.mtimeMs)
} catch {
// ignore
}
}
return latest
}
+122
View File
@@ -8,42 +8,102 @@
* @module
*/
import type * as appMeta from "../appMeta.js";
import type * as auth from "../auth.js";
import type * as commentModeration from "../commentModeration.js";
import type * as comments from "../comments.js";
import type * as crons from "../crons.js";
import type * as devSeed from "../devSeed.js";
import type * as devSeedExtra from "../devSeedExtra.js";
import type * as downloads from "../downloads.js";
import type * as functions from "../functions.js";
import type * as githubBackups from "../githubBackups.js";
import type * as githubBackupsNode from "../githubBackupsNode.js";
import type * as githubIdentity from "../githubIdentity.js";
import type * as githubImport from "../githubImport.js";
import type * as githubRestore from "../githubRestore.js";
import type * as githubRestoreMutations from "../githubRestoreMutations.js";
import type * as githubSoulBackups from "../githubSoulBackups.js";
import type * as githubSoulBackupsNode from "../githubSoulBackupsNode.js";
import type * as http from "../http.js";
import type * as httpApi from "../httpApi.js";
import type * as httpApiV1 from "../httpApiV1.js";
import type * as httpApiV1_packagesV1 from "../httpApiV1/packagesV1.js";
import type * as httpApiV1_shared from "../httpApiV1/shared.js";
import type * as httpApiV1_skillsV1 from "../httpApiV1/skillsV1.js";
import type * as httpApiV1_soulsV1 from "../httpApiV1/soulsV1.js";
import type * as httpApiV1_starsV1 from "../httpApiV1/starsV1.js";
import type * as httpApiV1_transfersV1 from "../httpApiV1/transfersV1.js";
import type * as httpApiV1_usersV1 from "../httpApiV1/usersV1.js";
import type * as httpApiV1_whoamiV1 from "../httpApiV1/whoamiV1.js";
import type * as httpPreflight from "../httpPreflight.js";
import type * as leaderboards from "../leaderboards.js";
import type * as lib_access from "../lib/access.js";
import type * as lib_apiTokenAuth from "../lib/apiTokenAuth.js";
import type * as lib_badges from "../lib/badges.js";
import type * as lib_batching from "../lib/batching.js";
import type * as lib_changelog from "../lib/changelog.js";
import type * as lib_commentScamPrompt from "../lib/commentScamPrompt.js";
import type * as lib_contentTypes from "../lib/contentTypes.js";
import type * as lib_embeddingVisibility from "../lib/embeddingVisibility.js";
import type * as lib_embeddings from "../lib/embeddings.js";
import type * as lib_githubAccount from "../lib/githubAccount.js";
import type * as lib_githubActionsOidc from "../lib/githubActionsOidc.js";
import type * as lib_githubBackup from "../lib/githubBackup.js";
import type * as lib_githubIdentity from "../lib/githubIdentity.js";
import type * as lib_githubImport from "../lib/githubImport.js";
import type * as lib_githubProfileSync from "../lib/githubProfileSync.js";
import type * as lib_githubRestoreHelpers from "../lib/githubRestoreHelpers.js";
import type * as lib_githubSoulBackup from "../lib/githubSoulBackup.js";
import type * as lib_globalStats from "../lib/globalStats.js";
import type * as lib_httpHeaders from "../lib/httpHeaders.js";
import type * as lib_httpRateLimit from "../lib/httpRateLimit.js";
import type * as lib_httpUtils from "../lib/httpUtils.js";
import type * as lib_leaderboards from "../lib/leaderboards.js";
import type * as lib_manualOverrides from "../lib/manualOverrides.js";
import type * as lib_moderation from "../lib/moderation.js";
import type * as lib_moderationEngine from "../lib/moderationEngine.js";
import type * as lib_moderationReasonCodes from "../lib/moderationReasonCodes.js";
import type * as lib_openaiResponse from "../lib/openaiResponse.js";
import type * as lib_packageRegistry from "../lib/packageRegistry.js";
import type * as lib_packageSearchDigest from "../lib/packageSearchDigest.js";
import type * as lib_packageSecurity from "../lib/packageSecurity.js";
import type * as lib_public from "../lib/public.js";
import type * as lib_publishLimits from "../lib/publishLimits.js";
import type * as lib_publishers from "../lib/publishers.js";
import type * as lib_reporting from "../lib/reporting.js";
import type * as lib_reservedHandles from "../lib/reservedHandles.js";
import type * as lib_reservedSlugs from "../lib/reservedSlugs.js";
import type * as lib_searchText from "../lib/searchText.js";
import type * as lib_securityPrompt from "../lib/securityPrompt.js";
import type * as lib_skillBackfill from "../lib/skillBackfill.js";
import type * as lib_skillCapabilityTags from "../lib/skillCapabilityTags.js";
import type * as lib_skillPublish from "../lib/skillPublish.js";
import type * as lib_skillQuality from "../lib/skillQuality.js";
import type * as lib_skillSafety from "../lib/skillSafety.js";
import type * as lib_skillSearchDigest from "../lib/skillSearchDigest.js";
import type * as lib_skillStats from "../lib/skillStats.js";
import type * as lib_skillSummary from "../lib/skillSummary.js";
import type * as lib_skillZip from "../lib/skillZip.js";
import type * as lib_skills from "../lib/skills.js";
import type * as lib_soulChangelog from "../lib/soulChangelog.js";
import type * as lib_soulPublish from "../lib/soulPublish.js";
import type * as lib_staticPublishScan from "../lib/staticPublishScan.js";
import type * as lib_tokens from "../lib/tokens.js";
import type * as lib_userSearch from "../lib/userSearch.js";
import type * as lib_webhooks from "../lib/webhooks.js";
import type * as llmEval from "../llmEval.js";
import type * as maintenance from "../maintenance.js";
import type * as packagePublishTokens from "../packagePublishTokens.js";
import type * as packages from "../packages.js";
import type * as publishers from "../publishers.js";
import type * as rateLimits from "../rateLimits.js";
import type * as search from "../search.js";
import type * as seed from "../seed.js";
import type * as seedDemo from "../seedDemo.js";
import type * as seedSouls from "../seedSouls.js";
import type * as skillStatEvents from "../skillStatEvents.js";
import type * as skillTransfers from "../skillTransfers.js";
import type * as skills from "../skills.js";
import type * as soulComments from "../soulComments.js";
import type * as soulDownloads from "../soulDownloads.js";
@@ -55,6 +115,7 @@ import type * as telemetry from "../telemetry.js";
import type * as tokens from "../tokens.js";
import type * as uploads from "../uploads.js";
import type * as users from "../users.js";
import type * as vt from "../vt.js";
import type * as webhooks from "../webhooks.js";
import type {
@@ -64,42 +125,102 @@ import type {
} from "convex/server";
declare const fullApi: ApiFromModules<{
appMeta: typeof appMeta;
auth: typeof auth;
commentModeration: typeof commentModeration;
comments: typeof comments;
crons: typeof crons;
devSeed: typeof devSeed;
devSeedExtra: typeof devSeedExtra;
downloads: typeof downloads;
functions: typeof functions;
githubBackups: typeof githubBackups;
githubBackupsNode: typeof githubBackupsNode;
githubIdentity: typeof githubIdentity;
githubImport: typeof githubImport;
githubRestore: typeof githubRestore;
githubRestoreMutations: typeof githubRestoreMutations;
githubSoulBackups: typeof githubSoulBackups;
githubSoulBackupsNode: typeof githubSoulBackupsNode;
http: typeof http;
httpApi: typeof httpApi;
httpApiV1: typeof httpApiV1;
"httpApiV1/packagesV1": typeof httpApiV1_packagesV1;
"httpApiV1/shared": typeof httpApiV1_shared;
"httpApiV1/skillsV1": typeof httpApiV1_skillsV1;
"httpApiV1/soulsV1": typeof httpApiV1_soulsV1;
"httpApiV1/starsV1": typeof httpApiV1_starsV1;
"httpApiV1/transfersV1": typeof httpApiV1_transfersV1;
"httpApiV1/usersV1": typeof httpApiV1_usersV1;
"httpApiV1/whoamiV1": typeof httpApiV1_whoamiV1;
httpPreflight: typeof httpPreflight;
leaderboards: typeof leaderboards;
"lib/access": typeof lib_access;
"lib/apiTokenAuth": typeof lib_apiTokenAuth;
"lib/badges": typeof lib_badges;
"lib/batching": typeof lib_batching;
"lib/changelog": typeof lib_changelog;
"lib/commentScamPrompt": typeof lib_commentScamPrompt;
"lib/contentTypes": typeof lib_contentTypes;
"lib/embeddingVisibility": typeof lib_embeddingVisibility;
"lib/embeddings": typeof lib_embeddings;
"lib/githubAccount": typeof lib_githubAccount;
"lib/githubActionsOidc": typeof lib_githubActionsOidc;
"lib/githubBackup": typeof lib_githubBackup;
"lib/githubIdentity": typeof lib_githubIdentity;
"lib/githubImport": typeof lib_githubImport;
"lib/githubProfileSync": typeof lib_githubProfileSync;
"lib/githubRestoreHelpers": typeof lib_githubRestoreHelpers;
"lib/githubSoulBackup": typeof lib_githubSoulBackup;
"lib/globalStats": typeof lib_globalStats;
"lib/httpHeaders": typeof lib_httpHeaders;
"lib/httpRateLimit": typeof lib_httpRateLimit;
"lib/httpUtils": typeof lib_httpUtils;
"lib/leaderboards": typeof lib_leaderboards;
"lib/manualOverrides": typeof lib_manualOverrides;
"lib/moderation": typeof lib_moderation;
"lib/moderationEngine": typeof lib_moderationEngine;
"lib/moderationReasonCodes": typeof lib_moderationReasonCodes;
"lib/openaiResponse": typeof lib_openaiResponse;
"lib/packageRegistry": typeof lib_packageRegistry;
"lib/packageSearchDigest": typeof lib_packageSearchDigest;
"lib/packageSecurity": typeof lib_packageSecurity;
"lib/public": typeof lib_public;
"lib/publishLimits": typeof lib_publishLimits;
"lib/publishers": typeof lib_publishers;
"lib/reporting": typeof lib_reporting;
"lib/reservedHandles": typeof lib_reservedHandles;
"lib/reservedSlugs": typeof lib_reservedSlugs;
"lib/searchText": typeof lib_searchText;
"lib/securityPrompt": typeof lib_securityPrompt;
"lib/skillBackfill": typeof lib_skillBackfill;
"lib/skillCapabilityTags": typeof lib_skillCapabilityTags;
"lib/skillPublish": typeof lib_skillPublish;
"lib/skillQuality": typeof lib_skillQuality;
"lib/skillSafety": typeof lib_skillSafety;
"lib/skillSearchDigest": typeof lib_skillSearchDigest;
"lib/skillStats": typeof lib_skillStats;
"lib/skillSummary": typeof lib_skillSummary;
"lib/skillZip": typeof lib_skillZip;
"lib/skills": typeof lib_skills;
"lib/soulChangelog": typeof lib_soulChangelog;
"lib/soulPublish": typeof lib_soulPublish;
"lib/staticPublishScan": typeof lib_staticPublishScan;
"lib/tokens": typeof lib_tokens;
"lib/userSearch": typeof lib_userSearch;
"lib/webhooks": typeof lib_webhooks;
llmEval: typeof llmEval;
maintenance: typeof maintenance;
packagePublishTokens: typeof packagePublishTokens;
packages: typeof packages;
publishers: typeof publishers;
rateLimits: typeof rateLimits;
search: typeof search;
seed: typeof seed;
seedDemo: typeof seedDemo;
seedSouls: typeof seedSouls;
skillStatEvents: typeof skillStatEvents;
skillTransfers: typeof skillTransfers;
skills: typeof skills;
soulComments: typeof soulComments;
soulDownloads: typeof soulDownloads;
@@ -111,6 +232,7 @@ declare const fullApi: ApiFromModules<{
tokens: typeof tokens;
uploads: typeof uploads;
users: typeof users;
vt: typeof vt;
webhooks: typeof webhooks;
}>;
+10
View File
@@ -0,0 +1,10 @@
import { api, internal } from "./_generated/api";
void internal.downloads.recordDownloadInternal;
void internal.soulDownloads.incrementInternal;
// @ts-expect-error download counters must not be publicly callable
void api.downloads.increment;
// @ts-expect-error soul download counters must not be publicly callable
void api.soulDownloads.increment;
+14
View File
@@ -0,0 +1,14 @@
import { query } from "./functions";
function normalizeEnv(value: string | undefined) {
const normalized = value?.trim();
return normalized ? normalized : null;
}
export const getDeploymentInfo = query({
args: {},
handler: async () => ({
appBuildSha: normalizeEnv(process.env.APP_BUILD_SHA),
deployedAt: normalizeEnv(process.env.APP_DEPLOYED_AT),
}),
});
+2 -2
View File
@@ -2,7 +2,7 @@ export default {
providers: [
{
domain: process.env.CONVEX_SITE_URL,
applicationID: 'convex',
applicationID: "convex",
},
],
}
};
+141
View File
@@ -0,0 +1,141 @@
import { describe, expect, it, vi } from "vitest";
import type { Id } from "./_generated/dataModel";
import {
BANNED_REAUTH_MESSAGE,
DELETED_ACCOUNT_REAUTH_MESSAGE,
handleDeletedUserSignIn,
} from "./auth";
function makeCtx({
user,
banRecords,
}: {
user: {
deletedAt?: number;
deactivatedAt?: number;
purgedAt?: number;
banReason?: string;
} | null;
banRecords?: Array<Record<string, unknown>>;
}) {
const query = {
withIndex: vi.fn().mockReturnValue({
collect: vi.fn().mockResolvedValue(banRecords ?? []),
}),
};
const ctx = {
db: {
get: vi.fn().mockResolvedValue(user),
patch: vi.fn().mockResolvedValue(null),
query: vi.fn().mockReturnValue(query),
},
};
return { ctx, query };
}
describe("handleDeletedUserSignIn", () => {
const userId = "users:1" as Id<"users">;
it("skips when user not found", async () => {
const { ctx } = makeCtx({ user: null });
await handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId });
expect(ctx.db.get).toHaveBeenCalledWith(userId);
expect(ctx.db.query).not.toHaveBeenCalled();
});
it("skips active users", async () => {
const { ctx } = makeCtx({ user: { deletedAt: undefined, deactivatedAt: undefined } });
await handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId });
expect(ctx.db.query).not.toHaveBeenCalled();
expect(ctx.db.patch).not.toHaveBeenCalled();
});
it("blocks sign-in for deactivated users", async () => {
const { ctx } = makeCtx({ user: { deactivatedAt: 123, purgedAt: 123 } });
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(DELETED_ACCOUNT_REAUTH_MESSAGE);
expect(ctx.db.query).not.toHaveBeenCalled();
expect(ctx.db.patch).not.toHaveBeenCalled();
});
it("migrates legacy self-deleted users and blocks sign-in", async () => {
const { ctx } = makeCtx({ user: { deletedAt: 123 }, banRecords: [] });
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(DELETED_ACCOUNT_REAUTH_MESSAGE);
expect(ctx.db.patch).toHaveBeenCalledWith(userId, {
deletedAt: undefined,
deactivatedAt: 123,
purgedAt: 123,
updatedAt: expect.any(Number),
});
});
it("migrates legacy users on fresh login (existingUserId is null)", async () => {
const { ctx } = makeCtx({ user: { deletedAt: 123 }, banRecords: [] });
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: null }),
).rejects.toThrow(DELETED_ACCOUNT_REAUTH_MESSAGE);
expect(ctx.db.patch).toHaveBeenCalledWith(userId, {
deletedAt: undefined,
deactivatedAt: 123,
purgedAt: 123,
updatedAt: expect.any(Number),
});
});
it("skips mutation when existingUserId does not match userId", async () => {
const otherUserId = "users:999" as Id<"users">;
const { ctx } = makeCtx({ user: { deletedAt: 123 } });
await handleDeletedUserSignIn(ctx as never, { userId, existingUserId: otherUserId });
expect(ctx.db.query).not.toHaveBeenCalled();
expect(ctx.db.patch).not.toHaveBeenCalled();
});
it("blocks banned users with a custom message", async () => {
const { ctx } = makeCtx({ user: { deletedAt: 123 }, banRecords: [{ action: "user.ban" }] });
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(BANNED_REAUTH_MESSAGE);
expect(ctx.db.patch).not.toHaveBeenCalled();
});
it("blocks users auto-banned for malware", async () => {
const { ctx } = makeCtx({
user: { deletedAt: 123, banReason: "malware auto-ban" },
banRecords: [{ action: "user.autoban.malware" }],
});
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(BANNED_REAUTH_MESSAGE);
expect(ctx.db.patch).not.toHaveBeenCalled();
});
it("includes the moderator ban reason in the sign-in error", async () => {
const { ctx } = makeCtx({
user: { deletedAt: 123, banReason: "Chargeback fraud" },
banRecords: [{ action: "user.ban" }],
});
await expect(
handleDeletedUserSignIn(ctx as never, { userId, existingUserId: userId }),
).rejects.toThrow(`${BANNED_REAUTH_MESSAGE} Reason: Chargeback fraud`);
});
});
+103 -6
View File
@@ -1,19 +1,116 @@
import GitHub from '@auth/core/providers/github'
import { convexAuth } from '@convex-dev/auth/server'
import GitHub from "@auth/core/providers/github";
import { convexAuth } from "@convex-dev/auth/server";
import type { GenericMutationCtx } from "convex/server";
import { ConvexError } from "convex/values";
import { internal } from "./_generated/api";
import type { DataModel, Id } from "./_generated/dataModel";
import { shouldScheduleGitHubProfileSync } from "./lib/githubProfileSync";
export const BANNED_REAUTH_MESSAGE =
"This account has been banned and cannot sign in. If you believe this is a mistake, please contact security@openclaw.ai and we will review it.";
export const DELETED_ACCOUNT_REAUTH_MESSAGE =
"This account has been permanently deleted and cannot be restored.";
const REAUTH_BLOCKING_BAN_ACTIONS = new Set(["user.ban", "user.autoban.malware"]);
function getBannedReauthMessage(reason: string | undefined) {
const normalizedReason = reason?.trim();
if (!normalizedReason || normalizedReason.toLowerCase() === "malware auto-ban") {
return BANNED_REAUTH_MESSAGE;
}
return `${BANNED_REAUTH_MESSAGE} Reason: ${normalizedReason}`;
}
export async function handleDeletedUserSignIn(
ctx: GenericMutationCtx<DataModel>,
args: { userId: Id<"users">; existingUserId: Id<"users"> | null },
userOverride?: {
deletedAt?: number;
deactivatedAt?: number;
purgedAt?: number;
banReason?: string;
} | null,
) {
const user = userOverride !== undefined ? userOverride : await ctx.db.get(args.userId);
if (!user?.deletedAt && !user?.deactivatedAt) return;
// Verify that the incoming identity matches the existing account to prevent bypass.
if (args.existingUserId && args.existingUserId !== args.userId) {
return;
}
if (user.deactivatedAt) {
throw new ConvexError(DELETED_ACCOUNT_REAUTH_MESSAGE);
}
const userId = args.userId;
const deletedAt = user.deletedAt ?? Date.now();
const banRecords = await ctx.db
.query("auditLogs")
.withIndex("by_target", (q) => q.eq("targetType", "user").eq("targetId", userId.toString()))
.collect();
const hasBlockingBan = banRecords.some((record) =>
REAUTH_BLOCKING_BAN_ACTIONS.has(record.action),
);
if (hasBlockingBan) {
throw new ConvexError(getBannedReauthMessage(user.banReason));
}
// Migrate legacy self-deleted accounts (stored in deletedAt) to the new
// irreversible state and reject sign-in.
await ctx.db.patch(userId, {
deletedAt: undefined,
deactivatedAt: deletedAt,
purgedAt: user.purgedAt ?? deletedAt,
updatedAt: Date.now(),
});
throw new ConvexError(DELETED_ACCOUNT_REAUTH_MESSAGE);
}
export const { auth, signIn, signOut, store, isAuthenticated } = convexAuth({
providers: [
GitHub({
clientId: process.env.AUTH_GITHUB_ID ?? '',
clientSecret: process.env.AUTH_GITHUB_SECRET ?? '',
clientId: process.env.AUTH_GITHUB_ID ?? "",
clientSecret: process.env.AUTH_GITHUB_SECRET ?? "",
profile(profile) {
return {
id: String(profile.id),
name: profile.login,
email: profile.email ?? undefined,
image: profile.avatar_url,
}
};
},
}),
],
})
callbacks: {
/**
* Block sign-in for deleted/deactivated users and sync GitHub profile.
*
* Performance note: This callback runs on every OAuth sign-in, but the
* audit log query ONLY executes when a legacy deleted user attempts to sign
* in (user.deletedAt is set). For active users, this is a single field check.
*
* The GitHub profile sync is scheduled as a background action to handle
* the case where a user renames their GitHub account (fixes #303).
*/
async afterUserCreatedOrUpdated(ctx, args) {
const user = await ctx.db.get(args.userId);
await handleDeletedUserSignIn(ctx, args, user);
await ctx.scheduler.runAfter(0, internal.publishers.ensurePersonalPublisherInternal, {
userId: args.userId,
});
// Schedule GitHub profile sync to handle username renames (fixes #303)
// This runs as a background action so it doesn't block sign-in
const now = Date.now();
if (shouldScheduleGitHubProfileSync(user, now)) {
await ctx.scheduler.runAfter(0, internal.users.syncGitHubProfileAction, {
userId: args.userId,
});
}
},
},
});
+290
View File
@@ -0,0 +1,290 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("./_generated/api", () => ({
internal: {
commentModeration: {
getCommentScamBackfillPageInternal: Symbol(
"commentModeration.getCommentScamBackfillPageInternal",
),
applyCommentScamResultInternal: Symbol("commentModeration.applyCommentScamResultInternal"),
backfillCommentScamModerationInternal: Symbol(
"commentModeration.backfillCommentScamModerationInternal",
),
continueCommentScamModerationJobInternal: Symbol(
"commentModeration.continueCommentScamModerationJobInternal",
),
},
llmEval: {
evaluateCommentForScam: Symbol("llmEval.evaluateCommentForScam"),
},
users: {
banUserInternal: Symbol("users.banUserInternal"),
},
},
}));
const { applyCommentScamResultInternalHandler, backfillCommentScamModerationInternalHandler } =
await import("./commentModeration");
const { internal } = await import("./_generated/api");
const previousOpenAiApiKey = process.env.OPENAI_API_KEY;
beforeEach(() => {
process.env.OPENAI_API_KEY = "test-key";
});
afterEach(() => {
if (previousOpenAiApiKey === undefined) {
delete process.env.OPENAI_API_KEY;
return;
}
process.env.OPENAI_API_KEY = previousOpenAiApiKey;
});
describe("commentModeration backfill", () => {
it("evaluates comments and bans on certain/high scams", async () => {
const runQuery = vi.fn().mockResolvedValueOnce({
items: [
{
commentId: "comments:1",
skillId: "skills:1",
userId: "users:2",
body: 'echo "mal" | base64 -D | bash',
softDeletedAt: undefined,
scamScanCheckedAt: undefined,
},
],
cursor: null,
isDone: true,
});
const runAction = vi.fn().mockResolvedValue({
ok: true,
model: "gpt-5-mini",
verdict: "certain_scam",
confidence: "high",
explanation: "Obfuscated shell execution payload.",
evidence: ["base64 decode piped to bash"],
});
const runMutation = vi.fn().mockResolvedValue({
ok: true,
shouldBan: true,
banned: true,
alreadyBanned: false,
protectedRole: false,
wouldBan: false,
});
const result = await backfillCommentScamModerationInternalHandler(
{ runQuery, runAction, runMutation } as never,
{
actorUserId: "users:admin",
dryRun: false,
batchSize: 10,
maxBatches: 1,
} as never,
);
expect(result.ok).toBe(true);
expect(result.stats.commentsScanned).toBe(1);
expect(result.stats.commentsEvaluated).toBe(1);
expect(result.stats.certainScams).toBe(1);
expect(result.stats.banCandidates).toBe(1);
expect(result.stats.usersBanned).toBe(1);
expect(runAction).toHaveBeenCalledWith(internal.llmEval.evaluateCommentForScam, {
commentId: "comments:1",
skillId: "skills:1",
userId: "users:2",
body: 'echo "mal" | base64 -D | bash',
});
expect(runMutation).toHaveBeenCalledWith(
internal.commentModeration.applyCommentScamResultInternal,
{
actorUserId: "users:admin",
commentId: "comments:1",
verdict: "certain_scam",
confidence: "high",
explanation: "Obfuscated shell execution payload.",
evidence: ["base64 decode piped to bash"],
model: "gpt-5-mini",
checkedAt: expect.any(Number),
dryRun: false,
},
);
});
it("skips previously scanned comments unless rescan=true", async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [
{
commentId: "comments:1",
skillId: "skills:1",
userId: "users:2",
body: "something",
softDeletedAt: undefined,
scamScanCheckedAt: 123,
},
],
cursor: null,
isDone: true,
});
const runAction = vi.fn();
const runMutation = vi.fn();
const result = await backfillCommentScamModerationInternalHandler(
{ runQuery, runAction, runMutation } as never,
{
actorUserId: "users:admin",
batchSize: 10,
maxBatches: 1,
} as never,
);
expect(result.stats.commentsScanned).toBe(1);
expect(result.stats.skippedAlreadyScanned).toBe(1);
expect(runAction).not.toHaveBeenCalled();
expect(runMutation).not.toHaveBeenCalled();
});
it("tracks dry-run ban candidates without banning", async () => {
const runQuery = vi.fn().mockResolvedValue({
items: [
{
commentId: "comments:9",
skillId: "skills:7",
userId: "users:5",
body: "run this update installer from random domain",
softDeletedAt: undefined,
scamScanCheckedAt: undefined,
},
],
cursor: null,
isDone: true,
});
const runAction = vi.fn().mockResolvedValue({
ok: true,
model: "gpt-5-mini",
verdict: "certain_scam",
confidence: "high",
explanation: "Social-engineering install command.",
evidence: ["unknown update domain"],
});
const runMutation = vi.fn().mockResolvedValue({
ok: true,
shouldBan: true,
banned: false,
alreadyBanned: false,
protectedRole: false,
wouldBan: true,
});
const result = await backfillCommentScamModerationInternalHandler(
{ runQuery, runAction, runMutation } as never,
{
actorUserId: "users:admin",
dryRun: true,
batchSize: 10,
maxBatches: 1,
} as never,
);
expect(result.stats.usersBanned).toBe(0);
expect(result.stats.usersWouldBeBanned).toBe(1);
});
});
describe("applyCommentScamResultInternalHandler", () => {
it("persists scan metadata and triggers ban with bounded reason", async () => {
const get = vi
.fn()
.mockResolvedValueOnce({
_id: "comments:1",
skillId: "skills:1",
userId: "users:2",
})
.mockResolvedValueOnce({
_id: "users:2",
role: "user",
});
const patch = vi.fn();
const insert = vi.fn();
const runMutation = vi
.fn()
.mockResolvedValue({ ok: true, alreadyBanned: false, deletedSkills: 0 });
const result = await applyCommentScamResultInternalHandler(
{ db: { get, patch, insert }, runMutation } as never,
{
actorUserId: "users:admin",
commentId: "comments:1",
verdict: "certain_scam",
confidence: "high",
explanation: "X".repeat(700),
evidence: ["Y".repeat(280), "Z".repeat(280)],
model: "gpt-5-mini",
checkedAt: 123,
} as never,
);
expect(result.banned).toBe(true);
expect(insert).toHaveBeenCalledWith("auditLogs", {
actorUserId: "users:admin",
action: "comment.scam_scan",
targetType: "comment",
targetId: "comments:1",
metadata: {
skillId: "skills:1",
commentAuthorId: "users:2",
verdict: "certain_scam",
confidence: "high",
shouldBan: true,
model: "gpt-5-mini",
},
createdAt: 123,
});
const banCall = runMutation.mock.calls.find(
(call) => call[0] === internal.users.banUserInternal,
);
expect(banCall).toBeTruthy();
if (!banCall) throw new Error("Expected ban mutation to be called");
expect((banCall[1] as { reason: string }).reason.length).toBeLessThanOrEqual(500);
expect(patch).toHaveBeenCalledWith("comments:1", {
scamBanTriggeredAt: 123,
});
});
it("skips banning moderator/admin accounts", async () => {
const get = vi
.fn()
.mockResolvedValueOnce({
_id: "comments:2",
skillId: "skills:2",
userId: "users:staff",
})
.mockResolvedValueOnce({
_id: "users:staff",
role: "moderator",
});
const patch = vi.fn();
const insert = vi.fn();
const runMutation = vi.fn();
const result = await applyCommentScamResultInternalHandler(
{ db: { get, patch, insert }, runMutation } as never,
{
actorUserId: "users:admin",
commentId: "comments:2",
verdict: "certain_scam",
confidence: "high",
explanation: "Malicious command spam.",
evidence: ["base64|bash"],
model: "gpt-5-mini",
checkedAt: 300,
} as never,
);
expect(result.protectedRole).toBe(true);
expect(runMutation).not.toHaveBeenCalled();
});
});
+479
View File
@@ -0,0 +1,479 @@
import { ConvexError, v } from "convex/values";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { ActionCtx, MutationCtx } from "./_generated/server";
import { action, internalAction, internalMutation, internalQuery } from "./functions";
import { assertRole, requireUserFromAction } from "./lib/access";
import {
buildCommentScamBanReason,
isCertainScam,
type CommentScamConfidence,
type CommentScamVerdict,
} from "./lib/commentScamPrompt";
const DEFAULT_BATCH_SIZE = 25;
const MAX_BATCH_SIZE = 100;
const DEFAULT_MAX_BATCHES = 10;
const MAX_MAX_BATCHES = 200;
type CommentBackfillPageItem = {
commentId: Id<"comments">;
skillId: Id<"skills">;
userId: Id<"users">;
body: string;
softDeletedAt?: number;
scamScanCheckedAt?: number;
};
type CommentBackfillPageResult = {
items: CommentBackfillPageItem[];
cursor: string | null;
isDone: boolean;
};
type ApplyCommentScamResult = {
ok: true;
shouldBan: boolean;
banned: boolean;
alreadyBanned: boolean;
protectedRole: boolean;
wouldBan: boolean;
};
export type CommentScamBackfillStats = {
commentsScanned: number;
commentsEvaluated: number;
certainScams: number;
banCandidates: number;
usersBanned: number;
usersAlreadyBanned: number;
usersWouldBeBanned: number;
protectedRoleSkips: number;
skippedSoftDeleted: number;
skippedAlreadyScanned: number;
skippedEmptyBody: number;
evalErrors: number;
};
export type CommentScamBackfillActionArgs = {
actorUserId: Id<"users">;
dryRun?: boolean;
batchSize?: number;
maxBatches?: number;
cursor?: string;
rescan?: boolean;
includeSoftDeleted?: boolean;
};
export type CommentScamBackfillActionResult = {
ok: true;
stats: CommentScamBackfillStats;
isDone: boolean;
cursor: string | null;
};
export const getCommentScamBackfillPageInternal = internalQuery({
args: {
cursor: v.optional(v.string()),
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<CommentBackfillPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const { page, isDone, continueCursor } = await ctx.db
.query("comments")
.order("asc")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
return {
items: page.map((comment) => ({
commentId: comment._id,
skillId: comment.skillId,
userId: comment.userId,
body: comment.body,
softDeletedAt: comment.softDeletedAt,
scamScanCheckedAt: comment.scamScanCheckedAt,
})),
cursor: continueCursor,
isDone,
};
},
});
export async function applyCommentScamResultInternalHandler(
ctx: MutationCtx,
args: {
actorUserId: Id<"users">;
commentId: Id<"comments">;
verdict: CommentScamVerdict;
confidence: CommentScamConfidence;
explanation: string;
evidence: string[];
model: string;
checkedAt: number;
dryRun?: boolean;
},
): Promise<ApplyCommentScamResult> {
const comment = await ctx.db.get(args.commentId);
if (!comment) {
throw new ConvexError("Comment not found");
}
const user = await ctx.db.get(comment.userId);
if (!user) {
throw new ConvexError("Comment author not found");
}
const dryRun = Boolean(args.dryRun);
const shouldBan = isCertainScam({
verdict: args.verdict,
confidence: args.confidence,
});
const explanation = args.explanation.trim().slice(0, 1200);
const evidence = args.evidence
.map((item) => item.trim())
.filter(Boolean)
.slice(0, 5);
if (!dryRun) {
await ctx.db.patch(comment._id, {
scamScanVerdict: args.verdict,
scamScanConfidence: args.confidence,
scamScanExplanation: explanation,
scamScanEvidence: evidence,
scamScanModel: args.model,
scamScanCheckedAt: args.checkedAt,
});
await ctx.db.insert("auditLogs", {
actorUserId: args.actorUserId,
action: "comment.scam_scan",
targetType: "comment",
targetId: comment._id,
metadata: {
skillId: comment.skillId,
commentAuthorId: comment.userId,
verdict: args.verdict,
confidence: args.confidence,
shouldBan,
model: args.model,
},
createdAt: args.checkedAt,
});
}
if (!shouldBan) {
return {
ok: true,
shouldBan,
banned: false,
alreadyBanned: false,
protectedRole: false,
wouldBan: false,
};
}
if (user.role === "admin" || user.role === "moderator") {
return {
ok: true,
shouldBan,
banned: false,
alreadyBanned: false,
protectedRole: true,
wouldBan: false,
};
}
if (user.deletedAt || user.deactivatedAt) {
return {
ok: true,
shouldBan,
banned: false,
alreadyBanned: true,
protectedRole: false,
wouldBan: false,
};
}
if (dryRun) {
return {
ok: true,
shouldBan,
banned: false,
alreadyBanned: false,
protectedRole: false,
wouldBan: true,
};
}
const reason = buildCommentScamBanReason({
commentId: String(comment._id),
skillId: String(comment.skillId),
explanation,
evidence,
});
const banResult = await ctx.runMutation(internal.users.banUserInternal, {
actorUserId: args.actorUserId,
targetUserId: comment.userId,
reason,
});
if (!banResult.alreadyBanned) {
await ctx.db.patch(comment._id, {
scamBanTriggeredAt: args.checkedAt,
});
}
return {
ok: true,
shouldBan,
banned: !banResult.alreadyBanned,
alreadyBanned: Boolean(banResult.alreadyBanned),
protectedRole: false,
wouldBan: false,
};
}
export const applyCommentScamResultInternal = internalMutation({
args: {
actorUserId: v.id("users"),
commentId: v.id("comments"),
verdict: v.union(v.literal("not_scam"), v.literal("likely_scam"), v.literal("certain_scam")),
confidence: v.union(v.literal("low"), v.literal("medium"), v.literal("high")),
explanation: v.string(),
evidence: v.array(v.string()),
model: v.string(),
checkedAt: v.number(),
dryRun: v.optional(v.boolean()),
},
handler: applyCommentScamResultInternalHandler,
});
export async function backfillCommentScamModerationInternalHandler(
ctx: ActionCtx,
args: CommentScamBackfillActionArgs,
): Promise<CommentScamBackfillActionResult> {
if (!process.env.OPENAI_API_KEY) {
throw new ConvexError("OPENAI_API_KEY not configured");
}
const dryRun = Boolean(args.dryRun);
const rescan = Boolean(args.rescan);
const includeSoftDeleted = Boolean(args.includeSoftDeleted);
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES);
let cursor: string | null = args.cursor ?? null;
let isDone = false;
const stats: CommentScamBackfillStats = {
commentsScanned: 0,
commentsEvaluated: 0,
certainScams: 0,
banCandidates: 0,
usersBanned: 0,
usersAlreadyBanned: 0,
usersWouldBeBanned: 0,
protectedRoleSkips: 0,
skippedSoftDeleted: 0,
skippedAlreadyScanned: 0,
skippedEmptyBody: 0,
evalErrors: 0,
};
for (let i = 0; i < maxBatches; i++) {
const page = (await ctx.runQuery(
internal.commentModeration.getCommentScamBackfillPageInternal,
{
cursor: cursor ?? undefined,
batchSize,
},
)) as CommentBackfillPageResult;
cursor = page.cursor;
isDone = page.isDone;
for (const comment of page.items) {
stats.commentsScanned++;
if (!includeSoftDeleted && comment.softDeletedAt) {
stats.skippedSoftDeleted++;
continue;
}
if (!rescan && comment.scamScanCheckedAt) {
stats.skippedAlreadyScanned++;
continue;
}
const body = comment.body.trim();
if (!body) {
stats.skippedEmptyBody++;
continue;
}
const evalResult = (await ctx.runAction(internal.llmEval.evaluateCommentForScam, {
commentId: comment.commentId,
skillId: comment.skillId,
userId: comment.userId,
body,
})) as
| {
ok: true;
model: string;
verdict: CommentScamVerdict;
confidence: CommentScamConfidence;
explanation: string;
evidence: string[];
}
| { ok: false; error: string };
if (!evalResult.ok) {
stats.evalErrors++;
continue;
}
stats.commentsEvaluated++;
const shouldBan = isCertainScam(evalResult);
if (evalResult.verdict === "certain_scam") {
stats.certainScams++;
}
if (shouldBan) {
stats.banCandidates++;
}
const applyResult = (await ctx.runMutation(
internal.commentModeration.applyCommentScamResultInternal,
{
actorUserId: args.actorUserId,
commentId: comment.commentId,
verdict: evalResult.verdict,
confidence: evalResult.confidence,
explanation: evalResult.explanation,
evidence: evalResult.evidence,
model: evalResult.model,
checkedAt: Date.now(),
dryRun,
},
)) as ApplyCommentScamResult;
if (applyResult.banned) stats.usersBanned++;
if (applyResult.alreadyBanned) stats.usersAlreadyBanned++;
if (applyResult.wouldBan) stats.usersWouldBeBanned++;
if (applyResult.protectedRole) stats.protectedRoleSkips++;
}
if (isDone) break;
}
return {
ok: true,
stats,
isDone,
cursor,
};
}
export const backfillCommentScamModerationInternal = internalAction({
args: {
actorUserId: v.id("users"),
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
cursor: v.optional(v.string()),
rescan: v.optional(v.boolean()),
includeSoftDeleted: v.optional(v.boolean()),
},
handler: backfillCommentScamModerationInternalHandler,
});
export const backfillCommentScamModeration: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
cursor: v.optional(v.string()),
rescan: v.optional(v.boolean()),
includeSoftDeleted: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<CommentScamBackfillActionResult> => {
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin", "moderator"]);
return ctx.runAction(internal.commentModeration.backfillCommentScamModerationInternal, {
actorUserId: user._id,
...args,
}) as Promise<CommentScamBackfillActionResult>;
},
});
export const continueCommentScamModerationJobInternal = internalAction({
args: {
actorUserId: v.id("users"),
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
cursor: v.optional(v.string()),
rescan: v.optional(v.boolean()),
includeSoftDeleted: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const result = await backfillCommentScamModerationInternalHandler(ctx, {
actorUserId: args.actorUserId,
dryRun: args.dryRun,
batchSize: args.batchSize,
cursor: args.cursor,
maxBatches: 1,
rescan: args.rescan,
includeSoftDeleted: args.includeSoftDeleted,
});
if (!result.isDone && result.cursor) {
await ctx.scheduler.runAfter(
2_000,
internal.commentModeration.continueCommentScamModerationJobInternal,
{
actorUserId: args.actorUserId,
dryRun: Boolean(args.dryRun),
batchSize: args.batchSize ?? DEFAULT_BATCH_SIZE,
cursor: result.cursor,
rescan: Boolean(args.rescan),
includeSoftDeleted: Boolean(args.includeSoftDeleted),
},
);
}
return result;
},
});
export const scheduleCommentScamModeration: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
rescan: v.optional(v.boolean()),
includeSoftDeleted: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<{ ok: true }> => {
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin", "moderator"]);
await ctx.scheduler.runAfter(
0,
internal.commentModeration.continueCommentScamModerationJobInternal,
{
actorUserId: user._id,
dryRun: Boolean(args.dryRun),
batchSize: clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE),
cursor: undefined,
rescan: Boolean(args.rescan),
includeSoftDeleted: Boolean(args.includeSoftDeleted),
},
);
return { ok: true as const };
},
});
function clampInt(value: number, min: number, max: number) {
return Math.min(Math.max(Math.trunc(value), min), max);
}
+151
View File
@@ -0,0 +1,151 @@
import type { Id } from "./_generated/dataModel";
import type { MutationCtx } from "./_generated/server";
import { assertModerator, requireUser } from "./lib/access";
import { requireGitHubAccountAge } from "./lib/githubAccount";
import {
AUTO_HIDE_REPORT_THRESHOLD,
MAX_ACTIVE_REPORTS_PER_USER,
MAX_REPORT_REASON_LENGTH,
} from "./lib/reporting";
import { insertStatEvent } from "./skillStatEvents";
export async function addHandler(ctx: MutationCtx, args: { skillId: Id<"skills">; body: string }) {
const { userId } = await requireUser(ctx);
await requireGitHubAccountAge(ctx, userId);
const body = args.body.trim();
if (!body) throw new Error("Comment body required");
const skill = await ctx.db.get(args.skillId);
if (!skill) throw new Error("Skill not found");
await ctx.db.insert("comments", {
skillId: args.skillId,
userId,
body,
createdAt: Date.now(),
softDeletedAt: undefined,
deletedBy: undefined,
});
await insertStatEvent(ctx, { skillId: skill._id, kind: "comment" });
}
export async function removeHandler(ctx: MutationCtx, args: { commentId: Id<"comments"> }) {
const { user } = await requireUser(ctx);
const comment = await ctx.db.get(args.commentId);
if (!comment) throw new Error("Comment not found");
if (comment.softDeletedAt) return;
const isOwner = comment.userId === user._id;
if (!isOwner) {
assertModerator(user);
}
await ctx.db.patch(comment._id, {
softDeletedAt: Date.now(),
deletedBy: user._id,
});
await insertStatEvent(ctx, { skillId: comment.skillId, kind: "uncomment" });
await ctx.db.insert("auditLogs", {
actorUserId: user._id,
action: "comment.delete",
targetType: "comment",
targetId: comment._id,
metadata: { skillId: comment.skillId },
createdAt: Date.now(),
});
}
async function countActiveReportsForUser(ctx: MutationCtx, userId: Id<"users">) {
const reports = await ctx.db
.query("commentReports")
.withIndex("by_user", (q) => q.eq("userId", userId))
.collect();
let count = 0;
for (const report of reports) {
const comment = await ctx.db.get(report.commentId);
if (!comment || comment.softDeletedAt) continue;
const skill = await ctx.db.get(comment.skillId);
if (!skill || skill.softDeletedAt || skill.moderationStatus === "removed") continue;
const owner = await ctx.db.get(comment.userId);
if (!owner || owner.deletedAt || owner.deactivatedAt) continue;
count += 1;
if (count >= MAX_ACTIVE_REPORTS_PER_USER) break;
}
return count;
}
export async function reportHandler(
ctx: MutationCtx,
args: { commentId: Id<"comments">; reason: string },
) {
const { userId } = await requireUser(ctx);
const comment = await ctx.db.get(args.commentId);
if (!comment || comment.softDeletedAt) {
throw new Error("Comment not found");
}
const skill = await ctx.db.get(comment.skillId);
if (!skill || skill.softDeletedAt || skill.moderationStatus === "removed") {
throw new Error("Comment not found");
}
const reason = args.reason.trim();
if (!reason) {
throw new Error("Report reason required.");
}
const existing = await ctx.db
.query("commentReports")
.withIndex("by_comment_user", (q) => q.eq("commentId", args.commentId).eq("userId", userId))
.unique();
if (existing) return { ok: true as const, reported: false, alreadyReported: true };
const activeReports = await countActiveReportsForUser(ctx, userId);
if (activeReports >= MAX_ACTIVE_REPORTS_PER_USER) {
throw new Error("Report limit reached. Please wait for moderation before reporting more.");
}
const now = Date.now();
await ctx.db.insert("commentReports", {
commentId: args.commentId,
skillId: comment.skillId,
userId,
reason: reason.slice(0, MAX_REPORT_REASON_LENGTH),
createdAt: now,
});
const nextReportCount = (comment.reportCount ?? 0) + 1;
const shouldAutoHide = nextReportCount > AUTO_HIDE_REPORT_THRESHOLD && !comment.softDeletedAt;
const updates: {
reportCount: number;
lastReportedAt: number;
softDeletedAt?: number;
} = {
reportCount: nextReportCount,
lastReportedAt: now,
};
if (shouldAutoHide) {
updates.softDeletedAt = now;
}
await ctx.db.patch(comment._id, updates);
if (shouldAutoHide) {
await insertStatEvent(ctx, { skillId: comment.skillId, kind: "uncomment" });
await ctx.db.insert("auditLogs", {
actorUserId: userId,
action: "comment.auto_hide",
targetType: "comment",
targetId: comment._id,
metadata: { skillId: comment.skillId, reportCount: nextReportCount },
createdAt: now,
});
}
return { ok: true as const, reported: true, alreadyReported: false };
}
+127
View File
@@ -0,0 +1,127 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
import { listBySkillHandler } from "./comments";
function makeCtx(args: {
comments: Array<Record<string, unknown>>;
usersById: Record<string, Record<string, unknown> | null>;
}) {
const get = async (id: string) => args.usersById[id] ?? null;
const take = async () => args.comments;
const order = () => ({ take });
const withIndex = () => ({ order });
const query = () => ({ withIndex });
return { db: { get, query } } as never;
}
describe("comments.listBySkill", () => {
it("skips soft-deleted comments", async () => {
const ctx = makeCtx({
comments: [
{
_id: "comments:live",
skillId: "skills:1",
userId: "users:live",
body: "hello",
},
{
_id: "comments:deleted",
skillId: "skills:1",
userId: "users:live",
body: "bye",
softDeletedAt: 123,
},
],
usersById: {
"users:live": {
_id: "users:live",
_creationTime: 1,
handle: "live",
name: "live",
displayName: "Live",
image: null,
bio: null,
},
},
});
const result = await listBySkillHandler(ctx, {
skillId: "skills:1",
limit: 50,
} as never);
expect(result).toHaveLength(1);
expect(result[0]?.comment._id).toBe("comments:live");
});
it("skips comments whose author is deleted/deactivated/missing", async () => {
const ctx = makeCtx({
comments: [
{
_id: "comments:ok",
skillId: "skills:1",
userId: "users:ok",
body: "ok",
},
{
_id: "comments:deleted-user",
skillId: "skills:1",
userId: "users:deleted",
body: "hidden",
},
{
_id: "comments:deactivated-user",
skillId: "skills:1",
userId: "users:deactivated",
body: "hidden",
},
{
_id: "comments:missing-user",
skillId: "skills:1",
userId: "users:missing",
body: "hidden",
},
],
usersById: {
"users:ok": {
_id: "users:ok",
_creationTime: 1,
handle: "ok",
name: "ok",
displayName: "Ok",
image: null,
bio: null,
},
"users:deleted": {
_id: "users:deleted",
_creationTime: 1,
handle: "deleted",
name: "deleted",
displayName: "Deleted",
image: null,
bio: null,
deletedAt: 123,
},
"users:deactivated": {
_id: "users:deactivated",
_creationTime: 1,
handle: "deactivated",
name: "deactivated",
displayName: "Deactivated",
image: null,
bio: null,
deactivatedAt: 456,
},
},
});
const result = await listBySkillHandler(ctx, {
skillId: "skills:1",
limit: 50,
} as never);
expect(result).toHaveLength(1);
expect(result[0]?.comment._id).toBe("comments:ok");
expect(result[0]?.user._id).toBe("users:ok");
});
});
+614
View File
@@ -0,0 +1,614 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from "vitest";
vi.mock("./lib/access", () => ({
assertModerator: vi.fn(),
requireUser: vi.fn(),
}));
vi.mock("./skillStatEvents", () => ({
insertStatEvent: vi.fn(),
}));
vi.mock("./lib/githubAccount", () => ({
requireGitHubAccountAge: vi.fn(),
}));
const { requireUser, assertModerator } = await import("./lib/access");
const { insertStatEvent } = await import("./skillStatEvents");
const { requireGitHubAccountAge } = await import("./lib/githubAccount");
const { addHandler, removeHandler, reportHandler } = await import("./comments.handlers");
describe("comments mutations", () => {
afterEach(() => {
vi.mocked(assertModerator).mockReset();
vi.mocked(requireUser).mockReset();
vi.mocked(insertStatEvent).mockReset();
vi.mocked(requireGitHubAccountAge).mockReset();
vi.restoreAllMocks();
});
it("add avoids direct skill patch and records stat event", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
vi.mocked(requireGitHubAccountAge).mockResolvedValue(undefined as never);
const get = vi.fn().mockResolvedValue({
_id: "skills:1",
});
const insert = vi.fn();
const patch = vi.fn();
const ctx = { db: { get, insert, patch } } as never;
await addHandler(ctx, { skillId: "skills:1", body: " hello " } as never);
expect(requireGitHubAccountAge).toHaveBeenCalledWith(ctx, "users:1");
expect(patch).not.toHaveBeenCalled();
expect(insertStatEvent).toHaveBeenCalledWith(ctx, {
skillId: "skills:1",
kind: "comment",
});
});
it("add blocks new comments when github account age gate fails", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:new",
user: { _id: "users:new", role: "user" },
} as never);
vi.mocked(requireGitHubAccountAge).mockRejectedValue(
new Error(
"GitHub account must be at least 14 days old to upload skills. Try again in 3 days.",
),
);
const get = vi.fn();
const insert = vi.fn();
const patch = vi.fn();
const ctx = { db: { get, insert, patch } } as never;
await expect(addHandler(ctx, { skillId: "skills:1", body: "hello" } as never)).rejects.toThrow(
/at least 14 days old/i,
);
expect(get).not.toHaveBeenCalled();
expect(insert).not.toHaveBeenCalled();
expect(patch).not.toHaveBeenCalled();
expect(insertStatEvent).not.toHaveBeenCalled();
});
it("remove keeps comment soft-delete patch free of updatedAt", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:2",
user: { _id: "users:2", role: "moderator" },
} as never);
const comment = {
_id: "comments:1",
skillId: "skills:1",
userId: "users:2",
softDeletedAt: undefined,
};
const get = vi.fn(async (id: string) => {
if (id === "comments:1") return comment;
if (id === "skills:1") {
return { _id: "skills:1", softDeletedAt: undefined, moderationStatus: "active" };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const ctx = { db: { get, insert, patch } } as never;
await removeHandler(ctx, { commentId: "comments:1" } as never);
expect(patch).toHaveBeenCalledTimes(1);
const deletePatch = vi.mocked(patch).mock.calls[0]?.[1] as Record<string, unknown>;
expect(deletePatch.updatedAt).toBeUndefined();
expect(insertStatEvent).toHaveBeenCalledWith(ctx, {
skillId: "skills:1",
kind: "uncomment",
});
});
it("remove rejects non-owner without moderator permission", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:3",
user: { _id: "users:3", role: "user" },
} as never);
vi.mocked(assertModerator).mockImplementation(() => {
throw new Error("Moderator role required");
});
const comment = {
_id: "comments:2",
skillId: "skills:2",
userId: "users:9",
softDeletedAt: undefined,
};
const get = vi.fn().mockResolvedValue(comment);
const insert = vi.fn();
const patch = vi.fn();
const ctx = { db: { get, insert, patch } } as never;
await expect(removeHandler(ctx, { commentId: "comments:2" } as never)).rejects.toThrow(
"Moderator role required",
);
expect(patch).not.toHaveBeenCalled();
expect(insertStatEvent).not.toHaveBeenCalled();
});
it("remove no-ops for soft-deleted comment", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:4",
user: { _id: "users:4", role: "moderator" },
} as never);
const comment = {
_id: "comments:3",
skillId: "skills:3",
userId: "users:4",
softDeletedAt: 123,
};
const get = vi.fn().mockResolvedValue(comment);
const insert = vi.fn();
const patch = vi.fn();
const ctx = { db: { get, insert, patch } } as never;
await removeHandler(ctx, { commentId: "comments:3" } as never);
expect(patch).not.toHaveBeenCalled();
expect(insert).not.toHaveBeenCalled();
expect(insertStatEvent).not.toHaveBeenCalled();
});
it("report increments count and stores reason", async () => {
vi.spyOn(Date, "now").mockReturnValue(1_700_000_000_000);
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
const comment = {
_id: "comments:1",
skillId: "skills:1",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 1,
};
const get = vi.fn(async (id: string) => {
if (id === "comments:1") return comment;
if (id === "skills:1") {
return { _id: "skills:1", softDeletedAt: undefined, moderationStatus: "active" };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn((table: string) => {
if (table === "commentReports") {
return {
withIndex: (index: string) => {
if (index === "by_comment_user") {
return { unique: vi.fn().mockResolvedValue(null) };
}
if (index === "by_user") {
return { collect: vi.fn().mockResolvedValue([]) };
}
throw new Error(`Unexpected index ${index}`);
},
};
}
throw new Error(`Unexpected table ${table}`);
});
const ctx = { db: { get, insert, patch, query } } as never;
const result = await reportHandler(ctx, {
commentId: "comments:1",
reason: " spam ",
} as never);
expect(result).toEqual({ ok: true, reported: true, alreadyReported: false });
expect(insert).toHaveBeenCalledWith("commentReports", {
commentId: "comments:1",
skillId: "skills:1",
userId: "users:1",
reason: "spam",
createdAt: 1_700_000_000_000,
});
expect(patch).toHaveBeenCalledWith("comments:1", {
reportCount: 2,
lastReportedAt: 1_700_000_000_000,
});
expect(insertStatEvent).not.toHaveBeenCalled();
});
it("report returns alreadyReported for duplicate reporter/comment pair", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
const comment = {
_id: "comments:dup",
skillId: "skills:1",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 0,
};
const get = vi.fn(async (id: string) => {
if (id === "comments:dup") return comment;
if (id === "skills:1") {
return { _id: "skills:1", softDeletedAt: undefined, moderationStatus: "active" };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn((table: string) => {
if (table !== "commentReports") throw new Error(`Unexpected table ${table}`);
return {
withIndex: (index: string) => {
if (index === "by_comment_user") {
return { unique: vi.fn().mockResolvedValue({ _id: "commentReports:existing" }) };
}
throw new Error(`Unexpected index ${index}`);
},
};
});
const ctx = { db: { get, insert, patch, query } } as never;
const result = await reportHandler(ctx, { commentId: "comments:dup", reason: "spam" } as never);
expect(result).toEqual({ ok: true, reported: false, alreadyReported: true });
expect(insert).not.toHaveBeenCalled();
expect(patch).not.toHaveBeenCalled();
});
it("report rejects empty reason", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
const comment = {
_id: "comments:empty",
skillId: "skills:1",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 0,
};
const get = vi.fn(async (id: string) => {
if (id === "comments:empty") return comment;
if (id === "skills:1") {
return { _id: "skills:1", softDeletedAt: undefined, moderationStatus: "active" };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn();
const ctx = { db: { get, insert, patch, query } } as never;
await expect(
reportHandler(ctx, { commentId: "comments:empty", reason: " " } as never),
).rejects.toThrow("Report reason required.");
expect(query).not.toHaveBeenCalled();
expect(insert).not.toHaveBeenCalled();
expect(patch).not.toHaveBeenCalled();
});
it("report rejects comment when parent skill is hidden/removed", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
const comment = {
_id: "comments:hidden-parent",
skillId: "skills:hidden",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 0,
};
const get = vi.fn(async (id: string) => {
if (id === "comments:hidden-parent") return comment;
if (id === "skills:hidden") {
return { _id: "skills:hidden", softDeletedAt: 123, moderationStatus: "removed" };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn();
const ctx = { db: { get, insert, patch, query } } as never;
await expect(
reportHandler(ctx, { commentId: "comments:hidden-parent", reason: "abuse" } as never),
).rejects.toThrow("Comment not found");
expect(query).not.toHaveBeenCalled();
expect(insert).not.toHaveBeenCalled();
expect(patch).not.toHaveBeenCalled();
});
it("report truncates long reason to 500 chars", async () => {
vi.spyOn(Date, "now").mockReturnValue(1_700_000_000_050);
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
const comment = {
_id: "comments:long",
skillId: "skills:1",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 0,
};
const get = vi.fn(async (id: string) => {
if (id === "comments:long") return comment;
if (id === "skills:1") {
return { _id: "skills:1", softDeletedAt: undefined, moderationStatus: "active" };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn((table: string) => {
if (table !== "commentReports") throw new Error(`Unexpected table ${table}`);
return {
withIndex: (index: string) => {
if (index === "by_comment_user") return { unique: vi.fn().mockResolvedValue(null) };
if (index === "by_user") return { collect: vi.fn().mockResolvedValue([]) };
throw new Error(`Unexpected index ${index}`);
},
};
});
const ctx = { db: { get, insert, patch, query } } as never;
await reportHandler(ctx, { commentId: "comments:long", reason: "x".repeat(700) } as never);
const reportInsert = vi.mocked(insert).mock.calls.find((call) => call[0] === "commentReports");
expect(reportInsert?.[1]).toMatchObject({
commentId: "comments:long",
reason: "x".repeat(500),
});
});
it("report active-count filter ignores stale/non-active report targets", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
const comment = {
_id: "comments:target2",
skillId: "skills:1",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 0,
};
const reports = [
{
_id: "commentReports:1",
commentId: "comments:deleted",
userId: "users:1",
skillId: "skills:1",
},
{
_id: "commentReports:2",
commentId: "comments:removed-skill",
userId: "users:1",
skillId: "skills:removed",
},
{
_id: "commentReports:3",
commentId: "comments:deleted-owner",
userId: "users:1",
skillId: "skills:active",
},
];
const get = vi.fn(async (id: string) => {
if (id === "comments:target2") return comment;
if (id === "skills:1") {
return { _id: "skills:1", softDeletedAt: undefined, moderationStatus: "active" };
}
if (id === "comments:deleted") {
return {
_id: "comments:deleted",
softDeletedAt: 123,
skillId: "skills:1",
userId: "users:2",
};
}
if (id === "comments:removed-skill") {
return {
_id: "comments:removed-skill",
softDeletedAt: undefined,
skillId: "skills:removed",
userId: "users:2",
};
}
if (id === "skills:removed") {
return { _id: "skills:removed", softDeletedAt: undefined, moderationStatus: "removed" };
}
if (id === "comments:deleted-owner") {
return {
_id: "comments:deleted-owner",
softDeletedAt: undefined,
skillId: "skills:active",
userId: "users:deleted-owner",
};
}
if (id === "skills:active") {
return { _id: "skills:active", softDeletedAt: undefined, moderationStatus: "active" };
}
if (id === "users:deleted-owner") {
return { _id: "users:deleted-owner", deletedAt: 1, deactivatedAt: undefined };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn((table: string) => {
if (table !== "commentReports") throw new Error(`Unexpected table ${table}`);
return {
withIndex: (index: string) => {
if (index === "by_comment_user") return { unique: vi.fn().mockResolvedValue(null) };
if (index === "by_user") return { collect: vi.fn().mockResolvedValue(reports) };
throw new Error(`Unexpected index ${index}`);
},
};
});
const ctx = { db: { get, insert, patch, query } } as never;
const result = await reportHandler(ctx, {
commentId: "comments:target2",
reason: "still allowed",
} as never);
expect(result).toEqual({ ok: true, reported: true, alreadyReported: false });
expect(insert).toHaveBeenCalledWith(
"commentReports",
expect.objectContaining({ commentId: "comments:target2", userId: "users:1" }),
);
});
it("report rejects when active report limit is reached", async () => {
vi.mocked(requireUser).mockResolvedValue({
userId: "users:1",
user: { _id: "users:1", role: "user" },
} as never);
const comment = {
_id: "comments:target",
skillId: "skills:1",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 0,
};
const reportedComment = {
_id: "comments:reported",
skillId: "skills:active",
userId: "users:owner",
softDeletedAt: undefined,
};
const reports = Array.from({ length: 20 }, (_, i) => ({
_id: `commentReports:${i + 1}`,
commentId: `comments:reported-${i + 1}`,
userId: "users:1",
skillId: "skills:active",
createdAt: i + 1,
}));
const get = vi.fn(async (id: string) => {
if (id === "comments:target") return comment;
if (id === "skills:1") {
return { _id: "skills:1", softDeletedAt: undefined, moderationStatus: "active" };
}
if (String(id).startsWith("comments:reported-")) return reportedComment;
if (id === "skills:active") {
return { _id: "skills:active", softDeletedAt: undefined, moderationStatus: "active" };
}
if (id === "users:owner") {
return { _id: "users:owner", deletedAt: undefined, deactivatedAt: undefined };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn((table: string) => {
if (table === "commentReports") {
return {
withIndex: (index: string) => {
if (index === "by_comment_user") {
return { unique: vi.fn().mockResolvedValue(null) };
}
if (index === "by_user") {
return { collect: vi.fn().mockResolvedValue(reports) };
}
throw new Error(`Unexpected index ${index}`);
},
};
}
throw new Error(`Unexpected table ${table}`);
});
const ctx = { db: { get, insert, patch, query } } as never;
await expect(
reportHandler(ctx, { commentId: "comments:target", reason: "abuse" } as never),
).rejects.toThrow("Report limit reached. Please wait for moderation before reporting more.");
expect(insert).not.toHaveBeenCalled();
expect(patch).not.toHaveBeenCalled();
});
it("report auto-hides comment after fourth unique report", async () => {
vi.spyOn(Date, "now").mockReturnValue(1_700_000_000_100);
vi.mocked(requireUser).mockResolvedValue({
userId: "users:3",
user: { _id: "users:3", role: "user" },
} as never);
const comment = {
_id: "comments:4",
skillId: "skills:9",
userId: "users:2",
softDeletedAt: undefined,
reportCount: 3,
};
const get = vi.fn(async (id: string) => {
if (id === "comments:4") return comment;
if (id === "skills:9") {
return { _id: "skills:9", softDeletedAt: undefined, moderationStatus: "active" };
}
return null;
});
const insert = vi.fn();
const patch = vi.fn();
const query = vi.fn((table: string) => {
if (table === "commentReports") {
return {
withIndex: (index: string) => {
if (index === "by_comment_user") {
return { unique: vi.fn().mockResolvedValue(null) };
}
if (index === "by_user") {
return { collect: vi.fn().mockResolvedValue([]) };
}
throw new Error(`Unexpected index ${index}`);
},
};
}
throw new Error(`Unexpected table ${table}`);
});
const ctx = { db: { get, insert, patch, query } } as never;
const result = await reportHandler(ctx, {
commentId: "comments:4",
reason: " hate ",
} as never);
expect(result).toEqual({ ok: true, reported: true, alreadyReported: false });
expect(patch).toHaveBeenCalledWith("comments:4", {
reportCount: 4,
lastReportedAt: 1_700_000_000_100,
softDeletedAt: 1_700_000_000_100,
});
expect(insertStatEvent).toHaveBeenCalledWith(ctx, {
skillId: "skills:9",
kind: "uncomment",
});
expect(insert).toHaveBeenCalledWith("auditLogs", {
actorUserId: "users:3",
action: "comment.auto_hide",
targetType: "comment",
targetId: "comments:4",
metadata: { skillId: "skills:9", reportCount: 4 },
createdAt: 1_700_000_000_100,
});
});
});
+41 -79
View File
@@ -1,87 +1,49 @@
import { v } from 'convex/values'
import type { Doc } from './_generated/dataModel'
import { mutation, query } from './_generated/server'
import { assertRole, requireUser } from './lib/access'
import { v } from "convex/values";
import type { Doc } from "./_generated/dataModel";
import { addHandler, removeHandler, reportHandler } from "./comments.handlers";
import { mutation, query } from "./functions";
import { type PublicUser, toPublicUser } from "./lib/public";
export const listBySkill = query({
args: { skillId: v.id('skills'), limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = args.limit ?? 50
const comments = await ctx.db
.query('comments')
.withIndex('by_skill', (q) => q.eq('skillId', args.skillId))
.order('desc')
.take(limit)
args: { skillId: v.id("skills"), limit: v.optional(v.number()) },
handler: listBySkillHandler,
});
const results: Array<{ comment: Doc<'comments'>; user: Doc<'users'> | null }> = []
for (const comment of comments) {
if (comment.softDeletedAt) continue
const user = await ctx.db.get(comment.userId)
results.push({ comment, user })
}
return results
},
})
export async function listBySkillHandler(
ctx: import("./_generated/server").QueryCtx,
args: { skillId: import("./_generated/dataModel").Id<"skills">; limit?: number },
) {
const limit = args.limit ?? 50;
const comments = await ctx.db
.query("comments")
.withIndex("by_skill", (q) => q.eq("skillId", args.skillId))
.order("desc")
.take(limit);
const rows = await Promise.all(
comments.map(
async (comment): Promise<{ comment: Doc<"comments">; user: PublicUser } | null> => {
if (comment.softDeletedAt) return null;
const user = toPublicUser(await ctx.db.get(comment.userId));
if (!user) return null;
return { comment, user };
},
),
);
return rows.filter((row): row is { comment: Doc<"comments">; user: PublicUser } => row !== null);
}
export const add = mutation({
args: { skillId: v.id('skills'), body: v.string() },
handler: async (ctx, args) => {
const { userId } = await requireUser(ctx)
const body = args.body.trim()
if (!body) throw new Error('Comment body required')
const skill = await ctx.db.get(args.skillId)
if (!skill) throw new Error('Skill not found')
await ctx.db.insert('comments', {
skillId: args.skillId,
userId,
body,
createdAt: Date.now(),
softDeletedAt: undefined,
deletedBy: undefined,
})
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, comments: skill.stats.comments + 1 },
updatedAt: Date.now(),
})
},
})
args: { skillId: v.id("skills"), body: v.string() },
handler: addHandler,
});
export const remove = mutation({
args: { commentId: v.id('comments') },
handler: async (ctx, args) => {
const { user } = await requireUser(ctx)
const comment = await ctx.db.get(args.commentId)
if (!comment) throw new Error('Comment not found')
if (comment.softDeletedAt) return
args: { commentId: v.id("comments") },
handler: removeHandler,
});
const isOwner = comment.userId === user._id
if (!isOwner) {
assertRole(user, ['admin', 'moderator'])
}
await ctx.db.patch(comment._id, {
softDeletedAt: Date.now(),
deletedBy: user._id,
})
const skill = await ctx.db.get(comment.skillId)
if (skill) {
await ctx.db.patch(skill._id, {
stats: { ...skill.stats, comments: Math.max(0, skill.stats.comments - 1) },
updatedAt: Date.now(),
})
}
await ctx.db.insert('auditLogs', {
actorUserId: user._id,
action: 'comment.delete',
targetType: 'comment',
targetId: comment._id,
metadata: { skillId: comment.skillId },
createdAt: Date.now(),
})
},
})
export const report = mutation({
args: { commentId: v.id("comments"), reason: v.string() },
handler: reportHandler,
});
+63 -12
View File
@@ -1,27 +1,78 @@
import { cronJobs } from 'convex/server'
import { internal } from './_generated/api'
import { cronJobs } from "convex/server";
import { internal } from "./_generated/api";
const crons = cronJobs()
const crons = cronJobs();
crons.interval(
'github-backup-sync',
"github-backup-sync",
{ minutes: 30 },
internal.githubBackupsNode.syncGitHubBackupsInternal,
{ batchSize: 50, maxBatches: 5 },
)
);
crons.interval(
'trending-leaderboard',
"trending-leaderboard",
{ minutes: 60 },
internal.leaderboards.rebuildTrendingLeaderboardInternal,
internal.leaderboards.rebuildTrendingLeaderboardAction,
{ limit: 200 },
)
);
crons.interval(
'skill-stats-backfill',
{ minutes: 10 },
"skill-stats-backfill",
{ hours: 6 },
internal.statsMaintenance.runSkillStatBackfillInternal,
{ batchSize: 200, maxBatches: 5 },
)
);
export default crons
// Runs frequently to keep dailyStats/trending accurate,
// but does NOT patch skill documents (only writes to skillDailyStats).
crons.interval(
"skill-stat-events",
{ minutes: 15 },
internal.skillStatEvents.processSkillStatEventsAction,
{},
);
// Syncs accumulated stat deltas to skill documents every 6 hours.
// Runs infrequently to avoid thundering-herd reactive query invalidation.
// Uses processedAt field to track progress (independent of the action cursor).
crons.interval(
"skill-doc-stat-sync",
{ hours: 6 },
internal.skillStatEvents.processSkillStatEventsInternal,
{ batchSize: 500 },
);
crons.interval(
"global-stats-update",
{ hours: 24 },
internal.statsMaintenance.updateGlobalStatsAction,
{},
);
crons.interval("vt-pending-scans", { minutes: 5 }, internal.vt.pollPendingScans, {
batchSize: 100,
});
crons.interval("vt-cache-backfill", { minutes: 30 }, internal.vt.backfillActiveSkillsVTCache, {
batchSize: 100,
});
crons.interval(
"package-scan-backfill",
{ minutes: 30 },
internal.packages.backfillPackageReleaseScansInternal,
{ batchSize: 100 },
);
// Daily re-scan of all active skills at 3am UTC
crons.daily("vt-daily-rescan", { hourUTC: 3, minuteUTC: 0 }, internal.vt.rescanActiveSkills, {});
crons.interval(
"download-dedupe-prune",
{ hours: 24 },
internal.downloads.pruneDownloadDedupesInternal,
{},
);
export default crons;
+219 -123
View File
@@ -1,33 +1,45 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import { internalAction, internalMutation } from './_generated/server'
import { EMBEDDING_DIMENSIONS } from './lib/embeddings'
import { parseClawdisMetadata, parseFrontmatter } from './lib/skills'
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { ActionCtx } from "./_generated/server";
import { internalAction, internalMutation } from "./functions";
import { EMBEDDING_DIMENSIONS } from "./lib/embeddings";
import { parseClawdisMetadata, parseFrontmatter } from "./lib/skills";
type SeedSkillSpec = {
slug: string
displayName: string
summary: string
version: string
metadata: Record<string, unknown>
rawSkillMd: string
}
slug: string;
displayName: string;
summary: string;
version: string;
metadata: Record<string, unknown>;
rawSkillMd: string;
};
type SeedActionArgs = {
reset?: boolean;
};
type SeedActionResult = {
ok: true;
results: Array<Record<string, unknown> & { slug: string }>;
};
type SeedMutationResult = Record<string, unknown>;
const SEED_SKILLS: SeedSkillSpec[] = [
{
slug: 'padel',
displayName: 'Padel',
summary: 'Check padel court availability and manage bookings via Playtomic.',
version: '0.1.0',
slug: "padel",
displayName: "Padel",
summary: "Check padel court availability and manage bookings via Playtomic.",
version: "0.1.0",
metadata: {
clawdbot: {
nix: {
plugin: 'github:joshp123/padel-cli',
systems: ['aarch64-darwin', 'x86_64-linux'],
plugin: "github:joshp123/padel-cli",
systems: ["aarch64-darwin", "x86_64-linux"],
},
config: {
requiredEnv: ['PADEL_AUTH_FILE'],
stateDirs: ['.config/padel'],
requiredEnv: ["PADEL_AUTH_FILE"],
stateDirs: [".config/padel"],
example:
'config = { env = { PADEL_AUTH_FILE = "/run/agenix/padel-auth"; }; stateDirs = [ ".config/padel" ]; };',
},
@@ -93,18 +105,18 @@ Only the authorized booker can confirm bookings. If the requester is not authori
`,
},
{
slug: 'gohome',
displayName: 'GoHome',
summary: 'Operate GoHome via gRPC discovery, metrics, and Grafana dashboards.',
version: '0.1.0',
slug: "gohome",
displayName: "GoHome",
summary: "Operate GoHome via gRPC discovery, metrics, and Grafana dashboards.",
version: "0.1.0",
metadata: {
clawdbot: {
nix: {
plugin: 'github:joshp123/gohome',
systems: ['x86_64-linux', 'aarch64-linux'],
plugin: "github:joshp123/gohome",
systems: ["x86_64-linux", "aarch64-linux"],
},
config: {
requiredEnv: ['GOHOME_GRPC_ADDR', 'GOHOME_HTTP_BASE'],
requiredEnv: ["GOHOME_GRPC_ADDR", "GOHOME_HTTP_BASE"],
example:
'config = { env = { GOHOME_GRPC_ADDR = "gohome:9000"; GOHOME_HTTP_BASE = "http://gohome:8080"; }; };',
},
@@ -166,19 +178,19 @@ Only call write RPCs after explicit user approval.
`,
},
{
slug: 'xuezh',
displayName: 'Xuezh',
summary: 'Teach Mandarin with the xuezh engine for review, speaking, and audits.',
version: '0.1.0',
slug: "xuezh",
displayName: "Xuezh",
summary: "Teach Mandarin with the xuezh engine for review, speaking, and audits.",
version: "0.1.0",
metadata: {
clawdbot: {
nix: {
plugin: 'github:joshp123/xuezh',
systems: ['aarch64-darwin', 'x86_64-linux'],
plugin: "github:joshp123/xuezh",
systems: ["aarch64-darwin", "x86_64-linux"],
},
config: {
requiredEnv: ['XUEZH_AZURE_SPEECH_KEY_FILE', 'XUEZH_AZURE_SPEECH_REGION'],
stateDirs: ['.config/xuezh'],
requiredEnv: ["XUEZH_AZURE_SPEECH_KEY_FILE", "XUEZH_AZURE_SPEECH_REGION"],
stateDirs: [".config/xuezh"],
example:
'config = { env = { XUEZH_AZURE_SPEECH_KEY_FILE = "/run/agenix/xuezh-azure-speech-key"; XUEZH_AZURE_SPEECH_REGION = "westeurope"; }; stateDirs = [ ".config/xuezh" ]; };',
},
@@ -227,63 +239,97 @@ xuezh audio process-voice --file ./utterance.wav
\`\`\`
`,
},
]
{
slug: "hanzi-helper",
displayName: "汉字助手",
summary: "汉字学习与分析工具,支持笔画查询、部首检索和组词生成。",
version: "0.1.0",
metadata: {
clawdbot: {
nix: {
plugin: "github:example/hanzi-helper",
systems: ["aarch64-darwin", "x86_64-linux"],
},
config: {
requiredEnv: ["HANZI_DB_PATH"],
stateDirs: [".config/hanzi"],
example:
'config = { env = { HANZI_DB_PATH = ".config/hanzi/db"; }; stateDirs = [ ".config/hanzi" ]; };',
},
cliHelp: `汉字助手 - Chinese character learning and analysis
Usage:
hanzi-helper [command]
Available Commands:
lookup 查询汉字信息(笔画、部首、释义)
radical 按部首检索汉字
stroke 按笔画数筛选汉字
words 生成汉字组词
practice 练习汉字书写
quiz 汉字听写测试
Flags:
-h, --help help for hanzi-helper
--json Output JSON
`,
},
},
rawSkillMd: `---
name: hanzi-helper
description: 汉字学习与分析工具,提供笔画查询、部首检索、组词生成和汉字听写练习功能。
---
# 汉字助手
## 功能介绍
汉字助手是一个强大的中文汉字学习工具,帮助用户深入了解每个汉字的结构和含义。
## CLI
\`\`\`bash
hanzi-helper lookup --char 学
hanzi-helper radical --name 木
hanzi-helper stroke --count 8
hanzi-helper words --char 大 --limit 20
\`\`\`
## 使用场景
- **汉字查询**:输入任意汉字,查看笔画数、部首、繁体形式和基本释义
- **部首检索**:按部首浏览相关汉字,了解汉字的分类规律
- **组词生成**:输入一个汉字,自动生成常用词语和成语
- **听写练习**:随机生成汉字听写测试,巩固学习效果
## 学习建议
建议每天学习五个新汉字,结合组词和例句加深记忆。坚持使用听写练习功能可以有效提高汉字识别能力。
`,
},
];
function injectMetadata(rawSkillMd: string, metadata: Record<string, unknown>) {
const frontmatterEnd = rawSkillMd.indexOf('\n---', 3)
if (frontmatterEnd === -1) return rawSkillMd
const frontmatterEnd = rawSkillMd.indexOf("\n---", 3);
if (frontmatterEnd === -1) return rawSkillMd;
return `${rawSkillMd.slice(0, frontmatterEnd)}\nmetadata: ${JSON.stringify(
metadata,
)}${rawSkillMd.slice(frontmatterEnd)}`
)}${rawSkillMd.slice(frontmatterEnd)}`;
}
export const seedNixSkills = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const results = []
async function seedNixSkillsHandler(
ctx: ActionCtx,
args: SeedActionArgs,
): Promise<SeedActionResult> {
const results: Array<Record<string, unknown> & { slug: string }> = [];
for (const spec of SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
for (const spec of SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata);
const frontmatter = parseFrontmatter(skillMd);
const clawdis = parseClawdisMetadata(frontmatter);
const storageId = await ctx.storage.store(new Blob([skillMd], { type: "text/markdown" }));
const result = await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})
results.push({ slug: spec.slug, ...result })
}
return { ok: true, results }
},
})
export const seedPadelSkill = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
const spec = SEED_SKILLS.find((entry) => entry.slug === 'padel')
if (!spec) throw new Error('padel seed spec missing')
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata)
const frontmatter = parseFrontmatter(skillMd)
const clawdis = parseClawdisMetadata(frontmatter)
const storageId = await ctx.storage.store(new Blob([skillMd], { type: 'text/markdown' }))
return ctx.runMutation(internal.devSeed.seedSkillMutation, {
const result: SeedMutationResult = await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
@@ -294,14 +340,58 @@ export const seedPadelSkill = internalAction({
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})
});
results.push({ slug: spec.slug, ...result });
}
return { ok: true, results };
}
export const seedNixSkills: ReturnType<typeof internalAction> = internalAction({
args: {
reset: v.optional(v.boolean()),
},
})
handler: seedNixSkillsHandler,
});
async function seedPadelSkillHandler(
ctx: ActionCtx,
args: SeedActionArgs,
): Promise<SeedMutationResult> {
const spec = SEED_SKILLS.find((entry) => entry.slug === "padel");
if (!spec) throw new Error("padel seed spec missing");
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata);
const frontmatter = parseFrontmatter(skillMd);
const clawdis = parseClawdisMetadata(frontmatter);
const storageId = await ctx.storage.store(new Blob([skillMd], { type: "text/markdown" }));
return (await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})) as SeedMutationResult;
}
export const seedPadelSkill: ReturnType<typeof internalAction> = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: seedPadelSkillHandler,
});
export const seedSkillMutation = internalMutation({
args: {
reset: v.optional(v.boolean()),
storageId: v.id('_storage'),
storageId: v.id("_storage"),
metadata: v.any(),
frontmatter: v.any(),
clawdis: v.any(),
@@ -313,49 +403,49 @@ export const seedSkillMutation = internalMutation({
},
handler: async (ctx, args) => {
const existing = await ctx.db
.query('skills')
.withIndex('by_slug', (q) => q.eq('slug', args.slug))
.unique()
.query("skills")
.withIndex("by_slug", (q) => q.eq("slug", args.slug))
.unique();
if (existing && !args.reset) {
return { ok: true, skipped: true, skillId: existing._id }
return { ok: true, skipped: true, skillId: existing._id };
}
if (existing && args.reset) {
const versions = await ctx.db
.query('skillVersions')
.withIndex('by_skill', (q) => q.eq('skillId', existing._id))
.collect()
.query("skillVersions")
.withIndex("by_skill", (q) => q.eq("skillId", existing._id))
.collect();
for (const version of versions) {
await ctx.db.delete(version._id)
await ctx.db.delete(version._id);
}
const embeddings = await ctx.db
.query('skillEmbeddings')
.withIndex('by_skill', (q) => q.eq('skillId', existing._id))
.collect()
.query("skillEmbeddings")
.withIndex("by_skill", (q) => q.eq("skillId", existing._id))
.collect();
for (const embedding of embeddings) {
await ctx.db.delete(embedding._id)
await ctx.db.delete(embedding._id);
}
await ctx.db.delete(existing._id)
await ctx.db.delete(existing._id);
}
const now = Date.now()
const now = Date.now();
const existingUsers = await ctx.db
.query('users')
.withIndex('handle', (q) => q.eq('handle', 'local'))
.collect()
.query("users")
.withIndex("handle", (q) => q.eq("handle", "local"))
.collect();
const userId =
existingUsers[0]?._id ??
(await ctx.db.insert('users', {
handle: 'local',
displayName: 'Local Dev',
role: 'admin',
(await ctx.db.insert("users", {
handle: "local",
displayName: "Local Dev",
role: "admin",
createdAt: now,
updatedAt: now,
}))
}));
const skillId = await ctx.db.insert('skills', {
const skillId = await ctx.db.insert("skills", {
slug: args.slug,
displayName: args.displayName,
summary: args.summary,
@@ -378,19 +468,24 @@ export const seedSkillMutation = internalMutation({
},
createdAt: now,
updatedAt: now,
})
});
await ctx.db.patch(userId, {
publishedSkills: 1,
totalStars: 0,
totalDownloads: 0,
});
const versionId = await ctx.db.insert('skillVersions', {
const versionId = await ctx.db.insert("skillVersions", {
skillId,
version: args.version,
changelog: 'Seeded local version for screenshots.',
changelog: "Seeded local version for screenshots.",
files: [
{
path: 'SKILL.md',
path: "SKILL.md",
size: args.skillMd.length,
storageId: args.storageId,
sha256: 'seeded',
contentType: 'text/markdown',
sha256: "seeded",
contentType: "text/markdown",
},
],
parsed: {
@@ -401,18 +496,19 @@ export const seedSkillMutation = internalMutation({
createdBy: userId,
createdAt: now,
softDeletedAt: undefined,
})
});
const embeddingId = await ctx.db.insert('skillEmbeddings', {
const embeddingId = await ctx.db.insert("skillEmbeddings", {
skillId,
versionId,
ownerId: userId,
embedding: Array.from({ length: EMBEDDING_DIMENSIONS }, () => 0),
isLatest: true,
isApproved: true,
visibility: 'latest-approved',
visibility: "latest-approved",
updatedAt: now,
})
});
await ctx.db.insert("embeddingSkillMap", { embeddingId, skillId });
await ctx.db.patch(skillId, {
latestVersionId: versionId,
@@ -430,8 +526,8 @@ export const seedSkillMutation = internalMutation({
comments: 0,
},
updatedAt: now,
})
});
return { ok: true, skillId, versionId, embeddingId }
return { ok: true, skillId, versionId, embeddingId };
},
})
});
+557
View File
@@ -0,0 +1,557 @@
/**
* Extra seed skills for pagination testing.
*
* This file contains 50 placeholder skills to test pagination behavior.
* Run with: bunx convex run internal.devSeedExtra.seedExtraSkillsInternal
* Or with reset: bunx convex run internal.devSeedExtra.seedExtraSkillsInternal '{"reset": true}'
*/
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { internalAction, internalMutation } from "./functions";
import { parseClawdisMetadata, parseFrontmatter } from "./lib/skills";
type SeedSkillSpec = {
slug: string;
displayName: string;
summary: string;
version: string;
metadata: Record<string, unknown>;
rawSkillMd: string;
};
function makeSkill(
slug: string,
displayName: string,
summary: string,
envVars: string[] = [],
commands: string[] = ["help", "status", "run"],
): SeedSkillSpec {
const cliHelp = `${slug} - ${summary}
Usage:
${slug} [command]
Commands:
${commands.map((cmd) => ` ${cmd.padEnd(12)} Run ${cmd} operation`).join("\n")}
Flags:
-h, --help Show help
--json Output as JSON
`;
const rawSkillMd = `---
name: ${slug}
description: ${summary}
---
# ${displayName}
## CLI
\`\`\`bash
${commands.map((cmd) => `${slug} ${cmd}`).join("\n")}
\`\`\`
## Usage
Use this skill to ${summary.toLowerCase()}.
`;
return {
slug,
displayName,
summary,
version: "0.1.0",
metadata: {
clawdbot: {
nix: {
plugin: `github:example/${slug}`,
systems: ["aarch64-darwin", "x86_64-linux"],
},
config: {
requiredEnv: envVars,
},
cliHelp,
},
},
rawSkillMd,
};
}
// 50 placeholder skills for pagination testing
const EXTRA_SEED_SKILLS: SeedSkillSpec[] = [
// DevOps & Infrastructure (10)
makeSkill(
"kubectl-helper",
"Kubectl Helper",
"Simplified kubectl commands for common Kubernetes operations.",
["KUBECONFIG"],
["pods", "logs", "exec", "describe", "apply"],
),
makeSkill(
"terraform-runner",
"Terraform Runner",
"Execute Terraform plans and applies with safety checks.",
["TF_VAR_region", "AWS_PROFILE"],
["plan", "apply", "destroy", "output", "state"],
),
makeSkill(
"ansible-exec",
"Ansible Exec",
"Run Ansible playbooks and ad-hoc commands.",
["ANSIBLE_INVENTORY"],
["playbook", "adhoc", "inventory", "facts", "vault"],
),
makeSkill(
"docker-compose-mgr",
"Docker Compose Manager",
"Manage Docker Compose stacks and services.",
["DOCKER_HOST"],
["up", "down", "logs", "ps", "restart"],
),
makeSkill(
"k9s-wrapper",
"K9s Wrapper",
"Interactive Kubernetes cluster management via K9s.",
["KUBECONFIG"],
["launch", "contexts", "namespaces", "pods", "logs"],
),
makeSkill(
"helm-charts",
"Helm Charts",
"Manage Helm chart deployments and releases.",
["KUBECONFIG", "HELM_REPO"],
["install", "upgrade", "rollback", "list", "search"],
),
makeSkill(
"prometheus-alerts",
"Prometheus Alerts",
"Query Prometheus metrics and manage alerting rules.",
["PROMETHEUS_URL"],
["query", "alerts", "rules", "targets", "status"],
),
makeSkill(
"grafana-dash",
"Grafana Dashboards",
"Create and manage Grafana dashboards programmatically.",
["GRAFANA_URL", "GRAFANA_API_KEY"],
["list", "export", "import", "create", "delete"],
),
makeSkill(
"nginx-config",
"Nginx Config",
"Generate and validate Nginx configuration files.",
["NGINX_CONF_DIR"],
["generate", "validate", "reload", "test", "sites"],
),
makeSkill(
"jenkins-jobs",
"Jenkins Jobs",
"Manage Jenkins jobs and pipelines.",
["JENKINS_URL", "JENKINS_TOKEN"],
["list", "build", "status", "logs", "config"],
),
// Productivity (8)
makeSkill(
"todoist-sync",
"Todoist Sync",
"Sync and manage Todoist tasks from the command line.",
["TODOIST_API_TOKEN"],
["list", "add", "complete", "projects", "labels"],
),
makeSkill(
"notion-backup",
"Notion Backup",
"Export and backup Notion workspaces.",
["NOTION_TOKEN"],
["export", "backup", "restore", "pages", "databases"],
),
makeSkill(
"gcal-manager",
"Google Calendar Manager",
"Manage Google Calendar events and schedules.",
["GOOGLE_CREDENTIALS_FILE"],
["events", "create", "delete", "calendars", "reminders"],
),
makeSkill(
"time-tracker",
"Time Tracker",
"Track time spent on projects and tasks.",
["TIMETRACK_DB"],
["start", "stop", "status", "report", "projects"],
),
makeSkill(
"email-digest",
"Email Digest",
"Generate email digests and summaries.",
["IMAP_SERVER", "IMAP_USER"],
["fetch", "digest", "search", "folders", "unread"],
),
makeSkill(
"habit-tracker",
"Habit Tracker",
"Track daily habits and streaks.",
["HABITS_DB"],
["log", "streak", "stats", "habits", "remind"],
),
makeSkill(
"bookmark-sync",
"Bookmark Sync",
"Sync bookmarks across browsers and devices.",
["BOOKMARKS_DIR"],
["sync", "export", "import", "search", "tags"],
),
makeSkill(
"notes-export",
"Notes Export",
"Export notes to various formats.",
["NOTES_DIR"],
["export", "convert", "search", "list", "tags"],
),
// Media & Entertainment (6)
makeSkill(
"spotify-ctl",
"Spotify Control",
"Control Spotify playback from the terminal.",
["SPOTIFY_CLIENT_ID", "SPOTIFY_CLIENT_SECRET"],
["play", "pause", "next", "prev", "search"],
),
makeSkill(
"plex-manager",
"Plex Manager",
"Manage Plex media libraries and playback.",
["PLEX_URL", "PLEX_TOKEN"],
["libraries", "scan", "search", "play", "sessions"],
),
makeSkill(
"ytdl-wrapper",
"YouTube Downloader",
"Download videos from YouTube and other platforms.",
["YTDL_OUTPUT_DIR"],
["download", "info", "playlist", "audio", "formats"],
),
makeSkill(
"podcast-dl",
"Podcast Downloader",
"Download and manage podcast episodes.",
["PODCAST_DIR"],
["subscribe", "download", "list", "play", "search"],
),
makeSkill(
"audiobook-player",
"Audiobook Player",
"Manage and play audiobook collections.",
["AUDIOBOOK_DIR"],
["play", "pause", "bookmark", "list", "progress"],
),
makeSkill(
"music-lib",
"Music Library",
"Organize and query local music libraries.",
["MUSIC_DIR"],
["scan", "search", "play", "playlist", "stats"],
),
// Smart Home (8)
makeSkill(
"hass-control",
"Home Assistant Control",
"Control Home Assistant entities and automations.",
["HASS_URL", "HASS_TOKEN"],
["entities", "services", "automations", "scenes", "history"],
),
makeSkill(
"zigbee-mqtt",
"Zigbee2MQTT",
"Manage Zigbee devices via MQTT.",
["MQTT_BROKER", "ZIGBEE_TOPIC"],
["devices", "pair", "remove", "rename", "groups"],
),
makeSkill(
"tasmota-ctl",
"Tasmota Control",
"Control Tasmota-flashed devices.",
["TASMOTA_HOSTS"],
["status", "power", "config", "update", "backup"],
),
makeSkill(
"esphome-mgr",
"ESPHome Manager",
"Manage ESPHome device configurations.",
["ESPHOME_DIR"],
["compile", "upload", "logs", "dashboard", "config"],
),
makeSkill(
"mqtt-broker",
"MQTT Broker",
"Interact with MQTT brokers for IoT messaging.",
["MQTT_BROKER", "MQTT_USER"],
["pub", "sub", "topics", "clients", "stats"],
),
makeSkill(
"hue-lights",
"Philips Hue",
"Control Philips Hue lights and scenes.",
["HUE_BRIDGE_IP", "HUE_API_KEY"],
["lights", "scenes", "groups", "schedules", "sensors"],
),
makeSkill(
"smart-thermo",
"Smart Thermostat",
"Control smart thermostats and HVAC systems.",
["THERMOSTAT_API_KEY"],
["status", "set", "schedule", "history", "zones"],
),
makeSkill(
"cam-viewer",
"Camera Viewer",
"View and manage security camera feeds.",
["CAMERA_URLS"],
["list", "snapshot", "stream", "record", "events"],
),
// Finance (5)
makeSkill(
"budget-track",
"Budget Tracker",
"Track budgets and spending across categories.",
["BUDGET_DB"],
["summary", "add", "categories", "report", "goals"],
),
makeSkill(
"crypto-watch",
"Crypto Watcher",
"Monitor cryptocurrency prices and portfolios.",
["CRYPTO_API_KEY"],
["prices", "portfolio", "alerts", "history", "convert"],
),
makeSkill(
"stock-alerts",
"Stock Alerts",
"Set up stock price alerts and notifications.",
["STOCK_API_KEY"],
["quote", "watch", "alerts", "portfolio", "news"],
),
makeSkill(
"expense-cat",
"Expense Categorizer",
"Automatically categorize expenses.",
["EXPENSE_DB"],
["import", "categorize", "report", "rules", "export"],
),
makeSkill(
"invoice-gen",
"Invoice Generator",
"Generate and manage invoices.",
["INVOICE_DIR", "COMPANY_INFO"],
["create", "list", "send", "paid", "overdue"],
),
// Communication (5)
makeSkill(
"slack-bot",
"Slack Bot",
"Interact with Slack channels and messages.",
["SLACK_TOKEN"],
["send", "channels", "users", "search", "files"],
),
makeSkill(
"discord-mgr",
"Discord Manager",
"Manage Discord servers and messages.",
["DISCORD_TOKEN"],
["send", "servers", "channels", "members", "roles"],
),
makeSkill(
"telegram-bot",
"Telegram Bot",
"Send and receive Telegram messages.",
["TELEGRAM_BOT_TOKEN"],
["send", "receive", "chats", "files", "inline"],
),
makeSkill(
"matrix-cli",
"Matrix CLI",
"Interact with Matrix chat rooms.",
["MATRIX_HOMESERVER", "MATRIX_TOKEN"],
["send", "rooms", "join", "leave", "sync"],
),
makeSkill(
"irc-bridge",
"IRC Bridge",
"Bridge IRC channels to other platforms.",
["IRC_SERVER", "IRC_NICK"],
["connect", "join", "send", "channels", "users"],
),
// Data & Analytics (5)
makeSkill(
"pg-queries",
"PostgreSQL Queries",
"Execute PostgreSQL queries and manage databases.",
["DATABASE_URL"],
["query", "tables", "schema", "backup", "restore"],
),
makeSkill(
"clickhouse-ql",
"ClickHouse Queries",
"Run ClickHouse analytics queries.",
["CLICKHOUSE_URL"],
["query", "tables", "insert", "system", "optimize"],
),
makeSkill(
"redis-cli",
"Redis CLI",
"Interact with Redis cache and data structures.",
["REDIS_URL"],
["get", "set", "keys", "info", "flush"],
),
makeSkill(
"elastic-search",
"Elasticsearch",
"Search and manage Elasticsearch indices.",
["ELASTICSEARCH_URL"],
["search", "index", "mapping", "cluster", "aliases"],
),
makeSkill(
"mongo-shell",
"MongoDB Shell",
"Query and manage MongoDB collections.",
["MONGODB_URI"],
["find", "insert", "update", "delete", "aggregate"],
),
// Security (3)
makeSkill(
"vault-secrets",
"Vault Secrets",
"Manage secrets in HashiCorp Vault.",
["VAULT_ADDR", "VAULT_TOKEN"],
["read", "write", "list", "delete", "seal"],
),
makeSkill(
"gpg-keys",
"GPG Keys",
"Manage GPG keys and encryption.",
["GNUPGHOME"],
["list", "generate", "export", "import", "encrypt"],
),
makeSkill(
"ssh-rotate",
"SSH Key Rotator",
"Rotate and manage SSH keys.",
["SSH_KEY_DIR"],
["generate", "rotate", "deploy", "list", "revoke"],
),
// CJK Language Support (2)
makeSkill(
"nihongo-check",
"日本語チェッカー",
"日本語文章の文法チェックと翻訳支援ツール。Japanese grammar checker and translation assistant.",
["NIHONGO_API_KEY"],
["check", "translate", "kanji", "grammar", "vocabulary"],
),
makeSkill(
"hangukgeo-helper",
"한국어 도우미",
"한국어 학습 보조 도구입니다. Korean language learning assistant with vocabulary and grammar support.",
["HANGUL_API_KEY"],
["learn", "quiz", "vocabulary", "grammar", "pronunciation"],
),
];
function injectMetadata(rawSkillMd: string, metadata: Record<string, unknown>) {
const frontmatterEnd = rawSkillMd.indexOf("\n---", 3);
if (frontmatterEnd === -1) return rawSkillMd;
return `${rawSkillMd.slice(0, frontmatterEnd)}\nmetadata: ${JSON.stringify(
metadata,
)}${rawSkillMd.slice(frontmatterEnd)}`;
}
function randomStats() {
return {
downloads: Math.floor(Math.random() * 5000),
stars: Math.floor(Math.random() * 500),
installsCurrent: Math.floor(Math.random() * 200),
installsAllTime: Math.floor(Math.random() * 1000),
};
}
export const applyRandomStats = internalMutation({
args: {
skillId: v.id("skills"),
stats: v.object({
downloads: v.number(),
stars: v.number(),
installsCurrent: v.number(),
installsAllTime: v.number(),
}),
},
handler: async (ctx, args) => {
await ctx.db.patch(args.skillId, {
statsDownloads: args.stats.downloads,
statsStars: args.stats.stars,
statsInstallsCurrent: args.stats.installsCurrent,
statsInstallsAllTime: args.stats.installsAllTime,
stats: {
downloads: args.stats.downloads,
stars: args.stats.stars,
installsCurrent: args.stats.installsCurrent,
installsAllTime: args.stats.installsAllTime,
versions: 1,
comments: 0,
},
});
},
});
export const seedExtraSkillsInternal = internalAction({
args: {
reset: v.optional(v.boolean()),
},
handler: async (ctx: ActionCtx, args) => {
const results: Array<{ slug: string; ok: boolean; skipped?: boolean }> = [];
for (const spec of EXTRA_SEED_SKILLS) {
const skillMd = injectMetadata(spec.rawSkillMd, spec.metadata);
const frontmatter = parseFrontmatter(skillMd);
const clawdis = parseClawdisMetadata(frontmatter);
const storageId = await ctx.storage.store(new Blob([skillMd], { type: "text/markdown" }));
const result = (await ctx.runMutation(internal.devSeed.seedSkillMutation, {
reset: args.reset,
storageId,
metadata: spec.metadata,
frontmatter,
clawdis,
skillMd,
slug: spec.slug,
displayName: spec.displayName,
summary: spec.summary,
version: spec.version,
})) as { ok: boolean; skipped?: boolean; skillId?: string };
// Apply random stats after creation (only if not skipped)
if (result.skillId && !result.skipped) {
const stats = randomStats();
await ctx.runMutation(internal.devSeedExtra.applyRandomStats, {
skillId: result.skillId as Id<"skills">,
stats,
});
}
results.push({ slug: spec.slug, ok: result.ok, skipped: result.skipped });
}
const created = results.filter((r) => !r.skipped).length;
const skipped = results.filter((r) => r.skipped).length;
return { ok: true, total: results.length, created, skipped };
},
});
+139
View File
@@ -0,0 +1,139 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { ActionCtx } from "./_generated/server";
import { __test, downloadZipHandler } from "./downloads";
type RateLimitArgs = { key: string; limit: number; windowMs: number };
function isRateLimitArgs(args: unknown): args is RateLimitArgs {
if (!args || typeof args !== "object") return false;
const value = args as Record<string, unknown>;
return (
typeof value.key === "string" &&
typeof value.limit === "number" &&
typeof value.windowMs === "number"
);
}
const okRate = () => ({
allowed: true,
remaining: 10,
limit: 100,
resetAt: Date.now() + 60_000,
});
describe("downloads helpers", () => {
afterEach(() => {
vi.unstubAllEnvs();
vi.unstubAllGlobals();
});
it("calculates hour start boundaries", () => {
const hour = 3_600_000;
expect(__test.getHourStart(0)).toBe(0);
expect(__test.getHourStart(hour - 1)).toBe(0);
expect(__test.getHourStart(hour)).toBe(hour);
expect(__test.getHourStart(hour + 1)).toBe(hour);
});
it("prefers user identity when token user exists", () => {
const request = new Request("https://example.com", {
headers: { "cf-connecting-ip": "1.2.3.4" },
});
expect(__test.getDownloadIdentityValue(request, "users_123")).toBe("user:users_123");
});
it("uses cf-connecting-ip for anonymous identity", () => {
const request = new Request("https://example.com", {
headers: { "cf-connecting-ip": "1.2.3.4" },
});
expect(__test.getDownloadIdentityValue(request, null)).toBe("ip:1.2.3.4");
});
it("falls back to forwarded ip when explicitly enabled", () => {
vi.stubEnv("TRUST_FORWARDED_IPS", "true");
const request = new Request("https://example.com", {
headers: { "x-forwarded-for": "10.0.0.1, 10.0.0.2" },
});
expect(__test.getDownloadIdentityValue(request, null)).toBe("ip:10.0.0.1");
});
it("returns null when user and ip are missing", () => {
const request = new Request("https://example.com");
expect(__test.getDownloadIdentityValue(request, null)).toBeNull();
});
it("records zip downloads through the internal mutation path", async () => {
class MockResponse {
status: number;
headers: Headers;
constructor(_body?: BodyInit | null, init?: ResponseInit) {
this.status = init?.status ?? 200;
this.headers = new Headers(init?.headers);
}
}
vi.stubGlobal("Response", MockResponse as unknown as typeof Response);
const runQuery = vi.fn(async (_query: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
if ("slug" in args) {
return {
skill: {
_id: "skills:1",
ownerUserId: "users:1",
slug: "demo",
tags: {},
latestVersionId: "skillVersions:1",
},
moderationInfo: null,
};
}
if ("versionId" in args) {
return {
_id: "skillVersions:1",
version: "1.0.0",
createdAt: 3,
files: [{ path: "SKILL.md", storageId: "_storage:1" }],
softDeletedAt: undefined,
};
}
return null;
});
const runMutation = vi.fn(async (mutation: unknown, args: Record<string, unknown>) => {
if (isRateLimitArgs(args)) return okRate();
return { mutation, args };
});
const storageGet = vi.fn().mockResolvedValue(new Blob(["hello"], { type: "text/markdown" }));
const response = await downloadZipHandler(
{
runQuery,
runMutation,
storage: { get: storageGet },
} as unknown as ActionCtx,
new Request("https://example.com/api/v1/download?slug=demo", {
headers: { "cf-connecting-ip": "1.2.3.4" },
}),
);
expect(response.status).toBe(200);
expect(response.headers.get("Content-Type")).toBe("application/zip");
expect(storageGet).toHaveBeenCalledWith("_storage:1");
const recordCalls = runMutation.mock.calls.filter(([, args]) => {
if (!args || typeof args !== "object") return false;
const value = args as Record<string, unknown>;
return (
value.skillId === "skills:1" &&
typeof value.identityHash === "string" &&
typeof value.hourStart === "number"
);
});
expect(recordCalls).toHaveLength(1);
expect(recordCalls[0]?.[1]).toEqual({
skillId: "skills:1",
identityHash: expect.any(String),
hourStart: expect.any(Number),
});
});
});
+191 -51
View File
@@ -1,81 +1,221 @@
import { v } from 'convex/values'
import { zipSync } from 'fflate'
import { api } from './_generated/api'
import { httpAction, mutation } from './_generated/server'
import { applySkillStatDeltas, bumpDailySkillStats } from './lib/skillStats'
import { v } from "convex/values";
import { api, internal } from "./_generated/api";
import { httpAction, internalMutation } from "./functions";
import { getOptionalApiTokenUserId } from "./lib/apiTokenAuth";
import { corsHeaders, mergeHeaders } from "./lib/httpHeaders";
import { applyRateLimit, getClientIp } from "./lib/httpRateLimit";
import { buildDeterministicZip } from "./lib/skillZip";
import { hashToken } from "./lib/tokens";
import { insertStatEvent } from "./skillStatEvents";
export const downloadZip = httpAction(async (ctx, request) => {
const url = new URL(request.url)
const slug = url.searchParams.get('slug')?.trim().toLowerCase()
const versionParam = url.searchParams.get('version')?.trim()
const tagParam = url.searchParams.get('tag')?.trim()
const HOUR_MS = 3_600_000;
const DEDUPE_RETENTION_MS = 7 * 24 * HOUR_MS;
const PRUNE_BATCH_SIZE = 200;
const PRUNE_MAX_BATCHES = 50;
export async function downloadZipHandler(
ctx: Parameters<Parameters<typeof httpAction>[0]>[0],
request: Request,
) {
const url = new URL(request.url);
const slug = url.searchParams.get("slug")?.trim().toLowerCase();
const versionParam = url.searchParams.get("version")?.trim();
const tagParam = url.searchParams.get("tag")?.trim();
if (!slug) {
return new Response('Missing slug', { status: 400 })
return new Response("Missing slug", {
status: 400,
headers: corsHeaders(),
});
}
const skillResult = await ctx.runQuery(api.skills.getBySlug, { slug })
const rate = await applyRateLimit(ctx, request, "download");
if (!rate.ok) return rate.response;
const skillResult = await ctx.runQuery(api.skills.getBySlug, { slug });
if (!skillResult?.skill) {
return new Response('Skill not found', { status: 404 })
return new Response("Skill not found", {
status: 404,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
const skill = skillResult.skill
let version = skillResult.latestVersion
// Block downloads based on moderation status.
const mod = skillResult.moderationInfo;
if (mod?.isMalwareBlocked) {
return new Response(
"Blocked: this skill has been flagged as malicious by VirusTotal and cannot be downloaded.",
{
status: 403,
headers: mergeHeaders(rate.headers, corsHeaders()),
},
);
}
if (mod?.isPendingScan) {
return new Response(
"This skill is pending a security scan by VirusTotal. Please try again in a few minutes.",
{
status: 423,
headers: mergeHeaders(rate.headers, corsHeaders()),
},
);
}
if (mod?.isRemoved) {
return new Response("This skill has been removed by a moderator.", {
status: 410,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
if (mod?.isHiddenByMod) {
return new Response("This skill is currently unavailable.", {
status: 403,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
const skill = skillResult.skill;
let version = skill.latestVersionId
? await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: skill.latestVersionId,
})
: null;
if (versionParam) {
version = await ctx.runQuery(api.skills.getVersionBySkillAndVersion, {
version = await ctx.runQuery(internal.skills.getVersionBySkillAndVersionInternal, {
skillId: skill._id,
version: versionParam,
})
});
} else if (tagParam) {
const versionId = skill.tags[tagParam]
const versionId = skill.tags[tagParam];
if (versionId) {
version = await ctx.runQuery(api.skills.getVersionById, { versionId })
version = await ctx.runQuery(internal.skills.getVersionByIdInternal, { versionId });
}
}
if (!version) {
return new Response('Version not found', { status: 404 })
return new Response("Version not found", {
status: 404,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
if (version.softDeletedAt) {
return new Response('Version not available', { status: 410 })
return new Response("Version not available", {
status: 410,
headers: mergeHeaders(rate.headers, corsHeaders()),
});
}
const files: Record<string, Uint8Array> = {}
const entries: Array<{ path: string; bytes: Uint8Array }> = [];
for (const file of version.files) {
const blob = await ctx.storage.get(file.storageId)
if (!blob) continue
const buffer = new Uint8Array(await blob.arrayBuffer())
files[file.path] = buffer
const blob = await ctx.storage.get(file.storageId);
if (!blob) continue;
const buffer = new Uint8Array(await blob.arrayBuffer());
entries.push({ path: file.path, bytes: buffer });
}
const zipArray = buildDeterministicZip(entries, {
ownerId: String(skill.ownerUserId),
slug: skill.slug,
version: version.version,
publishedAt: version.createdAt,
});
const zipBlob = new Blob([zipArray], { type: "application/zip" });
const zipData = zipSync(files, { level: 6 })
const zipArray = Uint8Array.from(zipData)
const zipBlob = new Blob([zipArray], { type: 'application/zip' })
await ctx.runMutation(api.downloads.increment, { skillId: skill._id })
try {
const userId = await getOptionalApiTokenUserId(ctx, request);
const identity = getDownloadIdentityValue(request, userId ? String(userId) : null);
if (identity) {
await ctx.runMutation(internal.downloads.recordDownloadInternal, {
skillId: skill._id,
identityHash: await hashToken(identity),
hourStart: getHourStart(Date.now()),
});
}
} catch {
// Best-effort metric path; do not fail downloads.
}
return new Response(zipBlob, {
status: 200,
headers: {
'Content-Type': 'application/zip',
'Content-Disposition': `attachment; filename="${slug}-${version.version}.zip"`,
'Cache-Control': 'private, max-age=60',
},
})
})
headers: mergeHeaders(
rate.headers,
{
"Content-Type": "application/zip",
"Content-Disposition": `attachment; filename="${slug}-${version.version}.zip"`,
"Cache-Control": "private, max-age=60",
},
corsHeaders(),
),
});
}
export const increment = mutation({
args: { skillId: v.id('skills') },
handler: async (ctx, args) => {
const skill = await ctx.db.get(args.skillId)
if (!skill) return
const now = Date.now()
const patch = applySkillStatDeltas(skill, { downloads: 1 })
await ctx.db.patch(skill._id, {
...patch,
updatedAt: now,
})
await bumpDailySkillStats(ctx, { skillId: skill._id, now, downloads: 1 })
export const downloadZip = httpAction(downloadZipHandler);
export const recordDownloadInternal = internalMutation({
args: {
skillId: v.id("skills"),
identityHash: v.string(),
hourStart: v.number(),
},
})
handler: async (ctx, args) => {
const existing = await ctx.db
.query("downloadDedupes")
.withIndex("by_skill_identity_hour", (q) =>
q
.eq("skillId", args.skillId)
.eq("identityHash", args.identityHash)
.eq("hourStart", args.hourStart),
)
.unique();
if (existing) return;
await ctx.db.insert("downloadDedupes", {
skillId: args.skillId,
identityHash: args.identityHash,
hourStart: args.hourStart,
createdAt: Date.now(),
});
await insertStatEvent(ctx, {
skillId: args.skillId,
kind: "download",
});
},
});
export const pruneDownloadDedupesInternal = internalMutation({
args: {},
handler: async (ctx) => {
const cutoff = Date.now() - DEDUPE_RETENTION_MS;
for (let batches = 0; batches < PRUNE_MAX_BATCHES; batches += 1) {
const stale = await ctx.db
.query("downloadDedupes")
.withIndex("by_hour", (q) => q.lt("hourStart", cutoff))
.take(PRUNE_BATCH_SIZE);
if (stale.length === 0) break;
for (const entry of stale) {
await ctx.db.delete(entry._id);
}
if (stale.length < PRUNE_BATCH_SIZE) break;
}
},
});
export function getHourStart(timestamp: number) {
return Math.floor(timestamp / HOUR_MS) * HOUR_MS;
}
export function getDownloadIdentityValue(request: Request, userId: string | null) {
if (userId) return `user:${userId}`;
const ip = getClientIp(request);
if (!ip) return null;
return `ip:${ip}`;
}
export const __test = {
getHourStart,
getDownloadIdentityValue,
};
+526
View File
@@ -0,0 +1,526 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from "vitest";
import { internal } from "./_generated/api";
import {
repointPackageLatestRelease,
scheduleOwnerPublisherDigestSync,
syncPackageSearchDigestForPackageId,
syncPackageSearchDigestsForOwnerUserId,
} from "./functions";
describe("package digest sync", () => {
it("clears latestVersion when the current package release is soft-deleted", async () => {
const pkg = {
_id: "packages:demo",
name: "demo-plugin",
normalizedName: "demo-plugin",
displayName: "Demo Plugin",
family: "code-plugin",
channel: "community",
isOfficial: false,
ownerUserId: "users:owner",
summary: "demo",
capabilityTags: ["tools"],
executesCode: true,
runtimeId: null,
softDeletedAt: undefined,
createdAt: 1,
updatedAt: 2,
latestReleaseId: "packageReleases:demo-2",
latestVersionSummary: { version: "2.0.0" },
verification: { tier: "community" },
};
const latestRelease = {
_id: "packageReleases:demo-2",
version: "2.0.0",
softDeletedAt: 10,
};
const owner = {
_id: "users:owner",
handle: "owner",
deletedAt: undefined,
deactivatedAt: undefined,
};
const ctx = {
db: {
get: vi.fn(async (id: string) => {
if (id === "packages:demo") return pkg;
if (id === "packageReleases:demo-2") return latestRelease;
if (id === "users:owner") return owner;
return null;
}),
query: vi.fn(() => ({
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
})),
patch: vi.fn(),
insert: vi.fn(),
delete: vi.fn(),
},
};
await syncPackageSearchDigestForPackageId(ctx as never, "packages:demo" as never);
expect(ctx.db.insert).toHaveBeenCalledWith(
"packageSearchDigest",
expect.objectContaining({
packageId: "packages:demo",
latestVersion: undefined,
ownerHandle: "owner",
}),
);
});
it("preserves latestVersion when the current package release is active", async () => {
const pkg = {
_id: "packages:demo",
name: "demo-plugin",
normalizedName: "demo-plugin",
displayName: "Demo Plugin",
family: "code-plugin",
channel: "community",
isOfficial: false,
ownerUserId: "users:owner",
summary: "demo",
capabilityTags: ["tools"],
executesCode: true,
runtimeId: null,
softDeletedAt: undefined,
createdAt: 1,
updatedAt: 2,
latestReleaseId: "packageReleases:demo-2",
latestVersionSummary: { version: "2.0.0" },
verification: { tier: "community" },
};
const latestRelease = {
_id: "packageReleases:demo-2",
version: "2.0.0",
};
const owner = {
_id: "users:owner",
handle: "owner",
deletedAt: undefined,
deactivatedAt: undefined,
};
const ctx = {
db: {
get: vi.fn(async (id: string) => {
if (id === "packages:demo") return pkg;
if (id === "packageReleases:demo-2") return latestRelease;
if (id === "users:owner") return owner;
return null;
}),
query: vi.fn(() => ({
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
})),
patch: vi.fn(),
insert: vi.fn(),
delete: vi.fn(),
},
};
await syncPackageSearchDigestForPackageId(ctx as never, "packages:demo" as never);
expect(ctx.db.insert).toHaveBeenCalledWith(
"packageSearchDigest",
expect.objectContaining({
packageId: "packages:demo",
latestVersion: "2.0.0",
ownerHandle: "owner",
}),
);
});
it("repoints packages to the highest-version active release and restores its summary", async () => {
const pkg = {
_id: "packages:demo",
_creationTime: 1,
name: "demo-plugin",
normalizedName: "demo-plugin",
displayName: "Demo Plugin",
family: "code-plugin",
channel: "community",
isOfficial: false,
ownerUserId: "users:owner",
summary: "latest summary",
tags: {
latest: "packageReleases:demo-2",
stable: "packageReleases:demo-2",
},
latestReleaseId: "packageReleases:demo-2",
latestVersionSummary: { version: "2.0.0" },
capabilityTags: ["new"],
executesCode: true,
compatibility: { openclaw: "^2.0.0" },
capabilities: { capabilityTags: ["new"], executesCode: true },
verification: { tier: "community" },
runtimeId: null,
softDeletedAt: undefined,
createdAt: 1,
updatedAt: 2,
};
const fallbackRelease = {
_id: "packageReleases:demo-1",
_creationTime: 10,
packageId: "packages:demo",
version: "1.0.0",
changelog: "old stable",
summary: "stable summary",
compatibility: { openclaw: "^1.0.0" },
capabilities: { capabilityTags: ["stable"], executesCode: false },
verification: { tier: "verified" },
distTags: ["stable"],
createdAt: 10,
softDeletedAt: undefined,
};
const legacyHotfixRelease = {
_id: "packageReleases:demo-legacy",
_creationTime: 20,
packageId: "packages:demo",
version: "0.9.9",
changelog: "legacy hotfix",
summary: "legacy summary",
compatibility: { openclaw: "^0.9.0" },
capabilities: { capabilityTags: ["legacy"], executesCode: false },
verification: { tier: "verified" },
distTags: ["legacy"],
createdAt: 20,
softDeletedAt: undefined,
};
const owner = {
_id: "users:owner",
handle: "owner",
deletedAt: undefined,
deactivatedAt: undefined,
};
const ctx = {
db: {
get: vi.fn(async (id: string) => {
if (id === "packages:demo") return pkg;
if (id === "packageReleases:demo-1") return fallbackRelease;
if (id === "users:owner") return owner;
return null;
}),
query: vi.fn((table: string) => {
if (table === "packageReleases") {
return {
withIndex: vi.fn(() => ({
order: vi.fn(() => ({
paginate: vi.fn().mockResolvedValue({
page: [legacyHotfixRelease, fallbackRelease],
isDone: true,
continueCursor: "",
}),
})),
})),
};
}
if (table === "packageSearchDigest") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
};
}
if (table === "packageCapabilitySearchDigest") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
};
}
throw new Error(`Unexpected table ${table}`);
}),
patch: vi.fn(),
insert: vi.fn(),
delete: vi.fn(),
},
};
await repointPackageLatestRelease(
ctx as never,
"packages:demo" as never,
"packageReleases:demo-2" as never,
);
expect(ctx.db.patch).toHaveBeenCalledWith("packageReleases:demo-1", {
distTags: ["stable", "latest"],
});
expect(ctx.db.patch).toHaveBeenCalledWith(
"packages:demo",
expect.objectContaining({
latestReleaseId: "packageReleases:demo-1",
tags: { latest: "packageReleases:demo-1" },
latestVersionSummary: expect.objectContaining({ version: "1.0.0" }),
summary: "stable summary",
capabilityTags: ["stable"],
executesCode: false,
}),
);
expect(ctx.db.insert).toHaveBeenCalledWith(
"packageSearchDigest",
expect.objectContaining({
latestVersion: "1.0.0",
ownerHandle: "owner",
}),
);
});
it("repoints bundle packages to the newest surviving release, not semver-looking versions", async () => {
const pkg = {
_id: "packages:bundle",
_creationTime: 1,
name: "demo-bundle",
normalizedName: "demo-bundle",
displayName: "Demo Bundle",
family: "bundle-plugin",
channel: "community",
isOfficial: false,
ownerUserId: "users:owner",
summary: "latest summary",
tags: {
latest: "packageReleases:bundle-latest",
},
latestReleaseId: "packageReleases:bundle-latest",
latestVersionSummary: { version: "latest" },
capabilityTags: ["new"],
executesCode: false,
compatibility: { hosts: ["openclaw"] },
capabilities: { capabilityTags: ["new"], executesCode: false },
verification: { tier: "community" },
runtimeId: "bundle.runtime",
softDeletedAt: undefined,
createdAt: 1,
updatedAt: 2,
};
const semverLookingRelease = {
_id: "packageReleases:bundle-semver",
_creationTime: 10,
packageId: "packages:bundle",
version: "2.0.0",
changelog: "older semver",
summary: "older semver summary",
compatibility: { hosts: ["openclaw"] },
capabilities: { capabilityTags: ["semver"], executesCode: false },
verification: { tier: "verified" },
distTags: ["legacy"],
createdAt: 10,
softDeletedAt: undefined,
};
const newestRelease = {
_id: "packageReleases:bundle-newest",
_creationTime: 20,
packageId: "packages:bundle",
version: "2024-12",
changelog: "newest bundle build",
summary: "newest bundle summary",
compatibility: { hosts: ["openclaw"] },
capabilities: { capabilityTags: ["bundle"], executesCode: false },
verification: { tier: "verified" },
distTags: ["release-2024-12"],
createdAt: 20,
softDeletedAt: undefined,
};
const owner = {
_id: "users:owner",
handle: "owner",
deletedAt: undefined,
deactivatedAt: undefined,
};
const ctx = {
db: {
get: vi.fn(async (id: string) => {
if (id === "packages:bundle") return pkg;
if (id === "packageReleases:bundle-newest") return newestRelease;
if (id === "users:owner") return owner;
return null;
}),
query: vi.fn((table: string) => {
if (table === "packageReleases") {
return {
withIndex: vi.fn(() => ({
order: vi.fn(() => ({
paginate: vi.fn().mockResolvedValue({
page: [newestRelease, semverLookingRelease],
isDone: true,
continueCursor: "",
}),
})),
})),
};
}
if (table === "packageSearchDigest") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
};
}
if (table === "packageCapabilitySearchDigest") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
};
}
throw new Error(`Unexpected table ${table}`);
}),
patch: vi.fn(),
insert: vi.fn(),
delete: vi.fn(),
},
};
await repointPackageLatestRelease(
ctx as never,
"packages:bundle" as never,
"packageReleases:bundle-latest" as never,
);
expect(ctx.db.patch).toHaveBeenCalledWith("packageReleases:bundle-newest", {
distTags: ["release-2024-12", "latest"],
});
expect(ctx.db.patch).toHaveBeenCalledWith(
"packages:bundle",
expect.objectContaining({
latestReleaseId: "packageReleases:bundle-newest",
tags: { latest: "packageReleases:bundle-newest" },
latestVersionSummary: expect.objectContaining({ version: "2024-12" }),
summary: "newest bundle summary",
}),
);
expect(ctx.db.insert).toHaveBeenCalledWith(
"packageSearchDigest",
expect.objectContaining({
latestVersion: "2024-12",
ownerHandle: "owner",
}),
);
});
it("re-syncs package digests when an owner handle changes", async () => {
const owner = {
_id: "users:owner",
handle: "renamed",
deletedAt: undefined,
deactivatedAt: undefined,
};
const pkg = {
_id: "packages:demo",
_creationTime: 1,
name: "demo-plugin",
normalizedName: "demo-plugin",
displayName: "Demo Plugin",
family: "code-plugin",
channel: "community",
isOfficial: false,
ownerUserId: "users:owner",
summary: "demo",
tags: {},
latestReleaseId: undefined,
latestVersionSummary: undefined,
capabilityTags: [],
executesCode: false,
runtimeId: null,
softDeletedAt: undefined,
createdAt: 1,
updatedAt: 2,
verification: undefined,
};
const paginate = vi.fn().mockResolvedValueOnce({
page: [pkg],
isDone: true,
continueCursor: "",
});
const ctx = {
db: {
get: vi.fn(async (id: string) => {
if (id === "users:owner") return owner;
return null;
}),
query: vi.fn((table: string) => {
if (table === "packages") {
return {
withIndex: vi.fn(() => ({
paginate,
})),
};
}
if (table === "packageSearchDigest") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
};
}
if (table === "packageCapabilitySearchDigest") {
return {
withIndex: vi.fn(() => ({
unique: vi.fn().mockResolvedValue(null),
collect: vi.fn().mockResolvedValue([]),
})),
};
}
throw new Error(`Unexpected table ${table}`);
}),
patch: vi.fn(),
insert: vi.fn(),
delete: vi.fn(),
},
};
await syncPackageSearchDigestsForOwnerUserId(ctx as never, "users:owner" as never);
expect(paginate).toHaveBeenCalledWith({ cursor: null, numItems: 100 });
expect(ctx.db.insert).toHaveBeenCalledWith(
"packageSearchDigest",
expect.objectContaining({
packageId: "packages:demo",
ownerHandle: "renamed",
}),
);
});
});
describe("publisher digest scheduling", () => {
it("schedules package and skill digest sync in separate background mutations", async () => {
const ctx = {
scheduler: {
runAfter: vi.fn().mockResolvedValue(undefined),
},
};
await scheduleOwnerPublisherDigestSync(ctx as never, "publishers:demo" as never);
expect(ctx.scheduler.runAfter).toHaveBeenCalledTimes(2);
expect(ctx.scheduler.runAfter).toHaveBeenNthCalledWith(
1,
0,
internal.functions.syncPackageSearchDigestsForOwnerPublisherIdInternal,
{ ownerPublisherId: "publishers:demo" },
);
expect(ctx.scheduler.runAfter).toHaveBeenNthCalledWith(
2,
0,
internal.functions.syncSkillSearchDigestsForOwnerPublisherIdInternal,
{ ownerPublisherId: "publishers:demo" },
);
});
it("skips scheduling when the trigger context has no scheduler", async () => {
await expect(
scheduleOwnerPublisherDigestSync({} as never, "publishers:demo" as never),
).resolves.toBeUndefined();
});
});
+384
View File
@@ -0,0 +1,384 @@
import { customCtx, customMutation } from "convex-helpers/server/customFunctions";
import { Triggers } from "convex-helpers/server/triggers";
import { v } from "convex/values";
import semver from "semver";
import { internal } from "./_generated/api";
import type { DataModel, Doc, Id } from "./_generated/dataModel";
import {
mutation as rawMutation,
internalMutation as rawInternalMutation,
query,
internalQuery,
action,
internalAction,
httpAction,
} from "./_generated/server";
import type { MutationCtx } from "./_generated/server";
import {
deletePackageSearchDigests,
extractPackageDigestFields,
upsertPackageSearchDigest,
} from "./lib/packageSearchDigest";
import { getOwnerPublisher } from "./lib/publishers";
import { extractDigestFields, upsertSkillSearchDigest } from "./lib/skillSearchDigest";
const triggers = new Triggers<DataModel>();
function isMissingTableError(error: unknown, table: string) {
return (
error instanceof Error &&
new RegExp(`unexpected (query )?table:? ${table}`, "i").test(error.message)
);
}
type PackageDigestSyncCtx = Pick<MutationCtx, "db">;
type OwnerPublisherDigestScheduleCtx = Pick<Partial<MutationCtx>, "scheduler">;
type LatestPackageRelease = Pick<
Doc<"packageReleases">,
| "_id"
| "createdAt"
| "version"
| "changelog"
| "summary"
| "compatibility"
| "capabilities"
| "verification"
| "distTags"
> & {
scanStatus?: Doc<"packages">["scanStatus"];
};
function toPackageLatestVersionSummary(
release: LatestPackageRelease | null,
): Doc<"packages">["latestVersionSummary"] {
if (!release) return undefined;
return {
version: release.version,
createdAt: release.createdAt,
changelog: release.changelog,
compatibility: release.compatibility,
capabilities: release.capabilities,
verification: release.verification,
};
}
function compareFallbackReleases(
family: Doc<"packages">["family"],
a: LatestPackageRelease,
b: LatestPackageRelease,
) {
if (family === "bundle-plugin") {
if (a.createdAt !== b.createdAt) return a.createdAt - b.createdAt;
return a._id.localeCompare(b._id);
}
const aSemver = semver.valid(a.version);
const bSemver = semver.valid(b.version);
if (aSemver && bSemver) return semver.compare(aSemver, bSemver);
if (aSemver) return 1;
if (bSemver) return -1;
if (a.createdAt !== b.createdAt) return a.createdAt - b.createdAt;
return a._id.localeCompare(b._id);
}
async function getPreferredFallbackPackageRelease(
ctx: PackageDigestSyncCtx,
packageId: Id<"packages">,
family: Doc<"packages">["family"],
): Promise<LatestPackageRelease | null> {
let cursor: string | null = null;
let best: LatestPackageRelease | null = null;
while (true) {
const page = await ctx.db
.query("packageReleases")
.withIndex("by_package_active_created", (q) =>
q.eq("packageId", packageId).eq("softDeletedAt", undefined),
)
.order("desc")
.paginate({ cursor, numItems: 100 });
for (const release of page.page) {
const candidate: LatestPackageRelease = {
_id: release._id,
createdAt: release.createdAt,
version: release.version,
changelog: release.changelog,
summary: release.summary,
compatibility: release.compatibility,
capabilities: release.capabilities,
verification: release.verification,
scanStatus: release.verification?.scanStatus,
distTags: release.distTags,
};
if (!best || compareFallbackReleases(family, candidate, best) > 0) best = candidate;
}
if (page.isDone) return best;
cursor = page.continueCursor;
}
}
async function syncPackageSearchDigest(
ctx: PackageDigestSyncCtx,
pkg: Doc<"packages"> | null | undefined,
) {
if (!pkg) return;
const latestRelease = pkg.latestReleaseId ? await ctx.db.get(pkg.latestReleaseId) : null;
const fields = extractPackageDigestFields(pkg);
const owner = await getOwnerPublisher(ctx, {
ownerPublisherId: pkg.ownerPublisherId,
ownerUserId: pkg.ownerUserId,
});
await upsertPackageSearchDigest(ctx, {
...fields,
latestVersion:
latestRelease && !latestRelease.softDeletedAt ? latestRelease.version : undefined,
ownerHandle: owner?.handle ?? "",
ownerKind: owner?.kind,
});
}
export async function syncPackageSearchDigestForPackageId(
ctx: PackageDigestSyncCtx,
packageId: Id<"packages"> | null | undefined,
) {
if (!packageId) return;
const pkg = await ctx.db.get(packageId);
if (!pkg) return;
await syncPackageSearchDigest(ctx, pkg);
}
export async function syncPackageSearchDigestsForOwnerUserId(
ctx: PackageDigestSyncCtx,
ownerUserId: Id<"users"> | null | undefined,
) {
if (!ownerUserId) return;
let cursor: string | null = null;
try {
while (true) {
const page = await ctx.db
.query("packages")
.withIndex("by_owner", (q) => q.eq("ownerUserId", ownerUserId))
.paginate({ cursor, numItems: 100 });
for (const pkg of page.page) {
await syncPackageSearchDigest(ctx, pkg);
}
if (page.isDone) break;
cursor = page.continueCursor;
}
} catch (error) {
if (isMissingTableError(error, "packages")) return;
throw error;
}
}
export async function syncPackageSearchDigestsForOwnerPublisherId(
ctx: PackageDigestSyncCtx,
ownerPublisherId: Id<"publishers"> | null | undefined,
) {
if (!ownerPublisherId) return;
let cursor: string | null = null;
try {
while (true) {
const page = await ctx.db
.query("packages")
.withIndex("by_owner_publisher", (q) => q.eq("ownerPublisherId", ownerPublisherId))
.paginate({ cursor, numItems: 100 });
for (const pkg of page.page) {
await syncPackageSearchDigest(ctx, pkg);
}
if (page.isDone) break;
cursor = page.continueCursor;
}
} catch (error) {
if (isMissingTableError(error, "packages")) return;
throw error;
}
}
async function syncSkillSearchDigestForSkill(
ctx: PackageDigestSyncCtx,
skill: Doc<"skills"> | null | undefined,
) {
if (!skill) return;
const fields = extractDigestFields(skill);
const owner = await getOwnerPublisher(ctx, {
ownerPublisherId: skill.ownerPublisherId,
ownerUserId: skill.ownerUserId,
});
await upsertSkillSearchDigest(ctx, {
...fields,
ownerHandle: owner?.handle ?? "",
ownerKind: owner?.kind,
ownerName: owner?.linkedUserId ? owner.handle : undefined,
ownerDisplayName: owner?.displayName,
ownerImage: owner?.image,
});
}
export async function syncSkillSearchDigestsForOwnerPublisherId(
ctx: PackageDigestSyncCtx,
ownerPublisherId: Id<"publishers"> | null | undefined,
) {
if (!ownerPublisherId) return;
let cursor: string | null = null;
try {
while (true) {
const page = await ctx.db
.query("skills")
.withIndex("by_owner_publisher", (q) => q.eq("ownerPublisherId", ownerPublisherId))
.paginate({ cursor, numItems: 100 });
for (const skill of page.page) {
await syncSkillSearchDigestForSkill(ctx, skill);
}
if (page.isDone) break;
cursor = page.continueCursor;
}
} catch (error) {
if (isMissingTableError(error, "skills")) return;
throw error;
}
}
export async function scheduleOwnerPublisherDigestSync(
ctx: OwnerPublisherDigestScheduleCtx,
ownerPublisherId: Id<"publishers"> | null | undefined,
) {
if (!ownerPublisherId || !ctx.scheduler) return;
await ctx.scheduler.runAfter(
0,
internal.functions.syncPackageSearchDigestsForOwnerPublisherIdInternal,
{ ownerPublisherId },
);
await ctx.scheduler.runAfter(
0,
internal.functions.syncSkillSearchDigestsForOwnerPublisherIdInternal,
{ ownerPublisherId },
);
}
export const syncPackageSearchDigestsForOwnerPublisherIdInternal = rawInternalMutation({
args: {
ownerPublisherId: v.id("publishers"),
},
handler: async (ctx, args) => {
await syncPackageSearchDigestsForOwnerPublisherId(ctx, args.ownerPublisherId);
},
});
export const syncSkillSearchDigestsForOwnerPublisherIdInternal = rawInternalMutation({
args: {
ownerPublisherId: v.id("publishers"),
},
handler: async (ctx, args) => {
await syncSkillSearchDigestsForOwnerPublisherId(ctx, args.ownerPublisherId);
},
});
export async function repointPackageLatestRelease(
ctx: PackageDigestSyncCtx,
packageId: Id<"packages"> | null | undefined,
affectedReleaseId: Id<"packageReleases"> | null | undefined,
) {
if (!packageId || !affectedReleaseId) return;
const pkg = await ctx.db.get(packageId);
if (!pkg) return;
const nextTags = Object.fromEntries(
Object.entries(pkg.tags).filter(([, releaseId]) => releaseId !== affectedReleaseId),
) as Doc<"packages">["tags"];
const latestPointerAffected =
pkg.latestReleaseId === affectedReleaseId || pkg.tags.latest === affectedReleaseId;
if (!latestPointerAffected && Object.keys(nextTags).length === Object.keys(pkg.tags).length) {
return;
}
const nextLatest = latestPointerAffected
? await getPreferredFallbackPackageRelease(ctx, packageId, pkg.family)
: null;
if (latestPointerAffected && nextLatest && !(nextLatest.distTags ?? []).includes("latest")) {
await ctx.db.patch(nextLatest._id, {
distTags: [...(nextLatest.distTags ?? []), "latest"],
});
}
const patch: Partial<Doc<"packages">> = {
tags: latestPointerAffected && nextLatest ? { ...nextTags, latest: nextLatest._id } : nextTags,
updatedAt: Date.now(),
};
if (latestPointerAffected) {
patch.latestReleaseId = nextLatest?._id;
patch.latestVersionSummary = toPackageLatestVersionSummary(nextLatest);
patch.summary = nextLatest?.summary;
patch.capabilityTags = nextLatest?.capabilities?.capabilityTags;
patch.executesCode =
typeof nextLatest?.capabilities?.executesCode === "boolean"
? nextLatest.capabilities.executesCode
: undefined;
patch.compatibility = nextLatest?.compatibility;
patch.capabilities = nextLatest?.capabilities;
patch.verification = nextLatest?.verification;
patch.scanStatus = nextLatest?.scanStatus;
}
await ctx.db.patch(pkg._id, patch);
await syncPackageSearchDigest(ctx, { ...pkg, ...patch });
}
triggers.register("skills", async (ctx, change) => {
if (change.operation === "delete") {
const existing = await ctx.db
.query("skillSearchDigest")
.withIndex("by_skill", (q) => q.eq("skillId", change.id))
.unique();
if (existing) await ctx.db.delete(existing._id);
} else {
await syncSkillSearchDigestForSkill(ctx, change.newDoc);
}
});
triggers.register("packages", async (ctx, change) => {
if (change.operation === "delete") {
await deletePackageSearchDigests(ctx, change.id);
return;
}
await syncPackageSearchDigest(ctx, change.newDoc);
});
triggers.register("packageReleases", async (ctx, change) => {
if (change.operation === "insert") return;
if (
change.operation === "update" &&
change.oldDoc.softDeletedAt === change.newDoc.softDeletedAt
) {
return;
}
const packageId =
change.operation === "delete" ? change.oldDoc.packageId : change.newDoc.packageId;
const affectedReleaseId = change.operation === "delete" ? change.oldDoc._id : change.newDoc._id;
if (change.operation === "delete" || change.newDoc.softDeletedAt) {
await repointPackageLatestRelease(ctx, packageId, affectedReleaseId);
return;
}
await syncPackageSearchDigestForPackageId(ctx, packageId);
});
triggers.register("users", async (ctx, change) => {
if (
change.operation === "update" &&
change.oldDoc.handle === change.newDoc.handle &&
change.oldDoc.deletedAt === change.newDoc.deletedAt &&
change.oldDoc.deactivatedAt === change.newDoc.deactivatedAt
) {
return;
}
const ownerUserId = change.operation === "delete" ? change.id : change.newDoc._id;
await syncPackageSearchDigestsForOwnerUserId(ctx, ownerUserId);
});
triggers.register("publishers", async (ctx, change) => {
const ownerPublisherId = change.operation === "delete" ? change.id : change.newDoc._id;
await scheduleOwnerPublisherDigestSync(ctx, ownerPublisherId);
});
export const mutation = customMutation(rawMutation, customCtx(triggers.wrapDB));
export const internalMutation = customMutation(rawInternalMutation, customCtx(triggers.wrapDB));
export { query, internalQuery, action, internalAction, httpAction };
+158
View File
@@ -0,0 +1,158 @@
import { describe, expect, it, vi } from "vitest";
import { getGitHubBackupPageInternal } from "./githubBackups";
const handler = (getGitHubBackupPageInternal as unknown as { _handler: Function })._handler;
describe("githubBackups page filtering", () => {
it("skips non-public skills (soft-deleted, hidden, removed)", async () => {
const activeSkill = {
_id: "skills:active",
slug: "active-skill",
displayName: "Active Skill",
ownerUserId: "users:active",
latestVersionId: "skillVersions:active",
softDeletedAt: undefined,
moderationStatus: "active",
};
const hiddenSkill = {
_id: "skills:hidden",
slug: "hidden-skill",
displayName: "Hidden Skill",
ownerUserId: "users:hidden",
latestVersionId: "skillVersions:hidden",
softDeletedAt: undefined,
moderationStatus: "hidden",
};
const removedSkill = {
_id: "skills:removed",
slug: "removed-skill",
displayName: "Removed Skill",
ownerUserId: "users:removed",
latestVersionId: "skillVersions:removed",
softDeletedAt: undefined,
moderationStatus: "removed",
};
const softDeletedSkill = {
_id: "skills:soft",
slug: "soft-skill",
displayName: "Soft Skill",
ownerUserId: "users:soft",
latestVersionId: "skillVersions:soft",
softDeletedAt: 1,
moderationStatus: "active",
};
const get = vi.fn(async (id: string) => {
if (id === "skillVersions:active") {
return {
_id: "skillVersions:active",
version: "1.0.0",
files: [{ path: "SKILL.md", size: 10, storageId: "storage:1", sha256: "abc" }],
createdAt: 1_700_000_000_000,
};
}
if (id === "users:active") {
return {
_id: "users:active",
handle: "alice",
deletedAt: undefined,
deactivatedAt: undefined,
};
}
return null;
});
const paginate = vi.fn().mockResolvedValue({
page: [activeSkill, hiddenSkill, removedSkill, softDeletedSkill],
isDone: true,
continueCursor: null,
});
const order = vi.fn().mockReturnValue({ paginate });
const query = vi.fn().mockReturnValue({ order });
const result = await handler(
{
db: {
query,
get,
},
} as never,
{ batchSize: 50 },
);
expect(result).toMatchObject({
isDone: true,
cursor: null,
items: [
{
kind: "ok",
slug: "active-skill",
ownerHandle: "alice",
version: "1.0.0",
},
],
});
expect(get).toHaveBeenCalledTimes(2);
});
it("keeps legacy skills with undefined moderationStatus eligible", async () => {
const legacySkill = {
_id: "skills:legacy",
slug: "legacy-skill",
displayName: "Legacy Skill",
ownerUserId: "users:legacy",
latestVersionId: "skillVersions:legacy",
softDeletedAt: undefined,
moderationStatus: undefined,
};
const get = vi.fn(async (id: string) => {
if (id === "skillVersions:legacy") {
return {
_id: "skillVersions:legacy",
version: "2.0.0",
files: [{ path: "SKILL.md", size: 20, storageId: "storage:2", sha256: "def" }],
createdAt: 1_700_000_000_100,
};
}
if (id === "users:legacy") {
return {
_id: "users:legacy",
handle: null,
deletedAt: undefined,
deactivatedAt: undefined,
};
}
return null;
});
const paginate = vi.fn().mockResolvedValue({
page: [legacySkill],
isDone: true,
continueCursor: null,
});
const order = vi.fn().mockReturnValue({ paginate });
const query = vi.fn().mockReturnValue({ order });
const result = await handler(
{
db: {
query,
get,
},
} as never,
{},
);
expect(result.items).toHaveLength(1);
expect(result.items[0]).toMatchObject({
kind: "ok",
slug: "legacy-skill",
ownerHandle: "users:legacy",
version: "2.0.0",
});
});
});
+98 -77
View File
@@ -1,51 +1,54 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalMutation, internalQuery } from './_generated/server'
import { assertRole, requireUserFromAction } from './lib/access'
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc, Id } from "./_generated/dataModel";
import { action, internalMutation, internalQuery } from "./functions";
import { assertRole, requireUserFromAction } from "./lib/access";
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const SYNC_STATE_KEY = 'default'
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
const SYNC_STATE_KEY = "default";
type BackupPageItem =
| {
kind: 'ok'
skillId: Id<'skills'>
versionId: Id<'skillVersions'>
slug: string
displayName: string
version: string
ownerHandle: string
files: Doc<'skillVersions'>['files']
publishedAt: number
kind: "ok";
skillId: Id<"skills">;
versionId: Id<"skillVersions">;
slug: string;
displayName: string;
version: string;
ownerHandle: string;
files: Doc<"skillVersions">["files"];
publishedAt: number;
}
| { kind: 'missingLatestVersion'; skillId: Id<'skills'> }
| { kind: 'missingVersionDoc'; skillId: Id<'skills'>; versionId: Id<'skillVersions'> }
| { kind: 'missingOwner'; skillId: Id<'skills'>; ownerUserId: Id<'users'> }
| { kind: "missingLatestVersion"; skillId: Id<"skills"> }
| { kind: "missingVersionDoc"; skillId: Id<"skills">; versionId: Id<"skillVersions"> }
| { kind: "missingOwner"; skillId: Id<"skills">; ownerUserId: Id<"users"> };
type BackupPageResult = {
items: BackupPageItem[]
cursor: string | null
isDone: boolean
}
items: BackupPageItem[];
cursor: string | null;
isDone: boolean;
};
type BackupSyncState = {
cursor: string | null
}
cursor: string | null;
pruneCursor: string | null;
};
export type SyncGitHubBackupsResult = {
stats: {
skillsScanned: number
skillsSkipped: number
skillsBackedUp: number
skillsMissingVersion: number
skillsMissingOwner: number
errors: number
}
cursor: string | null
isDone: boolean
}
skillsScanned: number;
skillsSkipped: number;
skillsBackedUp: number;
skillsDeleted: number;
skillsMissingVersion: number;
skillsMissingOwner: number;
errors: number;
};
cursor: string | null;
pruneCursor: string | null;
isDone: boolean;
};
export const getGitHubBackupPageInternal = internalQuery({
args: {
@@ -53,38 +56,38 @@ export const getGitHubBackupPageInternal = internalQuery({
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BackupPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const { page, isDone, continueCursor } = await ctx.db
.query('skills')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
.query("skills")
.order("asc")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
const items: BackupPageItem[] = []
const items: BackupPageItem[] = [];
for (const skill of page) {
if (skill.softDeletedAt) continue
if (!isPubliclyAvailableSkill(skill)) continue;
if (!skill.latestVersionId) {
items.push({ kind: 'missingLatestVersion', skillId: skill._id })
continue
items.push({ kind: "missingLatestVersion", skillId: skill._id });
continue;
}
const version = await ctx.db.get(skill.latestVersionId)
const version = await ctx.db.get(skill.latestVersionId);
if (!version) {
items.push({
kind: 'missingVersionDoc',
kind: "missingVersionDoc",
skillId: skill._id,
versionId: skill.latestVersionId,
})
continue
});
continue;
}
const owner = await ctx.db.get(skill.ownerUserId)
if (!owner || owner.deletedAt) {
items.push({ kind: 'missingOwner', skillId: skill._id, ownerUserId: skill.ownerUserId })
continue
const owner = await ctx.db.get(skill.ownerUserId);
if (!owner || owner.deletedAt || owner.deactivatedAt) {
items.push({ kind: "missingOwner", skillId: skill._id, ownerUserId: skill.ownerUserId });
continue;
}
items.push({
kind: 'ok',
kind: "ok",
skillId: skill._id,
versionId: version._id,
slug: skill.slug,
@@ -93,78 +96,96 @@ export const getGitHubBackupPageInternal = internalQuery({
ownerHandle: owner.handle ?? owner._id,
files: version.files,
publishedAt: version.createdAt,
})
});
}
return { items, cursor: continueCursor, isDone }
return { items, cursor: continueCursor, isDone };
},
})
});
function isPubliclyAvailableSkill(skill: {
softDeletedAt?: number;
moderationStatus?: string | null;
}) {
if (skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
}
export const getGitHubBackupSyncStateInternal = internalQuery({
args: {},
handler: async (ctx): Promise<BackupSyncState> => {
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
return { cursor: state?.cursor ?? null }
.query("githubBackupSyncState")
.withIndex("by_key", (q) => q.eq("key", SYNC_STATE_KEY))
.unique();
return { cursor: state?.cursor ?? null, pruneCursor: state?.pruneCursor ?? null };
},
})
});
export const setGitHubBackupSyncStateInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
pruneCursor: v.optional(v.string()),
},
handler: async (ctx, args) => {
const now = Date.now()
const now = Date.now();
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
.query("githubBackupSyncState")
.withIndex("by_key", (q) => q.eq("key", SYNC_STATE_KEY))
.unique();
if (!state) {
await ctx.db.insert('githubBackupSyncState', {
await ctx.db.insert("githubBackupSyncState", {
key: SYNC_STATE_KEY,
cursor: args.cursor,
pruneCursor: args.pruneCursor,
updatedAt: now,
})
return { ok: true as const }
});
return { ok: true as const };
}
await ctx.db.patch(state._id, {
cursor: args.cursor,
pruneCursor: args.pruneCursor,
updatedAt: now,
})
});
return { ok: true as const }
return { ok: true as const };
},
})
});
export const syncGitHubBackups: ReturnType<typeof action> = action({
args: {
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
pruneBatchSize: v.optional(v.number()),
resetCursor: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SyncGitHubBackupsResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin"]);
if (args.resetCursor && !args.dryRun) {
await ctx.runMutation(internal.githubBackups.setGitHubBackupSyncStateInternal, {
cursor: undefined,
})
pruneCursor: undefined,
});
}
return ctx.runAction(internal.githubBackupsNode.syncGitHubBackupsInternal, {
dryRun: args.dryRun,
batchSize: args.batchSize,
maxBatches: args.maxBatches,
}) as Promise<SyncGitHubBackupsResult>
pruneBatchSize: args.pruneBatchSize,
}) as Promise<SyncGitHubBackupsResult>;
},
})
});
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
return Math.max(min, Math.min(max, Math.floor(value)));
}
+213 -74
View File
@@ -1,56 +1,64 @@
'use node'
"use node";
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction } from './_generated/server'
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { internalAction } from "./functions";
import {
backupSkillToGitHub,
deleteGitHubSkillBackup,
fetchGitHubSkillMeta,
getGitHubBackupContext,
isGitHubBackupConfigured,
} from './lib/githubBackup'
listGitHubSkillBackupEntries,
normalizeOwner,
} from "./lib/githubBackup";
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const DEFAULT_MAX_BATCHES = 5
const MAX_MAX_BATCHES = 200
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
const DEFAULT_MAX_BATCHES = 5;
const MAX_MAX_BATCHES = 200;
const DEFAULT_PRUNE_BATCH_SIZE = 10;
const MAX_PRUNE_BATCH_SIZE = 100;
type BackupPageItem =
| {
kind: 'ok'
slug: string
version: string
displayName: string
ownerHandle: string
files: Doc<'skillVersions'>['files']
publishedAt: number
kind: "ok";
slug: string;
version: string;
displayName: string;
ownerHandle: string;
files: Doc<"skillVersions">["files"];
publishedAt: number;
}
| { kind: 'missingLatestVersion' }
| { kind: 'missingVersionDoc' }
| { kind: 'missingOwner' }
| { kind: "missingLatestVersion" }
| { kind: "missingVersionDoc" }
| { kind: "missingOwner" };
export type GitHubBackupSyncStats = {
skillsScanned: number
skillsSkipped: number
skillsBackedUp: number
skillsMissingVersion: number
skillsMissingOwner: number
errors: number
}
skillsScanned: number;
skillsSkipped: number;
skillsBackedUp: number;
skillsDeleted: number;
skillsMissingVersion: number;
skillsMissingOwner: number;
errors: number;
};
export type SyncGitHubBackupsInternalArgs = {
dryRun?: boolean
batchSize?: number
maxBatches?: number
}
dryRun?: boolean;
batchSize?: number;
maxBatches?: number;
pruneBatchSize?: number;
};
export type SyncGitHubBackupsInternalResult = {
stats: GitHubBackupSyncStats
cursor: string | null
isDone: boolean
}
stats: GitHubBackupSyncStats;
cursor: string | null;
pruneCursor: string | null;
isDone: boolean;
};
export const backupSkillForPublishInternal = internalAction({
args: {
@@ -62,7 +70,7 @@ export const backupSkillForPublishInternal = internalAction({
v.object({
path: v.string(),
size: v.number(),
storageId: v.id('_storage'),
storageId: v.id("_storage"),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
@@ -71,69 +79,77 @@ export const backupSkillForPublishInternal = internalAction({
},
handler: async (ctx, args) => {
if (!isGitHubBackupConfigured()) {
return { skipped: true as const }
return { skipped: true as const };
}
await backupSkillToGitHub(ctx, args)
return { skipped: false as const }
await backupSkillToGitHub(ctx, args);
return { skipped: false as const };
},
})
});
export async function syncGitHubBackupsInternalHandler(
ctx: ActionCtx,
args: SyncGitHubBackupsInternalArgs,
): Promise<SyncGitHubBackupsInternalResult> {
const dryRun = Boolean(args.dryRun)
const dryRun = Boolean(args.dryRun);
const stats: GitHubBackupSyncStats = {
skillsScanned: 0,
skillsSkipped: 0,
skillsBackedUp: 0,
skillsDeleted: 0,
skillsMissingVersion: 0,
skillsMissingOwner: 0,
errors: 0,
}
};
if (!isGitHubBackupConfigured()) {
return { stats, cursor: null, isDone: true }
return { stats, cursor: null, pruneCursor: null, isDone: true };
}
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const context = await getGitHubBackupContext()
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES);
const pruneBatchSize = clampInt(
args.pruneBatchSize ?? DEFAULT_PRUNE_BATCH_SIZE,
1,
MAX_PRUNE_BATCH_SIZE,
);
const context = await getGitHubBackupContext();
const state = dryRun
? { cursor: null as string | null }
? { cursor: null as string | null, pruneCursor: null as string | null }
: ((await ctx.runQuery(internal.githubBackups.getGitHubBackupSyncStateInternal, {})) as {
cursor: string | null
})
cursor: string | null;
pruneCursor: string | null;
});
let cursor: string | null = state.cursor
let isDone = false
let cursor: string | null = state.cursor;
let pruneCursor: string | null = state.pruneCursor;
let isDone = false;
for (let batch = 0; batch < maxBatches; batch++) {
const page = (await ctx.runQuery(internal.githubBackups.getGitHubBackupPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as { items: BackupPageItem[]; cursor: string | null; isDone: boolean }
})) as { items: BackupPageItem[]; cursor: string | null; isDone: boolean };
cursor = page.cursor
isDone = page.isDone
cursor = page.cursor;
isDone = page.isDone;
for (const item of page.items) {
if (item.kind !== 'ok') {
if (item.kind === 'missingLatestVersion' || item.kind === 'missingVersionDoc') {
stats.skillsMissingVersion += 1
} else if (item.kind === 'missingOwner') {
stats.skillsMissingOwner += 1
if (item.kind !== "ok") {
if (item.kind === "missingLatestVersion" || item.kind === "missingVersionDoc") {
stats.skillsMissingVersion += 1;
} else if (item.kind === "missingOwner") {
stats.skillsMissingOwner += 1;
}
continue
continue;
}
stats.skillsScanned += 1
stats.skillsScanned += 1;
try {
const meta = await fetchGitHubSkillMeta(context, item.ownerHandle, item.slug)
const meta = await fetchGitHubSkillMeta(context, item.ownerHandle, item.slug);
if (meta?.latest?.version === item.version) {
stats.skillsSkipped += 1
continue
stats.skillsSkipped += 1;
continue;
}
if (!dryRun) {
@@ -148,25 +164,128 @@ export async function syncGitHubBackupsInternalHandler(
publishedAt: item.publishedAt,
},
context,
)
stats.skillsBackedUp += 1
);
stats.skillsBackedUp += 1;
}
} catch (error) {
console.error('GitHub backup sync failed', error)
stats.errors += 1
console.error("GitHub backup sync failed", error);
stats.errors += 1;
}
}
if (!dryRun) {
await ctx.runMutation(internal.githubBackups.setGitHubBackupSyncStateInternal, {
cursor: isDone ? undefined : (cursor ?? undefined),
})
pruneCursor: pruneCursor ?? undefined,
});
}
if (isDone) break
if (isDone) break;
}
return { stats, cursor, isDone }
pruneCursor = await pruneDeletedSkillBackups(
ctx,
context,
dryRun,
stats,
pruneCursor,
pruneBatchSize,
);
if (!dryRun) {
await ctx.runMutation(internal.githubBackups.setGitHubBackupSyncStateInternal, {
cursor: isDone ? undefined : (cursor ?? undefined),
pruneCursor: pruneCursor ?? undefined,
});
}
return { stats, cursor, pruneCursor, isDone };
}
async function pruneDeletedSkillBackups(
ctx: ActionCtx,
context: Awaited<ReturnType<typeof getGitHubBackupContext>>,
dryRun: boolean,
stats: GitHubBackupSyncStats,
pruneCursor: string | null,
pruneBatchSize: number,
): Promise<string | null> {
let entries: Awaited<ReturnType<typeof listGitHubSkillBackupEntries>>;
try {
entries = await listGitHubSkillBackupEntries(context);
} catch (error) {
console.error("GitHub backup cleanup list failed", error);
stats.errors += 1;
return pruneCursor;
}
if (!entries.length) return null;
const sortedEntries = [...entries].sort((a, b) => a.rootPath.localeCompare(b.rootPath));
const startIndex =
pruneCursor == null
? 0
: sortedEntries.findIndex((entry) => entry.rootPath.localeCompare(pruneCursor) > 0);
if (startIndex === -1) return null;
const chunk = sortedEntries.slice(startIndex, startIndex + pruneBatchSize);
if (!chunk.length) return null;
let lastProcessed = pruneCursor;
for (const entry of chunk) {
lastProcessed = entry.rootPath;
try {
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: entry.slug,
})) as Doc<"skills"> | null;
if (!isMirrorEligibleSkill(skill)) {
await deleteBackupIfNeeded(context, entry, dryRun, stats);
continue;
}
const owner = (await ctx.runQuery(internal.users.getByIdInternal, {
userId: skill.ownerUserId,
})) as Doc<"users"> | null;
if (!owner || owner.deletedAt || owner.deactivatedAt) {
await deleteBackupIfNeeded(context, entry, dryRun, stats);
continue;
}
const ownerHandle = normalizeOwner(owner.handle ?? owner._id);
if (ownerHandle !== entry.owner) {
await deleteBackupIfNeeded(context, entry, dryRun, stats);
}
} catch (error) {
console.error("GitHub backup cleanup failed", error);
stats.errors += 1;
}
}
const reachedEnd = startIndex + chunk.length >= sortedEntries.length;
return reachedEnd ? null : (lastProcessed ?? null);
}
function isMirrorEligibleSkill(skill: Doc<"skills"> | null): skill is Doc<"skills"> {
if (!skill || skill.softDeletedAt) return false;
return (
skill.moderationStatus === undefined ||
skill.moderationStatus === null ||
skill.moderationStatus === "active"
);
}
async function deleteBackupIfNeeded(
context: Awaited<ReturnType<typeof getGitHubBackupContext>>,
entry: Awaited<ReturnType<typeof listGitHubSkillBackupEntries>>[number],
dryRun: boolean,
stats: GitHubBackupSyncStats,
) {
const result = dryRun
? { deleted: true as const }
: await deleteGitHubSkillBackup(context, entry.owner, entry.slug);
if (result.deleted) {
stats.skillsDeleted += 1;
}
}
export const syncGitHubBackupsInternal = internalAction({
@@ -174,10 +293,30 @@ export const syncGitHubBackupsInternal = internalAction({
dryRun: v.optional(v.boolean()),
batchSize: v.optional(v.number()),
maxBatches: v.optional(v.number()),
pruneBatchSize: v.optional(v.number()),
},
handler: syncGitHubBackupsInternalHandler,
})
});
export const deleteGitHubBackupForSlugInternal = internalAction({
args: {
ownerHandle: v.string(),
slug: v.string(),
dryRun: v.optional(v.boolean()),
},
handler: async (_ctx, args) => {
if (!isGitHubBackupConfigured()) {
return { skipped: true as const, deleted: false as const };
}
if (args.dryRun) {
return { skipped: false as const, deleted: true as const, dryRun: true as const };
}
const context = await getGitHubBackupContext();
const result = await deleteGitHubSkillBackup(context, args.ownerHandle, args.slug);
return { skipped: false as const, ...result };
},
});
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
return Math.max(min, Math.min(max, Math.floor(value)));
}
+8
View File
@@ -0,0 +1,8 @@
import { v } from "convex/values";
import { internalQuery } from "./functions";
import { getGitHubProviderAccountId } from "./lib/githubIdentity";
export const getGitHubProviderAccountIdInternal = internalQuery({
args: { userId: v.id("users") },
handler: async (ctx, args) => getGitHubProviderAccountId(ctx, args.userId),
});
+36
View File
@@ -0,0 +1,36 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
import { __test } from "./githubImport";
import { buildGitHubZipForTests } from "./lib/githubImport";
describe("githubImport", () => {
it("formats storage failure message with file context", () => {
const message = __test.buildStoreFailureMessage("skill/SKILL.md", 123, new Error("disk full"));
expect(message).toBe('Failed to store file "skill/SKILL.md" (123 bytes). disk full');
});
it("formats publish failure message with fallback text", () => {
expect(__test.buildPublishFailureMessage(new Error("slug exists"))).toBe(
"Import failed during publish: slug exists. Check skill format, slug availability, and try again.",
);
expect(__test.buildPublishFailureMessage("unexpected")).toBe(
"Import failed during publish: unexpected. Check skill format, slug availability, and try again.",
);
});
it("filters mac junk files while unzipping archive entries", () => {
const zip = buildGitHubZipForTests({
"demo-repo/skill/SKILL.md": "# Demo",
"demo-repo/skill/notes.md": "notes",
"demo-repo/skill/.DS_Store": "junk",
"demo-repo/skill/._notes.md": "junk",
"demo-repo/__MACOSX/._SKILL.md": "junk",
});
const entries = __test.unzipToEntries(zip);
expect(Object.keys(entries).sort()).toEqual([
"demo-repo/skill/SKILL.md",
"demo-repo/skill/notes.md",
]);
});
});
+188 -165
View File
@@ -1,11 +1,11 @@
import { ConvexError, v } from 'convex/values'
import { unzipSync } from 'fflate'
import semver from 'semver'
import { api, internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { action } from './_generated/server'
import { requireUserFromAction } from './lib/access'
import { ConvexError, v } from "convex/values";
import { unzipSync } from "fflate";
import semver from "semver";
import { api, internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { action } from "./functions";
import { requireUserFromAction } from "./lib/access";
import {
buildGitHubImportFileList,
computeDefaultSelectedPaths,
@@ -18,29 +18,29 @@ import {
stripGitHubZipRoot,
suggestDisplayName,
suggestVersion,
} from './lib/githubImport'
import { publishVersionForUser } from './lib/skillPublish'
import { sanitizePath } from './lib/skills'
} from "./lib/githubImport";
import { publishVersionForUser } from "./lib/skillPublish";
import { isMacJunkPath, sanitizePath } from "./lib/skills";
const MAX_SELECTED_BYTES = 50 * 1024 * 1024
const MAX_UNZIPPED_BYTES = 80 * 1024 * 1024
const MAX_FILE_COUNT = 7_500
const MAX_SINGLE_FILE_BYTES = 10 * 1024 * 1024
const MAX_SELECTED_BYTES = 50 * 1024 * 1024;
const MAX_UNZIPPED_BYTES = 80 * 1024 * 1024;
const MAX_FILE_COUNT = 7_500;
const MAX_SINGLE_FILE_BYTES = 10 * 1024 * 1024;
export const previewGitHubImport = action({
args: { url: v.string() },
handler: async (ctx, args) => {
await requireUserFromAction(ctx)
await requireUserFromAction(ctx);
const parsed = parseGitHubImportUrl(args.url)
const resolved = await resolveGitHubCommit(parsed, fetch)
const zipBytes = await fetchGitHubZipBytes(resolved, fetch)
const entries = unzipToEntries(zipBytes)
const stripped = stripGitHubZipRoot(entries)
const parsed = parseGitHubImportUrl(args.url);
const resolved = await resolveGitHubCommit(parsed, fetch);
const zipBytes = await fetchGitHubZipBytes(resolved, fetch);
const entries = unzipToEntries(zipBytes);
const stripped = stripGitHubZipRoot(entries);
const candidates = detectGitHubImportCandidates(stripped).filter((candidate) =>
isCandidateUnderResolvedPath(candidate.path, resolved.path),
)
if (candidates.length === 0) throw new ConvexError('No SKILL.md found in this repo')
);
if (candidates.length === 0) throw new ConvexError("No SKILL.md found in this repo");
return {
resolved,
@@ -50,53 +50,53 @@ export const previewGitHubImport = action({
name: candidate.name ?? null,
description: candidate.description ?? null,
})),
}
};
},
})
});
export const previewGitHubImportCandidate = action({
args: { url: v.string(), candidatePath: v.string() },
handler: async (ctx, args) => {
const { userId } = await requireUserFromAction(ctx)
const { userId } = await requireUserFromAction(ctx);
const parsed = parseGitHubImportUrl(args.url)
const resolved = await resolveGitHubCommit(parsed, fetch)
const zipBytes = await fetchGitHubZipBytes(resolved, fetch)
const entries = unzipToEntries(zipBytes)
const stripped = stripGitHubZipRoot(entries)
const parsed = parseGitHubImportUrl(args.url);
const resolved = await resolveGitHubCommit(parsed, fetch);
const zipBytes = await fetchGitHubZipBytes(resolved, fetch);
const entries = unzipToEntries(zipBytes);
const stripped = stripGitHubZipRoot(entries);
const normalizedCandidatePath = normalizeRepoPath(args.candidatePath)
const normalizedCandidatePath = normalizeRepoPath(args.candidatePath);
if (!isCandidateUnderResolvedPath(normalizedCandidatePath, resolved.path)) {
throw new ConvexError('Candidate path is outside the requested import scope')
throw new ConvexError("Candidate path is outside the requested import scope");
}
const candidates = detectGitHubImportCandidates(stripped).filter((candidate) =>
isCandidateUnderResolvedPath(candidate.path, resolved.path),
)
);
const candidate = candidates.find((item) => item.path === normalizedCandidatePath)
if (!candidate) throw new ConvexError('Candidate not found')
const candidate = candidates.find((item) => item.path === normalizedCandidatePath);
if (!candidate) throw new ConvexError("Candidate not found");
const files = listTextFilesUnderCandidate(stripped, candidate.path)
const defaultSelectedPaths = computeDefaultSelectedPaths({ candidate, files })
const files = listTextFilesUnderCandidate(stripped, candidate.path);
const defaultSelectedPaths = computeDefaultSelectedPaths({ candidate, files });
const fileList = buildGitHubImportFileList({
candidate,
files,
defaultSelectedPaths,
})
});
const baseForNaming = candidate.path ? (candidate.path.split('/').at(-1) ?? '') : resolved.repo
const suggestedDisplayName = suggestDisplayName(candidate, baseForNaming)
const baseForNaming = candidate.path ? (candidate.path.split("/").at(-1) ?? "") : resolved.repo;
const suggestedDisplayName = suggestDisplayName(candidate, baseForNaming);
const rawSlugBase = sanitizeSlug(candidate.path ? baseForNaming : resolved.repo)
const suggestedSlug = await suggestAvailableSlug(ctx, userId, rawSlugBase)
const rawSlugBase = sanitizeSlug(candidate.path ? baseForNaming : resolved.repo);
const suggestedSlug = await suggestAvailableSlug(ctx, userId, rawSlugBase);
const existing = await ctx.runQuery(api.skills.getBySlug, { slug: suggestedSlug })
const existing = await ctx.runQuery(api.skills.getBySlug, { slug: suggestedSlug });
const existingLatest =
existing?.skill && existing.skill.ownerUserId === userId
? (existing.latestVersion?.version ?? null)
: null
const suggestedVersion = suggestVersion(existingLatest)
: null;
const suggestedVersion = suggestVersion(existingLatest);
return {
resolved,
@@ -111,12 +111,12 @@ export const previewGitHubImportCandidate = action({
slug: suggestedSlug,
displayName: suggestedDisplayName,
version: suggestedVersion,
tags: ['latest'],
tags: ["latest"],
},
files: fileList,
}
};
},
})
});
export const importGitHubSkill = action({
args: {
@@ -130,188 +130,211 @@ export const importGitHubSkill = action({
tags: v.optional(v.array(v.string())),
},
handler: async (ctx, args) => {
const { userId } = await requireUserFromAction(ctx)
const { userId } = await requireUserFromAction(ctx);
const parsed = parseGitHubImportUrl(args.url)
const resolved = await resolveGitHubCommit(parsed, fetch)
if (!/^[a-f0-9]{40}$/i.test(args.commit)) throw new ConvexError('Invalid commit')
const parsed = parseGitHubImportUrl(args.url);
const resolved = await resolveGitHubCommit(parsed, fetch);
if (!/^[a-f0-9]{40}$/i.test(args.commit)) throw new ConvexError("Invalid commit");
if (args.commit.toLowerCase() !== resolved.commit.toLowerCase()) {
throw new ConvexError('Import is out of date. Re-run preview.')
throw new ConvexError("Import is out of date. Re-run preview.");
}
const normalizedCandidatePath = normalizeRepoPath(args.candidatePath)
const normalizedCandidatePath = normalizeRepoPath(args.candidatePath);
if (!isCandidateUnderResolvedPath(normalizedCandidatePath, resolved.path)) {
throw new ConvexError('Candidate path is outside the requested import scope')
throw new ConvexError("Candidate path is outside the requested import scope");
}
const zipBytes = await fetchGitHubZipBytes(resolved, fetch)
const entries = stripGitHubZipRoot(unzipToEntries(zipBytes))
const zipBytes = await fetchGitHubZipBytes(resolved, fetch);
const entries = stripGitHubZipRoot(unzipToEntries(zipBytes));
const candidates = detectGitHubImportCandidates(entries).filter((candidate) =>
isCandidateUnderResolvedPath(candidate.path, resolved.path),
)
const candidate = candidates.find((item) => item.path === normalizedCandidatePath)
if (!candidate) throw new ConvexError('Candidate not found')
);
const candidate = candidates.find((item) => item.path === normalizedCandidatePath);
if (!candidate) throw new ConvexError("Candidate not found");
const filesUnderCandidate = listTextFilesUnderCandidate(entries, candidate.path)
const byPath = new Map(filesUnderCandidate.map((file) => [file.path, file.bytes]))
const filesUnderCandidate = listTextFilesUnderCandidate(entries, candidate.path);
const byPath = new Map(filesUnderCandidate.map((file) => [file.path, file.bytes]));
const selected = Array.from(
new Set(args.selectedPaths.map((path) => normalizeRepoPath(path)).filter(Boolean)),
)
if (selected.length === 0) throw new ConvexError('No files selected')
);
if (selected.length === 0) throw new ConvexError("No files selected");
const candidateRoot = candidate.path ? `${candidate.path}/` : ''
const normalizedReadmePath = normalizeRepoPath(candidate.readmePath)
const candidateRoot = candidate.path ? `${candidate.path}/` : "";
const normalizedReadmePath = normalizeRepoPath(candidate.readmePath);
if (!selected.includes(normalizedReadmePath)) {
throw new ConvexError('SKILL.md must be selected')
throw new ConvexError("SKILL.md must be selected");
}
let totalBytes = 0
let totalBytes = 0;
const storedFiles: Array<{
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}> = []
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
}> = [];
for (const path of selected.sort()) {
if (candidateRoot && !path.startsWith(candidateRoot)) {
throw new ConvexError('Selected file is outside the chosen skill folder')
throw new ConvexError("Selected file is outside the chosen skill folder");
}
const bytes = byPath.get(path)
if (!bytes) continue
totalBytes += bytes.byteLength
if (totalBytes > MAX_SELECTED_BYTES) throw new ConvexError('Selected files exceed 50MB limit')
const bytes = byPath.get(path);
if (!bytes) continue;
totalBytes += bytes.byteLength;
if (totalBytes > MAX_SELECTED_BYTES)
throw new ConvexError("Selected files exceed 50MB limit");
const relPath = candidateRoot ? path.slice(candidateRoot.length) : path
const sanitized = sanitizePath(relPath)
if (!sanitized) throw new ConvexError('Invalid file paths')
const relPath = candidateRoot ? path.slice(candidateRoot.length) : path;
const sanitized = sanitizePath(relPath);
if (!sanitized) throw new ConvexError("Invalid file paths");
const sha256 = await sha256Hex(bytes)
const safeBytes = new Uint8Array(bytes)
const storageId = await ctx.storage.store(new Blob([safeBytes], { type: 'text/plain' }))
const sha256 = await sha256Hex(bytes);
const safeBytes = new Uint8Array(bytes);
let storageId: Id<"_storage">;
try {
storageId = await ctx.storage.store(new Blob([safeBytes], { type: "text/plain" }));
} catch (error) {
throw new ConvexError(buildStoreFailureMessage(sanitized, bytes.byteLength, error));
}
storedFiles.push({
path: sanitized,
size: bytes.byteLength,
storageId,
sha256,
contentType: 'text/plain',
})
contentType: "text/plain",
});
}
if (storedFiles.length === 0) throw new ConvexError('No files selected')
if (storedFiles.length === 0) throw new ConvexError("No files selected");
const slugBase = (args.slug ?? '').trim().toLowerCase()
const displayName = (args.displayName ?? '').trim()
const tags = (args.tags ?? ['latest']).map((tag) => tag.trim()).filter(Boolean)
const version = (args.version ?? '').trim()
const slugBase = (args.slug ?? "").trim().toLowerCase();
const displayName = (args.displayName ?? "").trim();
const tags = (args.tags ?? ["latest"]).map((tag) => tag.trim()).filter(Boolean);
const version = (args.version ?? "").trim();
if (!slugBase) throw new ConvexError('Slug required')
if (!displayName) throw new ConvexError('Display name required')
if (!version || !semver.valid(version)) throw new ConvexError('Version must be valid semver')
if (!slugBase) throw new ConvexError("Slug required");
if (!displayName) throw new ConvexError("Display name required");
if (!version || !semver.valid(version)) throw new ConvexError("Version must be valid semver");
const result = await publishVersionForUser(ctx, userId, {
slug: slugBase,
displayName,
version,
changelog: '',
tags,
files: storedFiles,
source: {
kind: 'github',
url: resolved.originalUrl,
repo: `${resolved.owner}/${resolved.repo}`,
ref: resolved.ref,
commit: resolved.commit,
path: candidate.path,
importedAt: Date.now(),
},
})
let result: Awaited<ReturnType<typeof publishVersionForUser>>;
try {
result = await publishVersionForUser(ctx, userId, {
slug: slugBase,
displayName,
version,
changelog: "",
tags,
files: storedFiles,
source: {
kind: "github",
url: resolved.originalUrl,
repo: `${resolved.owner}/${resolved.repo}`,
ref: resolved.ref,
commit: resolved.commit,
path: candidate.path,
importedAt: Date.now(),
},
});
} catch (error) {
throw new ConvexError(buildPublishFailureMessage(error));
}
return { ok: true, slug: slugBase, version, ...result }
return { ok: true, slug: slugBase, version, ...result };
},
})
});
function unzipToEntries(zipBytes: Uint8Array) {
const entries = unzipSync(zipBytes)
const out: Record<string, Uint8Array> = {}
const rawPaths = Object.keys(entries)
if (rawPaths.length > MAX_FILE_COUNT) throw new ConvexError('Repo archive has too many files')
let totalBytes = 0
const entries = unzipSync(zipBytes);
const out: Record<string, Uint8Array> = {};
const rawPaths = Object.keys(entries);
if (rawPaths.length > MAX_FILE_COUNT) throw new ConvexError("Repo archive has too many files");
let totalBytes = 0;
for (const [rawPath, bytes] of Object.entries(entries)) {
const normalizedPath = normalizeZipPath(rawPath)
if (!normalizedPath) continue
if (isJunkPath(normalizedPath)) continue
if (!bytes) continue
if (bytes.byteLength > MAX_SINGLE_FILE_BYTES) continue
totalBytes += bytes.byteLength
if (totalBytes > MAX_UNZIPPED_BYTES) throw new ConvexError('Repo archive is too large')
out[normalizedPath] = bytes
const normalizedPath = normalizeZipPath(rawPath);
if (!normalizedPath) continue;
if (isMacJunkPath(normalizedPath)) continue;
if (!bytes) continue;
if (bytes.byteLength > MAX_SINGLE_FILE_BYTES) continue;
totalBytes += bytes.byteLength;
if (totalBytes > MAX_UNZIPPED_BYTES) throw new ConvexError("Repo archive is too large");
out[normalizedPath] = bytes;
}
return out
return out;
}
function isCandidateUnderResolvedPath(candidatePath: string, resolvedPath: string) {
const root = normalizeRepoPath(resolvedPath)
if (!root) return true
if (!candidatePath) return false
if (candidatePath === root) return true
return candidatePath.startsWith(`${root}/`)
const root = normalizeRepoPath(resolvedPath);
if (!root) return true;
if (!candidatePath) return false;
if (candidatePath === root) return true;
return candidatePath.startsWith(`${root}/`);
}
function sanitizeSlug(value: string) {
return value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]+/g, '-')
.replace(/^-+/, '')
.replace(/-+$/, '')
.replace(/--+/g, '-')
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+/, "")
.replace(/-+$/, "")
.replace(/--+/g, "-");
}
async function suggestAvailableSlug(ctx: ActionCtx, userId: Id<'users'>, base: string) {
const cleaned = sanitizeSlug(base)
if (!cleaned) throw new ConvexError('Could not derive slug')
async function suggestAvailableSlug(ctx: ActionCtx, userId: Id<"users">, base: string) {
const cleaned = sanitizeSlug(base);
if (!cleaned) throw new ConvexError("Could not derive slug");
for (let i = 0; i < 50; i += 1) {
const candidate = i === 0 ? cleaned : `${cleaned}-${i + 1}`
const existing = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug: candidate })
if (!existing) return candidate
if (existing.ownerUserId === userId) return candidate
const candidate = i === 0 ? cleaned : `${cleaned}-${i + 1}`;
const existing = await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: candidate,
});
if (!existing) return candidate;
if (existing.ownerUserId === userId) return candidate;
}
throw new ConvexError('Could not find an available slug')
throw new ConvexError("Could not find an available slug");
}
async function sha256Hex(bytes: Uint8Array) {
const normalized = new Uint8Array(bytes)
const digest = await crypto.subtle.digest('SHA-256', normalized.buffer)
return toHex(new Uint8Array(digest))
const normalized = new Uint8Array(bytes);
const digest = await crypto.subtle.digest("SHA-256", normalized.buffer);
return toHex(new Uint8Array(digest));
}
function toHex(bytes: Uint8Array) {
let out = ''
for (const byte of bytes) out += byte.toString(16).padStart(2, '0')
return out
let out = "";
for (const byte of bytes) out += byte.toString(16).padStart(2, "0");
return out;
}
function normalizeZipPath(path: string) {
const normalized = path
.replaceAll('\u0000', '')
.replaceAll('\\', '/')
.replaceAll("\u0000", "")
.replaceAll("\\", "/")
.trim()
.replace(/^\.\/+/, '')
.replace(/^\/+/, '')
if (!normalized) return ''
if (normalized.includes('..')) return ''
return normalized
.replace(/^\.\/+/, "")
.replace(/^\/+/, "");
if (!normalized) return "";
if (normalized.includes("..")) return "";
return normalized;
}
function isJunkPath(path: string) {
const normalized = path.toLowerCase()
if (normalized.startsWith('__macosx/')) return true
if (normalized.endsWith('/.ds_store')) return true
if (normalized === '.ds_store') return true
return false
function toErrorMessage(error: unknown) {
return error instanceof Error ? error.message : String(error);
}
function buildStoreFailureMessage(path: string, sizeBytes: number, error: unknown) {
return `Failed to store file "${path}" (${sizeBytes} bytes). ${toErrorMessage(error)}`;
}
function buildPublishFailureMessage(error: unknown) {
return `Import failed during publish: ${toErrorMessage(error)}. Check skill format, slug availability, and try again.`;
}
export const __test = {
buildPublishFailureMessage,
buildStoreFailureMessage,
unzipToEntries,
};
+225
View File
@@ -0,0 +1,225 @@
"use node";
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc, Id } from "./_generated/dataModel";
import { internalAction } from "./functions";
import { assertAdmin } from "./lib/access";
import { guessContentTypeForPath } from "./lib/contentTypes";
import {
fetchGitHubSkillMeta,
getGitHubBackupContext,
isGitHubBackupConfigured,
} from "./lib/githubBackup";
import { listGitHubBackupFiles, readGitHubBackupFile } from "./lib/githubRestoreHelpers";
import { publishVersionForUser } from "./lib/skillPublish";
type RestoreResult = {
slug: string;
status: "restored" | "slug_conflict" | "already_exists" | "no_backup" | "error";
detail?: string;
};
type BulkRestoreResult = {
results: RestoreResult[];
totalRestored: number;
totalConflicts: number;
totalSkipped: number;
totalErrors: number;
};
/**
* Admin-only: restore a single skill from GitHub backup.
* Reads the backup files from the GitHub repo and re-creates the skill in the database.
*/
export const restoreSkillFromBackup = internalAction({
args: {
actorUserId: v.id("users"),
ownerHandle: v.string(),
ownerUserId: v.id("users"),
slug: v.string(),
forceOverwriteSquatter: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<RestoreResult> => {
try {
const actor = await ctx.runQuery(internal.users.getByIdInternal, {
userId: args.actorUserId,
});
if (!actor || actor.deletedAt || actor.deactivatedAt) {
return { slug: args.slug, status: "error", detail: "Actor not found" };
}
assertAdmin(actor as Doc<"users">);
if (!isGitHubBackupConfigured()) {
return { slug: args.slug, status: "error", detail: "GitHub backup not configured" };
}
const ghContext = await getGitHubBackupContext();
// Check if skill already exists in the DB
const existingSkill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: args.slug,
})) as Doc<"skills"> | null;
if (existingSkill) {
if (existingSkill.ownerUserId === args.ownerUserId) {
return {
slug: args.slug,
status: "already_exists",
detail: "Skill already owned by user",
};
}
if (!args.forceOverwriteSquatter) {
return {
slug: args.slug,
status: "slug_conflict",
detail: `Slug occupied by another user. Set forceOverwriteSquatter=true to reclaim.`,
};
}
// Free the slug in-transaction by renaming the squatter, then enqueue cleanup.
await ctx.runMutation(
internal.githubRestoreMutations.evictSquatterSkillForRestoreInternal,
{
actorUserId: args.actorUserId,
slug: args.slug,
rightfulOwnerUserId: args.ownerUserId,
},
);
}
// Fetch metadata from GitHub backup
const meta = await fetchGitHubSkillMeta(ghContext, args.ownerHandle, args.slug);
if (!meta) {
return { slug: args.slug, status: "no_backup", detail: "No backup found in GitHub repo" };
}
// Read the actual files from the backup
const backupFiles = await listGitHubBackupFiles(ghContext, args.ownerHandle, args.slug);
if (backupFiles.length === 0) {
return { slug: args.slug, status: "no_backup", detail: "Backup has no files" };
}
// Download and store each file in Convex storage
const storedFiles: Array<{
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType: string;
}> = [];
for (const filePath of backupFiles) {
const fileContent = await readGitHubBackupFile(
ghContext,
args.ownerHandle,
args.slug,
filePath,
);
if (!fileContent) continue;
const sha256 = await sha256Hex(fileContent);
const contentType = guessContentTypeForPath(filePath);
const blob = new Blob([Buffer.from(fileContent)], { type: contentType });
const storageId = await ctx.storage.store(blob);
storedFiles.push({
path: filePath,
size: fileContent.byteLength,
storageId,
sha256,
contentType,
});
}
if (storedFiles.length === 0) {
return { slug: args.slug, status: "error", detail: "Could not download any backup files" };
}
await publishVersionForUser(
ctx,
args.ownerUserId,
{
slug: args.slug,
displayName: meta.displayName,
version: meta.latest.version,
changelog: "Restored from GitHub backup",
files: storedFiles,
},
{
bypassGitHubAccountAge: true,
bypassNewSkillRateLimit: true,
bypassQualityGate: true,
skipBackup: true,
skipWebhook: true,
},
);
return { slug: args.slug, status: "restored" };
} catch (error) {
const message = error instanceof Error ? error.message : "Unknown error";
console.error(`[restore] Failed to restore ${args.slug}:`, message);
return { slug: args.slug, status: "error", detail: message };
}
},
});
/**
* Admin-only: bulk restore all skills for a user from GitHub backup.
*/
export const restoreUserSkillsFromBackup = internalAction({
args: {
actorUserId: v.id("users"),
ownerHandle: v.string(),
ownerUserId: v.id("users"),
slugs: v.array(v.string()),
forceOverwriteSquatter: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<BulkRestoreResult> => {
const results: RestoreResult[] = [];
let totalRestored = 0;
let totalConflicts = 0;
let totalSkipped = 0;
let totalErrors = 0;
for (const slug of args.slugs) {
const result = (await ctx.runAction(internal.githubRestore.restoreSkillFromBackup, {
actorUserId: args.actorUserId,
ownerHandle: args.ownerHandle,
ownerUserId: args.ownerUserId,
slug,
forceOverwriteSquatter: args.forceOverwriteSquatter,
})) as RestoreResult;
results.push(result);
switch (result.status) {
case "restored":
totalRestored += 1;
break;
case "slug_conflict":
totalConflicts += 1;
break;
case "already_exists":
case "no_backup":
totalSkipped += 1;
break;
case "error":
totalErrors += 1;
break;
}
}
return { results, totalRestored, totalConflicts, totalSkipped, totalErrors };
},
});
async function sha256Hex(bytes: Uint8Array) {
const { createHash } = await import("node:crypto");
const hash = createHash("sha256");
hash.update(bytes);
return hash.digest("hex");
}
// guessContentTypeForPath in lib/contentTypes.ts
+83
View File
@@ -0,0 +1,83 @@
import { v } from "convex/values";
import { internal } from "./_generated/api";
import { internalMutation } from "./functions";
import { assertAdmin } from "./lib/access";
export const evictSquatterSkillForRestoreInternal = internalMutation({
args: {
actorUserId: v.id("users"),
slug: v.string(),
rightfulOwnerUserId: v.id("users"),
},
handler: async (ctx, args) => {
const actor = await ctx.db.get(args.actorUserId);
if (!actor || actor.deletedAt || actor.deactivatedAt) throw new Error("Actor not found");
assertAdmin(actor);
const slug = args.slug.trim().toLowerCase();
if (!slug) throw new Error("Slug required");
const now = Date.now();
const existingSkill = await ctx.db
.query("skills")
.withIndex("by_slug", (q) => q.eq("slug", slug))
.unique();
if (!existingSkill) return { ok: true as const, action: "noop" as const };
if (existingSkill.ownerUserId === args.rightfulOwnerUserId) {
return { ok: true as const, action: "already_owned" as const };
}
const evictedSlug = buildEvictedSlug(slug, now);
// Free the slug immediately (same transaction) by renaming the squatter's skill.
await ctx.db.patch(existingSkill._id, {
slug: evictedSlug,
softDeletedAt: now,
hiddenAt: existingSkill.hiddenAt ?? now,
hiddenBy: existingSkill.hiddenBy ?? actor._id,
updatedAt: now,
});
// Remove from vector search ASAP.
const embeddings = await ctx.db
.query("skillEmbeddings")
.withIndex("by_skill", (q) => q.eq("skillId", existingSkill._id))
.collect();
for (const embedding of embeddings) {
await ctx.db.patch(embedding._id, {
visibility: "deleted",
updatedAt: now,
});
}
// Cleanup the rest asynchronously (versions, fingerprints, installs, etc.)
await ctx.scheduler.runAfter(0, internal.skills.hardDeleteInternal, {
skillId: existingSkill._id,
actorUserId: actor._id,
phase: "versions",
});
await ctx.db.insert("auditLogs", {
actorUserId: actor._id,
action: "slug.reclaim.sync",
targetType: "skill",
targetId: existingSkill._id,
metadata: {
slug,
evictedSlug,
squatterUserId: existingSkill.ownerUserId,
rightfulOwnerUserId: args.rightfulOwnerUserId,
reason: "Synchronous eviction during GitHub restore",
},
createdAt: now,
});
return { ok: true as const, action: "evicted" as const, evictedSlug };
},
});
function buildEvictedSlug(slug: string, now: number) {
const suffix = now.toString(36);
return `${slug}-evicted-${suffix}`;
}
+77 -77
View File
@@ -1,51 +1,51 @@
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc, Id } from './_generated/dataModel'
import { action, internalMutation, internalQuery } from './_generated/server'
import { assertRole, requireUserFromAction } from './lib/access'
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc, Id } from "./_generated/dataModel";
import { action, internalMutation, internalQuery } from "./functions";
import { assertRole, requireUserFromAction } from "./lib/access";
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const SYNC_STATE_KEY = 'souls'
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
const SYNC_STATE_KEY = "souls";
type BackupPageItem =
| {
kind: 'ok'
soulId: Id<'souls'>
versionId: Id<'soulVersions'>
slug: string
displayName: string
version: string
ownerHandle: string
files: Doc<'soulVersions'>['files']
publishedAt: number
kind: "ok";
soulId: Id<"souls">;
versionId: Id<"soulVersions">;
slug: string;
displayName: string;
version: string;
ownerHandle: string;
files: Doc<"soulVersions">["files"];
publishedAt: number;
}
| { kind: 'missingLatestVersion'; soulId: Id<'souls'> }
| { kind: 'missingVersionDoc'; soulId: Id<'souls'>; versionId: Id<'soulVersions'> }
| { kind: 'missingOwner'; soulId: Id<'souls'>; ownerUserId: Id<'users'> }
| { kind: "missingLatestVersion"; soulId: Id<"souls"> }
| { kind: "missingVersionDoc"; soulId: Id<"souls">; versionId: Id<"soulVersions"> }
| { kind: "missingOwner"; soulId: Id<"souls">; ownerUserId: Id<"users"> };
type BackupPageResult = {
items: BackupPageItem[]
cursor: string | null
isDone: boolean
}
items: BackupPageItem[];
cursor: string | null;
isDone: boolean;
};
type BackupSyncState = {
cursor: string | null
}
cursor: string | null;
};
export type SyncGitHubSoulBackupsResult = {
stats: {
soulsScanned: number
soulsSkipped: number
soulsBackedUp: number
soulsMissingVersion: number
soulsMissingOwner: number
errors: number
}
cursor: string | null
isDone: boolean
}
soulsScanned: number;
soulsSkipped: number;
soulsBackedUp: number;
soulsMissingVersion: number;
soulsMissingOwner: number;
errors: number;
};
cursor: string | null;
isDone: boolean;
};
export const getGitHubSoulBackupPageInternal = internalQuery({
args: {
@@ -53,38 +53,38 @@ export const getGitHubSoulBackupPageInternal = internalQuery({
batchSize: v.optional(v.number()),
},
handler: async (ctx, args): Promise<BackupPageResult> => {
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const { page, isDone, continueCursor } = await ctx.db
.query('souls')
.order('asc')
.paginate({ cursor: args.cursor ?? null, numItems: batchSize })
.query("souls")
.order("asc")
.paginate({ cursor: args.cursor ?? null, numItems: batchSize });
const items: BackupPageItem[] = []
const items: BackupPageItem[] = [];
for (const soul of page) {
if (soul.softDeletedAt) continue
if (soul.softDeletedAt) continue;
if (!soul.latestVersionId) {
items.push({ kind: 'missingLatestVersion', soulId: soul._id })
continue
items.push({ kind: "missingLatestVersion", soulId: soul._id });
continue;
}
const version = await ctx.db.get(soul.latestVersionId)
const version = await ctx.db.get(soul.latestVersionId);
if (!version) {
items.push({
kind: 'missingVersionDoc',
kind: "missingVersionDoc",
soulId: soul._id,
versionId: soul.latestVersionId,
})
continue
});
continue;
}
const owner = await ctx.db.get(soul.ownerUserId)
if (!owner || owner.deletedAt) {
items.push({ kind: 'missingOwner', soulId: soul._id, ownerUserId: soul.ownerUserId })
continue
const owner = await ctx.db.get(soul.ownerUserId);
if (!owner || owner.deletedAt || owner.deactivatedAt) {
items.push({ kind: "missingOwner", soulId: soul._id, ownerUserId: soul.ownerUserId });
continue;
}
items.push({
kind: 'ok',
kind: "ok",
soulId: soul._id,
versionId: version._id,
slug: soul.slug,
@@ -93,52 +93,52 @@ export const getGitHubSoulBackupPageInternal = internalQuery({
ownerHandle: owner.handle ?? owner._id,
files: version.files,
publishedAt: version.createdAt,
})
});
}
return { items, cursor: continueCursor, isDone }
return { items, cursor: continueCursor, isDone };
},
})
});
export const getGitHubSoulBackupSyncStateInternal = internalQuery({
args: {},
handler: async (ctx): Promise<BackupSyncState> => {
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
return { cursor: state?.cursor ?? null }
.query("githubBackupSyncState")
.withIndex("by_key", (q) => q.eq("key", SYNC_STATE_KEY))
.unique();
return { cursor: state?.cursor ?? null };
},
})
});
export const setGitHubSoulBackupSyncStateInternal = internalMutation({
args: {
cursor: v.optional(v.string()),
},
handler: async (ctx, args) => {
const now = Date.now()
const now = Date.now();
const state = await ctx.db
.query('githubBackupSyncState')
.withIndex('by_key', (q) => q.eq('key', SYNC_STATE_KEY))
.unique()
.query("githubBackupSyncState")
.withIndex("by_key", (q) => q.eq("key", SYNC_STATE_KEY))
.unique();
if (!state) {
await ctx.db.insert('githubBackupSyncState', {
await ctx.db.insert("githubBackupSyncState", {
key: SYNC_STATE_KEY,
cursor: args.cursor,
updatedAt: now,
})
return { ok: true as const }
});
return { ok: true as const };
}
await ctx.db.patch(state._id, {
cursor: args.cursor,
updatedAt: now,
})
});
return { ok: true as const }
return { ok: true as const };
},
})
});
export const syncGitHubSoulBackups: ReturnType<typeof action> = action({
args: {
@@ -148,23 +148,23 @@ export const syncGitHubSoulBackups: ReturnType<typeof action> = action({
resetCursor: v.optional(v.boolean()),
},
handler: async (ctx, args): Promise<SyncGitHubSoulBackupsResult> => {
const { user } = await requireUserFromAction(ctx)
assertRole(user, ['admin'])
const { user } = await requireUserFromAction(ctx);
assertRole(user, ["admin"]);
if (args.resetCursor && !args.dryRun) {
await ctx.runMutation(internal.githubSoulBackups.setGitHubSoulBackupSyncStateInternal, {
cursor: undefined,
})
});
}
return ctx.runAction(internal.githubSoulBackupsNode.syncGitHubSoulBackupsInternal, {
dryRun: args.dryRun,
batchSize: args.batchSize,
maxBatches: args.maxBatches,
}) as Promise<SyncGitHubSoulBackupsResult>
}) as Promise<SyncGitHubSoulBackupsResult>;
},
})
});
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
return Math.max(min, Math.min(max, Math.floor(value)));
}
+73 -73
View File
@@ -1,56 +1,56 @@
'use node'
"use node";
import { v } from 'convex/values'
import { internal } from './_generated/api'
import type { Doc } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { internalAction } from './_generated/server'
import { v } from "convex/values";
import { internal } from "./_generated/api";
import type { Doc } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { internalAction } from "./functions";
import {
backupSoulToGitHub,
fetchGitHubSoulMeta,
getGitHubSoulBackupContext,
isGitHubSoulBackupConfigured,
} from './lib/githubSoulBackup'
} from "./lib/githubSoulBackup";
const DEFAULT_BATCH_SIZE = 50
const MAX_BATCH_SIZE = 200
const DEFAULT_MAX_BATCHES = 5
const MAX_MAX_BATCHES = 200
const DEFAULT_BATCH_SIZE = 50;
const MAX_BATCH_SIZE = 200;
const DEFAULT_MAX_BATCHES = 5;
const MAX_MAX_BATCHES = 200;
type BackupPageItem =
| {
kind: 'ok'
slug: string
version: string
displayName: string
ownerHandle: string
files: Doc<'soulVersions'>['files']
publishedAt: number
kind: "ok";
slug: string;
version: string;
displayName: string;
ownerHandle: string;
files: Doc<"soulVersions">["files"];
publishedAt: number;
}
| { kind: 'missingLatestVersion' }
| { kind: 'missingVersionDoc' }
| { kind: 'missingOwner' }
| { kind: "missingLatestVersion" }
| { kind: "missingVersionDoc" }
| { kind: "missingOwner" };
export type GitHubSoulBackupSyncStats = {
soulsScanned: number
soulsSkipped: number
soulsBackedUp: number
soulsMissingVersion: number
soulsMissingOwner: number
errors: number
}
soulsScanned: number;
soulsSkipped: number;
soulsBackedUp: number;
soulsMissingVersion: number;
soulsMissingOwner: number;
errors: number;
};
export type SyncGitHubSoulBackupsInternalArgs = {
dryRun?: boolean
batchSize?: number
maxBatches?: number
}
dryRun?: boolean;
batchSize?: number;
maxBatches?: number;
};
export type SyncGitHubSoulBackupsInternalResult = {
stats: GitHubSoulBackupSyncStats
cursor: string | null
isDone: boolean
}
stats: GitHubSoulBackupSyncStats;
cursor: string | null;
isDone: boolean;
};
export const backupSoulForPublishInternal = internalAction({
args: {
@@ -62,7 +62,7 @@ export const backupSoulForPublishInternal = internalAction({
v.object({
path: v.string(),
size: v.number(),
storageId: v.id('_storage'),
storageId: v.id("_storage"),
sha256: v.string(),
contentType: v.optional(v.string()),
}),
@@ -71,18 +71,18 @@ export const backupSoulForPublishInternal = internalAction({
},
handler: async (ctx, args) => {
if (!isGitHubSoulBackupConfigured()) {
return { skipped: true as const }
return { skipped: true as const };
}
await backupSoulToGitHub(ctx, args)
return { skipped: false as const }
await backupSoulToGitHub(ctx, args);
return { skipped: false as const };
},
})
});
export async function syncGitHubSoulBackupsInternalHandler(
ctx: ActionCtx,
args: SyncGitHubSoulBackupsInternalArgs,
): Promise<SyncGitHubSoulBackupsInternalResult> {
const dryRun = Boolean(args.dryRun)
const dryRun = Boolean(args.dryRun);
const stats: GitHubSoulBackupSyncStats = {
soulsScanned: 0,
soulsSkipped: 0,
@@ -90,15 +90,15 @@ export async function syncGitHubSoulBackupsInternalHandler(
soulsMissingVersion: 0,
soulsMissingOwner: 0,
errors: 0,
}
};
if (!isGitHubSoulBackupConfigured()) {
return { stats, cursor: null, isDone: true }
return { stats, cursor: null, isDone: true };
}
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE)
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES)
const context = await getGitHubSoulBackupContext()
const batchSize = clampInt(args.batchSize ?? DEFAULT_BATCH_SIZE, 1, MAX_BATCH_SIZE);
const maxBatches = clampInt(args.maxBatches ?? DEFAULT_MAX_BATCHES, 1, MAX_MAX_BATCHES);
const context = await getGitHubSoulBackupContext();
const state = dryRun
? { cursor: null as string | null }
@@ -106,37 +106,37 @@ export async function syncGitHubSoulBackupsInternalHandler(
internal.githubSoulBackups.getGitHubSoulBackupSyncStateInternal,
{},
)) as {
cursor: string | null
})
cursor: string | null;
});
let cursor: string | null = state.cursor
let isDone = false
let cursor: string | null = state.cursor;
let isDone = false;
for (let batch = 0; batch < maxBatches; batch++) {
const page = (await ctx.runQuery(internal.githubSoulBackups.getGitHubSoulBackupPageInternal, {
cursor: cursor ?? undefined,
batchSize,
})) as { items: BackupPageItem[]; cursor: string | null; isDone: boolean }
})) as { items: BackupPageItem[]; cursor: string | null; isDone: boolean };
cursor = page.cursor
isDone = page.isDone
cursor = page.cursor;
isDone = page.isDone;
for (const item of page.items) {
if (item.kind !== 'ok') {
if (item.kind === 'missingLatestVersion' || item.kind === 'missingVersionDoc') {
stats.soulsMissingVersion += 1
} else if (item.kind === 'missingOwner') {
stats.soulsMissingOwner += 1
if (item.kind !== "ok") {
if (item.kind === "missingLatestVersion" || item.kind === "missingVersionDoc") {
stats.soulsMissingVersion += 1;
} else if (item.kind === "missingOwner") {
stats.soulsMissingOwner += 1;
}
continue
continue;
}
stats.soulsScanned += 1
stats.soulsScanned += 1;
try {
const meta = await fetchGitHubSoulMeta(context, item.ownerHandle, item.slug)
const meta = await fetchGitHubSoulMeta(context, item.ownerHandle, item.slug);
if (meta?.latest?.version === item.version) {
stats.soulsSkipped += 1
continue
stats.soulsSkipped += 1;
continue;
}
if (!dryRun) {
@@ -151,25 +151,25 @@ export async function syncGitHubSoulBackupsInternalHandler(
publishedAt: item.publishedAt,
},
context,
)
stats.soulsBackedUp += 1
);
stats.soulsBackedUp += 1;
}
} catch (error) {
console.error('GitHub soul backup sync failed', error)
stats.errors += 1
console.error("GitHub soul backup sync failed", error);
stats.errors += 1;
}
}
if (!dryRun) {
await ctx.runMutation(internal.githubSoulBackups.setGitHubSoulBackupSyncStateInternal, {
cursor: isDone ? undefined : (cursor ?? undefined),
})
});
}
if (isDone) break
if (isDone) break;
}
return { stats, cursor, isDone }
return { stats, cursor, isDone };
}
export const syncGitHubSoulBackupsInternal = internalAction({
@@ -179,8 +179,8 @@ export const syncGitHubSoulBackupsInternal = internalAction({
maxBatches: v.optional(v.number()),
},
handler: syncGitHubSoulBackupsInternalHandler,
})
});
function clampInt(value: number, min: number, max: number) {
return Math.max(min, Math.min(max, Math.floor(value)))
return Math.max(min, Math.min(max, Math.floor(value)));
}
+159 -61
View File
@@ -1,7 +1,7 @@
import { ApiRoutes, LegacyApiRoutes } from 'clawdhub-schema'
import { httpRouter } from 'convex/server'
import { auth } from './auth'
import { downloadZip } from './downloads'
import { ApiRoutes, LegacyApiRoutes } from "clawhub-schema";
import { httpRouter } from "convex/server";
import { auth } from "./auth";
import { downloadZip } from "./downloads";
import {
cliPublishHttp,
cliSkillDeleteHttp,
@@ -12,10 +12,20 @@ import {
getSkillHttp,
resolveSkillVersionHttp,
searchSkillsHttp,
} from './httpApi'
} from "./httpApi";
import {
listBundlePluginsV1Http,
listCodePluginsV1Http,
listPackagesV1Http,
listPluginsV1Http,
listSkillsV1Http,
listSoulsV1Http,
mintPublishTokenV1Http,
packagesDeleteRouterV1Http,
packagesGetRouterV1Http,
packagesPostRouterV1Http,
pluginsGetRouterV1Http,
publishPackageV1Http,
publishSkillV1Http,
publishSoulV1Http,
resolveSkillVersionV1Http,
@@ -28,167 +38,255 @@ import {
soulsPostRouterV1Http,
starsDeleteRouterV1Http,
starsPostRouterV1Http,
transfersGetRouterV1Http,
usersListV1Http,
usersPostRouterV1Http,
whoamiV1Http,
} from './httpApiV1'
} from "./httpApiV1";
import { preflightHandler } from "./httpPreflight";
const http = httpRouter()
const http = httpRouter();
auth.addHttpRoutes(http)
auth.addHttpRoutes(http);
http.route({
path: ApiRoutes.download,
method: 'GET',
method: "GET",
handler: downloadZip,
})
});
http.route({
path: ApiRoutes.search,
method: 'GET',
method: "GET",
handler: searchSkillsV1Http,
})
});
http.route({
path: ApiRoutes.resolve,
method: 'GET',
method: "GET",
handler: resolveSkillVersionV1Http,
})
});
http.route({
path: ApiRoutes.skills,
method: 'GET',
method: "GET",
handler: listSkillsV1Http,
})
});
http.route({
path: ApiRoutes.packages,
method: "GET",
handler: listPackagesV1Http,
});
http.route({
path: ApiRoutes.plugins,
method: "GET",
handler: listPluginsV1Http,
});
http.route({
path: ApiRoutes.codePlugins,
method: "GET",
handler: listCodePluginsV1Http,
});
http.route({
path: ApiRoutes.bundlePlugins,
method: "GET",
handler: listBundlePluginsV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.skills}/`,
method: 'GET',
method: "GET",
handler: skillsGetRouterV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.packages}/`,
method: "GET",
handler: packagesGetRouterV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.plugins}/`,
method: "GET",
handler: pluginsGetRouterV1Http,
});
http.route({
path: ApiRoutes.skills,
method: 'POST',
method: "POST",
handler: publishSkillV1Http,
})
});
http.route({
path: ApiRoutes.packages,
method: "POST",
handler: publishPackageV1Http,
});
http.route({
path: ApiRoutes.publishTokenMint,
method: "POST",
handler: mintPublishTokenV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.packages}/`,
method: "POST",
handler: packagesPostRouterV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.packages}/`,
method: "DELETE",
handler: packagesDeleteRouterV1Http,
});
http.route({
pathPrefix: `${ApiRoutes.skills}/`,
method: 'POST',
method: "POST",
handler: skillsPostRouterV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.skills}/`,
method: 'DELETE',
method: "DELETE",
handler: skillsDeleteRouterV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.stars}/`,
method: 'POST',
method: "POST",
handler: starsPostRouterV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.stars}/`,
method: 'DELETE',
method: "DELETE",
handler: starsDeleteRouterV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.transfers}/`,
method: "GET",
handler: transfersGetRouterV1Http,
});
http.route({
path: ApiRoutes.whoami,
method: 'GET',
method: "GET",
handler: whoamiV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.users}/`,
method: "POST",
handler: usersPostRouterV1Http,
});
http.route({
path: ApiRoutes.users,
method: "GET",
handler: usersListV1Http,
});
http.route({
path: ApiRoutes.souls,
method: 'GET',
method: "GET",
handler: listSoulsV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: 'GET',
method: "GET",
handler: soulsGetRouterV1Http,
})
});
http.route({
path: ApiRoutes.souls,
method: 'POST',
method: "POST",
handler: publishSoulV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: 'POST',
method: "POST",
handler: soulsPostRouterV1Http,
})
});
http.route({
pathPrefix: `${ApiRoutes.souls}/`,
method: 'DELETE',
method: "DELETE",
handler: soulsDeleteRouterV1Http,
})
});
http.route({
pathPrefix: "/api/",
method: "OPTIONS",
handler: preflightHandler,
});
// TODO: remove legacy /api routes after deprecation window.
http.route({
path: LegacyApiRoutes.download,
method: 'GET',
method: "GET",
handler: downloadZip,
})
});
http.route({
path: LegacyApiRoutes.search,
method: 'GET',
method: "GET",
handler: searchSkillsHttp,
})
});
http.route({
path: LegacyApiRoutes.skill,
method: 'GET',
method: "GET",
handler: getSkillHttp,
})
});
http.route({
path: LegacyApiRoutes.skillResolve,
method: 'GET',
method: "GET",
handler: resolveSkillVersionHttp,
})
});
http.route({
path: LegacyApiRoutes.cliWhoami,
method: 'GET',
method: "GET",
handler: cliWhoamiHttp,
})
});
http.route({
path: LegacyApiRoutes.cliUploadUrl,
method: 'POST',
method: "POST",
handler: cliUploadUrlHttp,
})
});
http.route({
path: LegacyApiRoutes.cliPublish,
method: 'POST',
method: "POST",
handler: cliPublishHttp,
})
});
http.route({
path: LegacyApiRoutes.cliTelemetrySync,
method: 'POST',
method: "POST",
handler: cliTelemetrySyncHttp,
})
});
http.route({
path: LegacyApiRoutes.cliSkillDelete,
method: 'POST',
method: "POST",
handler: cliSkillDeleteHttp,
})
});
http.route({
path: LegacyApiRoutes.cliSkillUndelete,
method: 'POST',
method: "POST",
handler: cliSkillUndeleteHttp,
})
});
export default http
export default http;
+421 -314
View File
@@ -1,96 +1,154 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from 'vitest'
import { afterEach, describe, expect, it, vi } from "vitest";
vi.mock('./lib/apiTokenAuth', () => ({
vi.mock("./lib/apiTokenAuth", () => ({
requireApiTokenUser: vi.fn(),
}))
}));
vi.mock('./skills', () => ({
vi.mock("./skills", () => ({
publishVersionForUser: vi.fn(),
}))
}));
const { requireApiTokenUser } = await import('./lib/apiTokenAuth')
const { publishVersionForUser } = await import('./skills')
const { __handlers, cliSkillDeleteHttp, cliSkillUndeleteHttp } = await import('./httpApi')
const { hashSkillFiles } = await import('./lib/skills')
const { requireApiTokenUser } = await import("./lib/apiTokenAuth");
const { publishVersionForUser } = await import("./skills");
const { __handlers } = await import("./httpApi");
const { hashSkillFiles } = await import("./lib/skills");
function makeCtx(partial: Record<string, unknown>) {
return partial as unknown as import('./_generated/server').ActionCtx
return partial as unknown as import("./_generated/server").ActionCtx;
}
describe('httpApi handlers', () => {
describe("httpApi handlers", () => {
afterEach(() => {
vi.mocked(requireApiTokenUser).mockReset()
vi.mocked(publishVersionForUser).mockReset()
})
vi.mocked(requireApiTokenUser).mockReset();
vi.mocked(publishVersionForUser).mockReset();
});
it('searchSkillsHttp returns empty results for empty query', async () => {
it("searchSkillsHttp returns empty results for empty query", async () => {
const response = await __handlers.searchSkillsHandler(
makeCtx({ runAction: vi.fn() }),
new Request('https://example.com/api/search?q=%20%20'),
)
expect(response.status).toBe(200)
expect(await response.json()).toEqual({ results: [] })
})
new Request("https://example.com/api/search?q=%20%20"),
);
expect(response.status).toBe(200);
expect(await response.json()).toEqual({ results: [] });
});
it('searchSkillsHttp forwards args', async () => {
it("searchSkillsHttp forwards args (approvedOnly alias)", async () => {
const runAction = vi.fn().mockResolvedValue([
{
score: 1,
skill: { slug: 'a', displayName: 'A', summary: null, updatedAt: 1 },
skill: { slug: "a", displayName: "A", summary: null, updatedAt: 1 },
version: null,
},
])
]);
const response = await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request('https://example.com/api/search?q=test&approvedOnly=true&limit=5'),
)
new Request("https://example.com/api/search?q=test&approvedOnly=true&limit=5"),
);
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
query: "test",
limit: 5,
approvedOnly: true,
})
expect(response.status).toBe(200)
const json = await response.json()
expect(json.results[0].slug).toBe('a')
})
highlightedOnly: true,
nonSuspiciousOnly: undefined,
});
expect(response.status).toBe(200);
const json = await response.json();
expect(json.results[0].slug).toBe("a");
});
it('searchSkillsHttp omits approvedOnly when false', async () => {
const runAction = vi.fn().mockResolvedValue([])
it("searchSkillsHttp forwards highlightedOnly", async () => {
const runAction = vi.fn().mockResolvedValue([]);
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request('https://example.com/api/search?q=test&approvedOnly=false'),
)
new Request("https://example.com/api/search?q=test&highlightedOnly=true"),
);
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: 'test',
query: "test",
limit: undefined,
approvedOnly: undefined,
})
})
highlightedOnly: true,
nonSuspiciousOnly: undefined,
});
});
it('getSkillHttp validates slug', async () => {
it("searchSkillsHttp omits highlightedOnly when approvedOnly is false", async () => {
const runAction = vi.fn().mockResolvedValue([]);
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request("https://example.com/api/search?q=test&approvedOnly=false"),
);
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: "test",
limit: undefined,
highlightedOnly: undefined,
nonSuspiciousOnly: undefined,
});
});
it("searchSkillsHttp forwards nonSuspiciousOnly", async () => {
const runAction = vi.fn().mockResolvedValue([]);
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request("https://example.com/api/search?q=test&nonSuspiciousOnly=1"),
);
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: "test",
limit: undefined,
highlightedOnly: undefined,
nonSuspiciousOnly: true,
});
});
it("searchSkillsHttp forwards legacy nonSuspicious alias", async () => {
const runAction = vi.fn().mockResolvedValue([]);
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request("https://example.com/api/search?q=test&nonSuspicious=1"),
);
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: "test",
limit: undefined,
highlightedOnly: undefined,
nonSuspiciousOnly: true,
});
});
it("searchSkillsHttp prefers canonical nonSuspiciousOnly over legacy alias", async () => {
const runAction = vi.fn().mockResolvedValue([]);
await __handlers.searchSkillsHandler(
makeCtx({ runAction }),
new Request("https://example.com/api/search?q=test&nonSuspiciousOnly=false&nonSuspicious=1"),
);
expect(runAction).toHaveBeenCalledWith(expect.anything(), {
query: "test",
limit: undefined,
highlightedOnly: undefined,
nonSuspiciousOnly: undefined,
});
});
it("getSkillHttp validates slug", async () => {
const response = await __handlers.getSkillHandler(
makeCtx({ runQuery: vi.fn() }),
new Request('https://example.com/api/skill'),
)
expect(response.status).toBe(400)
})
new Request("https://example.com/api/skill"),
);
expect(response.status).toBe(400);
});
it('getSkillHttp returns 404 when missing', async () => {
const runQuery = vi.fn().mockResolvedValue(null)
it("getSkillHttp returns 404 when missing", async () => {
const runQuery = vi.fn().mockResolvedValue(null);
const response = await __handlers.getSkillHandler(
makeCtx({ runQuery }),
new Request('https://example.com/api/skill?slug=missing'),
)
expect(response.status).toBe(404)
})
new Request("https://example.com/api/skill?slug=missing"),
);
expect(response.status).toBe(404);
});
it('getSkillHttp returns payload with owner and latestVersion', async () => {
it("getSkillHttp returns payload with owner and latestVersion", async () => {
const runQuery = vi.fn().mockResolvedValue({
skill: {
slug: 'demo',
displayName: 'Demo',
summary: 'x',
slug: "demo",
displayName: "Demo",
summary: "x",
tags: {},
stats: {
downloads: 0,
@@ -103,25 +161,25 @@ describe('httpApi handlers', () => {
createdAt: 1,
updatedAt: 2,
},
latestVersion: { version: '1.0.0', createdAt: 3, changelog: 'c' },
owner: { handle: 'p', displayName: 'Peter', image: null },
})
latestVersion: { version: "1.0.0", createdAt: 3, changelog: "c" },
owner: { handle: "p", displayName: "Peter", image: null },
});
const response = await __handlers.getSkillHandler(
makeCtx({ runQuery }),
new Request('https://example.com/api/skill?slug=demo'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.skill.slug).toBe('demo')
expect(json.latestVersion.version).toBe('1.0.0')
expect(json.owner.handle).toBe('p')
})
new Request("https://example.com/api/skill?slug=demo"),
);
expect(response.status).toBe(200);
const json = await response.json();
expect(json.skill.slug).toBe("demo");
expect(json.latestVersion.version).toBe("1.0.0");
expect(json.owner.handle).toBe("p");
});
it('getSkillHttp returns payload with null owner/latestVersion', async () => {
it("getSkillHttp returns payload with null owner/latestVersion", async () => {
const runQuery = vi.fn().mockResolvedValue({
skill: {
slug: 'demo',
displayName: 'Demo',
slug: "demo",
displayName: "Demo",
summary: null,
tags: {},
stats: {},
@@ -130,344 +188,393 @@ describe('httpApi handlers', () => {
},
latestVersion: null,
owner: null,
})
});
const response = await __handlers.getSkillHandler(
makeCtx({ runQuery }),
new Request('https://example.com/api/skill?slug=demo'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.latestVersion).toBeNull()
expect(json.owner).toBeNull()
})
new Request("https://example.com/api/skill?slug=demo"),
);
expect(response.status).toBe(200);
const json = await response.json();
expect(json.latestVersion).toBeNull();
expect(json.owner).toBeNull();
});
it('resolveSkillVersionHttp validates hash', async () => {
it("resolveSkillVersionHttp validates hash", async () => {
const response = await __handlers.resolveSkillVersionHandler(
makeCtx({ runQuery: vi.fn() }),
new Request('https://example.com/api/skill/resolve?slug=demo&hash=bad'),
)
expect(response.status).toBe(400)
})
new Request("https://example.com/api/skill/resolve?slug=demo&hash=bad"),
);
expect(response.status).toBe(400);
});
it('resolveSkillVersionHttp returns 404 when missing', async () => {
const runQuery = vi.fn().mockResolvedValue(null)
it("resolveSkillVersionHttp returns 404 when missing", async () => {
const runQuery = vi.fn().mockResolvedValue(null);
const response = await __handlers.resolveSkillVersionHandler(
makeCtx({ runQuery }),
new Request(
'https://example.com/api/skill/resolve?slug=missing&hash=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
"https://example.com/api/skill/resolve?slug=missing&hash=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
),
)
expect(response.status).toBe(404)
})
);
expect(response.status).toBe(404);
});
it('resolveSkillVersionHttp returns match and latestVersion', async () => {
const matchHash = await hashSkillFiles([{ path: 'SKILL.md', sha256: 'abc' }])
it("resolveSkillVersionHttp returns match and latestVersion", async () => {
const matchHash = await hashSkillFiles([{ path: "SKILL.md", sha256: "abc" }]);
const runQuery = vi.fn().mockResolvedValueOnce({
match: { version: '1.0.0' },
latestVersion: { version: '2.0.0' },
})
match: { version: "1.0.0" },
latestVersion: { version: "2.0.0" },
});
const response = await __handlers.resolveSkillVersionHandler(
makeCtx({ runQuery }),
new Request(`https://example.com/api/skill/resolve?slug=demo&hash=${matchHash}`),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.match.version).toBe('1.0.0')
expect(json.latestVersion.version).toBe('2.0.0')
})
);
expect(response.status).toBe(200);
const json = await response.json();
expect(json.match.version).toBe("1.0.0");
expect(json.latestVersion.version).toBe("2.0.0");
});
it('cliWhoamiHttp returns 401 on auth failure', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
it("cliWhoamiHttp returns 401 on auth failure", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
const response = await __handlers.cliWhoamiHandler(
makeCtx({}),
new Request('https://x/api/cli/whoami'),
)
expect(response.status).toBe(401)
})
new Request("https://x/api/cli/whoami"),
);
expect(response.status).toBe(401);
});
it('cliWhoamiHttp returns user payload on success', async () => {
it("cliWhoamiHttp returns user payload on success", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({
user: { handle: 'p', displayName: 'Peter', image: 'x' },
} as never)
user: { handle: "p", displayName: "Peter", image: "x" },
} as never);
const response = await __handlers.cliWhoamiHandler(
makeCtx({}),
new Request('https://x/api/cli/whoami'),
)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.user.handle).toBe('p')
})
new Request("https://x/api/cli/whoami"),
);
expect(response.status).toBe(200);
const json = await response.json();
expect(json.user.handle).toBe("p");
});
it('cliTelemetrySyncHttp forwards roots and returns ok', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'users:1' } as never)
const runMutation = vi.fn().mockResolvedValue(null)
it("cliTelemetrySyncHttp forwards roots and returns ok", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const runMutation = vi.fn().mockResolvedValue(null);
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
roots: [
{
rootId: 'abc',
label: '~/skills',
skills: [{ slug: 'weather', version: null }],
rootId: "abc",
label: "~/skills",
skills: [{ slug: "weather", version: null }],
},
],
}),
}),
)
expect(response.status).toBe(200)
expect(await response.json()).toEqual({ ok: true })
expect(runMutation).toHaveBeenCalledTimes(1)
})
);
expect(response.status).toBe(200);
expect(await response.json()).toEqual({ ok: true });
expect(runMutation).toHaveBeenCalledTimes(1);
});
it('cliTelemetrySyncHttp returns 400 on invalid payload', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'users:1' } as never)
it("cliTelemetrySyncHttp returns 400 on invalid payload", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation: vi.fn() }),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ roots: 'nope' }),
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ roots: "nope" }),
}),
)
expect(response.status).toBe(400)
})
);
expect(response.status).toBe(400);
});
it('cliTelemetrySyncHttp forwards skill versions when provided', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'users:1' } as never)
const runMutation = vi.fn().mockResolvedValue(null)
it("cliTelemetrySyncHttp forwards skill versions when provided", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "users:1" } as never);
const runMutation = vi.fn().mockResolvedValue(null);
await __handlers.cliTelemetrySyncHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
roots: [
{
rootId: 'abc',
label: '~/skills',
skills: [{ slug: 'weather', version: '1.0.0' }],
rootId: "abc",
label: "~/skills",
skills: [{ slug: "weather", version: "1.0.0" }],
},
],
}),
}),
)
);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'users:1',
userId: "users:1",
roots: [
{ rootId: 'abc', label: '~/skills', skills: [{ slug: 'weather', version: '1.0.0' }] },
{ rootId: "abc", label: "~/skills", skills: [{ slug: "weather", version: "1.0.0" }] },
],
})
})
});
});
it('cliTelemetrySyncHttp returns 400 on invalid json', async () => {
const request = new Request('https://x/api/cli/telemetry/sync', { method: 'POST', body: '{' })
const response = await __handlers.cliTelemetrySyncHandler(makeCtx({}), request)
expect(response.status).toBe(400)
})
it("cliTelemetrySyncHttp returns 400 on invalid json", async () => {
const request = new Request("https://x/api/cli/telemetry/sync", { method: "POST", body: "{" });
const response = await __handlers.cliTelemetrySyncHandler(makeCtx({}), request);
expect(response.status).toBe(400);
});
it('cliTelemetrySyncHttp returns 401 when unauthorized', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
it("cliTelemetrySyncHttp returns 401 when unauthorized", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
const response = await __handlers.cliTelemetrySyncHandler(
makeCtx({}),
new Request('https://x/api/cli/telemetry/sync', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
new Request("https://x/api/cli/telemetry/sync", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ roots: [] }),
}),
)
expect(response.status).toBe(401)
})
);
expect(response.status).toBe(401);
});
it('cliUploadUrlHttp returns uploadUrl', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue('https://upload.local')
it("cliUploadUrlHttp returns uploadUrl", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const runMutation = vi.fn().mockResolvedValue("https://upload.local");
const response = await __handlers.cliUploadUrlHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/upload-url', { method: 'POST' }),
)
expect(response.status).toBe(200)
expect(await response.json()).toEqual({ uploadUrl: 'https://upload.local' })
})
new Request("https://x/api/cli/upload-url", { method: "POST" }),
);
expect(response.status).toBe(200);
expect(await response.json()).toEqual({ uploadUrl: "https://upload.local" });
});
it('cliUploadUrlHttp returns 401 when unauthorized', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
it("cliUploadUrlHttp returns 401 when unauthorized", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
const response = await __handlers.cliUploadUrlHandler(
makeCtx({}),
new Request('https://x/api/cli/upload-url', { method: 'POST' }),
)
expect(response.status).toBe(401)
})
new Request("https://x/api/cli/upload-url", { method: "POST" }),
);
expect(response.status).toBe(401);
});
it('cliPublishHttp returns 400 on invalid json', async () => {
const request = new Request('https://x/api/cli/publish', { method: 'POST', body: '{' })
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(400)
})
it("cliPublishHttp returns 400 on invalid json", async () => {
const request = new Request("https://x/api/cli/publish", { method: "POST", body: "{" });
const response = await __handlers.cliPublishHandler(makeCtx({}), request);
expect(response.status).toBe(400);
});
it('cliPublishHttp returns 401 when unauthorized', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const request = new Request('https://x/api/cli/publish', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
it("cliPublishHttp returns 401 when unauthorized", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
const request = new Request("https://x/api/cli/publish", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({}),
})
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(401)
})
});
const response = await __handlers.cliPublishHandler(makeCtx({}), request);
expect(response.status).toBe(401);
});
it('cliPublishHttp returns 400 on publish error', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
vi.mocked(publishVersionForUser).mockRejectedValueOnce(new Error('Nope'))
const request = new Request('https://x/api/cli/publish', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
it("cliPublishHttp returns 400 on publish error", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
vi.mocked(publishVersionForUser).mockRejectedValueOnce(new Error("Nope"));
const request = new Request("https://x/api/cli/publish", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
slug: 'cool-skill',
displayName: 'Cool Skill',
version: '1.2.3',
changelog: 'c',
files: [{ path: 'SKILL.md', size: 1, storageId: 'id', sha256: 'a' }],
slug: "cool-skill",
displayName: "Cool Skill",
version: "1.2.3",
changelog: "c",
acceptLicenseTerms: true,
files: [{ path: "SKILL.md", size: 1, storageId: "id", sha256: "a" }],
}),
})
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(400)
})
});
const response = await __handlers.cliPublishHandler(makeCtx({}), request);
expect(response.status).toBe(400);
});
it('cliPublishHttp returns 200 on success', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
it("cliPublishHttp returns 200 on success", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
vi.mocked(publishVersionForUser).mockResolvedValueOnce({
skillId: 's',
versionId: 'v',
embeddingId: 'e',
} as never)
const request = new Request('https://x/api/cli/publish', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
skillId: "s",
versionId: "v",
embeddingId: "e",
} as never);
const request = new Request("https://x/api/cli/publish", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
slug: 'cool-skill',
displayName: 'Cool Skill',
version: '1.2.3',
changelog: 'c',
files: [{ path: 'SKILL.md', size: 1, storageId: 'id', sha256: 'a' }],
slug: "cool-skill",
displayName: "Cool Skill",
version: "1.2.3",
changelog: "c",
acceptLicenseTerms: true,
files: [{ path: "SKILL.md", size: 1, storageId: "id", sha256: "a" }],
}),
})
const response = await __handlers.cliPublishHandler(makeCtx({}), request)
expect(response.status).toBe(200)
const json = await response.json()
expect(json.ok).toBe(true)
expect(json.skillId).toBe('s')
})
});
const response = await __handlers.cliPublishHandler(makeCtx({}), request);
expect(response.status).toBe(200);
const json = await response.json();
expect(json.ok).toBe(true);
expect(json.skillId).toBe("s");
});
it('cliSkillDeleteHandler returns 401 when unauthorized', async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error('Unauthorized'))
const request = new Request('https://x/api/cli/skill/delete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
})
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true)
expect(response.status).toBe(401)
})
it("cliPublishHttp accepts legacy clients that omit license terms", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
vi.mocked(publishVersionForUser).mockResolvedValueOnce({
skillId: "s",
versionId: "v",
embeddingId: "e",
} as never);
const request = new Request("https://x/api/cli/publish", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
slug: "cool-skill",
displayName: "Cool Skill",
version: "1.2.3",
changelog: "c",
files: [{ path: "SKILL.md", size: 1, storageId: "id", sha256: "a" }],
}),
});
const response = await __handlers.cliPublishHandler(makeCtx({}), request);
expect(response.status).toBe(200);
});
it('cliSkillDeleteHandler calls mutation and returns ok', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const request = new Request('https://x/api/cli/skill/delete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
})
const response = await __handlers.cliSkillDeleteHandler(makeCtx({ runMutation }), request, true)
expect(response.status).toBe(200)
it("cliPublishHttp rejects explicit license refusal", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const request = new Request("https://x/api/cli/publish", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
slug: "cool-skill",
displayName: "Cool Skill",
version: "1.2.3",
changelog: "c",
acceptLicenseTerms: false,
files: [{ path: "SKILL.md", size: 1, storageId: "id", sha256: "a" }],
}),
});
const response = await __handlers.cliPublishHandler(makeCtx({}), request);
expect(response.status).toBe(400);
expect(await response.text()).toMatch(/license terms must be accepted/i);
});
it("cliSkillDeleteHandler returns 401 when unauthorized", async () => {
vi.mocked(requireApiTokenUser).mockRejectedValueOnce(new Error("Unauthorized"));
const request = new Request("https://x/api/cli/skill/delete", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ slug: "demo" }),
});
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true);
expect(response.status).toBe(401);
});
it("cliSkillDeleteHandler calls mutation and returns ok", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const runMutation = vi.fn().mockResolvedValue({ ok: true });
const request = new Request("https://x/api/cli/skill/delete", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ slug: "demo" }),
});
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
request,
true,
);
expect(response.status).toBe(200);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'user1',
slug: 'demo',
userId: "user1",
slug: "demo",
deleted: true,
})
expect(await response.json()).toEqual({ ok: true })
})
});
expect(await response.json()).toEqual({ ok: true });
});
it('cliSkillDeleteHandler supports undelete', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const request = new Request('https://x/api/cli/skill/undelete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
})
it("cliSkillDeleteHandler supports undelete", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const runMutation = vi.fn().mockResolvedValue({ ok: true });
const request = new Request("https://x/api/cli/skill/undelete", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ slug: "demo" }),
});
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
request,
false,
)
expect(response.status).toBe(200)
);
expect(response.status).toBe(200);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'user1',
slug: 'demo',
userId: "user1",
slug: "demo",
deleted: false,
})
})
});
});
it('cliSkillUndeleteHttp calls delete handler with deleted=false', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const response = await cliSkillUndeleteHttp(
it("cliSkillUndeleteHttp calls delete handler with deleted=false", async () => {
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const runMutation = vi.fn().mockResolvedValue({ ok: true });
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/skill/undelete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
new Request("https://x/api/cli/skill/undelete", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ slug: "demo" }),
}),
)
expect(response.status).toBe(200)
false,
);
expect(response.status).toBe(200);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'user1',
slug: 'demo',
userId: "user1",
slug: "demo",
deleted: false,
})
warnSpy.mockRestore()
})
});
warnSpy.mockRestore();
});
it('cliSkillDeleteHttp calls delete handler with deleted=true', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const runMutation = vi.fn().mockResolvedValue({ ok: true })
const response = await cliSkillDeleteHttp(
it("cliSkillDeleteHttp calls delete handler with deleted=true", async () => {
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const runMutation = vi.fn().mockResolvedValue({ ok: true });
const response = await __handlers.cliSkillDeleteHandler(
makeCtx({ runMutation }),
new Request('https://x/api/cli/skill/delete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ slug: 'demo' }),
new Request("https://x/api/cli/skill/delete", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ slug: "demo" }),
}),
)
expect(response.status).toBe(200)
true,
);
expect(response.status).toBe(200);
expect(runMutation).toHaveBeenCalledWith(expect.anything(), {
userId: 'user1',
slug: 'demo',
userId: "user1",
slug: "demo",
deleted: true,
})
warnSpy.mockRestore()
})
});
warnSpy.mockRestore();
});
it('cliSkillDeleteHandler returns 400 on invalid json', async () => {
const request = new Request('https://x/api/cli/skill/delete', { method: 'POST', body: '{' })
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true)
expect(response.status).toBe(400)
})
it("cliSkillDeleteHandler returns 400 on invalid json", async () => {
const request = new Request("https://x/api/cli/skill/delete", { method: "POST", body: "{" });
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true);
expect(response.status).toBe(400);
});
it('cliSkillDeleteHandler returns 400 on invalid payload', async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: 'user1' } as never)
const request = new Request('https://x/api/cli/skill/delete', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
it("cliSkillDeleteHandler returns 400 on invalid payload", async () => {
vi.mocked(requireApiTokenUser).mockResolvedValueOnce({ userId: "user1" } as never);
const request = new Request("https://x/api/cli/skill/delete", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({}),
})
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true)
expect(response.status).toBe(400)
})
})
});
const response = await __handlers.cliSkillDeleteHandler(makeCtx({}), request, true);
expect(response.status).toBe(400);
});
});
+48 -48
View File
@@ -1,70 +1,70 @@
/* @vitest-environment node */
import { describe, expect, it } from 'vitest'
import { __test } from './httpApi'
import { describe, expect, it } from "vitest";
import { __test } from "./httpApi";
describe('httpApi', () => {
it('parses publish payload', () => {
describe("httpApi", () => {
it("parses publish payload", () => {
const parsed = __test.parsePublishBody({
slug: 'cool-skill',
displayName: 'Cool Skill',
version: '1.2.3',
changelog: 'stuff',
tags: ['latest', 'beta'],
slug: "cool-skill",
displayName: "Cool Skill",
version: "1.2.3",
changelog: "stuff",
tags: ["latest", "beta"],
files: [
{
path: 'SKILL.md',
path: "SKILL.md",
size: 5,
storageId: 'fakeStorageId',
sha256: 'abcd',
contentType: 'text/markdown',
storageId: "fakeStorageId",
sha256: "abcd",
contentType: "text/markdown",
},
],
})
expect(parsed.slug).toBe('cool-skill')
expect(parsed.tags).toEqual(['latest', 'beta'])
expect(parsed.files[0]?.path).toBe('SKILL.md')
})
});
expect(parsed.slug).toBe("cool-skill");
expect(parsed.tags).toEqual(["latest", "beta"]);
expect(parsed.files[0]?.path).toBe("SKILL.md");
});
it('normalizes optional fields in publish payload', () => {
it("normalizes optional fields in publish payload", () => {
const parsed = __test.parsePublishBody({
slug: 'cool-skill',
displayName: 'Cool Skill',
version: '1.2.3',
changelog: '',
slug: "cool-skill",
displayName: "Cool Skill",
version: "1.2.3",
changelog: "",
tags: [],
forkOf: { slug: 'base-skill' },
forkOf: { slug: "base-skill" },
files: [
{
path: 'SKILL.md',
path: "SKILL.md",
size: 5,
storageId: 'fakeStorageId',
sha256: 'abcd',
contentType: 'text/markdown',
storageId: "fakeStorageId",
sha256: "abcd",
contentType: "text/markdown",
},
],
})
expect(parsed.tags).toBeUndefined()
expect(parsed.source).toBeUndefined()
expect(parsed.forkOf).toEqual({ slug: 'base-skill', version: undefined })
})
});
expect(parsed.tags).toBeUndefined();
expect(parsed.source).toBeUndefined();
expect(parsed.forkOf).toEqual({ slug: "base-skill", version: undefined });
});
it('rejects invalid publish payloads', () => {
expect(() => __test.parsePublishBody(null)).toThrow(/Publish payload/i)
it("rejects invalid publish payloads", () => {
expect(() => __test.parsePublishBody(null)).toThrow(/Publish payload/i);
expect(() =>
__test.parsePublishBody({
slug: 'x',
displayName: 'X',
version: '1.0.0',
changelog: 'c',
slug: "x",
displayName: "X",
version: "1.0.0",
changelog: "c",
files: [],
}),
).toThrow(/files required/i)
})
).toThrow(/files required/i);
});
it('parses optional numbers', () => {
expect(__test.toOptionalNumber(null)).toBeUndefined()
expect(__test.toOptionalNumber('')).toBeUndefined()
expect(__test.toOptionalNumber('10')).toBe(10)
expect(__test.toOptionalNumber('nope')).toBeUndefined()
})
})
it("parses optional numbers", () => {
expect(__test.toOptionalNumber(null)).toBeUndefined();
expect(__test.toOptionalNumber("")).toBeUndefined();
expect(__test.toOptionalNumber("10")).toBe(10);
expect(__test.toOptionalNumber("nope")).toBeUndefined();
});
});
+140 -117
View File
@@ -5,53 +5,62 @@ import {
CliSkillDeleteRequestSchema,
CliTelemetrySyncRequestSchema,
parseArk,
} from 'clawdhub-schema'
import { api, internal } from './_generated/api'
import type { Id } from './_generated/dataModel'
import type { ActionCtx } from './_generated/server'
import { httpAction } from './_generated/server'
import { requireApiTokenUser } from './lib/apiTokenAuth'
import { publishVersionForUser } from './skills'
} from "clawhub-schema";
import { api, internal } from "./_generated/api";
import type { Id } from "./_generated/dataModel";
import type { ActionCtx } from "./_generated/server";
import { httpAction } from "./functions";
import { requireApiTokenUser } from "./lib/apiTokenAuth";
import { corsHeaders, mergeHeaders } from "./lib/httpHeaders";
import { parseBooleanQueryParam, resolveBooleanQueryParam } from "./lib/httpUtils";
import { publishVersionForUser } from "./skills";
type SearchSkillEntry = {
score: number
score: number;
skill: {
slug?: string
displayName?: string
summary?: string | null
updatedAt?: number
} | null
version: { version?: string } | null
}
slug?: string;
displayName?: string;
summary?: string | null;
updatedAt?: number;
} | null;
version: { version?: string } | null;
};
type GetBySlugResult = {
skill: {
_id: Id<'skills'>
slug: string
displayName: string
summary?: string
tags: Record<string, string>
stats: unknown
createdAt: number
updatedAt: number
} | null
latestVersion: { version: string; createdAt: number; changelog: string } | null
owner: { handle?: string; displayName?: string; image?: string } | null
} | null
_id: Id<"skills">;
slug: string;
displayName: string;
summary?: string;
tags: Record<string, string>;
stats: unknown;
createdAt: number;
updatedAt: number;
} | null;
latestVersion: { version: string; createdAt: number; changelog: string } | null;
owner: { handle?: string; displayName?: string; image?: string } | null;
} | null;
async function searchSkillsHandler(ctx: ActionCtx, request: Request) {
const url = new URL(request.url)
const query = url.searchParams.get('q')?.trim() ?? ''
const limit = toOptionalNumber(url.searchParams.get('limit'))
const approvedOnly = url.searchParams.get('approvedOnly') === 'true'
const url = new URL(request.url);
const query = url.searchParams.get("q")?.trim() ?? "";
const limit = toOptionalNumber(url.searchParams.get("limit"));
const approvedOnly = parseBooleanQueryParam(url.searchParams.get("approvedOnly"));
const highlightedOnly =
parseBooleanQueryParam(url.searchParams.get("highlightedOnly")) || approvedOnly;
const nonSuspiciousOnly = resolveBooleanQueryParam(
url.searchParams.get("nonSuspiciousOnly"),
url.searchParams.get("nonSuspicious"),
);
if (!query) return json({ results: [] })
if (!query) return json({ results: [] });
const results = (await ctx.runAction(api.search.searchSkills, {
query,
limit,
approvedOnly: approvedOnly || undefined,
})) as SearchSkillEntry[]
highlightedOnly: highlightedOnly || undefined,
nonSuspiciousOnly: nonSuspiciousOnly || undefined,
})) as SearchSkillEntry[];
return json({
results: results.map((result) => ({
@@ -62,18 +71,18 @@ async function searchSkillsHandler(ctx: ActionCtx, request: Request) {
version: result.version?.version ?? null,
updatedAt: result.skill?.updatedAt,
})),
})
});
}
export const searchSkillsHttp = httpAction(searchSkillsHandler)
export const searchSkillsHttp = httpAction(searchSkillsHandler);
async function getSkillHandler(ctx: ActionCtx, request: Request) {
const url = new URL(request.url)
const slug = url.searchParams.get('slug')?.trim().toLowerCase()
if (!slug) return text('Missing slug', 400)
const url = new URL(request.url);
const slug = url.searchParams.get("slug")?.trim().toLowerCase();
if (!slug) return text("Missing slug", 400);
const result = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult
if (!result?.skill) return text('Skill not found', 404)
const result = (await ctx.runQuery(api.skills.getBySlug, { slug })) as GetBySlugResult;
if (!result?.skill) return text("Skill not found", 404);
return json({
skill: {
@@ -99,122 +108,129 @@ async function getSkillHandler(ctx: ActionCtx, request: Request) {
image: result.owner.image ?? null,
}
: null,
})
});
}
export const getSkillHttp = httpAction(getSkillHandler)
export const getSkillHttp = httpAction(getSkillHandler);
async function resolveSkillVersionHandler(ctx: ActionCtx, request: Request) {
const url = new URL(request.url)
const slug = url.searchParams.get('slug')?.trim().toLowerCase()
const hash = url.searchParams.get('hash')?.trim().toLowerCase()
if (!slug || !hash) return text('Missing slug or hash', 400)
if (!/^[a-f0-9]{64}$/.test(hash)) return text('Invalid hash', 400)
const url = new URL(request.url);
const slug = url.searchParams.get("slug")?.trim().toLowerCase();
const hash = url.searchParams.get("hash")?.trim().toLowerCase();
if (!slug || !hash) return text("Missing slug or hash", 400);
if (!/^[a-f0-9]{64}$/.test(hash)) return text("Invalid hash", 400);
const resolved = await ctx.runQuery(api.skills.resolveVersionByHash, { slug, hash })
if (!resolved) return text('Skill not found', 404)
const resolved = await ctx.runQuery(api.skills.resolveVersionByHash, { slug, hash });
if (!resolved) return text("Skill not found", 404);
return json({ slug, match: resolved.match, latestVersion: resolved.latestVersion })
return json({ slug, match: resolved.match, latestVersion: resolved.latestVersion });
}
export const resolveSkillVersionHttp = httpAction(resolveSkillVersionHandler)
export const resolveSkillVersionHttp = httpAction(resolveSkillVersionHandler);
async function cliWhoamiHandler(ctx: ActionCtx, request: Request) {
try {
const { user } = await requireApiTokenUser(ctx, request)
const { user } = await requireApiTokenUser(ctx, request);
return json({
user: {
handle: user.handle ?? null,
displayName: user.displayName ?? null,
image: user.image ?? null,
},
})
});
} catch {
return text('Unauthorized', 401)
return text("Unauthorized", 401);
}
}
export const cliWhoamiHttp = httpAction(cliWhoamiHandler)
export const cliWhoamiHttp = httpAction(cliWhoamiHandler);
async function cliUploadUrlHandler(ctx: ActionCtx, request: Request) {
try {
const { userId } = await requireApiTokenUser(ctx, request)
const { userId } = await requireApiTokenUser(ctx, request);
const uploadUrl = await ctx.runMutation(internal.uploads.generateUploadUrlForUserInternal, {
userId,
})
return json({ uploadUrl })
});
return json({ uploadUrl });
} catch {
return text('Unauthorized', 401)
return text("Unauthorized", 401);
}
}
export const cliUploadUrlHttp = httpAction(cliUploadUrlHandler)
export const cliUploadUrlHttp = httpAction(cliUploadUrlHandler);
async function cliPublishHandler(ctx: ActionCtx, request: Request) {
let body: unknown
let body: unknown;
try {
body = await request.json()
body = await request.json();
} catch {
return text('Invalid JSON', 400)
return text("Invalid JSON", 400);
}
try {
const { userId } = await requireApiTokenUser(ctx, request)
const args = parsePublishBody(body)
const result = await publishVersionForUser(ctx, userId, args)
return json({ ok: true, ...result })
const { userId } = await requireApiTokenUser(ctx, request);
const args = parsePublishBody(body);
if (!hasAcceptedLegacyLicenseTerms(args.acceptLicenseTerms)) {
return text("MIT-0 license terms must be accepted to publish skills", 400);
}
const result = await publishVersionForUser(ctx, userId, args);
return json({ ok: true, ...result });
} catch (error) {
const message = error instanceof Error ? error.message : 'Publish failed'
if (message.toLowerCase().includes('unauthorized')) return text('Unauthorized', 401)
return text(message, 400)
const message = error instanceof Error ? error.message : "Publish failed";
if (message.toLowerCase().includes("unauthorized")) return text("Unauthorized", 401);
return text(message, 400);
}
}
export const cliPublishHttp = httpAction(cliPublishHandler)
function hasAcceptedLegacyLicenseTerms(acceptLicenseTerms: boolean | undefined) {
return acceptLicenseTerms !== false;
}
export const cliPublishHttp = httpAction(cliPublishHandler);
async function cliSkillDeleteHandler(ctx: ActionCtx, request: Request, deleted: boolean) {
let body: unknown
let body: unknown;
try {
body = await request.json()
body = await request.json();
} catch {
return text('Invalid JSON', 400)
return text("Invalid JSON", 400);
}
try {
const { userId } = await requireApiTokenUser(ctx, request)
const args = parseArk(CliSkillDeleteRequestSchema, body, 'Delete payload')
const { userId } = await requireApiTokenUser(ctx, request);
const args = parseArk(CliSkillDeleteRequestSchema, body, "Delete payload");
await ctx.runMutation(internal.skills.setSkillSoftDeletedInternal, {
userId,
slug: args.slug,
deleted,
})
const ok = parseArk(ApiCliSkillDeleteResponseSchema, { ok: true }, 'Delete response')
return json(ok)
});
const ok = parseArk(ApiCliSkillDeleteResponseSchema, { ok: true }, "Delete response");
return json(ok);
} catch (error) {
const message = error instanceof Error ? error.message : 'Delete failed'
if (message.toLowerCase().includes('unauthorized')) return text('Unauthorized', 401)
return text(message, 400)
const message = error instanceof Error ? error.message : "Delete failed";
if (message.toLowerCase().includes("unauthorized")) return text("Unauthorized", 401);
return text(message, 400);
}
}
export const cliSkillDeleteHttp = httpAction((ctx, request) =>
cliSkillDeleteHandler(ctx, request, true),
)
);
export const cliSkillUndeleteHttp = httpAction((ctx, request) =>
cliSkillDeleteHandler(ctx, request, false),
)
);
async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
let body: unknown
let body: unknown;
try {
body = await request.json()
body = await request.json();
} catch {
return text('Invalid JSON', 400)
return text("Invalid JSON", 400);
}
try {
const { userId } = await requireApiTokenUser(ctx, request)
const args = parseArk(CliTelemetrySyncRequestSchema, body, 'Telemetry payload')
const { userId } = await requireApiTokenUser(ctx, request);
const args = parseArk(CliTelemetrySyncRequestSchema, body, "Telemetry payload");
await ctx.runMutation(internal.telemetry.reportCliSyncInternal, {
userId,
roots: args.roots.map((root) => ({
@@ -225,53 +241,60 @@ async function cliTelemetrySyncHandler(ctx: ActionCtx, request: Request) {
version: skill.version ?? undefined,
})),
})),
})
const ok = parseArk(ApiCliTelemetrySyncResponseSchema, { ok: true }, 'Telemetry response')
return json(ok)
});
const ok = parseArk(ApiCliTelemetrySyncResponseSchema, { ok: true }, "Telemetry response");
return json(ok);
} catch (error) {
const message = error instanceof Error ? error.message : 'Telemetry failed'
if (message.toLowerCase().includes('unauthorized')) return text('Unauthorized', 401)
return text(message, 400)
const message = error instanceof Error ? error.message : "Telemetry failed";
if (message.toLowerCase().includes("unauthorized")) return text("Unauthorized", 401);
return text(message, 400);
}
}
export const cliTelemetrySyncHttp = httpAction(cliTelemetrySyncHandler)
export const cliTelemetrySyncHttp = httpAction(cliTelemetrySyncHandler);
function json(value: unknown, status = 200) {
return new Response(JSON.stringify(value), {
status,
headers: {
'Content-Type': 'application/json',
'Cache-Control': 'no-store',
},
})
headers: mergeHeaders(
{
"Content-Type": "application/json",
"Cache-Control": "no-store",
},
corsHeaders(),
),
});
}
function text(value: string, status: number) {
return new Response(value, {
status,
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'no-store',
},
})
headers: mergeHeaders(
{
"Content-Type": "text/plain; charset=utf-8",
"Cache-Control": "no-store",
},
corsHeaders(),
),
});
}
function toOptionalNumber(value: string | null) {
if (!value) return undefined
const parsed = Number.parseInt(value, 10)
return Number.isFinite(parsed) ? parsed : undefined
if (!value) return undefined;
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) ? parsed : undefined;
}
function parsePublishBody(body: unknown) {
const parsed = parseArk(CliPublishRequestSchema, body, 'Publish payload')
if (parsed.files.length === 0) throw new Error('files required')
const tags = parsed.tags && parsed.tags.length > 0 ? parsed.tags : undefined
const parsed = parseArk(CliPublishRequestSchema, body, "Publish payload");
if (parsed.files.length === 0) throw new Error("files required");
const tags = parsed.tags && parsed.tags.length > 0 ? parsed.tags : undefined;
return {
slug: parsed.slug,
displayName: parsed.displayName,
version: parsed.version,
changelog: parsed.changelog,
acceptLicenseTerms: parsed.acceptLicenseTerms,
tags,
source: parsed.source ?? undefined,
forkOf: parsed.forkOf
@@ -282,15 +305,15 @@ function parsePublishBody(body: unknown) {
: undefined,
files: parsed.files.map((file) => ({
...file,
storageId: file.storageId as Id<'_storage'>,
storageId: file.storageId as Id<"_storage">,
})),
}
};
}
export const __test = {
parsePublishBody,
toOptionalNumber,
}
};
export const __handlers = {
searchSkillsHandler,
@@ -301,4 +324,4 @@ export const __handlers = {
cliPublishHandler,
cliSkillDeleteHandler,
cliTelemetrySyncHandler,
}
};
File diff suppressed because it is too large Load Diff
+80 -1129
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+348
View File
@@ -0,0 +1,348 @@
import { CliPublishRequestSchema, normalizeTextContentType, parseArk } from "clawhub-schema";
import { internal } from "../_generated/api";
import type { Doc, Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { assertAdmin } from "../lib/access";
import { requireApiTokenUser, requirePackagePublishAuth } from "../lib/apiTokenAuth";
import { corsHeaders, mergeHeaders } from "../lib/httpHeaders";
import { getPublishFileSizeError, MAX_PUBLISH_FILE_BYTES } from "../lib/publishLimits";
import { isMacJunkPath } from "../lib/skills";
export const MAX_RAW_FILE_BYTES = 200 * 1024;
const SAFE_TEXT_FILE_CSP =
"default-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'";
function isSvgLike(contentType: string | undefined, path: string) {
return contentType?.toLowerCase().includes("svg") || path.toLowerCase().endsWith(".svg");
}
export function safeTextFileResponse(params: {
textContent: string;
path: string;
contentType?: string;
sha256: string;
size: number;
headers?: HeadersInit;
}) {
const contentType =
normalizeTextContentType(params.path, params.contentType) ?? params.contentType;
const isSvg = isSvgLike(contentType, params.path);
// For any text response that a browser might try to render, lock it down.
// In particular, this prevents SVG <foreignObject> script execution from reading
// localStorage tokens on this origin.
const headers = mergeHeaders(
params.headers,
{
"Content-Type": contentType
? `${contentType}; charset=utf-8`
: "text/plain; charset=utf-8",
"Cache-Control": "private, max-age=60",
ETag: params.sha256,
"X-Content-SHA256": params.sha256,
"X-Content-Size": String(params.size),
"X-Content-Type-Options": "nosniff",
"X-Frame-Options": "DENY",
"Content-Security-Policy": SAFE_TEXT_FILE_CSP,
...(isSvg ? { "Content-Disposition": "attachment" } : {}),
},
corsHeaders(),
);
return new Response(params.textContent, { status: 200, headers });
}
export function json(value: unknown, status = 200, headers?: HeadersInit) {
return new Response(JSON.stringify(value), {
status,
headers: mergeHeaders(
{
"Content-Type": "application/json",
"Cache-Control": "no-store",
},
headers,
corsHeaders(),
),
});
}
export function text(value: string, status: number, headers?: HeadersInit) {
return new Response(value, {
status,
headers: mergeHeaders(
{
"Content-Type": "text/plain; charset=utf-8",
"Cache-Control": "no-store",
},
headers,
corsHeaders(),
),
});
}
export async function parseJsonPayload(request: Request, headers: HeadersInit) {
try {
const payload = (await request.json()) as Record<string, unknown>;
return { ok: true as const, payload };
} catch {
return { ok: false as const, response: text("Invalid JSON", 400, headers) };
}
}
export async function requireApiTokenUserOrResponse(
ctx: ActionCtx,
request: Request,
headers: HeadersInit,
) {
try {
const auth = await requireApiTokenUser(ctx, request);
return { ok: true as const, userId: auth.userId, user: auth.user as Doc<"users"> };
} catch {
return { ok: false as const, response: text("Unauthorized", 401, headers) };
}
}
export async function requirePackagePublishAuthOrResponse(
ctx: ActionCtx,
request: Request,
headers: HeadersInit,
) {
try {
return { ok: true as const, auth: await requirePackagePublishAuth(ctx, request) };
} catch {
return { ok: false as const, response: text("Unauthorized", 401, headers) };
}
}
export function requireAdminOrResponse(user: Doc<"users">, headers: HeadersInit) {
try {
assertAdmin(user);
return { ok: true as const };
} catch {
return { ok: false as const, response: text("Forbidden", 403, headers) };
}
}
export function getPathSegments(request: Request, prefix: string) {
const pathname = new URL(request.url).pathname;
if (!pathname.startsWith(prefix)) return [];
const rest = pathname.slice(prefix.length);
return rest
.split("/")
.map((segment) => segment.trim())
.filter(Boolean)
.map((segment) => decodeURIComponent(segment));
}
export function toOptionalNumber(value: string | null) {
if (!value) return undefined;
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) ? parsed : undefined;
}
/**
* Batch resolve soul version tags to version strings.
* Collects all version IDs, fetches them in a single query, then maps back.
* Reduces N sequential queries to 1 batch query.
*/
export async function resolveSoulTagsBatch(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<"soulVersions">>>,
): Promise<Array<Record<string, string>>> {
return resolveVersionTagsBatch(ctx, tagsList, internal.souls.getVersionsByIdsInternal);
}
export async function resolveTagsBatch(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<"skillVersions">>>,
): Promise<Array<Record<string, string>>> {
return resolveVersionTagsBatch(ctx, tagsList, internal.skills.getVersionsByIdsInternal);
}
/**
* Batch resolve version tags to version strings.
* Collects all version IDs, fetches them in a single query, then maps back.
*
* Notes:
* - Uses `internal.*` queries to avoid expanding the public Convex API surface.
* - Sorts ids for stable query args (helps caching/log diffs).
*/
export async function resolveVersionTagsBatch<TTable extends "skillVersions" | "soulVersions">(
ctx: ActionCtx,
tagsList: Array<Record<string, Id<TTable>>>,
getVersionsByIdsQuery: unknown,
): Promise<Array<Record<string, string>>> {
const allVersionIds = new Set<Id<TTable>>();
for (const tags of tagsList) {
for (const versionId of Object.values(tags)) allVersionIds.add(versionId);
}
if (allVersionIds.size === 0) return tagsList.map(() => ({}));
const versionIds = [...allVersionIds].sort() as Array<Id<TTable>>;
const versions =
((await ctx.runQuery(getVersionsByIdsQuery as never, { versionIds } as never)) as Array<{
_id: Id<TTable>;
version: string;
softDeletedAt?: unknown;
}> | null) ?? [];
const versionMap = new Map<Id<TTable>, string>();
for (const v of versions) {
if (!v?.softDeletedAt) versionMap.set(v._id, v.version);
}
return tagsList.map((tags) => {
const resolved: Record<string, string> = {};
for (const [tag, versionId] of Object.entries(tags)) {
const version = versionMap.get(versionId);
if (version) resolved[tag] = version;
}
return resolved;
});
}
async function sha256Hex(bytes: Uint8Array) {
const data = new Uint8Array(bytes);
const digest = await crypto.subtle.digest("SHA-256", data);
return toHex(new Uint8Array(digest));
}
function toHex(bytes: Uint8Array) {
let out = "";
for (const byte of bytes) out += byte.toString(16).padStart(2, "0");
return out;
}
type FileLike = {
name: string;
size: number;
type: string;
arrayBuffer: () => Promise<ArrayBuffer>;
};
type FileLikeEntry = FormDataEntryValue & FileLike;
function toFileLike(entry: FormDataEntryValue): FileLikeEntry | null {
if (typeof entry === "string") return null;
const candidate = entry as Partial<FileLike>;
if (typeof candidate.name !== "string") return null;
if (typeof candidate.size !== "number") return null;
if (typeof candidate.arrayBuffer !== "function") return null;
return entry as FileLikeEntry;
}
export async function parseMultipartPublish(
ctx: ActionCtx,
request: Request,
): Promise<{
slug: string;
displayName: string;
version: string;
changelog: string;
acceptLicenseTerms?: boolean;
tags?: string[];
forkOf?: { slug: string; version?: string };
files: Array<{
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
}>;
}> {
const form = await request.formData();
const payloadRaw = form.get("payload");
if (!payloadRaw || typeof payloadRaw !== "string") {
throw new Error("Missing payload");
}
let payload: Record<string, unknown>;
try {
payload = JSON.parse(payloadRaw) as Record<string, unknown>;
} catch {
throw new Error("Invalid JSON payload");
}
const files: Array<{
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
}> = [];
for (const entry of form.getAll("files")) {
const file = toFileLike(entry);
if (!file) continue;
const path = file.name;
if (isMacJunkPath(path)) continue;
const size = file.size;
if (size > MAX_PUBLISH_FILE_BYTES) {
throw new Error(getPublishFileSizeError(path));
}
const contentType = file.type || undefined;
const buffer = new Uint8Array(await file.arrayBuffer());
const sha256 = await sha256Hex(buffer);
const storageId = await ctx.storage.store(file as Blob);
files.push({ path, size, storageId, sha256, contentType });
}
const forkOf = payload.forkOf && typeof payload.forkOf === "object" ? payload.forkOf : undefined;
const hasAcceptLicenseTerms = Object.prototype.hasOwnProperty.call(payload, "acceptLicenseTerms");
const body = {
slug: payload.slug,
displayName: payload.displayName,
version: payload.version,
changelog: typeof payload.changelog === "string" ? payload.changelog : "",
...(hasAcceptLicenseTerms ? { acceptLicenseTerms: payload.acceptLicenseTerms } : {}),
tags: Array.isArray(payload.tags) ? payload.tags : undefined,
...(payload.source ? { source: payload.source } : {}),
files,
...(forkOf ? { forkOf } : {}),
};
return parsePublishBody(body);
}
export function parsePublishBody(body: unknown) {
const parsed = parseArk(CliPublishRequestSchema, body, "Publish payload");
if (parsed.files.length === 0) throw new Error("files required");
const tags = parsed.tags && parsed.tags.length > 0 ? parsed.tags : undefined;
return {
slug: parsed.slug,
displayName: parsed.displayName,
version: parsed.version,
changelog: parsed.changelog,
acceptLicenseTerms: parsed.acceptLicenseTerms,
tags,
source: parsed.source ?? undefined,
forkOf: parsed.forkOf
? {
slug: parsed.forkOf.slug,
version: parsed.forkOf.version ?? undefined,
}
: undefined,
files: parsed.files.map((file) => ({
...file,
storageId: file.storageId as Id<"_storage">,
})),
};
}
export function softDeleteErrorToResponse(
entity: "skill" | "soul",
error: unknown,
headers: HeadersInit,
) {
const message = error instanceof Error ? error.message : `${entity} delete failed`;
const lower = message.toLowerCase();
if (lower.includes("unauthorized")) return text("Unauthorized", 401, headers);
if (lower.includes("forbidden")) return text("Forbidden", 403, headers);
if (lower.includes("not found")) return text(message, 404, headers);
if (lower.includes("slug required")) return text("Slug required", 400, headers);
// Unknown: server-side failure. Keep body generic.
return text("Internal Server Error", 500, headers);
}
File diff suppressed because it is too large Load Diff
+355
View File
@@ -0,0 +1,355 @@
import { api, internal } from "../_generated/api";
import type { Doc, Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { requireApiTokenUser } from "../lib/apiTokenAuth";
import { applyRateLimit, parseBearerToken } from "../lib/httpRateLimit";
import { publishSoulVersionForUser } from "../souls";
import {
MAX_RAW_FILE_BYTES,
getPathSegments,
json,
parseMultipartPublish,
parsePublishBody,
resolveSoulTagsBatch,
safeTextFileResponse,
softDeleteErrorToResponse,
text,
toOptionalNumber,
} from "./shared";
type ListSoulsResult = {
items: Array<{
soul: {
_id: Id<"souls">;
slug: string;
displayName: string;
summary?: string;
tags: Record<string, Id<"soulVersions">>;
stats: unknown;
createdAt: number;
updatedAt: number;
latestVersionId?: Id<"soulVersions">;
};
latestVersion: { version: string; createdAt: number; changelog: string } | null;
}>;
nextCursor: string | null;
};
type GetSoulBySlugResult = {
soul: {
_id: Id<"souls">;
slug: string;
displayName: string;
summary?: string;
tags: Record<string, Id<"soulVersions">>;
stats: unknown;
createdAt: number;
updatedAt: number;
} | null;
latestVersion: PublicSoulVersion | null;
owner: { handle?: string; displayName?: string; image?: string } | null;
} | null;
type ListSoulVersionsResult = {
items: PublicSoulVersion[];
nextCursor: string | null;
};
type PublicSoulVersion = Pick<
Doc<"soulVersions">,
| "_id"
| "_creationTime"
| "soulId"
| "version"
| "fingerprint"
| "changelog"
| "changelogSource"
| "createdBy"
| "createdAt"
| "softDeletedAt"
> & {
files: Array<{
path: string;
size: number;
sha256: string;
contentType?: string;
}>;
parsed?: {
clawdis?: Doc<"soulVersions">["parsed"]["clawdis"];
};
};
type SoulFile = PublicSoulVersion["files"][number];
export async function listSoulsV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const url = new URL(request.url);
const limit = toOptionalNumber(url.searchParams.get("limit"));
const cursor = url.searchParams.get("cursor")?.trim() || undefined;
const result = (await ctx.runQuery(api.souls.listPublicPage, {
limit,
cursor,
})) as ListSoulsResult;
// Batch resolve all tags in a single query instead of N queries
const resolvedTagsList = await resolveSoulTagsBatch(
ctx,
result.items.map((item) => item.soul.tags),
);
const items = result.items.map((item, idx) => ({
slug: item.soul.slug,
displayName: item.soul.displayName,
summary: item.soul.summary ?? null,
tags: resolvedTagsList[idx],
stats: item.soul.stats,
createdAt: item.soul.createdAt,
updatedAt: item.soul.updatedAt,
latestVersion: item.latestVersion
? {
version: item.latestVersion.version,
createdAt: item.latestVersion.createdAt,
changelog: item.latestVersion.changelog,
}
: null,
}));
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers);
}
export async function soulsGetRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/souls/");
if (segments.length === 0) return text("Missing slug", 400, rate.headers);
const slug = segments[0]?.trim().toLowerCase() ?? "";
const second = segments[1];
const third = segments[2];
if (segments.length === 1) {
const result = (await ctx.runQuery(api.souls.getBySlug, { slug })) as GetSoulBySlugResult;
if (!result?.soul) return text("Soul not found", 404, rate.headers);
const [tags] = await resolveSoulTagsBatch(ctx, [result.soul.tags]);
return json(
{
soul: {
slug: result.soul.slug,
displayName: result.soul.displayName,
summary: result.soul.summary ?? null,
tags,
stats: result.soul.stats,
createdAt: result.soul.createdAt,
updatedAt: result.soul.updatedAt,
},
latestVersion: result.latestVersion
? {
version: result.latestVersion.version,
createdAt: result.latestVersion.createdAt,
changelog: result.latestVersion.changelog,
}
: null,
owner: result.owner
? {
handle: result.owner.handle ?? null,
displayName: result.owner.displayName ?? null,
image: result.owner.image ?? null,
}
: null,
},
200,
rate.headers,
);
}
if (second === "versions" && segments.length === 2) {
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug });
if (!soul || soul.softDeletedAt) return text("Soul not found", 404, rate.headers);
const url = new URL(request.url);
const limit = toOptionalNumber(url.searchParams.get("limit"));
const cursor = url.searchParams.get("cursor")?.trim() || undefined;
const result = (await ctx.runQuery(api.souls.listVersionsPage, {
soulId: soul._id,
limit,
cursor,
})) as ListSoulVersionsResult;
const items = result.items
.filter((version) => !version.softDeletedAt)
.map((version) => ({
version: version.version,
createdAt: version.createdAt,
changelog: version.changelog,
changelogSource: version.changelogSource ?? null,
}));
return json({ items, nextCursor: result.nextCursor ?? null }, 200, rate.headers);
}
if (second === "versions" && third && segments.length === 3) {
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug });
if (!soul || soul.softDeletedAt) return text("Soul not found", 404, rate.headers);
const version = await ctx.runQuery(api.souls.getVersionBySoulAndVersion, {
soulId: soul._id,
version: third,
});
if (!version) return text("Version not found", 404, rate.headers);
if (version.softDeletedAt) return text("Version not available", 410, rate.headers);
return json(
{
soul: { slug: soul.slug, displayName: soul.displayName },
version: {
version: version.version,
createdAt: version.createdAt,
changelog: version.changelog,
changelogSource: version.changelogSource ?? null,
files: version.files.map((file: SoulFile) => ({
path: file.path,
size: file.size,
sha256: file.sha256,
contentType: file.contentType ?? null,
})),
},
},
200,
rate.headers,
);
}
if (second === "file" && segments.length === 2) {
const url = new URL(request.url);
const path = url.searchParams.get("path")?.trim();
if (!path) return text("Missing path", 400, rate.headers);
const versionParam = url.searchParams.get("version")?.trim();
const tagParam = url.searchParams.get("tag")?.trim();
const soul = await ctx.runQuery(internal.souls.getSoulBySlugInternal, { slug });
if (!soul || soul.softDeletedAt) return text("Soul not found", 404, rate.headers);
let version = soul.latestVersionId
? await ctx.runQuery(internal.souls.getVersionByIdInternal, {
versionId: soul.latestVersionId,
})
: null;
if (versionParam) {
version = await ctx.runQuery(internal.souls.getVersionBySoulAndVersionInternal, {
soulId: soul._id,
version: versionParam,
});
} else if (tagParam) {
const versionId = soul.tags[tagParam];
if (versionId) {
version = await ctx.runQuery(internal.souls.getVersionByIdInternal, { versionId });
}
}
if (!version) return text("Version not found", 404, rate.headers);
if (version.softDeletedAt) return text("Version not available", 410, rate.headers);
const normalized = path.trim();
const normalizedLower = normalized.toLowerCase();
const file =
version.files.find((entry) => entry.path === normalized) ??
version.files.find((entry) => entry.path.toLowerCase() === normalizedLower);
if (!file) return text("File not found", 404, rate.headers);
if (file.size > MAX_RAW_FILE_BYTES) return text("File exceeds 200KB limit", 413, rate.headers);
const blob = await ctx.storage.get(file.storageId);
if (!blob) return text("File missing in storage", 410, rate.headers);
const textContent = await blob.text();
void ctx.runMutation(internal.soulDownloads.incrementInternal, { soulId: soul._id });
return safeTextFileResponse({
textContent,
path: file.path,
contentType: file.contentType ?? undefined,
sha256: file.sha256,
size: file.size,
headers: rate.headers,
});
}
return text("Not found", 404, rate.headers);
}
export async function publishSoulV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
try {
if (!parseBearerToken(request)) return text("Unauthorized", 401, rate.headers);
} catch {
return text("Unauthorized", 401, rate.headers);
}
const { userId } = await requireApiTokenUser(ctx, request);
const contentType = request.headers.get("content-type") ?? "";
try {
if (contentType.includes("application/json")) {
const body = await request.json();
const payload = parsePublishBody(body);
const result = await publishSoulVersionForUser(ctx, userId, payload);
return json({ ok: true, ...result }, 200, rate.headers);
}
if (contentType.includes("multipart/form-data")) {
const payload = await parseMultipartPublish(ctx, request);
const result = await publishSoulVersionForUser(ctx, userId, payload);
return json({ ok: true, ...result }, 200, rate.headers);
}
} catch (error) {
const message = error instanceof Error ? error.message : "Publish failed";
return text(message, 400, rate.headers);
}
return text("Unsupported content type", 415, rate.headers);
}
export async function soulsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/souls/");
if (segments.length !== 2 || segments[1] !== "undelete") {
return text("Not found", 404, rate.headers);
}
const slug = segments[0]?.trim().toLowerCase() ?? "";
try {
const { userId } = await requireApiTokenUser(ctx, request);
await ctx.runMutation(internal.souls.setSoulSoftDeletedInternal, {
userId,
slug,
deleted: false,
});
return json({ ok: true }, 200, rate.headers);
} catch (error) {
return softDeleteErrorToResponse("soul", error, rate.headers);
}
}
export async function soulsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/souls/");
if (segments.length !== 1) return text("Not found", 404, rate.headers);
const slug = segments[0]?.trim().toLowerCase() ?? "";
try {
const { userId } = await requireApiTokenUser(ctx, request);
await ctx.runMutation(internal.souls.setSoulSoftDeletedInternal, {
userId,
slug,
deleted: true,
});
return json({ ok: true }, 200, rate.headers);
} catch (error) {
return softDeleteErrorToResponse("soul", error, rate.headers);
}
}
+54
View File
@@ -0,0 +1,54 @@
import { internal } from "../_generated/api";
import type { ActionCtx } from "../_generated/server";
import { requireApiTokenUser } from "../lib/apiTokenAuth";
import { applyRateLimit } from "../lib/httpRateLimit";
import { getPathSegments, json, text } from "./shared";
export async function starsPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/stars/");
if (segments.length !== 1) return text("Not found", 404, rate.headers);
const slug = segments[0]?.trim().toLowerCase() ?? "";
try {
const { userId } = await requireApiTokenUser(ctx, request);
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug });
if (!skill) return text("Skill not found", 404, rate.headers);
const result = await ctx.runMutation(internal.stars.addStarInternal, {
userId,
skillId: skill._id,
});
return json(result, 200, rate.headers);
} catch (e) {
if (e instanceof Error && e.message === "Skill not found") {
return text("Skill not found", 404, rate.headers);
}
return text("Unauthorized", 401, rate.headers);
}
}
export async function starsDeleteRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/stars/");
if (segments.length !== 1) return text("Not found", 404, rate.headers);
const slug = segments[0]?.trim().toLowerCase() ?? "";
try {
const { userId } = await requireApiTokenUser(ctx, request);
const skill = await ctx.runQuery(internal.skills.getSkillBySlugInternal, { slug });
if (!skill) return text("Skill not found", 404, rate.headers);
const result = await ctx.runMutation(internal.stars.removeStarInternal, {
userId,
skillId: skill._id,
});
return json(result, 200, rate.headers);
} catch {
return text("Unauthorized", 401, rate.headers);
}
}
+24
View File
@@ -0,0 +1,24 @@
import { internal } from "../_generated/api";
import type { ActionCtx } from "../_generated/server";
import { applyRateLimit } from "../lib/httpRateLimit";
import { getPathSegments, json, requireApiTokenUserOrResponse, text } from "./shared";
export async function transfersGetRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/transfers/");
const direction = segments[0]?.trim().toLowerCase() ?? "";
if (segments.length !== 1 || (direction !== "incoming" && direction !== "outgoing")) {
return text("Not found", 404, rate.headers);
}
const auth = await requireApiTokenUserOrResponse(ctx, request, rate.headers);
if (!auth.ok) return auth.response;
const transfers =
direction === "incoming"
? await ctx.runQuery(internal.skillTransfers.listIncomingInternal, { userId: auth.userId })
: await ctx.runQuery(internal.skillTransfers.listOutgoingInternal, { userId: auth.userId });
return json({ transfers }, 200, rate.headers);
}
+297
View File
@@ -0,0 +1,297 @@
import { api, internal } from "../_generated/api";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { requireApiTokenUser } from "../lib/apiTokenAuth";
import { applyRateLimit } from "../lib/httpRateLimit";
import {
getPathSegments,
json,
parseJsonPayload,
requireAdminOrResponse,
requireApiTokenUserOrResponse,
text,
toOptionalNumber,
} from "./shared";
export async function usersPostRouterV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "write");
if (!rate.ok) return rate.response;
const segments = getPathSegments(request, "/api/v1/users/");
if (segments.length !== 1) {
return text("Not found", 404, rate.headers);
}
const action = segments[0];
if (
action !== "ban" &&
action !== "role" &&
action !== "restore" &&
action !== "reclaim" &&
action !== "publisher"
) {
return text("Not found", 404, rate.headers);
}
const payloadResult = await parseJsonPayload(request, rate.headers);
if (!payloadResult.ok) return payloadResult.response;
const payload = payloadResult.payload;
const authResult = await requireApiTokenUserOrResponse(ctx, request, rate.headers);
if (!authResult.ok) return authResult.response;
const actorUserId = authResult.userId;
const actorUser = authResult.user;
// Restore and reclaim have different parameter shapes, handle them separately
if (action === "restore") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminRestore(ctx, request, payload, actorUserId, rate.headers);
}
if (action === "reclaim") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminReclaim(ctx, request, payload, actorUserId, rate.headers);
}
if (action === "publisher") {
const admin = requireAdminOrResponse(actorUser, rate.headers);
if (!admin.ok) return admin.response;
return handleAdminEnsurePublisher(ctx, payload, actorUserId, rate.headers);
}
const handleRaw = typeof payload.handle === "string" ? payload.handle.trim() : "";
const userIdRaw = typeof payload.userId === "string" ? payload.userId.trim() : "";
const reasonRaw = typeof payload.reason === "string" ? payload.reason.trim() : "";
if (!handleRaw && !userIdRaw) {
return text("Missing userId or handle", 400, rate.headers);
}
const roleRaw = typeof payload.role === "string" ? payload.role.trim().toLowerCase() : "";
if (action === "role" && !roleRaw) {
return text("Missing role", 400, rate.headers);
}
const role =
roleRaw === "user" || roleRaw === "moderator" || roleRaw === "admin" ? roleRaw : null;
if (action === "role" && !role) {
return text("Invalid role", 400, rate.headers);
}
let targetUserId: Id<"users"> | null = userIdRaw ? (userIdRaw as Id<"users">) : null;
if (!targetUserId) {
const handle = handleRaw.toLowerCase();
const user = await ctx.runQuery(api.users.getByHandle, { handle });
if (!user?._id) return text("User not found", 404, rate.headers);
targetUserId = user._id;
}
if (action === "ban") {
const reason = reasonRaw.length > 0 ? reasonRaw : undefined;
if (reason && reason.length > 500) {
return text("Reason too long (max 500 chars)", 400, rate.headers);
}
try {
const result = await ctx.runMutation(internal.users.banUserInternal, {
actorUserId,
targetUserId,
reason,
});
return json(result, 200, rate.headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Ban failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, rate.headers);
}
if (message.toLowerCase().includes("not found")) {
return text(message, 404, rate.headers);
}
return text(message, 400, rate.headers);
}
}
if (!role) {
return text("Invalid role", 400, rate.headers);
}
try {
const result = await ctx.runMutation(internal.users.setRoleInternal, {
actorUserId,
targetUserId,
role,
});
return json({ ok: true, role: result.role ?? role }, 200, rate.headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Role change failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, rate.headers);
}
if (message.toLowerCase().includes("not found")) {
return text(message, 404, rate.headers);
}
return text(message, 400, rate.headers);
}
}
/**
* POST /api/v1/users/restore
* Admin-only: restore skills from GitHub backup for a user.
* Body: { handle: string, slugs: string[], forceOverwriteSquatter?: boolean }
*/
async function handleAdminRestore(
ctx: ActionCtx,
_request: Request,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
if (!handle) return text("Missing handle", 400, headers);
const slugs = Array.isArray(payload.slugs)
? payload.slugs.filter((s): s is string => typeof s === "string")
: [];
if (slugs.length === 0) return text("Missing slugs array", 400, headers);
if (slugs.length > 100) return text("Too many slugs (max 100)", 400, headers);
const forceOverwriteSquatter = Boolean(payload.forceOverwriteSquatter);
const targetUser = await ctx.runQuery(api.users.getByHandle, { handle });
if (!targetUser?._id) return text("User not found", 404, headers);
try {
const result = await ctx.runAction(internal.githubRestore.restoreUserSkillsFromBackup, {
actorUserId,
ownerHandle: handle,
ownerUserId: targetUser._id,
slugs,
forceOverwriteSquatter,
});
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Restore failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, headers);
}
return text(message, 400, headers);
}
}
/**
* POST /api/v1/users/reclaim
* Admin-only: reclaim root slugs for the rightful owner.
* Default behavior is non-destructive owner transfer for existing skills
* (preserves versions/stats/metadata) and leaves missing slugs untouched.
* Body: { handle: string, slugs: string[], reason?: string }
*/
async function handleAdminReclaim(
ctx: ActionCtx,
_request: Request,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
if (!handle) return text("Missing handle", 400, headers);
const slugs = Array.isArray(payload.slugs)
? payload.slugs.filter((s): s is string => typeof s === "string")
: [];
if (slugs.length === 0) return text("Missing slugs array", 400, headers);
if (slugs.length > 200) return text("Too many slugs (max 200)", 400, headers);
const reason = typeof payload.reason === "string" ? payload.reason.trim() : undefined;
const targetUser = await ctx.runQuery(api.users.getByHandle, { handle });
if (!targetUser?._id) return text("User not found", 404, headers);
const results: Array<{ slug: string; ok: boolean; action?: string; error?: string }> = [];
for (const slug of slugs) {
try {
const result = (await ctx.runMutation(internal.skills.reclaimSlugInternal, {
actorUserId,
slug: slug.trim().toLowerCase(),
rightfulOwnerUserId: targetUser._id,
reason,
transferRootSlugOnly: true,
})) as { action?: string };
results.push({ slug, ok: true, action: result.action });
} catch (error) {
const message = error instanceof Error ? error.message : "Reclaim failed";
results.push({ slug, ok: false, error: message });
}
}
const succeeded = results.filter((r) => r.ok).length;
const failed = results.filter((r) => !r.ok).length;
return json({ ok: true, results, succeeded, failed }, 200, headers);
}
async function handleAdminEnsurePublisher(
ctx: ActionCtx,
payload: Record<string, unknown>,
actorUserId: Id<"users">,
headers: HeadersInit,
) {
const handle = typeof payload.handle === "string" ? payload.handle.trim().toLowerCase() : "";
if (!handle) return text("Missing handle", 400, headers);
const displayName =
typeof payload.displayName === "string" ? payload.displayName.trim() : undefined;
const trusted = typeof payload.trusted === "boolean" ? payload.trusted : true;
try {
const result = await ctx.runMutation(internal.publishers.ensureOrgPublisherHandleInternal, {
actorUserId,
handle,
displayName,
trusted,
});
return json(result, 200, headers);
} catch (error) {
const message = error instanceof Error ? error.message : "Publisher ensure failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, headers);
}
if (message.toLowerCase().includes("not found")) {
return text(message, 404, headers);
}
return text(message, 400, headers);
}
}
export async function usersListV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
const url = new URL(request.url);
const limitRaw = toOptionalNumber(url.searchParams.get("limit"));
const query = url.searchParams.get("q") ?? url.searchParams.get("query") ?? "";
let actorUserId: Id<"users">;
try {
const auth = await requireApiTokenUser(ctx, request);
actorUserId = auth.userId;
} catch {
return text("Unauthorized", 401, rate.headers);
}
const limit = Math.min(Math.max(limitRaw ?? 20, 1), 200);
try {
const result = await ctx.runQuery(internal.users.searchInternal, {
actorUserId,
query,
limit,
});
return json(result, 200, rate.headers);
} catch (error) {
const message = error instanceof Error ? error.message : "User search failed";
if (message.toLowerCase().includes("forbidden")) {
return text("Forbidden", 403, rate.headers);
}
if (message.toLowerCase().includes("unauthorized")) {
return text("Unauthorized", 401, rate.headers);
}
return text(message, 400, rate.headers);
}
}
+26
View File
@@ -0,0 +1,26 @@
import type { ActionCtx } from "../_generated/server";
import { requireApiTokenUser } from "../lib/apiTokenAuth";
import { applyRateLimit } from "../lib/httpRateLimit";
import { json, text } from "./shared";
export async function whoamiV1Handler(ctx: ActionCtx, request: Request) {
const rate = await applyRateLimit(ctx, request, "read");
if (!rate.ok) return rate.response;
try {
const { user } = await requireApiTokenUser(ctx, request);
return json(
{
user: {
handle: user.handle ?? null,
displayName: user.displayName ?? null,
image: user.image ?? null,
},
},
200,
rate.headers,
);
} catch {
return text("Unauthorized", 401, rate.headers);
}
}
+33
View File
@@ -0,0 +1,33 @@
import { httpAction } from "./functions";
import { corsHeaders, mergeHeaders } from "./lib/httpHeaders";
function getHeader(request: Request, name: string) {
return request.headers.get(name) ?? request.headers.get(name.toLowerCase());
}
export function buildPreflightHeaders(request: Request) {
const requestedHeaders = getHeader(request, "Access-Control-Request-Headers")?.trim() || null;
const requestedMethod = getHeader(request, "Access-Control-Request-Method")?.trim() || null;
const vary = [
...(requestedMethod ? ["Access-Control-Request-Method"] : []),
...(requestedHeaders ? ["Access-Control-Request-Headers"] : []),
].join(", ");
return mergeHeaders(corsHeaders(), {
"Access-Control-Allow-Methods": "GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD",
"Access-Control-Allow-Headers":
requestedHeaders ?? "Content-Type, Authorization, Digest, X-Clawhub-Version",
"Access-Control-Max-Age": "86400",
...(vary ? { Vary: vary } : {}),
});
}
export const preflightHandler = httpAction(async (_ctx, request) => {
// No cookies/credentials supported; allow any origin for simple browser access.
// If we ever add cookie auth, this must switch to reflecting origin + Allow-Credentials.
return new Response(null, {
status: 204,
headers: buildPreflightHeaders(request),
});
});
+40
View File
@@ -0,0 +1,40 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from "vitest";
import { rebuildTrendingLeaderboardInternal } from "./leaderboards";
const handler = (
rebuildTrendingLeaderboardInternal as unknown as {
_handler: (ctx: unknown, args: { limit?: number }) => Promise<unknown>;
}
)._handler;
describe("leaderboards.rebuildTrendingLeaderboardInternal", () => {
it("schedules the action-based rebuild instead of reading daily stats inline", async () => {
const runAfter = vi.fn().mockResolvedValue("job-1");
const ctx = {
db: {
get: vi.fn(),
insert: vi.fn(),
normalizeId: vi.fn(),
patch: vi.fn(),
query: vi.fn(),
replace: vi.fn(),
delete: vi.fn(),
system: {
get: vi.fn(),
query: vi.fn(),
},
},
scheduler: {
runAfter,
},
} as never;
const result = await handler(ctx, { limit: 500 });
expect(runAfter).toHaveBeenCalledTimes(1);
expect(runAfter.mock.calls[0]?.[0]).toBe(0);
expect(runAfter.mock.calls[0]?.[2]).toEqual({ limit: 200 });
expect(result).toEqual({ ok: true, count: 0, scheduled: true });
});
});
+124 -22
View File
@@ -1,39 +1,141 @@
import { v } from 'convex/values'
import { internalMutation } from './_generated/server'
import { buildTrendingLeaderboard } from './lib/leaderboards'
import { v } from "convex/values";
import { internal } from "./_generated/api";
import { internalAction, internalMutation, internalQuery } from "./functions";
import {
buildTrendingEntriesFromDailyRows,
getTrendingRange,
queryDailyStats,
takeTopNonSuspiciousTrendingEntries,
takeTopTrendingEntries,
TRENDING_LEADERBOARD_KIND,
TRENDING_NON_SUSPICIOUS_LEADERBOARD_KIND,
} from "./lib/leaderboards";
const MAX_TRENDING_LIMIT = 200
const KEEP_LEADERBOARD_ENTRIES = 3
const MAX_TRENDING_LIMIT = 200;
const KEEP_LEADERBOARD_ENTRIES = 3;
export const rebuildTrendingLeaderboardInternal = internalMutation({
args: { limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = clampInt(args.limit ?? MAX_TRENDING_LIMIT, 1, MAX_TRENDING_LIMIT)
const now = Date.now()
const { startDay, endDay, items } = await buildTrendingLeaderboard(ctx, { limit, now })
// ---------------------------------------------------------------------------
// Action → Query → Mutation pattern (avoids 32K document-read limit)
// ---------------------------------------------------------------------------
await ctx.db.insert('skillLeaderboards', {
kind: 'trending',
/** Reads a single day's skillDailyStats in its own query transaction. */
export const getDailyStats = internalQuery({
args: { day: v.number() },
handler: async (ctx, { day }) => {
const rows = await queryDailyStats(ctx, day);
return rows.map((r) => ({ skillId: r.skillId, installs: r.installs, downloads: r.downloads }));
},
});
export const filterTopNonSuspiciousTrendingEntries = internalQuery({
args: {
entries: v.array(
v.object({
skillId: v.id("skills"),
score: v.number(),
installs: v.number(),
downloads: v.number(),
}),
),
limit: v.number(),
},
handler: async (ctx, { entries, limit }) => {
return takeTopNonSuspiciousTrendingEntries(ctx, entries, limit);
},
});
/** Writes the pre-computed leaderboard and prunes old entries. */
export const writeTrendingLeaderboard = internalMutation({
args: {
kind: v.string(),
items: v.array(
v.object({
skillId: v.id("skills"),
score: v.number(),
installs: v.number(),
downloads: v.number(),
}),
),
startDay: v.number(),
endDay: v.number(),
},
handler: async (ctx, { kind, items, startDay, endDay }) => {
const now = Date.now();
await ctx.db.insert("skillLeaderboards", {
kind,
generatedAt: now,
rangeStartDay: startDay,
rangeEndDay: endDay,
items,
})
});
const recent = await ctx.db
.query('skillLeaderboards')
.withIndex('by_kind', (q) => q.eq('kind', 'trending'))
.order('desc')
.take(KEEP_LEADERBOARD_ENTRIES + 5)
.query("skillLeaderboards")
.withIndex("by_kind", (q) => q.eq("kind", kind))
.order("desc")
.take(KEEP_LEADERBOARD_ENTRIES + 5);
for (const entry of recent.slice(KEEP_LEADERBOARD_ENTRIES)) {
await ctx.db.delete(entry._id)
await ctx.db.delete(entry._id);
}
return { ok: true as const, count: items.length }
return { ok: true as const, count: items.length };
},
})
});
/** Orchestrates the rebuild: queries each day separately, aggregates, writes. */
export const rebuildTrendingLeaderboardAction = internalAction({
args: { limit: v.optional(v.number()) },
handler: async (ctx, args): Promise<{ ok: true; count: number }> => {
const limit = clampInt(args.limit ?? MAX_TRENDING_LIMIT, 1, MAX_TRENDING_LIMIT);
const now = Date.now();
const { startDay, endDay } = getTrendingRange(now);
const dayKeys = Array.from({ length: endDay - startDay + 1 }, (_, i) => startDay + i);
const perDayRows = await Promise.all(
dayKeys.map((day) => ctx.runQuery(internal.leaderboards.getDailyStats, { day })),
);
const entries = buildTrendingEntriesFromDailyRows(perDayRows);
const items = takeTopTrendingEntries(entries, limit);
const nonSuspicious = await ctx.runQuery(
internal.leaderboards.filterTopNonSuspiciousTrendingEntries,
{ entries, limit },
);
await ctx.runMutation(internal.leaderboards.writeTrendingLeaderboard, {
kind: TRENDING_LEADERBOARD_KIND,
items,
startDay,
endDay,
});
await ctx.runMutation(internal.leaderboards.writeTrendingLeaderboard, {
kind: TRENDING_NON_SUSPICIOUS_LEADERBOARD_KIND,
items: nonSuspicious,
startDay,
endDay,
});
return { ok: true as const, count: items.length };
},
});
// ---------------------------------------------------------------------------
// Legacy single-mutation entrypoint kept as a compatibility shim.
// Old callers may still invoke this function name directly, but the
// rebuild itself must happen in the action/query/mutation pipeline so each
// daily read happens in its own transaction.
// ---------------------------------------------------------------------------
export const rebuildTrendingLeaderboardInternal = internalMutation({
args: { limit: v.optional(v.number()) },
handler: async (ctx, args) => {
const limit = clampInt(args.limit ?? MAX_TRENDING_LIMIT, 1, MAX_TRENDING_LIMIT);
await ctx.scheduler.runAfter(0, internal.leaderboards.rebuildTrendingLeaderboardAction, {
limit,
});
return { ok: true as const, count: 0, scheduled: true as const };
},
});
function clampInt(value: number, min: number, max: number) {
return Math.min(Math.max(value, min), max)
return Math.min(Math.max(value, min), max);
}
+131
View File
@@ -0,0 +1,131 @@
import { describe, expect, it, vi } from "vitest";
vi.mock("@convex-dev/auth/server", () => ({
getAuthUserId: vi.fn(),
authTables: {},
}));
const { getAuthUserId } = await import("@convex-dev/auth/server");
const { assertAdmin, assertModerator, assertRole, requireUser, requireUserFromAction } =
await import("./access");
describe("access.requireUser", () => {
it("throws when auth is missing", async () => {
vi.mocked(getAuthUserId).mockResolvedValue(null);
await expect(
requireUser({
db: { get: vi.fn() },
} as never),
).rejects.toThrow("Unauthorized");
});
it("throws when user is deleted/deactivated/missing", async () => {
vi.mocked(getAuthUserId).mockResolvedValue("users:1" as never);
for (const value of [
null,
{ _id: "users:1", deletedAt: Date.now() },
{ _id: "users:1", deactivatedAt: Date.now() },
]) {
const dbGet = vi.fn().mockResolvedValue(value as never);
await expect(
requireUser({
db: { get: dbGet },
} as never),
).rejects.toThrow("User not found");
}
});
it("throws when auth resolves to an invalid user id", async () => {
vi.mocked(getAuthUserId).mockResolvedValue("users:broken" as never);
const dbGet = vi.fn().mockRejectedValue(new Error("Table mismatch"));
await expect(
requireUser({
db: { get: dbGet },
} as never),
).rejects.toThrow("User not found");
});
it("returns auth user when active", async () => {
vi.mocked(getAuthUserId).mockResolvedValue("users:2" as never);
const user = { _id: "users:2", role: "user" };
const dbGet = vi.fn().mockResolvedValue(user as never);
const result = await requireUser({
db: { get: dbGet },
} as never);
expect(dbGet).toHaveBeenCalledWith("users:2");
expect(result).toEqual({ userId: "users:2", user });
});
});
describe("access.requireUserFromAction", () => {
it("throws when auth is missing", async () => {
vi.mocked(getAuthUserId).mockResolvedValue(null);
await expect(
requireUserFromAction({
runQuery: vi.fn(),
} as never),
).rejects.toThrow("Unauthorized");
});
it("throws when action lookup returns deleted/deactivated/missing user", async () => {
vi.mocked(getAuthUserId).mockResolvedValue("users:1" as never);
for (const value of [
null,
{ _id: "users:1", deletedAt: Date.now() },
{ _id: "users:1", deactivatedAt: Date.now() },
]) {
const runQuery = vi.fn().mockResolvedValue(value as never);
await expect(
requireUserFromAction({
runQuery,
} as never),
).rejects.toThrow("User not found");
}
});
it("throws when action auth resolves to an invalid user id", async () => {
vi.mocked(getAuthUserId).mockResolvedValue("users:broken" as never);
const runQuery = vi.fn().mockRejectedValue(new Error("Table mismatch"));
await expect(
requireUserFromAction({
runQuery,
} as never),
).rejects.toThrow("User not found");
});
it("returns active user from action query", async () => {
vi.mocked(getAuthUserId).mockResolvedValue("users:9" as never);
const user = { _id: "users:9", role: "admin" };
const runQuery = vi.fn().mockResolvedValue(user as never);
const result = await requireUserFromAction({
runQuery,
} as never);
expect(runQuery).toHaveBeenCalledTimes(1);
expect(result).toEqual({ userId: "users:9", user });
});
});
describe("access role assertions", () => {
it("assertRole allows matching roles and rejects missing role", () => {
expect(() => assertRole({ role: "admin" } as never, ["admin"])).not.toThrow();
expect(() => assertRole({ role: undefined } as never, ["admin"])).toThrow("Forbidden");
expect(() => assertRole({ role: "user" } as never, ["admin"])).toThrow("Forbidden");
});
it("assertAdmin/assertModerator enforce expected policy", () => {
expect(() => assertAdmin({ role: "admin" } as never)).not.toThrow();
expect(() => assertAdmin({ role: "moderator" } as never)).toThrow("Forbidden");
expect(() => assertModerator({ role: "admin" } as never)).not.toThrow();
expect(() => assertModerator({ role: "moderator" } as never)).not.toThrow();
expect(() => assertModerator({ role: "user" } as never)).toThrow("Forbidden");
});
});
+72 -24
View File
@@ -1,28 +1,76 @@
import { getAuthUserId } from '@convex-dev/auth/server'
import { api } from '../_generated/api'
import type { Doc } from '../_generated/dataModel'
import type { ActionCtx, MutationCtx, QueryCtx } from '../_generated/server'
import { getAuthUserId } from "@convex-dev/auth/server";
import { internal } from "../_generated/api";
import type { Doc, Id } from "../_generated/dataModel";
import type { ActionCtx, MutationCtx, QueryCtx } from "../_generated/server";
export type Role = 'admin' | 'moderator' | 'user'
export type Role = "admin" | "moderator" | "user";
export async function requireUser(ctx: MutationCtx | QueryCtx) {
const userId = await getAuthUserId(ctx)
if (!userId) throw new Error('Unauthorized')
const user = await ctx.db.get(userId)
if (!user || user.deletedAt) throw new Error('User not found')
return { userId, user }
}
export async function requireUserFromAction(ctx: ActionCtx) {
const userId = await getAuthUserId(ctx)
if (!userId) throw new Error('Unauthorized')
const user = await ctx.runQuery(api.users.getById, { userId })
if (!user || user.deletedAt) throw new Error('User not found')
return { userId, user: user as Doc<'users'> }
}
export function assertRole(user: Doc<'users'>, allowed: Role[]) {
if (!user.role || !allowed.includes(user.role as Role)) {
throw new Error('Forbidden')
export async function getOptionalActiveAuthUserId(
ctx: MutationCtx | QueryCtx,
): Promise<Id<"users"> | undefined> {
try {
const userId = await getAuthUserId(ctx);
if (!userId) return undefined;
const user = await ctx.db.get(userId);
if (!user || user.deletedAt || user.deactivatedAt) return undefined;
return userId;
} catch {
return undefined;
}
}
export async function getOptionalActiveAuthUserIdFromAction(
ctx: ActionCtx,
): Promise<Id<"users"> | undefined> {
try {
const userId = await getAuthUserId(ctx);
if (!userId) return undefined;
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId });
if (!user || user.deletedAt || user.deactivatedAt) return undefined;
return userId;
} catch {
return undefined;
}
}
export async function requireUser(ctx: MutationCtx | QueryCtx) {
const userId = await getAuthUserId(ctx);
if (!userId) throw new Error("Unauthorized");
let user: Doc<"users"> | null;
try {
user = await ctx.db.get(userId);
} catch {
throw new Error("User not found");
}
if (!user || user.deletedAt || user.deactivatedAt) throw new Error("User not found");
return { userId, user };
}
export async function requireUserFromAction(
ctx: ActionCtx,
): Promise<{ userId: Id<"users">; user: Doc<"users"> }> {
const userId = await getAuthUserId(ctx);
if (!userId) throw new Error("Unauthorized");
let user: Doc<"users"> | null;
try {
user = await ctx.runQuery(internal.users.getByIdInternal, { userId });
} catch {
throw new Error("User not found");
}
if (!user || user.deletedAt || user.deactivatedAt) throw new Error("User not found");
return { userId, user: user as Doc<"users"> };
}
export function assertRole(user: Doc<"users">, allowed: Role[]) {
if (!user.role || !allowed.includes(user.role as Role)) {
throw new Error("Forbidden");
}
}
export function assertAdmin(user: Doc<"users">) {
assertRole(user, ["admin"]);
}
export function assertModerator(user: Doc<"users">) {
assertRole(user, ["admin", "moderator"]);
}
+110
View File
@@ -0,0 +1,110 @@
import { describe, expect, it, vi } from "vitest";
import { getOptionalApiTokenUserId } from "./apiTokenAuth";
import { hashToken } from "./tokens";
describe("getOptionalApiTokenUserId", () => {
it("returns null when auth header is missing", async () => {
const ctx = {
runQuery: vi.fn(),
};
const request = new Request("https://example.com");
const userId = await getOptionalApiTokenUserId(ctx as never, request);
expect(userId).toBeNull();
expect(ctx.runQuery).not.toHaveBeenCalled();
});
it("returns null for unknown token", async () => {
const ctx = {
runQuery: vi.fn().mockResolvedValue(null),
};
const request = new Request("https://example.com", {
headers: { authorization: "Bearer token-1" },
});
const userId = await getOptionalApiTokenUserId(ctx as never, request);
expect(userId).toBeNull();
expect(ctx.runQuery).toHaveBeenCalledTimes(1);
expect(ctx.runQuery.mock.calls[0]?.[1]).toEqual({
tokenHash: await hashToken("token-1"),
});
});
it("returns user id when token and user are valid", async () => {
const tokenId = "apiTokens_1";
const expectedUserId = "users_1";
const ctx = {
runQuery: vi
.fn()
.mockImplementation(async (_fn, args: { tokenHash?: string; tokenId?: string }) => {
if (args.tokenHash) {
return { _id: tokenId, revokedAt: undefined };
}
if (args.tokenId) {
return { _id: expectedUserId, deletedAt: undefined };
}
return null;
}),
};
const request = new Request("https://example.com", {
headers: { authorization: "Bearer token-2" },
});
const userId = await getOptionalApiTokenUserId(ctx as never, request);
expect(userId).toBe(expectedUserId);
expect(ctx.runQuery).toHaveBeenCalledTimes(2);
});
it("returns null when user is deleted", async () => {
const tokenId = "apiTokens_2";
const ctx = {
runQuery: vi
.fn()
.mockImplementation(async (_fn, args: { tokenHash?: string; tokenId?: string }) => {
if (args.tokenHash) {
return { _id: tokenId, revokedAt: undefined };
}
if (args.tokenId) {
return { _id: "users_deleted", deletedAt: Date.now() };
}
return null;
}),
};
const request = new Request("https://example.com", {
headers: { authorization: "Bearer token-3" },
});
const userId = await getOptionalApiTokenUserId(ctx as never, request);
expect(userId).toBeNull();
expect(ctx.runQuery).toHaveBeenCalledTimes(2);
});
it("returns null when user is deactivated", async () => {
const tokenId = "apiTokens_3";
const ctx = {
runQuery: vi
.fn()
.mockImplementation(async (_fn, args: { tokenHash?: string; tokenId?: string }) => {
if (args.tokenHash) {
return { _id: tokenId, revokedAt: undefined };
}
if (args.tokenId) {
return { _id: "users_deactivated", deactivatedAt: Date.now() };
}
return null;
}),
};
const request = new Request("https://example.com", {
headers: { authorization: "Bearer token-4" },
});
const userId = await getOptionalApiTokenUserId(ctx as never, request);
expect(userId).toBeNull();
expect(ctx.runQuery).toHaveBeenCalledTimes(2);
});
});
+115 -24
View File
@@ -1,36 +1,127 @@
import { ConvexError } from 'convex/values'
import { internal } from '../_generated/api'
import type { Doc } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { hashToken } from './tokens'
import { ConvexError } from "convex/values";
import { internal } from "../_generated/api";
import type { Doc } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { hashToken } from "./tokens";
type TokenAuthResult = { user: Doc<'users'>; userId: Doc<'users'>['_id'] }
type TokenAuthResult = { user: Doc<"users">; userId: Doc<"users">["_id"] };
type ApiTokenDoc = Doc<"apiTokens">;
type PackagePublishTokenAuthResult = {
kind: "github-actions";
publishToken: Doc<"packagePublishTokens">;
};
type PackagePublishTokenDoc = Doc<"packagePublishTokens">;
type UserPackagePublishAuthResult = {
kind: "user";
user: Doc<"users">;
userId: Doc<"users">["_id"];
};
const internalRefs = internal as unknown as {
tokens: {
getByHashInternal: unknown;
getUserForTokenInternal: unknown;
touchInternal: unknown;
};
packagePublishTokens: {
getByHashInternal: unknown;
touchInternal: unknown;
};
};
export async function requireApiTokenUser(
ctx: ActionCtx,
request: Request,
): Promise<TokenAuthResult> {
const header = request.headers.get('authorization') ?? request.headers.get('Authorization')
const token = parseBearerToken(header)
if (!token) throw new ConvexError('Unauthorized')
const header = request.headers.get("authorization") ?? request.headers.get("Authorization");
const token = parseBearerToken(header);
if (!token) throw new ConvexError("Unauthorized");
const tokenHash = await hashToken(token)
const apiToken = await ctx.runQuery(internal.tokens.getByHashInternal, { tokenHash })
if (!apiToken || apiToken.revokedAt) throw new ConvexError('Unauthorized')
const tokenHash = await hashToken(token);
const apiToken = (await ctx.runQuery(
internalRefs.tokens.getByHashInternal as never,
{
tokenHash,
} as never,
)) as ApiTokenDoc | null;
if (!apiToken || apiToken.revokedAt) throw new ConvexError("Unauthorized");
const user = await ctx.runQuery(internal.tokens.getUserForTokenInternal, {
tokenId: apiToken._id,
})
if (!user || user.deletedAt) throw new ConvexError('Unauthorized')
const user = (await ctx.runQuery(
internalRefs.tokens.getUserForTokenInternal as never,
{
tokenId: apiToken._id,
} as never,
)) as Doc<"users"> | null;
if (!user || user.deletedAt || user.deactivatedAt) throw new ConvexError("Unauthorized");
await ctx.runMutation(internal.tokens.touchInternal, { tokenId: apiToken._id })
return { user, userId: user._id }
await ctx.runMutation(
internalRefs.tokens.touchInternal as never,
{ tokenId: apiToken._id } as never,
);
return { user, userId: user._id };
}
function parseBearerToken(header: string | null) {
if (!header) return null
const trimmed = header.trim()
if (!trimmed.toLowerCase().startsWith('bearer ')) return null
const token = trimmed.slice(7).trim()
return token || null
export async function getOptionalApiTokenUserId(
ctx: ActionCtx,
request: Request,
): Promise<Doc<"users">["_id"] | null> {
const header = request.headers.get("authorization") ?? request.headers.get("Authorization");
const token = parseBearerToken(header);
if (!token) return null;
const tokenHash = await hashToken(token);
const apiToken = (await ctx.runQuery(
internalRefs.tokens.getByHashInternal as never,
{
tokenHash,
} as never,
)) as ApiTokenDoc | null;
if (!apiToken || apiToken.revokedAt) return null;
const user = (await ctx.runQuery(
internalRefs.tokens.getUserForTokenInternal as never,
{
tokenId: apiToken._id,
} as never,
)) as Doc<"users"> | null;
if (!user || user.deletedAt || user.deactivatedAt) return null;
return user._id;
}
export async function requirePackagePublishAuth(
ctx: ActionCtx,
request: Request,
): Promise<UserPackagePublishAuthResult | PackagePublishTokenAuthResult> {
const header = request.headers.get("authorization") ?? request.headers.get("Authorization");
const token = parseBearerToken(header);
if (!token) throw new ConvexError("Unauthorized");
const tokenHash = await hashToken(token);
const publishToken = (await ctx.runQuery(
internalRefs.packagePublishTokens.getByHashInternal as never,
{
tokenHash,
} as never,
)) as PackagePublishTokenDoc | null;
if (publishToken && !publishToken.revokedAt && publishToken.expiresAt > Date.now()) {
await ctx.runMutation(
internalRefs.packagePublishTokens.touchInternal as never,
{
tokenId: publishToken._id,
} as never,
);
return { kind: "github-actions", publishToken };
}
const auth = await requireApiTokenUser(ctx, request);
return { kind: "user", user: auth.user, userId: auth.userId };
}
export function parseBearerToken(header: string | null) {
if (!header) return null;
const trimmed = header.trim();
if (!trimmed.toLowerCase().startsWith("bearer ")) return null;
const token = trimmed.slice(7).trim();
return token || null;
}
+50
View File
@@ -0,0 +1,50 @@
import type { Doc, Id } from "../_generated/dataModel";
import type { QueryCtx } from "../_generated/server";
type BadgeKind = Doc<"skillBadges">["kind"];
export type SkillBadgeMap = Partial<Record<BadgeKind, { byUserId: Id<"users">; at: number }>>;
export type SkillBadgeSource = { badges?: SkillBadgeMap | null };
type BadgeCtx = Pick<QueryCtx, "db">;
export function isSkillHighlighted(skill: SkillBadgeSource) {
return Boolean(skill.badges?.highlighted);
}
export function isSkillOfficial(skill: SkillBadgeSource) {
return Boolean(skill.badges?.official);
}
export function isSkillDeprecated(skill: SkillBadgeSource) {
return Boolean(skill.badges?.deprecated);
}
export function buildBadgeMap(records: Doc<"skillBadges">[]): SkillBadgeMap {
return records.reduce<SkillBadgeMap>((acc, record) => {
acc[record.kind] = { byUserId: record.byUserId, at: record.at };
return acc;
}, {});
}
export async function getSkillBadgeMap(
ctx: BadgeCtx,
skillId: Id<"skills">,
): Promise<SkillBadgeMap> {
const records = await ctx.db
.query("skillBadges")
.withIndex("by_skill", (q) => q.eq("skillId", skillId))
.take(10);
return buildBadgeMap(records);
}
export async function getSkillBadgeMaps(
ctx: BadgeCtx,
skillIds: Array<Id<"skills">>,
): Promise<Map<Id<"skills">, SkillBadgeMap>> {
const entries = await Promise.all(
skillIds.map(async (skillId) => [skillId, await getSkillBadgeMap(ctx, skillId)] as const),
);
return new Map(entries);
}
+19
View File
@@ -0,0 +1,19 @@
import type { Scheduler } from "convex/server";
export function scheduleNextBatchIfNeeded<TArgs extends { cursor?: string }>(
scheduler: Scheduler,
fn: unknown,
args: TArgs,
isDone: boolean,
continueCursor: string | null,
) {
if (isDone) return;
void scheduler.runAfter(
0,
fn as never,
{
...args,
cursor: continueCursor ?? undefined,
} as never,
);
}
+23 -23
View File
@@ -1,34 +1,34 @@
import { describe, expect, it } from 'vitest'
import { __test } from './changelog'
import { describe, expect, it } from "vitest";
import { __test } from "./changelog";
describe('changelog utils', () => {
it('summarizes file diffs', () => {
describe("changelog utils", () => {
it("summarizes file diffs", () => {
const diff = __test.summarizeFileDiff(
[
{ path: 'a.txt', sha256: 'aaa' },
{ path: 'b.txt', sha256: 'bbb' },
{ path: "a.txt", sha256: "aaa" },
{ path: "b.txt", sha256: "bbb" },
],
[
{ path: 'a.txt', sha256: 'aaa' },
{ path: 'b.txt', sha256: 'ccc' },
{ path: 'c.txt', sha256: 'ddd' },
{ path: "a.txt", sha256: "aaa" },
{ path: "b.txt", sha256: "ccc" },
{ path: "c.txt", sha256: "ddd" },
],
)
);
expect(diff.added).toEqual(['c.txt'])
expect(diff.removed).toEqual([])
expect(diff.changed).toEqual(['b.txt'])
expect(__test.formatDiffSummary(diff)).toBe('1 added, 1 changed')
})
expect(diff.added).toEqual(["c.txt"]);
expect(diff.removed).toEqual([]);
expect(diff.changed).toEqual(["b.txt"]);
expect(__test.formatDiffSummary(diff)).toBe("1 added, 1 changed");
});
it('generates a fallback initial release note', () => {
it("generates a fallback initial release note", () => {
const text = __test.generateFallback({
slug: 'demo',
version: '1.0.0',
slug: "demo",
version: "1.0.0",
oldReadme: null,
nextReadme: 'hi',
nextReadme: "hi",
fileDiff: null,
})
expect(text).toMatch(/Initial release/i)
})
})
});
expect(text).toMatch(/Initial release/i);
});
});
+110 -130
View File
@@ -1,161 +1,141 @@
import { internal } from '../_generated/api'
import type { Doc, Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { internal } from "../_generated/api";
import type { Doc, Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { extractResponseText } from "./openaiResponse";
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? 'gpt-4.1'
const MAX_README_CHARS = 8_000
const MAX_PATHS_IN_PROMPT = 30
const CHANGELOG_MODEL = process.env.OPENAI_CHANGELOG_MODEL ?? "gpt-4.1";
const MAX_README_CHARS = 8_000;
const MAX_PATHS_IN_PROMPT = 30;
type FileMeta = { path: string; sha256?: string }
type FileMeta = { path: string; sha256?: string };
type FileDiffSummary = {
added: string[]
removed: string[]
changed: string[]
}
added: string[];
removed: string[];
changed: string[];
};
function clampText(value: string, maxChars: number) {
const trimmed = value.trim()
if (trimmed.length <= maxChars) return trimmed
return `${trimmed.slice(0, maxChars).trimEnd()}\n…`
const trimmed = value.trim();
if (trimmed.length <= maxChars) return trimmed;
return `${trimmed.slice(0, maxChars).trimEnd()}\n…`;
}
function summarizeFileDiff(oldFiles: FileMeta[], nextFiles: FileMeta[]): FileDiffSummary {
const oldByPath = new Map(oldFiles.map((f) => [f.path, f] as const))
const nextByPath = new Map(nextFiles.map((f) => [f.path, f] as const))
const oldByPath = new Map(oldFiles.map((f) => [f.path, f] as const));
const nextByPath = new Map(nextFiles.map((f) => [f.path, f] as const));
const added: string[] = []
const removed: string[] = []
const changed: string[] = []
const added: string[] = [];
const removed: string[] = [];
const changed: string[] = [];
for (const [path, file] of nextByPath.entries()) {
const prev = oldByPath.get(path)
const prev = oldByPath.get(path);
if (!prev) {
added.push(path)
continue
added.push(path);
continue;
}
if (file.sha256 && prev.sha256 && file.sha256 !== prev.sha256) changed.push(path)
if (file.sha256 && prev.sha256 && file.sha256 !== prev.sha256) changed.push(path);
}
for (const path of oldByPath.keys()) {
if (!nextByPath.has(path)) removed.push(path)
if (!nextByPath.has(path)) removed.push(path);
}
added.sort()
removed.sort()
changed.sort()
return { added, removed, changed }
added.sort();
removed.sort();
changed.sort();
return { added, removed, changed };
}
function formatDiffSummary(diff: FileDiffSummary) {
const parts: string[] = []
if (diff.added.length) parts.push(`${diff.added.length} added`)
if (diff.changed.length) parts.push(`${diff.changed.length} changed`)
if (diff.removed.length) parts.push(`${diff.removed.length} removed`)
return parts.join(', ') || 'no file changes detected'
const parts: string[] = [];
if (diff.added.length) parts.push(`${diff.added.length} added`);
if (diff.changed.length) parts.push(`${diff.changed.length} changed`);
if (diff.removed.length) parts.push(`${diff.removed.length} removed`);
return parts.join(", ") || "no file changes detected";
}
function pickPaths(values: string[]) {
if (values.length <= MAX_PATHS_IN_PROMPT) return values
return values.slice(0, MAX_PATHS_IN_PROMPT)
}
function extractResponseText(payload: unknown) {
if (!payload || typeof payload !== 'object') return null
const output = (payload as { output?: unknown }).output
if (!Array.isArray(output)) return null
const chunks: string[] = []
for (const item of output) {
if (!item || typeof item !== 'object') continue
if ((item as { type?: unknown }).type !== 'message') continue
const content = (item as { content?: unknown }).content
if (!Array.isArray(content)) continue
for (const part of content) {
if (!part || typeof part !== 'object') continue
if ((part as { type?: unknown }).type !== 'output_text') continue
const text = (part as { text?: unknown }).text
if (typeof text === 'string' && text.trim()) chunks.push(text)
}
}
const joined = chunks.join('\n').trim()
return joined || null
if (values.length <= MAX_PATHS_IN_PROMPT) return values;
return values.slice(0, MAX_PATHS_IN_PROMPT);
}
async function generateWithOpenAI(args: {
slug: string
version: string
oldReadme: string | null
nextReadme: string
fileDiff: FileDiffSummary | null
slug: string;
version: string;
oldReadme: string | null;
nextReadme: string;
fileDiff: FileDiffSummary | null;
}) {
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) return null
const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) return null;
const oldReadme = args.oldReadme ? clampText(args.oldReadme, MAX_README_CHARS) : ''
const nextReadme = clampText(args.nextReadme, MAX_README_CHARS)
const oldReadme = args.oldReadme ? clampText(args.oldReadme, MAX_README_CHARS) : "";
const nextReadme = clampText(args.nextReadme, MAX_README_CHARS);
const fileDiff = args.fileDiff
const diffSummary = fileDiff ? formatDiffSummary(fileDiff) : 'unknown'
const changedPaths = fileDiff ? pickPaths(fileDiff.changed) : []
const addedPaths = fileDiff ? pickPaths(fileDiff.added) : []
const removedPaths = fileDiff ? pickPaths(fileDiff.removed) : []
const fileDiff = args.fileDiff;
const diffSummary = fileDiff ? formatDiffSummary(fileDiff) : "unknown";
const changedPaths = fileDiff ? pickPaths(fileDiff.changed) : [];
const addedPaths = fileDiff ? pickPaths(fileDiff.added) : [];
const removedPaths = fileDiff ? pickPaths(fileDiff.removed) : [];
const input = [
`Skill: ${args.slug}`,
`Version: ${args.version}`,
`File changes: ${diffSummary}`,
changedPaths.length ? `Changed files (sample): ${changedPaths.join(', ')}` : null,
addedPaths.length ? `Added files (sample): ${addedPaths.join(', ')}` : null,
removedPaths.length ? `Removed files (sample): ${removedPaths.join(', ')}` : null,
changedPaths.length ? `Changed files (sample): ${changedPaths.join(", ")}` : null,
addedPaths.length ? `Added files (sample): ${addedPaths.join(", ")}` : null,
removedPaths.length ? `Removed files (sample): ${removedPaths.join(", ")}` : null,
oldReadme ? `Previous SKILL.md:\n${oldReadme}` : null,
`New SKILL.md:\n${nextReadme}`,
]
.filter(Boolean)
.join('\n\n')
.join("\n\n");
const response = await fetch('https://api.openai.com/v1/responses', {
method: 'POST',
const response = await fetch("https://api.openai.com/v1/responses", {
method: "POST",
headers: {
'Content-Type': 'application/json',
"Content-Type": "application/json",
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: CHANGELOG_MODEL,
instructions:
'Write a concise changelog for this skill version. Audience: everyone. Output plain text. Prefer 26 bullet points. If it is a big change, include a short 1-line summary first, then bullets. Dont mention that you are AI. Dont invent details; only use the inputs.',
"Write a concise changelog for this skill version. Audience: everyone. Output plain text. Prefer 26 bullet points. If it is a big change, include a short 1-line summary first, then bullets. Dont mention that you are AI. Dont invent details; only use the inputs.",
input,
max_output_tokens: 220,
}),
})
});
if (!response.ok) return null
const payload = (await response.json()) as unknown
return extractResponseText(payload)
if (!response.ok) return null;
const payload = (await response.json()) as unknown;
return extractResponseText(payload);
}
function generateFallback(args: {
slug: string
version: string
oldReadme: string | null
nextReadme: string
fileDiff: FileDiffSummary | null
slug: string;
version: string;
oldReadme: string | null;
nextReadme: string;
fileDiff: FileDiffSummary | null;
}) {
const lines: string[] = []
const lines: string[] = [];
if (!args.oldReadme) {
lines.push(`- Initial release.`)
return lines.join('\n')
lines.push(`- Initial release.`);
return lines.join("\n");
}
const diff = args.fileDiff
const diff = args.fileDiff;
if (diff) {
const parts: string[] = []
if (diff.added.length) parts.push(`added ${diff.added.length}`)
if (diff.changed.length) parts.push(`updated ${diff.changed.length}`)
if (diff.removed.length) parts.push(`removed ${diff.removed.length}`)
if (parts.length) lines.push(`- ${parts.join(', ')} file(s).`)
const parts: string[] = [];
if (diff.added.length) parts.push(`added ${diff.added.length}`);
if (diff.changed.length) parts.push(`updated ${diff.changed.length}`);
if (diff.removed.length) parts.push(`removed ${diff.removed.length}`);
if (parts.length) lines.push(`- ${parts.join(", ")} file(s).`);
}
lines.push(`- Updated SKILL.md and bundle contents.`)
return lines.join('\n')
lines.push(`- Updated SKILL.md and bundle contents.`);
return lines.join("\n");
}
export async function generateChangelogForPublish(
@@ -165,21 +145,21 @@ export async function generateChangelogForPublish(
try {
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: args.slug,
})) as Doc<'skills'> | null
const previous: Doc<'skillVersions'> | null =
})) as Doc<"skills"> | null;
const previous: Doc<"skillVersions"> | null =
skill?.latestVersionId && !skill.softDeletedAt
? ((await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: skill.latestVersionId,
})) as Doc<'skillVersions'> | null)
: null
})) as Doc<"skillVersions"> | null)
: null;
const oldReadmeText: string | null = previous
? await readReadmeFromVersion(ctx, previous)
: null
: null;
const oldFiles = previous
? previous.files.map((file) => ({ path: file.path, sha256: file.sha256 }))
: []
const fileDiff = previous ? summarizeFileDiff(oldFiles, args.files) : null
: [];
const fileDiff = previous ? summarizeFileDiff(oldFiles, args.files) : null;
const ai = await generateWithOpenAI({
slug: args.slug,
@@ -187,7 +167,7 @@ export async function generateChangelogForPublish(
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
}).catch(() => null)
}).catch(() => null);
return (
ai ??
@@ -198,42 +178,42 @@ export async function generateChangelogForPublish(
nextReadme: args.readmeText,
fileDiff,
})
)
);
} catch {
return '- Updated skill.'
return "- Updated skill.";
}
}
export async function generateChangelogPreview(
ctx: ActionCtx,
args: {
slug: string
version: string
readmeText: string
filePaths?: string[]
slug: string;
version: string;
readmeText: string;
filePaths?: string[];
},
): Promise<string> {
try {
const skill = (await ctx.runQuery(internal.skills.getSkillBySlugInternal, {
slug: args.slug,
})) as Doc<'skills'> | null
const previous: Doc<'skillVersions'> | null =
})) as Doc<"skills"> | null;
const previous: Doc<"skillVersions"> | null =
skill?.latestVersionId && !skill.softDeletedAt
? ((await ctx.runQuery(internal.skills.getVersionByIdInternal, {
versionId: skill.latestVersionId,
})) as Doc<'skillVersions'> | null)
: null
})) as Doc<"skillVersions"> | null)
: null;
const oldReadmeText: string | null = previous
? await readReadmeFromVersion(ctx, previous)
: null
: null;
const fileDiff =
previous && args.filePaths
? summarizeFileDiff(
previous.files.map((file) => ({ path: file.path, sha256: file.sha256 })),
args.filePaths.map((path) => ({ path })),
)
: null
: null;
const ai = await generateWithOpenAI({
slug: args.slug,
@@ -241,7 +221,7 @@ export async function generateChangelogPreview(
oldReadme: oldReadmeText,
nextReadme: args.readmeText,
fileDiff,
}).catch(() => null)
}).catch(() => null);
return (
ai ??
@@ -252,21 +232,21 @@ export async function generateChangelogPreview(
nextReadme: args.readmeText,
fileDiff,
})
)
);
} catch {
return '- Updated skill.'
return "- Updated skill.";
}
}
async function readReadmeFromVersion(ctx: ActionCtx, version: Doc<'skillVersions'>) {
async function readReadmeFromVersion(ctx: ActionCtx, version: Doc<"skillVersions">) {
const readmeFile = version.files.find((file) => {
const lower = file.path.toLowerCase()
return lower === 'skill.md' || lower === 'skills.md'
})
if (!readmeFile) return null
const blob = await ctx.storage.get(readmeFile.storageId as Id<'_storage'>)
if (!blob) return null
return blob.text()
const lower = file.path.toLowerCase();
return lower === "skill.md" || lower === "skills.md";
});
if (!readmeFile) return null;
const blob = await ctx.storage.get(readmeFile.storageId as Id<"_storage">);
if (!blob) return null;
return blob.text();
}
export const __test = {
@@ -275,4 +255,4 @@ export const __test = {
formatDiffSummary,
summarizeFileDiff,
generateFallback,
}
};
+77
View File
@@ -0,0 +1,77 @@
/* @vitest-environment node */
import { describe, expect, it } from "vitest";
import {
assembleCommentScamEvalUserMessage,
buildCommentScamBanReason,
isCertainScam,
parseCommentScamEvalResponse,
} from "./commentScamPrompt";
describe("commentScamPrompt", () => {
it("parses valid JSON response", () => {
const parsed = parseCommentScamEvalResponse(
JSON.stringify({
verdict: "certain_scam",
confidence: "high",
explanation: "Comment instructs users to decode base64 and pipe to bash.",
evidence: ["echo + base64 -D | bash", "fake update-service domain"],
}),
);
expect(parsed).toEqual({
verdict: "certain_scam",
confidence: "high",
explanation: "Comment instructs users to decode base64 and pipe to bash.",
evidence: ["echo + base64 -D | bash", "fake update-service domain"],
});
});
it("parses markdown-fenced JSON", () => {
const parsed = parseCommentScamEvalResponse(`\`\`\`json
{"verdict":"likely_scam","confidence":"medium","explanation":"Suspicious terminal one-liner.","evidence":["curl | bash"]}
\`\`\``);
expect(parsed).toMatchObject({
verdict: "likely_scam",
confidence: "medium",
});
});
it("rejects invalid response payloads", () => {
expect(parseCommentScamEvalResponse('{"verdict":"ban"}')).toBeNull();
expect(parseCommentScamEvalResponse("not-json")).toBeNull();
});
it("builds bounded ban reason", () => {
const reason = buildCommentScamBanReason({
commentId: "comments:1",
skillId: "skills:1",
explanation: "A".repeat(700),
evidence: ["B".repeat(300), "C".repeat(300), "D".repeat(300), "E".repeat(300)],
});
expect(reason.length).toBeLessThanOrEqual(500);
expect(reason).toContain("commentId=comments:1");
expect(reason).toContain("skillId=skills:1");
});
it("marks certainty only for high-confidence certain_scam", () => {
expect(isCertainScam({ verdict: "certain_scam", confidence: "high" })).toBe(true);
expect(isCertainScam({ verdict: "certain_scam", confidence: "medium" })).toBe(false);
expect(isCertainScam({ verdict: "likely_scam", confidence: "high" })).toBe(false);
});
it("builds compact user message with context", () => {
const message = assembleCommentScamEvalUserMessage({
commentId: "comments:1",
skillId: "skills:3",
userId: "users:9",
body: " test ",
});
expect(message).toContain("Comment ID: comments:1");
expect(message).toContain("Skill ID: skills:3");
expect(message).toContain("Author User ID: users:9");
expect(message).toContain("test");
});
});
+155
View File
@@ -0,0 +1,155 @@
export type CommentScamVerdict = "not_scam" | "likely_scam" | "certain_scam";
export type CommentScamConfidence = "low" | "medium" | "high";
export type CommentScamEvalResponse = {
verdict: CommentScamVerdict;
confidence: CommentScamConfidence;
explanation: string;
evidence: string[];
};
export const COMMENT_SCAM_EVAL_MAX_OUTPUT_TOKENS = 1200;
const MAX_COMMENT_CHARS = 4000;
const MAX_EXPLANATION_CHARS = 1200;
const MAX_EVIDENCE_ITEMS = 5;
const MAX_EVIDENCE_ITEM_CHARS = 160;
const MAX_BAN_REASON_CHARS = 500;
const VALID_VERDICTS = new Set<CommentScamVerdict>(["not_scam", "likely_scam", "certain_scam"]);
const VALID_CONFIDENCES = new Set<CommentScamConfidence>(["low", "medium", "high"]);
export const COMMENT_SCAM_EVALUATOR_SYSTEM_PROMPT = `You are a trust and safety classifier for user comments on a software registry.
Goal: detect comment scams with high precision.
A "certain_scam" verdict is only allowed when the comment clearly attempts fraud, credential theft, malware delivery, or social-engineering abuse.
High-confidence scam patterns include:
- Instructing users to run suspicious shell commands (especially obfuscated/base64/piped-to-bash/curl installer tricks).
- Fake support/update instructions pointing to unknown domains, executables, or terminal one-liners.
- Requests for private keys, seed phrases, passwords, API keys, session tokens, or wallet recovery data.
- Impersonation or urgent pressure language to bypass trust checks.
- Known scam payload structure (e.g. echo+base64+decode+bash, hidden downloader chains).
Important anti-false-positive rules:
- Do NOT mark legitimate troubleshooting or normal install instructions as "certain_scam" unless the malicious intent is explicit.
- If suspicious but ambiguous, use "likely_scam".
- If benign/unclear, use "not_scam".
Output JSON only:
{
"verdict": "not_scam" | "likely_scam" | "certain_scam",
"confidence": "low" | "medium" | "high",
"explanation": "short plain-language rationale",
"evidence": ["short concrete signal", "..."]
}`;
export function getCommentScamEvalModel(): string {
return process.env.OPENAI_COMMENT_EVAL_MODEL ?? process.env.OPENAI_EVAL_MODEL ?? "gpt-5-mini";
}
export function assembleCommentScamEvalUserMessage(args: {
commentId: string;
skillId: string;
userId: string;
body: string;
}): string {
const trimmed = args.body.trim();
const body =
trimmed.length > MAX_COMMENT_CHARS
? `${trimmed.slice(0, MAX_COMMENT_CHARS)}\n…[truncated]`
: trimmed;
return [
`Comment ID: ${args.commentId}`,
`Skill ID: ${args.skillId}`,
`Author User ID: ${args.userId}`,
"Comment body:",
"```",
body,
"```",
"Respond with a single JSON object.",
].join("\n");
}
function stripCodeFence(raw: string): string {
const text = raw.trim();
if (!text.startsWith("```")) return text;
const firstNewline = text.indexOf("\n");
if (firstNewline === -1) return text;
const withoutOpening = text.slice(firstNewline + 1);
const lastFence = withoutOpening.lastIndexOf("```");
if (lastFence === -1) return withoutOpening.trim();
return withoutOpening.slice(0, lastFence).trim();
}
function truncate(value: string, max: number): string {
if (value.length <= max) return value;
if (max <= 3) return value.slice(0, max);
return `${value.slice(0, max - 3)}...`;
}
export function parseCommentScamEvalResponse(raw: string): CommentScamEvalResponse | null {
let parsed: unknown;
try {
parsed = JSON.parse(stripCodeFence(raw));
} catch {
return null;
}
if (!parsed || typeof parsed !== "object") return null;
const obj = parsed as Record<string, unknown>;
const verdict =
typeof obj.verdict === "string" ? (obj.verdict.toLowerCase() as CommentScamVerdict) : null;
if (!verdict || !VALID_VERDICTS.has(verdict)) return null;
const confidence =
typeof obj.confidence === "string"
? (obj.confidence.toLowerCase() as CommentScamConfidence)
: null;
if (!confidence || !VALID_CONFIDENCES.has(confidence)) return null;
const rawExplanation = typeof obj.explanation === "string" ? obj.explanation.trim() : "";
if (!rawExplanation) return null;
const rawEvidence = Array.isArray(obj.evidence) ? obj.evidence : [];
const evidence = rawEvidence
.map((item) => (typeof item === "string" ? item.trim() : ""))
.filter(Boolean)
.slice(0, MAX_EVIDENCE_ITEMS)
.map((item) => truncate(item, MAX_EVIDENCE_ITEM_CHARS));
return {
verdict,
confidence,
explanation: truncate(rawExplanation, MAX_EXPLANATION_CHARS),
evidence,
};
}
export function isCertainScam(result: {
verdict: CommentScamVerdict;
confidence: CommentScamConfidence;
}): boolean {
return result.verdict === "certain_scam" && result.confidence === "high";
}
export function buildCommentScamBanReason(args: {
commentId: string;
skillId: string;
explanation: string;
evidence: string[];
}): string {
const explanation = args.explanation.trim();
const evidence = args.evidence
.map((item) => item.trim())
.filter(Boolean)
.slice(0, 3);
const suffix = ` commentId=${args.commentId} skillId=${args.skillId}`;
const evidenceSegment = evidence.length > 0 ? ` evidence: ${evidence.join("; ")}.` : "";
const core = `comment scam auto-ban. ${explanation}.${evidenceSegment}`;
const maxCoreChars = Math.max(0, MAX_BAN_REASON_CHARS - suffix.length);
return `${truncate(core, maxCoreChars)}${suffix}`;
}
+17
View File
@@ -0,0 +1,17 @@
const EXT_TO_TYPE: Record<string, string> = {
md: "text/markdown",
mdx: "text/markdown",
json: "application/json",
json5: "application/json",
yaml: "application/yaml",
yml: "application/yaml",
toml: "application/toml",
svg: "image/svg+xml",
};
export function guessContentTypeForPath(path: string) {
const trimmed = path.trim().toLowerCase();
if (!trimmed) return "application/octet-stream";
const ext = trimmed.split(".").at(-1) ?? "";
return EXT_TO_TYPE[ext] ?? "application/octet-stream";
}
+16
View File
@@ -0,0 +1,16 @@
export type EmbeddingVisibility =
| "latest"
| "latest-approved"
| "archived"
| "archived-approved"
| "deleted";
export function embeddingVisibilityFor(
isLatest: boolean,
isApproved: boolean,
): Exclude<EmbeddingVisibility, "deleted"> {
if (isLatest && isApproved) return "latest-approved";
if (isLatest) return "latest";
if (isApproved) return "archived-approved";
return "archived";
}
+95
View File
@@ -0,0 +1,95 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { EMBEDDING_DIMENSIONS, generateEmbedding } from "./embeddings";
const fetchMock = vi.fn<typeof fetch>();
const consoleWarnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
const originalFetch = globalThis.fetch;
const originalApiKey = process.env.OPENAI_API_KEY;
function jsonResponse(payload: unknown, init?: ResponseInit) {
return new Response(JSON.stringify(payload), {
status: 200,
headers: {
"content-type": "application/json",
},
...init,
});
}
beforeEach(() => {
fetchMock.mockReset();
globalThis.fetch = fetchMock as typeof fetch;
process.env.OPENAI_API_KEY = "test-key";
consoleWarnSpy.mockClear();
});
afterEach(() => {
globalThis.fetch = originalFetch;
if (originalApiKey === undefined) {
delete process.env.OPENAI_API_KEY;
} else {
process.env.OPENAI_API_KEY = originalApiKey;
}
vi.useRealTimers();
});
describe("generateEmbedding", () => {
it("returns zero embedding when OPENAI_API_KEY is missing", async () => {
delete process.env.OPENAI_API_KEY;
const result = await generateEmbedding("hello world");
expect(result).toHaveLength(EMBEDDING_DIMENSIONS);
expect(result.every((value) => value === 0)).toBe(true);
expect(fetchMock).not.toHaveBeenCalled();
});
it("retries on 429 responses and then succeeds", async () => {
vi.useFakeTimers();
fetchMock.mockResolvedValueOnce(new Response("rate limited", { status: 429 }));
fetchMock.mockResolvedValueOnce(jsonResponse({ data: [{ embedding: [0.25, 0.75] }] }));
const promise = generateEmbedding("retry me");
await vi.runAllTimersAsync();
await expect(promise).resolves.toEqual([0.25, 0.75]);
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it("does not retry non-retryable 4xx responses", async () => {
fetchMock.mockResolvedValueOnce(new Response("bad request", { status: 400 }));
await expect(generateEmbedding("bad")).rejects.toThrow("Embedding failed: bad request");
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it("retries on network failures and then succeeds", async () => {
vi.useFakeTimers();
fetchMock.mockRejectedValueOnce(new TypeError("fetch failed"));
fetchMock.mockResolvedValueOnce(jsonResponse({ data: [{ embedding: [1, 2, 3] }] }));
const promise = generateEmbedding("network retry");
await vi.runAllTimersAsync();
await expect(promise).resolves.toEqual([1, 2, 3]);
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it("retries timeouts up to max attempts and preserves timeout error", async () => {
vi.useFakeTimers();
fetchMock.mockRejectedValue(new DOMException("aborted", "AbortError"));
const promise = generateEmbedding("always timeout");
const rejection = expect(promise).rejects.toThrow(
"OpenAI API request timed out after 10 seconds",
);
await vi.runAllTimersAsync();
await rejection;
expect(fetchMock).toHaveBeenCalledTimes(3);
});
});
+135 -26
View File
@@ -1,38 +1,147 @@
export const EMBEDDING_MODEL = 'text-embedding-3-small'
export const EMBEDDING_DIMENSIONS = 1536
export const EMBEDDING_MODEL = "text-embedding-3-small";
export const EMBEDDING_DIMENSIONS = 1536;
const EMBEDDING_ENDPOINT = "https://api.openai.com/v1/embeddings";
const REQUEST_TIMEOUT_MS = 10_000;
const MAX_ATTEMPTS = 3;
const BASE_RETRY_DELAY_MS = 1_000;
class RetryableEmbeddingError extends Error {
constructor(message: string, options?: { cause?: unknown }) {
super(message, options);
this.name = "RetryableEmbeddingError";
}
}
function emptyEmbedding() {
return Array.from({ length: EMBEDDING_DIMENSIONS }, () => 0)
return Array.from({ length: EMBEDDING_DIMENSIONS }, () => 0);
}
function parseRetryAfterMs(retryAfterHeader: string | null) {
if (!retryAfterHeader) return null;
const seconds = Number(retryAfterHeader);
if (Number.isFinite(seconds) && seconds >= 0) {
return Math.round(seconds * 1000);
}
const dateMs = Date.parse(retryAfterHeader);
if (Number.isFinite(dateMs)) {
return Math.max(0, dateMs - Date.now());
}
return null;
}
function getRetryDelayMs(attempt: number, retryAfterMs: number | null) {
const exponentialDelayMs = BASE_RETRY_DELAY_MS * 2 ** attempt;
if (retryAfterMs == null) return exponentialDelayMs;
return Math.max(exponentialDelayMs, retryAfterMs);
}
function normalizeRetryableNetworkError(error: unknown) {
if (!(error instanceof Error)) return null;
if (error.name === "AbortError") {
return new RetryableEmbeddingError(
`OpenAI API request timed out after ${Math.floor(REQUEST_TIMEOUT_MS / 1000)} seconds`,
{ cause: error },
);
}
if (error instanceof TypeError) {
return new RetryableEmbeddingError(`Embedding request failed: ${error.message}`, {
cause: error,
});
}
return null;
}
function sleep(ms: number) {
return new Promise<void>((resolve) => {
setTimeout(resolve, ms);
});
}
export async function generateEmbedding(text: string) {
const apiKey = process.env.OPENAI_API_KEY
const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) {
console.warn('OPENAI_API_KEY is not configured; using zero embeddings')
return emptyEmbedding()
console.warn("OPENAI_API_KEY is not configured; using zero embeddings");
return emptyEmbedding();
}
const response = await fetch('https://api.openai.com/v1/embeddings', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: EMBEDDING_MODEL,
input: text,
}),
})
let lastRetryableError: RetryableEmbeddingError | null = null;
if (!response.ok) {
const message = await response.text()
throw new Error(`Embedding failed: ${message}`)
for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) {
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
try {
const response = await fetch(EMBEDDING_ENDPOINT, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify({
model: EMBEDDING_MODEL,
input: text,
}),
signal: controller.signal,
});
if (!response.ok) {
const message = await response.text();
const isRetryableStatus = response.status === 429 || response.status >= 500;
if (isRetryableStatus) {
const retryableError = new RetryableEmbeddingError(
`Embedding failed (${response.status}): ${message}`,
);
lastRetryableError = retryableError;
if (attempt < MAX_ATTEMPTS - 1) {
const retryAfterMs = parseRetryAfterMs(response.headers.get("retry-after"));
const delayMs = getRetryDelayMs(attempt, retryAfterMs);
console.warn(
`OpenAI embeddings retry in ${delayMs}ms (attempt ${attempt + 1}/${MAX_ATTEMPTS})`,
);
await sleep(delayMs);
continue;
}
throw retryableError;
}
throw new Error(`Embedding failed: ${message}`);
}
const payload = (await response.json()) as {
data?: Array<{ embedding: number[] }>;
};
const embedding = payload.data?.[0]?.embedding;
if (!embedding) throw new Error("Embedding missing from response");
return embedding;
} catch (error) {
const retryableNetworkError = normalizeRetryableNetworkError(error);
if (retryableNetworkError) {
lastRetryableError = retryableNetworkError;
if (attempt < MAX_ATTEMPTS - 1) {
const delayMs = getRetryDelayMs(attempt, null);
console.warn(
`OpenAI embeddings network retry in ${delayMs}ms (attempt ${attempt + 1}/${MAX_ATTEMPTS})`,
);
await sleep(delayMs);
continue;
}
throw retryableNetworkError;
}
throw error;
} finally {
clearTimeout(timeoutId);
}
}
const payload = (await response.json()) as {
data?: Array<{ embedding: number[] }>
}
const embedding = payload.data?.[0]?.embedding
if (!embedding) throw new Error('Embedding missing from response')
return embedding
throw lastRetryableError ?? new Error("Embedding failed after retries");
}
+405
View File
@@ -0,0 +1,405 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { internal } from "../_generated/api";
import { requireGitHubAccountAge, syncGitHubProfile } from "./githubAccount";
vi.mock("../_generated/api", () => ({
internal: {
githubIdentity: {
getGitHubProviderAccountIdInternal: Symbol("getGitHubProviderAccountIdInternal"),
},
users: {
getByIdInternal: Symbol("getByIdInternal"),
setGitHubCreatedAtInternal: Symbol("setGitHubCreatedAtInternal"),
syncGitHubProfileInternal: Symbol("syncGitHubProfileInternal"),
},
},
}));
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
describe("requireGitHubAccountAge", () => {
beforeEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
vi.unstubAllGlobals();
});
afterEach(() => {
vi.useRealTimers();
vi.unstubAllEnvs();
vi.unstubAllGlobals();
});
it("uses cached githubCreatedAt when present", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const runQuery = vi.fn().mockResolvedValue({
_id: "users:1",
githubCreatedAt: now.getTime() - 20 * ONE_DAY_MS,
});
const runMutation = vi.fn();
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
await requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never);
expect(fetchMock).not.toHaveBeenCalled();
expect(runMutation).not.toHaveBeenCalled();
expect(runQuery).toHaveBeenCalledWith(internal.users.getByIdInternal, { userId: "users:1" });
expect(runQuery).not.toHaveBeenCalledWith(
internal.githubIdentity.getGitHubProviderAccountIdInternal,
{ userId: "users:1" },
);
});
it("rejects deactivated users", async () => {
const runQuery = vi.fn().mockResolvedValue({
_id: "users:1",
deactivatedAt: Date.now(),
});
const runMutation = vi.fn();
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/User not found/i);
expect(fetchMock).not.toHaveBeenCalled();
});
it("rejects accounts younger than 14 days", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const runQuery = vi.fn().mockResolvedValue({
_id: "users:1",
githubCreatedAt: now.getTime() - 2 * ONE_DAY_MS,
});
const runMutation = vi.fn();
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/GitHub account must be at least 14 days old/i);
});
it("fetches githubCreatedAt when missing (by providerAccountId)", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
created_at: "2020-01-01T00:00:00Z",
}),
});
vi.stubGlobal("fetch", fetchMock);
await requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never);
expect(fetchMock).toHaveBeenCalledWith(
"https://api.github.com/user/12345",
expect.objectContaining({
headers: expect.objectContaining({ "User-Agent": "clawhub" }),
}),
);
expect(runMutation).toHaveBeenCalledWith(internal.users.setGitHubCreatedAtInternal, {
userId: "users:1",
githubCreatedAt: Date.parse("2020-01-01T00:00:00Z"),
});
});
it("rejects when providerAccountId is missing", async () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce(null);
const runMutation = vi.fn();
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/GitHub account required/i);
expect(fetchMock).not.toHaveBeenCalled();
});
it("rejects when providerAccountId is invalid", async () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("abc123");
const runMutation = vi.fn();
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/GitHub account lookup failed/i);
expect(fetchMock).not.toHaveBeenCalled();
});
it("throws when GitHub lookup fails", async () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 404 });
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/GitHub account lookup failed/i);
});
it("throws rate-limit error on 403", async () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 403 });
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/rate limit exceeded/i);
});
it("throws rate-limit error on 429", async () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 429 });
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/rate limit exceeded/i);
});
it("throws when GitHub returns an invalid payload", async () => {
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({}),
});
vi.stubGlobal("fetch", fetchMock);
await expect(
requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never),
).rejects.toThrow(/GitHub account lookup failed/i);
});
it("includes Authorization header when GITHUB_TOKEN is set", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
vi.stubEnv("GITHUB_TOKEN", "ghp_test123");
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
githubCreatedAt: undefined,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
created_at: "2020-01-01T00:00:00Z",
}),
});
vi.stubGlobal("fetch", fetchMock);
await requireGitHubAccountAge({ runQuery, runMutation } as never, "users:1" as never);
expect(fetchMock).toHaveBeenCalledWith(
"https://api.github.com/user/12345",
expect.objectContaining({
headers: {
"User-Agent": "clawhub",
Authorization: "Bearer ghp_test123",
},
}),
);
});
});
describe("syncGitHubProfile", () => {
beforeEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
vi.unstubAllGlobals();
});
afterEach(() => {
vi.useRealTimers();
vi.unstubAllEnvs();
vi.unstubAllGlobals();
});
it("skips recent syncs (throttle)", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const runQuery = vi.fn().mockResolvedValueOnce({
_id: "users:1",
name: "oldname",
githubProfileSyncedAt: now.getTime(),
});
const runMutation = vi.fn();
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
await syncGitHubProfile({ runQuery, runMutation } as never, "users:1" as never);
expect(fetchMock).not.toHaveBeenCalled();
expect(runMutation).not.toHaveBeenCalled();
});
it("updates profile even when only avatar changes", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
name: "same",
image: "https://avatars.githubusercontent.com/u/1?v=3",
githubProfileSyncedAt: now.getTime() - 10 * ONE_DAY_MS,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
login: "same",
avatar_url: "https://avatars.githubusercontent.com/u/1?v=4",
}),
});
vi.stubGlobal("fetch", fetchMock);
await syncGitHubProfile({ runQuery, runMutation } as never, "users:1" as never);
expect(runMutation).toHaveBeenCalledWith(internal.users.syncGitHubProfileInternal, {
userId: "users:1",
name: "same",
image: "https://avatars.githubusercontent.com/u/1?v=4",
syncedAt: now.getTime(),
});
});
it("updates name and records sync timestamp", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
name: "old",
githubProfileSyncedAt: now.getTime() - 10 * ONE_DAY_MS,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
login: "new",
avatar_url: "https://avatars.githubusercontent.com/u/1?v=1",
}),
});
vi.stubGlobal("fetch", fetchMock);
await syncGitHubProfile({ runQuery, runMutation } as never, "users:1" as never);
expect(runMutation).toHaveBeenCalledWith(internal.users.syncGitHubProfileInternal, {
userId: "users:1",
name: "new",
image: "https://avatars.githubusercontent.com/u/1?v=1",
syncedAt: now.getTime(),
});
});
it("forwards GitHub profile name (full name) when present", async () => {
vi.useFakeTimers();
const now = new Date("2026-02-02T12:00:00Z");
vi.setSystemTime(now);
const runQuery = vi
.fn()
.mockResolvedValueOnce({
_id: "users:1",
name: "same",
githubProfileSyncedAt: now.getTime() - 10 * ONE_DAY_MS,
})
.mockResolvedValueOnce("12345");
const runMutation = vi.fn();
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
login: "same",
name: "Real Name",
avatar_url: "https://avatars.githubusercontent.com/u/1?v=1",
}),
});
vi.stubGlobal("fetch", fetchMock);
await syncGitHubProfile({ runQuery, runMutation } as never, "users:1" as never);
expect(runMutation).toHaveBeenCalledWith(internal.users.syncGitHubProfileInternal, {
userId: "users:1",
name: "same",
image: "https://avatars.githubusercontent.com/u/1?v=1",
profileName: "Real Name",
syncedAt: now.getTime(),
});
});
});
+143
View File
@@ -0,0 +1,143 @@
import { ConvexError } from "convex/values";
import { internal } from "../_generated/api";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
import { GITHUB_PROFILE_SYNC_WINDOW_MS } from "./githubProfileSync";
const GITHUB_API = "https://api.github.com";
const MIN_ACCOUNT_AGE_MS = 14 * 24 * 60 * 60 * 1000;
type GitHubAccountGateCtx = Pick<ActionCtx, "runQuery" | "runMutation">;
type GitHubUser = {
login?: string;
name?: string;
avatar_url?: string;
created_at?: string;
};
function assertGitHubNumericId(providerAccountId: string) {
if (!/^[0-9]+$/.test(providerAccountId)) {
throw new ConvexError("GitHub account lookup failed");
}
}
function buildGitHubHeaders() {
const headers: Record<string, string> = { "User-Agent": "clawhub" };
const token = process.env.GITHUB_TOKEN;
if (token) {
headers.Authorization = `Bearer ${token}`;
}
return headers;
}
export async function requireGitHubAccountAge(ctx: GitHubAccountGateCtx, userId: Id<"users">) {
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId });
if (!user || user.deletedAt || user.deactivatedAt) throw new ConvexError("User not found");
const now = Date.now();
let createdAt = user.githubCreatedAt ?? null;
if (!createdAt) {
const providerAccountId = await ctx.runQuery(
internal.githubIdentity.getGitHubProviderAccountIdInternal,
{ userId },
);
if (!providerAccountId) {
// Invariant: GitHub is our only auth provider, so this should never happen.
throw new ConvexError("GitHub account required");
}
assertGitHubNumericId(providerAccountId);
// Fetch by immutable GitHub numeric ID to avoid username swap attacks entirely.
const response = await fetch(`${GITHUB_API}/user/${providerAccountId}`, {
headers: buildGitHubHeaders(),
});
if (!response.ok) {
if (response.status === 403 || response.status === 429) {
throw new ConvexError("GitHub API rate limit exceeded — please try again in a few minutes");
}
throw new ConvexError("GitHub account lookup failed");
}
const payload = (await response.json()) as GitHubUser;
const parsed = payload.created_at ? Date.parse(payload.created_at) : Number.NaN;
if (!Number.isFinite(parsed)) throw new ConvexError("GitHub account lookup failed");
createdAt = parsed;
await ctx.runMutation(internal.users.setGitHubCreatedAtInternal, {
userId,
githubCreatedAt: createdAt,
});
}
if (!createdAt) throw new ConvexError("GitHub account lookup failed");
const ageMs = now - createdAt;
if (ageMs < MIN_ACCOUNT_AGE_MS) {
const remainingMs = MIN_ACCOUNT_AGE_MS - ageMs;
const remainingDays = Math.max(1, Math.ceil(remainingMs / (24 * 60 * 60 * 1000)));
throw new ConvexError(
`GitHub account must be at least 14 days old to publish skills or post comments. Try again in ${remainingDays} day${
remainingDays === 1 ? "" : "s"
}.`,
);
}
}
/**
* Sync the user's GitHub profile (username, avatar) from the GitHub API.
* This handles the case where a user renames their GitHub account.
* Uses the immutable GitHub numeric ID to fetch the current profile.
*/
export async function syncGitHubProfile(ctx: ActionCtx, userId: Id<"users">) {
const user = await ctx.runQuery(internal.users.getByIdInternal, { userId });
if (!user || user.deletedAt || user.deactivatedAt) return;
const now = Date.now();
const lastSyncedAt = user.githubProfileSyncedAt ?? null;
if (lastSyncedAt && now - lastSyncedAt < GITHUB_PROFILE_SYNC_WINDOW_MS) return;
const providerAccountId = await ctx.runQuery(
internal.githubIdentity.getGitHubProviderAccountIdInternal,
{ userId },
);
if (!providerAccountId) return;
assertGitHubNumericId(providerAccountId);
const response = await fetch(`${GITHUB_API}/user/${providerAccountId}`, {
headers: buildGitHubHeaders(),
});
if (!response.ok) {
// Silently fail - this is a best-effort sync, not critical path
console.warn(`[syncGitHubProfile] GitHub API error for user ${userId}: ${response.status}`);
return;
}
const payload = (await response.json()) as GitHubUser;
const newLogin = payload.login?.trim();
const newImage = payload.avatar_url?.trim();
const profileName = payload.name?.trim();
if (!newLogin) return;
const args: {
userId: Id<"users">;
name: string;
image?: string;
syncedAt: number;
profileName?: string;
} = {
userId,
name: newLogin,
image: newImage,
syncedAt: now,
};
if (profileName && profileName !== newLogin) {
args.profileName = profileName;
}
await ctx.runMutation(internal.users.syncGitHubProfileInternal, args);
}
+334
View File
@@ -0,0 +1,334 @@
/* @vitest-environment node */
import { describe, expect, it, vi } from "vitest";
import {
extractWorkflowFilenameFromWorkflowRef,
verifyGitHubActionsTrustedPublishJwt,
type TrustedGitHubActionsPublisher,
} from "./githubActionsOidc";
const trustedPublisher: TrustedGitHubActionsPublisher = {
repository: "openclaw/openclaw",
repositoryId: "123456",
repositoryOwner: "openclaw",
repositoryOwnerId: "7890",
workflowFilename: "plugin-clawhub-release.yml",
environment: "clawhub-plugin-release",
};
const trustedPublisherWithoutEnvironment: TrustedGitHubActionsPublisher = {
...trustedPublisher,
environment: undefined,
};
const signingKeyPairPromise = crypto.subtle.generateKey(
{
name: "RSASSA-PKCS1-v1_5",
modulusLength: 2048,
publicExponent: new Uint8Array([1, 0, 1]),
hash: "SHA-256",
},
true,
["sign", "verify"],
);
describe("extractWorkflowFilenameFromWorkflowRef", () => {
it("extracts the workflow filename from workflow_ref", () => {
expect(
extractWorkflowFilenameFromWorkflowRef(
"openclaw/openclaw/.github/workflows/plugin-clawhub-release.yml@refs/heads/main",
"openclaw/openclaw",
),
).toBe("plugin-clawhub-release.yml");
});
});
describe("verifyGitHubActionsTrustedPublishJwt", () => {
it("accepts a valid GitHub Actions token", async () => {
const { token, jwks } = await createSignedToken({
repository: trustedPublisher.repository,
repository_id: trustedPublisher.repositoryId,
repository_owner: trustedPublisher.repositoryOwner,
repository_owner_id: trustedPublisher.repositoryOwnerId,
workflow_ref:
"openclaw/openclaw/.github/workflows/plugin-clawhub-release.yml@refs/heads/main",
runner_environment: "github-hosted",
environment: trustedPublisher.environment,
event_name: "workflow_dispatch",
workflow: "Plugin ClawHub Release",
sha: "deadbeef",
ref: "refs/heads/main",
ref_type: "branch",
actor: "onur",
actor_id: "42",
run_id: "100",
run_attempt: "2",
iss: "https://token.actions.githubusercontent.com",
aud: "clawhub",
exp: Math.floor(Date.now() / 1000) + 300,
iat: Math.floor(Date.now() / 1000) - 5,
});
const identity = await verifyGitHubActionsTrustedPublishJwt(token, trustedPublisher, {
fetchImpl: async () =>
new Response(JSON.stringify({ keys: [jwks] }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
});
expect(identity).toMatchObject({
repository: trustedPublisher.repository,
repositoryId: trustedPublisher.repositoryId,
repositoryOwner: trustedPublisher.repositoryOwner,
repositoryOwnerId: trustedPublisher.repositoryOwnerId,
workflowFilename: trustedPublisher.workflowFilename,
environment: trustedPublisher.environment,
runId: "100",
runAttempt: "2",
sha: "deadbeef",
});
});
it("accepts a valid GitHub Actions token when no environment is pinned", async () => {
const { token, jwks } = await createSignedToken({
repository: trustedPublisher.repository,
repository_id: trustedPublisher.repositoryId,
repository_owner: trustedPublisher.repositoryOwner,
repository_owner_id: trustedPublisher.repositoryOwnerId,
workflow_ref:
"openclaw/openclaw/.github/workflows/plugin-clawhub-release.yml@refs/heads/main",
runner_environment: "github-hosted",
event_name: "workflow_dispatch",
workflow: "Plugin ClawHub Release",
sha: "deadbeef",
ref: "refs/heads/main",
ref_type: "branch",
actor: "onur",
actor_id: "42",
run_id: "100",
run_attempt: "2",
iss: "https://token.actions.githubusercontent.com",
aud: "clawhub",
exp: Math.floor(Date.now() / 1000) + 300,
iat: Math.floor(Date.now() / 1000) - 5,
});
const identity = await verifyGitHubActionsTrustedPublishJwt(
token,
trustedPublisherWithoutEnvironment,
{
fetchImpl: async () =>
new Response(JSON.stringify({ keys: [jwks] }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
},
);
expect(identity).toMatchObject({
repository: trustedPublisher.repository,
repositoryId: trustedPublisher.repositoryId,
repositoryOwner: trustedPublisher.repositoryOwner,
repositoryOwnerId: trustedPublisher.repositoryOwnerId,
workflowFilename: trustedPublisher.workflowFilename,
runId: "100",
runAttempt: "2",
sha: "deadbeef",
});
expect(identity.environment).toBeUndefined();
});
it("rejects reusable workflow tokens", async () => {
const { token, jwks } = await createSignedToken({
repository: trustedPublisher.repository,
repository_id: trustedPublisher.repositoryId,
repository_owner: trustedPublisher.repositoryOwner,
repository_owner_id: trustedPublisher.repositoryOwnerId,
workflow_ref:
"openclaw/openclaw/.github/workflows/plugin-clawhub-release.yml@refs/heads/main",
job_workflow_ref:
"openclaw/shared/.github/workflows/reusable-plugin-release.yml@refs/heads/main",
runner_environment: "github-hosted",
environment: trustedPublisher.environment,
event_name: "workflow_dispatch",
workflow: "Plugin ClawHub Release",
sha: "deadbeef",
ref: "refs/heads/main",
run_id: "100",
run_attempt: "1",
iss: "https://token.actions.githubusercontent.com",
aud: "clawhub",
exp: Math.floor(Date.now() / 1000) + 300,
iat: Math.floor(Date.now() / 1000) - 5,
});
await expect(
verifyGitHubActionsTrustedPublishJwt(token, trustedPublisher, {
fetchImpl: async () =>
new Response(JSON.stringify({ keys: [jwks] }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
}),
).rejects.toThrow("Only the official ClawHub reusable workflow is supported");
});
it("accepts the official ClawHub reusable workflow", async () => {
const { token, jwks } = await createSignedToken({
repository: trustedPublisher.repository,
repository_id: trustedPublisher.repositoryId,
repository_owner: trustedPublisher.repositoryOwner,
repository_owner_id: trustedPublisher.repositoryOwnerId,
workflow_ref:
"openclaw/openclaw/.github/workflows/plugin-clawhub-release.yml@refs/heads/main",
job_workflow_ref: "openclaw/clawhub/.github/workflows/package-publish.yml@refs/heads/main",
runner_environment: "github-hosted",
environment: trustedPublisher.environment,
event_name: "workflow_dispatch",
workflow: "Plugin ClawHub Release",
sha: "deadbeef",
ref: "refs/heads/main",
run_id: "100",
run_attempt: "1",
iss: "https://token.actions.githubusercontent.com",
aud: "clawhub",
exp: Math.floor(Date.now() / 1000) + 300,
iat: Math.floor(Date.now() / 1000) - 5,
});
await expect(
verifyGitHubActionsTrustedPublishJwt(token, trustedPublisher, {
fetchImpl: async () =>
new Response(JSON.stringify({ keys: [jwks] }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
}),
).resolves.toMatchObject({
repository: trustedPublisher.repository,
workflowFilename: trustedPublisher.workflowFilename,
jobWorkflowRef: "openclaw/clawhub/.github/workflows/package-publish.yml@refs/heads/main",
});
});
it("rejects environment mismatches", async () => {
const { token, jwks } = await createSignedToken({
repository: trustedPublisher.repository,
repository_id: trustedPublisher.repositoryId,
repository_owner: trustedPublisher.repositoryOwner,
repository_owner_id: trustedPublisher.repositoryOwnerId,
workflow_ref:
"openclaw/openclaw/.github/workflows/plugin-clawhub-release.yml@refs/heads/main",
runner_environment: "github-hosted",
environment: "other-environment",
event_name: "workflow_dispatch",
workflow: "Plugin ClawHub Release",
sha: "deadbeef",
ref: "refs/heads/main",
run_id: "100",
run_attempt: "1",
iss: "https://token.actions.githubusercontent.com",
aud: "clawhub",
exp: Math.floor(Date.now() / 1000) + 300,
iat: Math.floor(Date.now() / 1000) - 5,
});
await expect(
verifyGitHubActionsTrustedPublishJwt(token, trustedPublisher, {
fetchImpl: async () =>
new Response(JSON.stringify({ keys: [jwks] }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
}),
).rejects.toThrow("GitHub OIDC environment mismatch");
});
it("refreshes JWKS on signing-key cache misses", async () => {
const now = Date.now() + 10 * 60_000;
const { token, jwks } = await createSignedToken(
{
repository: trustedPublisher.repository,
repository_id: trustedPublisher.repositoryId,
repository_owner: trustedPublisher.repositoryOwner,
repository_owner_id: trustedPublisher.repositoryOwnerId,
workflow_ref:
"openclaw/openclaw/.github/workflows/plugin-clawhub-release.yml@refs/heads/main",
runner_environment: "github-hosted",
environment: trustedPublisher.environment,
event_name: "workflow_dispatch",
workflow: "Plugin ClawHub Release",
sha: "deadbeef",
ref: "refs/heads/main",
ref_type: "branch",
actor: "onur",
actor_id: "42",
run_id: "100",
run_attempt: "2",
iss: "https://token.actions.githubusercontent.com",
aud: "clawhub",
exp: Math.floor(now / 1000) + 300,
iat: Math.floor(now / 1000) - 5,
},
"rotated-key",
);
const staleJwk = { ...jwks, kid: "stale-key" };
const fetchMock = vi
.fn()
.mockResolvedValueOnce(
new Response(JSON.stringify({ keys: [staleJwk] }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
)
.mockResolvedValueOnce(
new Response(JSON.stringify({ keys: [jwks] }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
);
await expect(
verifyGitHubActionsTrustedPublishJwt(token, trustedPublisher, {
fetchImpl: fetchMock,
now: () => now,
}),
).resolves.toMatchObject({
repository: trustedPublisher.repository,
workflowFilename: trustedPublisher.workflowFilename,
});
expect(fetchMock).toHaveBeenCalledTimes(2);
});
});
async function createSignedToken(payload: Record<string, unknown>, kid = "test-key") {
const keyPair = await signingKeyPairPromise;
const header = { alg: "RS256", kid, typ: "JWT" };
const encodedHeader = base64UrlEncodeJson(header);
const encodedPayload = base64UrlEncodeJson(payload);
const signingInput = `${encodedHeader}.${encodedPayload}`;
const signature = new Uint8Array(
await crypto.subtle.sign(
"RSASSA-PKCS1-v1_5",
keyPair.privateKey,
new TextEncoder().encode(signingInput),
),
);
const publicJwk = (await crypto.subtle.exportKey("jwk", keyPair.publicKey)) as JsonWebKey & {
kid?: string;
};
publicJwk.kid = kid;
return {
token: `${signingInput}.${base64UrlEncodeBytes(signature)}`,
jwks: publicJwk,
};
}
function base64UrlEncodeJson(value: unknown) {
return base64UrlEncodeBytes(new TextEncoder().encode(JSON.stringify(value)));
}
function base64UrlEncodeBytes(bytes: Uint8Array) {
let binary = "";
for (const byte of bytes) binary += String.fromCharCode(byte);
return btoa(binary).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/g, "");
}
+394
View File
@@ -0,0 +1,394 @@
type JwtHeader = {
alg?: unknown;
kid?: unknown;
typ?: unknown;
};
type JwtPayload = Record<string, unknown>;
type JwkSet = {
keys?: Array<JsonWebKey & { kid?: string; alg?: string; use?: string; kty?: string }>;
};
export type TrustedGitHubActionsPublisher = {
repository: string;
repositoryId: string;
repositoryOwner: string;
repositoryOwnerId: string;
workflowFilename: string;
environment?: string;
};
export type VerifiedGitHubActionsIdentity = {
repository: string;
repositoryId: string;
repositoryOwner: string;
repositoryOwnerId: string;
workflowFilename: string;
workflowName: string;
workflowRef: string;
jobWorkflowRef?: string;
environment?: string;
runnerEnvironment: string;
eventName: string;
sha: string;
ref: string;
refType?: string;
actor?: string;
actorId?: string;
runId: string;
runAttempt: string;
};
type VerifyGitHubActionsOidcOptions = {
fetchImpl?: typeof fetch;
now?: () => number;
};
type GitHubRepositoryIdentity = {
repository: string;
repositoryId: string;
repositoryOwner: string;
repositoryOwnerId: string;
};
type ParsedWorkflowRef = {
repository: string;
workflowFilename: string;
};
const GITHUB_ACTIONS_ISSUER = "https://token.actions.githubusercontent.com";
const GITHUB_ACTIONS_JWKS_URL = `${GITHUB_ACTIONS_ISSUER}/.well-known/jwks`;
const TRUSTED_AUDIENCE = "clawhub";
const CLOCK_SKEW_MS = 60_000;
const JWKS_CACHE_TTL_MS = 5 * 60_000;
const OFFICIAL_REUSABLE_WORKFLOW_REPOSITORY = "openclaw/clawhub";
const OFFICIAL_REUSABLE_WORKFLOW_FILENAME = "package-publish.yml";
let cachedJwks: { value: JwkSet; fetchedAt: number } | null = null;
export async function verifyGitHubActionsTrustedPublishJwt(
jwt: string,
trustedPublisher: TrustedGitHubActionsPublisher,
options: VerifyGitHubActionsOidcOptions = {},
): Promise<VerifiedGitHubActionsIdentity> {
const fetchImpl = options.fetchImpl ?? fetch;
const now = (options.now ?? Date.now)();
const { signingInput, signature, header, payload } = decodeJwt(jwt);
if (header.alg !== "RS256") {
throw new Error(
`Unsupported GitHub OIDC signing algorithm: ${formatClaimValue(header.alg ?? "<missing>")}`,
);
}
const keyId = requireString(header.kid, "kid");
let jwks = await fetchGitHubActionsJwks(fetchImpl, now);
let jwk = jwks.keys?.find((entry) => entry.kid === keyId);
if (!jwk) {
jwks = await fetchGitHubActionsJwks(fetchImpl, now, true);
jwk = jwks.keys?.find((entry) => entry.kid === keyId);
}
if (!jwk) throw new Error(`Unknown GitHub OIDC signing key: ${keyId}`);
const key = await crypto.subtle.importKey(
"jwk",
jwk,
{ name: "RSASSA-PKCS1-v1_5", hash: "SHA-256" },
false,
["verify"],
);
const verified = await crypto.subtle.verify(
"RSASSA-PKCS1-v1_5",
key,
signature,
new TextEncoder().encode(signingInput),
);
if (!verified) throw new Error("Invalid GitHub OIDC signature");
const issuer = requireString(payload.iss, "iss");
if (issuer !== GITHUB_ACTIONS_ISSUER) {
throw new Error(`Unexpected GitHub OIDC issuer: ${issuer}`);
}
if (!claimContainsAudience(payload.aud, TRUSTED_AUDIENCE)) {
throw new Error(`Unexpected GitHub OIDC audience: ${formatAudience(payload.aud)}`);
}
assertTokenTimeWindow(payload, now);
const repository = requireString(payload.repository, "repository");
const repositoryId = requireClaimString(payload.repository_id, "repository_id");
const repositoryOwner = requireString(payload.repository_owner, "repository_owner");
const repositoryOwnerId = requireClaimString(payload.repository_owner_id, "repository_owner_id");
const workflowRef = requireString(payload.workflow_ref, "workflow_ref");
const workflow = parseWorkflowRef(workflowRef, repository);
const jobWorkflowRef = optionalString(payload.job_workflow_ref);
const runnerEnvironment = requireString(payload.runner_environment, "runner_environment");
const environment = optionalString(payload.environment);
const eventName = requireString(payload.event_name, "event_name");
const workflowName = requireString(payload.workflow, "workflow");
const sha = requireString(payload.sha, "sha");
const ref = requireString(payload.ref, "ref");
const runId = requireClaimString(payload.run_id, "run_id");
const runAttempt = requireClaimString(payload.run_attempt, "run_attempt");
const refType = optionalString(payload.ref_type);
const actor = optionalString(payload.actor);
const actorId = optionalStringValue(payload.actor_id);
if (repository !== trustedPublisher.repository) {
throw new Error(
`GitHub OIDC repository mismatch: expected ${trustedPublisher.repository}, got ${repository}`,
);
}
if (repositoryId !== trustedPublisher.repositoryId) {
throw new Error(
`GitHub OIDC repository_id mismatch: expected ${trustedPublisher.repositoryId}, got ${repositoryId}`,
);
}
if (repositoryOwner !== trustedPublisher.repositoryOwner) {
throw new Error(
`GitHub OIDC repository_owner mismatch: expected ${trustedPublisher.repositoryOwner}, got ${repositoryOwner}`,
);
}
if (repositoryOwnerId !== trustedPublisher.repositoryOwnerId) {
throw new Error(
`GitHub OIDC repository_owner_id mismatch: expected ${trustedPublisher.repositoryOwnerId}, got ${repositoryOwnerId}`,
);
}
if (workflow.workflowFilename !== trustedPublisher.workflowFilename) {
throw new Error(
`GitHub OIDC workflow mismatch: expected ${trustedPublisher.workflowFilename}, got ${workflow.workflowFilename}`,
);
}
if (jobWorkflowRef) {
const reusableWorkflow = parseWorkflowRef(jobWorkflowRef);
const usesOfficialReusableWorkflow =
reusableWorkflow.repository === OFFICIAL_REUSABLE_WORKFLOW_REPOSITORY &&
reusableWorkflow.workflowFilename === OFFICIAL_REUSABLE_WORKFLOW_FILENAME;
if (!usesOfficialReusableWorkflow) {
throw new Error(
"Only the official ClawHub reusable workflow is supported for trusted publishing",
);
}
}
if (runnerEnvironment !== "github-hosted") {
throw new Error(
`Only GitHub-hosted runners may mint trusted publish tokens, got ${runnerEnvironment}`,
);
}
// v1 keeps secretless publishing behind a manual entry point. Environment
// pinning is optional, but if configured it must match exactly.
if (eventName !== "workflow_dispatch") {
throw new Error(`Trusted publishing requires workflow_dispatch, got ${eventName}`);
}
if (trustedPublisher.environment && environment !== trustedPublisher.environment) {
throw new Error(
`GitHub OIDC environment mismatch: expected ${trustedPublisher.environment}, got ${formatClaimValue(environment ?? "<missing>")}`,
);
}
return {
repository,
repositoryId,
repositoryOwner,
repositoryOwnerId,
workflowFilename: workflow.workflowFilename,
workflowName,
workflowRef,
...(jobWorkflowRef ? { jobWorkflowRef } : {}),
...(environment ? { environment } : {}),
runnerEnvironment,
eventName,
sha,
ref,
...(refType ? { refType } : {}),
...(actor ? { actor } : {}),
...(actorId ? { actorId } : {}),
runId,
runAttempt,
};
}
export async function fetchGitHubRepositoryIdentity(
repository: string,
fetchImpl: typeof fetch = fetch,
): Promise<GitHubRepositoryIdentity> {
const normalizedRepository = normalizeGitHubRepository(repository);
if (!normalizedRepository) {
throw new Error(`Invalid GitHub repository: ${repository}`);
}
const response = await fetchImpl(`https://api.github.com/repos/${normalizedRepository}`, {
headers: {
Accept: "application/vnd.github+json",
"User-Agent": "clawhub/package-trusted-publisher",
},
});
if (!response.ok) {
throw new Error(
`GitHub repository lookup failed for ${normalizedRepository}: ${response.status}`,
);
}
const body = (await response.json()) as {
id?: unknown;
full_name?: unknown;
owner?: { login?: unknown; id?: unknown };
};
const resolvedRepository = requireString(body.full_name, "full_name");
const ownerLogin = requireString(body.owner?.login, "owner.login");
return {
repository: resolvedRepository,
repositoryId: requireClaimString(body.id, "id"),
repositoryOwner: ownerLogin,
repositoryOwnerId: requireClaimString(body.owner?.id, "owner.id"),
};
}
export function normalizeGitHubRepository(repository: string) {
const trimmed = repository
.trim()
.replace(/^https?:\/\/github\.com\//i, "")
.replace(/\.git$/i, "");
const match = /^([A-Za-z0-9_.-]+)\/([A-Za-z0-9_.-]+)$/.exec(trimmed);
if (!match) return null;
return `${match[1]}/${match[2]}`;
}
export function extractWorkflowFilenameFromWorkflowRef(
workflowRef: string,
expectedRepository?: string,
) {
return parseWorkflowRef(workflowRef, expectedRepository).workflowFilename;
}
function parseWorkflowRef(workflowRef: string, expectedRepository?: string): ParsedWorkflowRef {
const match = /^([^/]+\/[^/]+)\/\.github\/workflows\/([^@/]+)@.+$/.exec(workflowRef.trim());
if (!match?.[1] || !match[2]) {
throw new Error(`Invalid GitHub workflow_ref claim: ${workflowRef}`);
}
if (expectedRepository && match[1] !== expectedRepository) {
throw new Error(
`GitHub workflow_ref repository mismatch: expected ${expectedRepository}, got ${match[1]}`,
);
}
return {
repository: match[1],
workflowFilename: match[2],
};
}
function decodeJwt(jwt: string) {
const parts = jwt.trim().split(".");
if (parts.length !== 3) throw new Error("Invalid GitHub OIDC token format");
const [encodedHeader, encodedPayload, encodedSignature] = parts;
const header = parseJsonSegment<JwtHeader>(encodedHeader, "header");
const payload = parseJsonSegment<JwtPayload>(encodedPayload, "payload");
return {
header,
payload,
signingInput: `${encodedHeader}.${encodedPayload}`,
signature: base64UrlToBytes(encodedSignature),
};
}
function parseJsonSegment<T>(segment: string, label: string) {
try {
return JSON.parse(new TextDecoder().decode(base64UrlToBytes(segment))) as T;
} catch {
throw new Error(`Invalid GitHub OIDC ${label}`);
}
}
function base64UrlToBytes(value: string) {
const normalized = value.replace(/-/g, "+").replace(/_/g, "/");
const padded = normalized.padEnd(Math.ceil(normalized.length / 4) * 4, "=");
const binary = atob(padded);
const bytes = new Uint8Array(binary.length);
for (let index = 0; index < binary.length; index += 1) {
bytes[index] = binary.charCodeAt(index);
}
return bytes;
}
async function fetchGitHubActionsJwks(fetchImpl: typeof fetch, now: number, forceRefresh = false) {
if (!forceRefresh && cachedJwks && now - cachedJwks.fetchedAt < JWKS_CACHE_TTL_MS) {
return cachedJwks.value;
}
const response = await fetchImpl(GITHUB_ACTIONS_JWKS_URL, {
headers: {
Accept: "application/json",
"User-Agent": "clawhub/github-actions-oidc",
},
});
if (!response.ok) {
throw new Error(`Failed to fetch GitHub OIDC JWKS: ${response.status}`);
}
const jwks = (await response.json()) as JwkSet;
cachedJwks = { value: jwks, fetchedAt: now };
return jwks;
}
function claimContainsAudience(audience: unknown, expected: string) {
if (typeof audience === "string") return audience === expected;
if (!Array.isArray(audience)) return false;
return audience.includes(expected);
}
function formatAudience(audience: unknown) {
if (typeof audience === "string") return audience;
if (Array.isArray(audience)) return audience.join(", ");
return formatClaimValue(audience ?? "<missing>");
}
function formatClaimValue(value: unknown) {
if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") {
return String(value);
}
return JSON.stringify(value);
}
function assertTokenTimeWindow(payload: JwtPayload, now: number) {
const expiresAt = requireNumericClaim(payload.exp, "exp") * 1000;
if (now - CLOCK_SKEW_MS >= expiresAt) {
throw new Error("GitHub OIDC token has expired");
}
const notBefore =
payload.nbf === undefined ? undefined : requireNumericClaim(payload.nbf, "nbf") * 1000;
if (typeof notBefore === "number" && now + CLOCK_SKEW_MS < notBefore) {
throw new Error("GitHub OIDC token is not active yet");
}
}
function requireClaimString(value: unknown, label: string) {
const normalized = optionalStringValue(value);
if (!normalized) throw new Error(`Missing GitHub OIDC claim: ${label}`);
return normalized;
}
function requireString(value: unknown, label: string) {
if (typeof value !== "string" || !value.trim()) {
throw new Error(`Missing GitHub OIDC claim: ${label}`);
}
return value;
}
function optionalString(value: unknown) {
return typeof value === "string" && value.trim() ? value : undefined;
}
function optionalStringValue(value: unknown) {
if (typeof value === "number" && Number.isFinite(value)) return String(value);
return optionalString(value);
}
function requireNumericClaim(value: unknown, label: string) {
if (typeof value === "number" && Number.isFinite(value)) return value;
if (typeof value === "string" && value.trim()) {
const parsed = Number(value);
if (Number.isFinite(parsed)) return parsed;
}
throw new Error(`Missing GitHub OIDC claim: ${label}`);
}
export const __test = {
base64UrlToBytes,
claimContainsAudience,
};
+407 -260
View File
@@ -1,96 +1,103 @@
'use node'
"use node";
import { createPrivateKey, createSign } from 'node:crypto'
import type { Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { createPrivateKey, createSign } from "node:crypto";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
const GITHUB_API = 'https://api.github.com'
const DEFAULT_REPO = 'clawdbot/skills'
const DEFAULT_ROOT = 'skills'
const META_FILENAME = '_meta.json'
const USER_AGENT = 'clawdhub/skills-backup'
const GITHUB_API = "https://api.github.com";
const DEFAULT_REPO = "clawdbot/skills";
const DEFAULT_ROOT = "skills";
const META_FILENAME = "_meta.json";
const USER_AGENT = "clawhub/skills-backup";
type BackupFile = {
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
};
type BackupParams = {
slug: string
version: string
displayName: string
ownerHandle: string
files: BackupFile[]
publishedAt: number
}
slug: string;
version: string;
displayName: string;
ownerHandle: string;
files: BackupFile[];
publishedAt: number;
};
type RepoInfo = {
default_branch?: string
}
default_branch?: string;
};
type GitRef = {
object: { sha: string }
}
object: { sha: string };
};
type GitCommit = {
sha: string
tree: { sha: string }
}
sha: string;
tree: { sha: string };
};
type GitTreeEntry = {
path?: string
type?: string
}
path?: string;
type?: string;
};
type GitTree = {
tree?: GitTreeEntry[]
}
tree?: GitTreeEntry[];
};
type MetaFile = {
owner: string
slug: string
displayName: string
owner: string;
slug: string;
displayName: string;
latest: {
version: string
publishedAt: number
commit: string | null
}
version: string;
publishedAt: number;
commit: string | null;
};
history: Array<{
version: string
publishedAt: number
commit: string
}>
}
version: string;
publishedAt: number;
commit: string;
}>;
};
export type GitHubBackupContext = {
token: string
repo: string
repoOwner: string
repoName: string
branch: string
root: string
}
token: string;
repo: string;
repoOwner: string;
repoName: string;
branch: string;
root: string;
};
export type GitHubSkillBackupEntry = {
owner: string;
slug: string;
rootPath: string;
metaPath: string;
};
export function isGitHubBackupConfigured() {
return Boolean(
process.env.GITHUB_APP_ID &&
process.env.GITHUB_APP_PRIVATE_KEY &&
process.env.GITHUB_APP_INSTALLATION_ID,
)
process.env.GITHUB_APP_PRIVATE_KEY &&
process.env.GITHUB_APP_INSTALLATION_ID,
);
}
export async function getGitHubBackupContext(): Promise<GitHubBackupContext> {
const repo = process.env.GITHUB_SKILLS_REPO ?? DEFAULT_REPO
const root = process.env.GITHUB_SKILLS_ROOT ?? DEFAULT_ROOT
const [repoOwner, repoName] = parseRepo(repo)
const token = await createInstallationToken()
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`)
const branch = repoInfo.default_branch ?? 'main'
const repo = process.env.GITHUB_SKILLS_REPO ?? DEFAULT_REPO;
const root = process.env.GITHUB_SKILLS_ROOT ?? DEFAULT_ROOT;
const [repoOwner, repoName] = parseRepo(repo);
const token = await createInstallationToken();
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`);
const branch = repoInfo.default_branch ?? "main";
return { token, repo, repoOwner, repoName, branch, root }
return { token, repo, repoOwner, repoName, branch, root };
}
export async function fetchGitHubSkillMeta(
@@ -98,14 +105,128 @@ export async function fetchGitHubSkillMeta(
ownerHandle: string,
slug: string,
): Promise<MetaFile | null> {
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug)
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug);
return fetchMetaFile(
context.token,
context.repoOwner,
context.repoName,
`${skillRoot}/${META_FILENAME}`,
context.branch,
)
);
}
export async function listGitHubSkillBackupEntries(
context: GitHubBackupContext,
): Promise<GitHubSkillBackupEntry[]> {
const ref = await githubGet<GitRef>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/ref/heads/${context.branch}`,
);
const baseCommit = await githubGet<GitCommit>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/commits/${ref.object.sha}`,
);
const tree = await githubGet<GitTree>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/trees/${baseCommit.tree.sha}?recursive=1`,
);
const prefix = context.root ? `${context.root}/` : "";
const entries: GitHubSkillBackupEntry[] = [];
for (const entry of tree.tree ?? []) {
if (entry.type !== "blob" || !entry.path) continue;
if (!entry.path.startsWith(prefix) || !entry.path.endsWith(`/${META_FILENAME}`)) continue;
const relative = entry.path.slice(prefix.length);
const segments = relative.split("/");
if (segments.length !== 3) continue;
const [owner, slug, file] = segments;
if (file !== META_FILENAME) continue;
const rootPath = prefix ? `${prefix}${owner}/${slug}` : `${owner}/${slug}`;
entries.push({ owner, slug, rootPath, metaPath: entry.path });
}
return entries;
}
const MAX_PUSH_RETRIES = 3;
export async function deleteGitHubSkillBackup(
context: GitHubBackupContext,
ownerHandle: string,
slug: string,
) {
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug);
for (let attempt = 0; attempt < MAX_PUSH_RETRIES; attempt++) {
const ref = await githubGet<GitRef>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/ref/heads/${context.branch}`,
);
const baseCommitSha = ref.object.sha;
const baseCommit = await githubGet<GitCommit>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/commits/${baseCommitSha}`,
);
const baseTreeSha = baseCommit.tree.sha;
const existingTree = await githubGet<GitTree>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/trees/${baseTreeSha}?recursive=1`,
);
const prefix = `${skillRoot}/`;
const pathsToDelete = (existingTree.tree ?? [])
.filter((entry) => entry.type === "blob" && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? "")
.filter(Boolean);
if (!pathsToDelete.length) return { deleted: false as const };
const treeEntries = pathsToDelete.map((path) => ({
path,
mode: "100644" as const,
type: "blob" as const,
sha: null,
}));
const newTree = await githubPost<{ sha: string }>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/trees`,
{
base_tree: baseTreeSha,
tree: treeEntries,
},
);
const commit = await githubPost<GitCommit>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/commits`,
{
message: `delete: ${skillRoot}`,
tree: newTree.sha,
parents: [baseCommitSha],
},
);
try {
await githubPatch(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/git/refs/heads/${context.branch}`,
{ sha: commit.sha },
);
return { deleted: true as const };
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
if (msg.includes("not a fast forward") && attempt < MAX_PUSH_RETRIES - 1) {
console.warn(
`GitHub backup delete push conflict for ${skillRoot}, retrying (attempt ${attempt + 1})`,
);
continue;
}
throw err;
}
}
return { deleted: false as const };
}
export async function backupSkillToGitHub(
@@ -113,124 +234,150 @@ export async function backupSkillToGitHub(
params: BackupParams,
context?: GitHubBackupContext,
) {
if (!isGitHubBackupConfigured()) return
if (!isGitHubBackupConfigured()) return;
const resolved = context ?? (await getGitHubBackupContext())
const skillRoot = buildSkillRoot(resolved.root, params.ownerHandle, params.slug)
const ref = await githubGet<GitRef>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/ref/heads/${resolved.branch}`,
)
const baseCommitSha = ref.object.sha
const baseCommit = await githubGet<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits/${baseCommitSha}`,
)
const baseTreeSha = baseCommit.tree.sha
const existingTree = await githubGet<GitTree>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees/${baseTreeSha}?recursive=1`,
)
const prefix = `${skillRoot}/`
const existingPaths = new Set(
(existingTree.tree ?? [])
.filter((entry) => entry.type === 'blob' && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? ''),
)
const newPaths = new Set<string>()
const treeEntries: Array<{
path: string
mode: '100644'
type: 'blob'
sha: string | null
}> = []
const resolved = context ?? (await getGitHubBackupContext());
const skillRoot = buildSkillRoot(resolved.root, params.ownerHandle, params.slug);
const metaPath = `${skillRoot}/${META_FILENAME}`;
// Phase 1: Create blobs (content-addressed, only needs to happen once).
// This is the expensive part — downloads files from Convex storage.
const fileBlobs: Array<{ path: string; blobSha: string }> = [];
for (const file of params.files) {
const content = await fetchStorageBase64(ctx, file.storageId)
const blobSha = await createBlob(resolved.token, resolved.repoOwner, resolved.repoName, content)
const path = `${skillRoot}/${file.path}`
newPaths.add(path)
treeEntries.push({ path, mode: '100644', type: 'blob', sha: blobSha })
const content = await fetchStorageBase64(ctx, file.storageId);
const blobSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
content,
);
fileBlobs.push({ path: `${skillRoot}/${file.path}`, blobSha });
}
const existingMeta = await fetchMetaFile(
resolved.token,
resolved.repoOwner,
resolved.repoName,
`${skillRoot}/${META_FILENAME}`,
resolved.branch,
)
const metaPath = `${skillRoot}/${META_FILENAME}`
const metaDraft = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, null)
const metaDraftContent = `${JSON.stringify(metaDraft, null, 2)}\n`
const metaDraftSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaDraftContent),
)
newPaths.add(metaPath)
treeEntries.push({ path: metaPath, mode: '100644', type: 'blob', sha: metaDraftSha })
// Phase 2: Build tree, commit, and push. Retry on conflict since
// a concurrent publish-time backup may have advanced the branch.
for (let attempt = 0; attempt < MAX_PUSH_RETRIES; attempt++) {
const ref = await githubGet<GitRef>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/ref/heads/${resolved.branch}`,
);
const baseCommitSha = ref.object.sha;
const baseCommit = await githubGet<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits/${baseCommitSha}`,
);
const baseTreeSha = baseCommit.tree.sha;
const existingTree = await githubGet<GitTree>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees/${baseTreeSha}?recursive=1`,
);
for (const path of existingPaths) {
if (newPaths.has(path)) continue
treeEntries.push({ path, mode: '100644', type: 'blob', sha: null })
const prefix = `${skillRoot}/`;
const existingPaths = new Set(
(existingTree.tree ?? [])
.filter((entry) => entry.type === "blob" && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? ""),
);
const newPaths = new Set<string>();
const treeEntries: Array<{
path: string;
mode: "100644";
type: "blob";
sha: string | null;
}> = [];
for (const { path, blobSha } of fileBlobs) {
newPaths.add(path);
treeEntries.push({ path, mode: "100644", type: "blob", sha: blobSha });
}
const existingMeta = await fetchMetaFile(
resolved.token,
resolved.repoOwner,
resolved.repoName,
metaPath,
resolved.branch,
);
const metaDraft = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, null);
const metaDraftContent = `${JSON.stringify(metaDraft, null, 2)}\n`;
const metaDraftSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaDraftContent),
);
newPaths.add(metaPath);
treeEntries.push({ path: metaPath, mode: "100644", type: "blob", sha: metaDraftSha });
for (const path of existingPaths) {
if (newPaths.has(path)) continue;
treeEntries.push({ path, mode: "100644", type: "blob", sha: null });
}
const newTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: baseTreeSha,
tree: treeEntries,
},
);
const commit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `skill: ${params.slug} v${params.version}`,
tree: newTree.sha,
parents: [baseCommitSha],
},
);
const metaFinal = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, commit.sha);
const metaFinalContent = `${JSON.stringify(metaFinal, null, 2)}\n`;
const metaFinalSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaFinalContent),
);
const metaTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: commit.tree.sha,
tree: [{ path: metaPath, mode: "100644", type: "blob", sha: metaFinalSha }],
},
);
const metaCommit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `meta: ${params.slug} v${params.version}`,
tree: metaTree.sha,
parents: [commit.sha],
},
);
try {
await githubPatch(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/refs/heads/${resolved.branch}`,
{ sha: metaCommit.sha },
);
return; // Success
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
if (msg.includes("not a fast forward") && attempt < MAX_PUSH_RETRIES - 1) {
console.warn(
`GitHub backup push conflict for ${params.slug}, retrying (attempt ${attempt + 1})`,
);
continue;
}
throw err;
}
}
const newTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: baseTreeSha,
tree: treeEntries,
},
)
const commit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `skill: ${params.slug} v${params.version}`,
tree: newTree.sha,
parents: [baseCommitSha],
},
)
const metaFinal = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, commit.sha)
const metaFinalContent = `${JSON.stringify(metaFinal, null, 2)}\n`
const metaFinalSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaFinalContent),
)
const metaTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: commit.tree.sha,
tree: [{ path: metaPath, mode: '100644', type: 'blob', sha: metaFinalSha }],
},
)
const metaCommit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
{
message: `meta: ${params.slug} v${params.version}`,
tree: metaTree.sha,
parents: [commit.sha],
},
)
await githubPatch(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/refs/heads/${resolved.branch}`,
{
sha: metaCommit.sha,
},
)
}
function buildMetaFile(
@@ -240,15 +387,15 @@ function buildMetaFile(
baseCommitSha: string,
latestCommitSha: string | null,
): MetaFile {
let history = [...(existing?.history ?? [])]
let history = [...(existing?.history ?? [])];
if (existing?.latest?.version) {
const previousCommit = existing.latest.commit ?? commitUrl(repo, baseCommitSha)
const previousCommit = existing.latest.commit ?? commitUrl(repo, baseCommitSha);
const previous = {
version: existing.latest.version,
publishedAt: existing.latest.publishedAt,
commit: previousCommit,
}
history = [previous, ...history.filter((entry) => entry.version !== previous.version)]
};
history = [previous, ...history.filter((entry) => entry.version !== previous.version)];
}
return {
@@ -261,7 +408,7 @@ function buildMetaFile(
commit: latestCommitSha ? commitUrl(repo, latestCommitSha) : null,
},
history: history.slice(0, 200),
}
};
}
async function fetchMetaFile(
@@ -275,63 +422,63 @@ async function fetchMetaFile(
const response = await githubGet<{ content?: string }>(
token,
`/repos/${repoOwner}/${repoName}/contents/${encodePath(path)}?ref=${branch}`,
)
if (!response.content) return null
const raw = fromBase64(response.content)
return JSON.parse(raw) as MetaFile
);
if (!response.content) return null;
const raw = fromBase64(response.content);
return JSON.parse(raw) as MetaFile;
} catch (error) {
if (isNotFoundError(error)) return null
throw error
if (isNotFoundError(error)) return null;
throw error;
}
}
async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<'_storage'>) {
const blob = await ctx.storage.get(storageId)
if (!blob) throw new Error('File missing in storage')
const buffer = Buffer.from(await blob.arrayBuffer())
return buffer.toString('base64')
async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<"_storage">) {
const blob = await ctx.storage.get(storageId);
if (!blob) throw new Error("File missing in storage");
const buffer = Buffer.from(await blob.arrayBuffer());
return buffer.toString("base64");
}
async function createInstallationToken() {
const appId = process.env.GITHUB_APP_ID
const installationId = process.env.GITHUB_APP_INSTALLATION_ID
const appId = process.env.GITHUB_APP_ID;
const installationId = process.env.GITHUB_APP_INSTALLATION_ID;
if (!appId || !installationId) {
throw new Error('GitHub App credentials missing')
throw new Error("GitHub App credentials missing");
}
const jwt = createAppJwt(appId)
const jwt = createAppJwt(appId);
const response = await fetch(`${GITHUB_API}/app/installations/${installationId}/access_tokens`, {
method: 'POST',
method: "POST",
headers: buildHeaders(jwt, true),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub App token failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub App token failed: ${message}`);
}
const payload = (await response.json()) as { token?: string }
if (!payload.token) throw new Error('GitHub App token missing')
return payload.token
const payload = (await response.json()) as { token?: string };
if (!payload.token) throw new Error("GitHub App token missing");
return payload.token;
}
function createAppJwt(appId: string) {
const privateKey = loadPrivateKey()
const now = Math.floor(Date.now() / 1000)
const header = { alg: 'RS256', typ: 'JWT' }
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId }
const encodedHeader = base64Url(JSON.stringify(header))
const encodedPayload = base64Url(JSON.stringify(payload))
const signingInput = `${encodedHeader}.${encodedPayload}`
const sign = createSign('RSA-SHA256')
sign.update(signingInput)
sign.end()
const signature = sign.sign(privateKey)
return `${signingInput}.${base64Url(signature)}`
const privateKey = loadPrivateKey();
const now = Math.floor(Date.now() / 1000);
const header = { alg: "RS256", typ: "JWT" };
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId };
const encodedHeader = base64Url(JSON.stringify(header));
const encodedPayload = base64Url(JSON.stringify(payload));
const signingInput = `${encodedHeader}.${encodedPayload}`;
const sign = createSign("RSA-SHA256");
sign.update(signingInput);
sign.end();
const signature = sign.sign(privateKey);
return `${signingInput}.${base64Url(signature)}`;
}
function loadPrivateKey() {
const raw = process.env.GITHUB_APP_PRIVATE_KEY
if (!raw) throw new Error('GITHUB_APP_PRIVATE_KEY is not configured')
const normalized = raw.replace(/\\n/g, '\n')
return createPrivateKey(normalized)
const raw = process.env.GITHUB_APP_PRIVATE_KEY;
if (!raw) throw new Error("GITHUB_APP_PRIVATE_KEY is not configured");
const normalized = raw.replace(/\\n/g, "\n");
return createPrivateKey(normalized);
}
async function createBlob(token: string, repoOwner: string, repoName: string, content: string) {
@@ -340,104 +487,104 @@ async function createBlob(token: string, repoOwner: string, repoName: string, co
`/repos/${repoOwner}/${repoName}/git/blobs`,
{
content,
encoding: 'base64',
encoding: "base64",
},
)
if (!result.sha) throw new Error('GitHub blob missing sha')
return result.sha
);
if (!result.sha) throw new Error("GitHub blob missing sha");
return result.sha;
}
async function githubGet<T>(token: string, path: string): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
headers: buildHeaders(token),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub GET ${path} failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub GET ${path} failed: ${message}`);
}
return (await response.json()) as T
return (await response.json()) as T;
}
async function githubPost<T>(token: string, path: string, body: unknown): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'POST',
method: "POST",
headers: buildHeaders(token),
body: JSON.stringify(body),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub POST ${path} failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub POST ${path} failed: ${message}`);
}
return (await response.json()) as T
return (await response.json()) as T;
}
async function githubPatch(token: string, path: string, body: unknown) {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'PATCH',
method: "PATCH",
headers: buildHeaders(token),
body: JSON.stringify(body),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub PATCH ${path} failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub PATCH ${path} failed: ${message}`);
}
}
function buildHeaders(token: string, isAppJwt = false) {
return {
Authorization: `${isAppJwt ? 'Bearer' : 'token'} ${token}`,
Accept: 'application/vnd.github+json',
'User-Agent': USER_AGENT,
}
Authorization: `${isAppJwt ? "Bearer" : "token"} ${token}`,
Accept: "application/vnd.github+json",
"User-Agent": USER_AGENT,
};
}
function parseRepo(repo: string) {
const [owner, name] = repo.split('/')
if (!owner || !name) throw new Error('GITHUB_SKILLS_REPO must be owner/repo')
return [owner, name] as const
const [owner, name] = repo.split("/");
if (!owner || !name) throw new Error("GITHUB_SKILLS_REPO must be owner/repo");
return [owner, name] as const;
}
function normalizeOwner(value: string) {
export function normalizeOwner(value: string) {
const normalized = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]/g, '-')
.replace(/-+/g, '-')
.replace(/^-+|-+$/g, '')
return normalized || 'unknown'
.replace(/[^a-z0-9-]/g, "-")
.replace(/-+/g, "-")
.replace(/^-+|-+$/g, "");
return normalized || "unknown";
}
function commitUrl(repo: string, sha: string) {
return `https://github.com/${repo}/commit/${sha}`
return `https://github.com/${repo}/commit/${sha}`;
}
function buildSkillRoot(root: string, ownerHandle: string, slug: string) {
const ownerSegment = normalizeOwner(ownerHandle)
return `${root}/${ownerSegment}/${slug}`
const ownerSegment = normalizeOwner(ownerHandle);
return `${root}/${ownerSegment}/${slug}`;
}
function encodePath(path: string) {
return path
.split('/')
.split("/")
.map((segment) => encodeURIComponent(segment))
.join('/')
.join("/");
}
function base64Url(value: string | Buffer) {
const buffer = typeof value === 'string' ? Buffer.from(value) : value
return buffer.toString('base64').replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/g, '')
function base64Url(value: string | Uint8Array) {
const buffer = typeof value === "string" ? Buffer.from(value) : Buffer.from(value);
return buffer.toString("base64").replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/g, "");
}
function toBase64(value: string) {
return Buffer.from(value).toString('base64')
return Buffer.from(value).toString("base64");
}
function fromBase64(value: string) {
return Buffer.from(value, 'base64').toString('utf8')
return Buffer.from(value, "base64").toString("utf8");
}
function isNotFoundError(error: unknown) {
return (
error instanceof Error && (error.message.includes('404') || error.message.includes('Not Found'))
)
error instanceof Error && (error.message.includes("404") || error.message.includes("Not Found"))
);
}
+19
View File
@@ -0,0 +1,19 @@
import { describe, expect, it } from "vitest";
import { canHealSkillOwnershipByGitHubProviderAccountId } from "./githubIdentity";
describe("canHealSkillOwnershipByGitHubProviderAccountId", () => {
it("denies when either providerAccountId is missing", () => {
expect(canHealSkillOwnershipByGitHubProviderAccountId(undefined, undefined)).toBe(false);
expect(canHealSkillOwnershipByGitHubProviderAccountId("123", undefined)).toBe(false);
expect(canHealSkillOwnershipByGitHubProviderAccountId(undefined, "123")).toBe(false);
expect(canHealSkillOwnershipByGitHubProviderAccountId(null, "123")).toBe(false);
});
it("denies when providerAccountId differs", () => {
expect(canHealSkillOwnershipByGitHubProviderAccountId("123", "456")).toBe(false);
});
it("allows when providerAccountId matches", () => {
expect(canHealSkillOwnershipByGitHubProviderAccountId("123", "123")).toBe(true);
});
});
+22
View File
@@ -0,0 +1,22 @@
import type { Id } from "../_generated/dataModel";
import type { QueryCtx } from "../_generated/server";
export function canHealSkillOwnershipByGitHubProviderAccountId(
ownerProviderAccountId: string | null | undefined,
callerProviderAccountId: string | null | undefined,
) {
// Security invariant: missing identity must never grant ownership.
if (!ownerProviderAccountId || !callerProviderAccountId) return false;
return ownerProviderAccountId === callerProviderAccountId;
}
export async function getGitHubProviderAccountId(
ctx: Pick<QueryCtx, "db">,
userId: Id<"users">,
): Promise<string | null> {
const account = await ctx.db
.query("authAccounts")
.withIndex("userIdAndProvider", (q) => q.eq("userId", userId).eq("provider", "github"))
.unique();
return account?.providerAccountId ?? null;
}
+171 -171
View File
@@ -1,7 +1,7 @@
/* @vitest-environment node */
import { unzipSync } from 'fflate'
import { describe, expect, it } from 'vitest'
import { unzipSync } from "fflate";
import { describe, expect, it } from "vitest";
import {
buildGitHubZipForTests,
computeDefaultSelectedPaths,
@@ -12,236 +12,236 @@ import {
resolveGitHubCommit,
resolveMarkdownTarget,
stripGitHubZipRoot,
} from './githubImport'
} from "./githubImport";
function requestInfoToUrlString(input: RequestInfo | URL): string {
if (typeof input === 'string') return input
if (input instanceof URL) return input.toString()
if (input instanceof Request) return input.url
if (typeof input === "string") return input;
if (input instanceof URL) return input.toString();
if (input instanceof Request) return input.url;
throw new Error('Unexpected fetch input type')
throw new Error("Unexpected fetch input type");
}
describe('github import', () => {
it('parses repo root urls', () => {
expect(parseGitHubImportUrl('https://github.com/visionik/ouracli')).toEqual({
owner: 'visionik',
repo: 'ouracli',
originalUrl: 'https://github.com/visionik/ouracli',
})
})
describe("github import", () => {
it("parses repo root urls", () => {
expect(parseGitHubImportUrl("https://github.com/visionik/ouracli")).toEqual({
owner: "visionik",
repo: "ouracli",
originalUrl: "https://github.com/visionik/ouracli",
});
});
it('rejects non-https and non-github urls', () => {
expect(() => parseGitHubImportUrl('http://github.com/a/b')).toThrow(/https/i)
expect(() => parseGitHubImportUrl('https://example.com/a/b')).toThrow(/github\.com/i)
expect(() => parseGitHubImportUrl('not-a-url')).toThrow(/Invalid URL/i)
})
it("rejects non-https and non-github urls", () => {
expect(() => parseGitHubImportUrl("http://github.com/a/b")).toThrow(/https/i);
expect(() => parseGitHubImportUrl("https://example.com/a/b")).toThrow(/github\.com/i);
expect(() => parseGitHubImportUrl("not-a-url")).toThrow(/Invalid URL/i);
});
it('rejects malformed tree/blob urls', () => {
expect(() => parseGitHubImportUrl('https://github.com/a/b/tree/')).toThrow(/Missing ref/i)
expect(() => parseGitHubImportUrl('https://github.com/a/b/blob/main')).toThrow(/Missing path/i)
expect(() => parseGitHubImportUrl('https://github.com/a/b/tree/main/bad%5cpath')).toThrow()
})
it("rejects malformed tree/blob urls", () => {
expect(() => parseGitHubImportUrl("https://github.com/a/b/tree/")).toThrow(/Missing ref/i);
expect(() => parseGitHubImportUrl("https://github.com/a/b/blob/main")).toThrow(/Missing path/i);
expect(() => parseGitHubImportUrl("https://github.com/a/b/tree/main/bad%5cpath")).toThrow();
});
it('parses tree urls with ref and path', () => {
expect(parseGitHubImportUrl('https://github.com/a/b/tree/main/skills/foo')).toEqual({
owner: 'a',
repo: 'b',
ref: 'main',
path: 'skills/foo',
originalUrl: 'https://github.com/a/b/tree/main/skills/foo',
})
})
it("parses tree urls with ref and path", () => {
expect(parseGitHubImportUrl("https://github.com/a/b/tree/main/skills/foo")).toEqual({
owner: "a",
repo: "b",
ref: "main",
path: "skills/foo",
originalUrl: "https://github.com/a/b/tree/main/skills/foo",
});
});
it('parses blob urls and derives folder path', () => {
expect(parseGitHubImportUrl('https://github.com/a/b/blob/main/skills/foo/SKILL.md')).toEqual({
owner: 'a',
repo: 'b',
ref: 'main',
path: 'skills/foo',
originalUrl: 'https://github.com/a/b/blob/main/skills/foo/SKILL.md',
})
})
it("parses blob urls and derives folder path", () => {
expect(parseGitHubImportUrl("https://github.com/a/b/blob/main/skills/foo/SKILL.md")).toEqual({
owner: "a",
repo: "b",
ref: "main",
path: "skills/foo",
originalUrl: "https://github.com/a/b/blob/main/skills/foo/SKILL.md",
});
});
it('strips single top-level folder from GitHub zip entries', () => {
it("strips single top-level folder from GitHub zip entries", () => {
const zip = buildGitHubZipForTests({
'repo-1/skill/SKILL.md': 'Body',
'repo-1/skill/a.txt': 'a',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
expect(Object.keys(stripped).sort()).toEqual(['skill/SKILL.md', 'skill/a.txt'])
})
"repo-1/skill/SKILL.md": "Body",
"repo-1/skill/a.txt": "a",
});
const stripped = stripGitHubZipRoot(unzipSync(zip));
expect(Object.keys(stripped).sort()).toEqual(["skill/SKILL.md", "skill/a.txt"]);
});
it('keeps paths when zip has multiple top-level roots', () => {
it("keeps paths when zip has multiple top-level roots", () => {
const zip = buildGitHubZipForTests({
'a/SKILL.md': 'Body',
'b/SKILL.md': 'Body',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
expect(Object.keys(stripped).sort()).toEqual(['a/SKILL.md', 'b/SKILL.md'])
})
"a/SKILL.md": "Body",
"b/SKILL.md": "Body",
});
const stripped = stripGitHubZipRoot(unzipSync(zip));
expect(Object.keys(stripped).sort()).toEqual(["a/SKILL.md", "b/SKILL.md"]);
});
it('detects candidates in a GitHub zip and strips the root folder', () => {
it("detects candidates in a GitHub zip and strips the root folder", () => {
const zip = buildGitHubZipForTests({
'ouracli-123/SKILL.md': `---\nname: demo\ndescription: Hello\n---\nBody`,
'ouracli-123/src/index.ts': 'export {}',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
const candidates = detectGitHubImportCandidates(stripped)
expect(candidates.map((c) => c.path)).toEqual([''])
expect(candidates[0]?.name).toBe('demo')
})
"ouracli-123/SKILL.md": `---\nname: demo\ndescription: Hello\n---\nBody`,
"ouracli-123/src/index.ts": "export {}",
});
const stripped = stripGitHubZipRoot(unzipSync(zip));
const candidates = detectGitHubImportCandidates(stripped);
expect(candidates.map((c) => c.path)).toEqual([""]);
expect(candidates[0]?.name).toBe("demo");
});
it('detects multiple candidates and supports skills.md', () => {
it("detects multiple candidates and supports skills.md", () => {
const zip = buildGitHubZipForTests({
'repo-1/alpha/SKILL.md': `---\nname: Alpha\n---\nBody`,
'repo-1/beta/skills.md': `---\nname: Beta\n---\nBody`,
'repo-1/readme.md': 'x',
})
const stripped = stripGitHubZipRoot(unzipSync(zip))
const candidates = detectGitHubImportCandidates(stripped)
expect(candidates.map((c) => c.path)).toEqual(['alpha', 'beta'])
expect(candidates.map((c) => c.name)).toEqual(['Alpha', 'Beta'])
})
"repo-1/alpha/SKILL.md": `---\nname: Alpha\n---\nBody`,
"repo-1/beta/skills.md": `---\nname: Beta\n---\nBody`,
"repo-1/readme.md": "x",
});
const stripped = stripGitHubZipRoot(unzipSync(zip));
const candidates = detectGitHubImportCandidates(stripped);
expect(candidates.map((c) => c.path)).toEqual(["alpha", "beta"]);
expect(candidates.map((c) => c.name)).toEqual(["Alpha", "Beta"]);
});
it('computes default selection via markdown references', () => {
it("computes default selection via markdown references", () => {
const entries = {
'skill/SKILL.md': `---\nname: demo\n---\nSee [usage](docs/usage.md) and ![logo](img/logo.svg).\nIgnore [web](https://example.com).`,
'skill/docs/usage.md': `See [more](more.md)`,
'skill/docs/more.md': `Ok`,
'skill/img/logo.svg': `<svg/>`,
'skill/extra.txt': 'not referenced',
}
"skill/SKILL.md": `---\nname: demo\n---\nSee [usage](docs/usage.md) and ![logo](img/logo.svg).\nIgnore [web](https://example.com).`,
"skill/docs/usage.md": `See [more](more.md)`,
"skill/docs/more.md": `Ok`,
"skill/img/logo.svg": `<svg/>`,
"skill/extra.txt": "not referenced",
};
const zip = buildGitHubZipForTests(
Object.fromEntries(Object.entries(entries).map(([k, v]) => [`repo-1/${k}`, v])),
)
const raw = unzipSync(zip)
const stripped = stripGitHubZipRoot(raw)
const candidates = detectGitHubImportCandidates(stripped)
const candidate = candidates.find((c) => c.path === 'skill')
expect(candidate).toBeTruthy()
if (!candidate) throw new Error('candidate not found')
);
const raw = unzipSync(zip);
const stripped = stripGitHubZipRoot(raw);
const candidates = detectGitHubImportCandidates(stripped);
const candidate = candidates.find((c) => c.path === "skill");
expect(candidate).toBeTruthy();
if (!candidate) throw new Error("candidate not found");
const files = Object.entries(stripped)
.filter(([path]) => path.startsWith('skill/'))
.map(([path, bytes]) => ({ path, bytes }))
const selected = computeDefaultSelectedPaths({ candidate, files })
expect(selected).toContain('skill/SKILL.md')
expect(selected).toContain('skill/docs/usage.md')
expect(selected).toContain('skill/docs/more.md')
expect(selected).toContain('skill/img/logo.svg')
expect(selected).not.toContain('skill/extra.txt')
})
.filter(([path]) => path.startsWith("skill/"))
.map(([path, bytes]) => ({ path, bytes }));
const selected = computeDefaultSelectedPaths({ candidate, files });
expect(selected).toContain("skill/SKILL.md");
expect(selected).toContain("skill/docs/usage.md");
expect(selected).toContain("skill/docs/more.md");
expect(selected).toContain("skill/img/logo.svg");
expect(selected).not.toContain("skill/extra.txt");
});
it('does not select files outside skill folder (even when referenced)', () => {
it("does not select files outside skill folder (even when referenced)", () => {
const entries = {
'skill/SKILL.md': `See [outside](../outside.md) and [abs](/abs.md) and [mail](mailto:test@example.com).`,
'outside.md': `secret`,
'skill/docs/usage.md': `Ok`,
}
"skill/SKILL.md": `See [outside](../outside.md) and [abs](/abs.md) and [mail](mailto:test@example.com).`,
"outside.md": `secret`,
"skill/docs/usage.md": `Ok`,
};
const zip = buildGitHubZipForTests(
Object.fromEntries(Object.entries(entries).map(([k, v]) => [`repo-1/${k}`, v])),
)
const stripped = stripGitHubZipRoot(unzipSync(zip))
const candidate = detectGitHubImportCandidates(stripped).find((c) => c.path === 'skill')
expect(candidate).toBeTruthy()
if (!candidate) throw new Error('candidate not found')
const files = Object.entries(stripped).map(([path, bytes]) => ({ path, bytes }))
const selected = computeDefaultSelectedPaths({ candidate, files })
expect(selected).toContain('skill/SKILL.md')
expect(selected).not.toContain('outside.md')
})
);
const stripped = stripGitHubZipRoot(unzipSync(zip));
const candidate = detectGitHubImportCandidates(stripped).find((c) => c.path === "skill");
expect(candidate).toBeTruthy();
if (!candidate) throw new Error("candidate not found");
const files = Object.entries(stripped).map(([path, bytes]) => ({ path, bytes }));
const selected = computeDefaultSelectedPaths({ candidate, files });
expect(selected).toContain("skill/SKILL.md");
expect(selected).not.toContain("outside.md");
});
it('extracts markdown targets with titles and angle brackets', () => {
it("extracts markdown targets with titles and angle brackets", () => {
const targets = extractMarkdownRelativeTargets(
`See [a](docs/usage.md "Title") and [b](<docs/my file.md>) and ![c](img/logo.svg)`,
)
expect(targets).toEqual(['docs/usage.md', 'docs/my file.md', 'img/logo.svg'])
})
);
expect(targets).toEqual(["docs/usage.md", "docs/my file.md", "img/logo.svg"]);
});
it('resolves markdown targets safely', () => {
expect(resolveMarkdownTarget('a/SKILL.md', 'docs/usage.md')).toBe('a/docs/usage.md')
expect(resolveMarkdownTarget('a/SKILL.md', '../oops.md')).toBeNull()
expect(resolveMarkdownTarget('a/SKILL.md', '/abs.md')).toBeNull()
expect(resolveMarkdownTarget('a/SKILL.md', 'docs/usage.md#section')).toBe('a/docs/usage.md')
expect(resolveMarkdownTarget('a/SKILL.md', 'docs/usage.md?x=1')).toBe('a/docs/usage.md')
})
it("resolves markdown targets safely", () => {
expect(resolveMarkdownTarget("a/SKILL.md", "docs/usage.md")).toBe("a/docs/usage.md");
expect(resolveMarkdownTarget("a/SKILL.md", "../oops.md")).toBeNull();
expect(resolveMarkdownTarget("a/SKILL.md", "/abs.md")).toBeNull();
expect(resolveMarkdownTarget("a/SKILL.md", "docs/usage.md#section")).toBe("a/docs/usage.md");
expect(resolveMarkdownTarget("a/SKILL.md", "docs/usage.md?x=1")).toBe("a/docs/usage.md");
});
it('resolves HEAD commit via redirect chain and refuses unexpected redirect hosts', async () => {
it("resolves HEAD commit via redirect chain and refuses unexpected redirect hosts", async () => {
const fetcher: typeof fetch = async (input) => {
const url = requestInfoToUrlString(input)
if (url.includes('/archive/HEAD.zip')) {
const url = requestInfoToUrlString(input);
if (url.includes("/archive/HEAD.zip")) {
return new Response(null, {
status: 302,
headers: {
location:
'https://codeload.github.com/a/b/zip/0123456789012345678901234567890123456789',
"https://codeload.github.com/a/b/zip/0123456789012345678901234567890123456789",
},
})
});
}
if (url.startsWith('https://codeload.github.com/a/b/zip/')) {
return new Response(null, { status: 200 })
if (url.startsWith("https://codeload.github.com/a/b/zip/")) {
return new Response(null, { status: 200 });
}
throw new Error(`Unexpected fetch: ${url}`)
}
throw new Error(`Unexpected fetch: ${url}`);
};
const resolved = await resolveGitHubCommit(
{ owner: 'a', repo: 'b', originalUrl: 'https://github.com/a/b' },
{ owner: "a", repo: "b", originalUrl: "https://github.com/a/b" },
fetcher,
)
expect(resolved.commit).toBe('0123456789012345678901234567890123456789')
);
expect(resolved.commit).toBe("0123456789012345678901234567890123456789");
const badFetcher: typeof fetch = async (input) => {
const url = requestInfoToUrlString(input)
if (url.includes('/archive/HEAD.zip')) {
const url = requestInfoToUrlString(input);
if (url.includes("/archive/HEAD.zip")) {
return new Response(null, {
status: 302,
headers: { location: 'https://evil.example/zip/abc' },
})
headers: { location: "https://evil.example/zip/abc" },
});
}
throw new Error(`Unexpected fetch: ${url}`)
}
throw new Error(`Unexpected fetch: ${url}`);
};
await expect(
resolveGitHubCommit(
{ owner: 'a', repo: 'b', originalUrl: 'https://github.com/a/b' },
{ owner: "a", repo: "b", originalUrl: "https://github.com/a/b" },
badFetcher,
),
).rejects.toThrow(/redirect/i)
})
).rejects.toThrow(/redirect/i);
});
it('resolves explicit ref commit via GitHub API', async () => {
it("resolves explicit ref commit via GitHub API", async () => {
const fetcher: typeof fetch = async (input) => {
const url = requestInfoToUrlString(input)
if (url.startsWith('https://api.github.com/repos/a/b/commits/')) {
return new Response(JSON.stringify({ sha: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }), {
const url = requestInfoToUrlString(input);
if (url.startsWith("https://api.github.com/repos/a/b/commits/")) {
return new Response(JSON.stringify({ sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }), {
status: 200,
})
});
}
throw new Error(`Unexpected fetch: ${url}`)
}
throw new Error(`Unexpected fetch: ${url}`);
};
const resolved = await resolveGitHubCommit(
{ owner: 'a', repo: 'b', ref: 'main', originalUrl: 'https://github.com/a/b' },
{ owner: "a", repo: "b", ref: "main", originalUrl: "https://github.com/a/b" },
fetcher,
)
expect(resolved.commit).toBe('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')
})
);
expect(resolved.commit).toBe("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
});
it('enforces zip byte cap when content-length is too large', async () => {
it("enforces zip byte cap when content-length is too large", async () => {
const resolved = {
owner: 'a',
repo: 'b',
ref: 'main',
commit: '0123456789012345678901234567890123456789',
path: '',
repoUrl: 'https://github.com/a/b',
originalUrl: 'https://github.com/a/b',
} as const
owner: "a",
repo: "b",
ref: "main",
commit: "0123456789012345678901234567890123456789",
path: "",
repoUrl: "https://github.com/a/b",
originalUrl: "https://github.com/a/b",
} as const;
const fetcher: typeof fetch = async () =>
new Response(new Blob([new Uint8Array([1, 2, 3])]), {
status: 200,
headers: { 'content-length': String(999_999_999) },
})
headers: { "content-length": String(999_999_999) },
});
await expect(fetchGitHubZipBytes(resolved, fetcher, { maxZipBytes: 10 })).rejects.toThrow(
/too large/i,
)
})
})
);
});
});
+242 -242
View File
@@ -1,106 +1,106 @@
import { TEXT_FILE_EXTENSION_SET } from 'clawdhub-schema'
import { zipSync } from 'fflate'
import semver from 'semver'
import { parseFrontmatter } from './skills'
import { TEXT_FILE_EXTENSION_SET } from "clawhub-schema";
import { zipSync } from "fflate";
import semver from "semver";
import { parseFrontmatter } from "./skills";
export type GitHubImportUrl = {
owner: string
repo: string
ref?: string
path?: string
originalUrl: string
}
owner: string;
repo: string;
ref?: string;
path?: string;
originalUrl: string;
};
export type GitHubImportResolved = {
owner: string
repo: string
ref: string
commit: string
path: string
repoUrl: string
originalUrl: string
}
owner: string;
repo: string;
ref: string;
commit: string;
path: string;
repoUrl: string;
originalUrl: string;
};
export type GitHubImportCandidate = {
path: string
readmePath: string
name?: string
description?: string
}
path: string;
readmePath: string;
name?: string;
description?: string;
};
export type GitHubImportFileEntry = {
path: string
size: number
defaultSelected: boolean
}
path: string;
size: number;
defaultSelected: boolean;
};
const MAX_REDIRECTS = 6
const GITHUB_HOST = 'github.com'
const CODELOAD_HOST = 'codeload.github.com'
const SKILL_FILENAMES = ['skill.md', 'skills.md']
const MAX_REDIRECTS = 6;
const GITHUB_HOST = "github.com";
const CODELOAD_HOST = "codeload.github.com";
const SKILL_FILENAMES = ["skill.md", "skills.md"];
export function parseGitHubImportUrl(input: string): GitHubImportUrl {
const originalUrl = input.trim()
let url: URL
const originalUrl = input.trim();
let url: URL;
try {
url = new URL(originalUrl)
url = new URL(originalUrl);
} catch {
throw new Error('Invalid URL')
throw new Error("Invalid URL");
}
if (url.protocol !== 'https:') throw new Error('Only https:// URLs are supported')
if (url.hostname !== GITHUB_HOST) throw new Error('Only github.com URLs are supported')
if (url.protocol !== "https:") throw new Error("Only https:// URLs are supported");
if (url.hostname !== GITHUB_HOST) throw new Error("Only github.com URLs are supported");
const segments = url.pathname
.split('/')
.split("/")
.map((segment) => segment.trim())
.filter(Boolean)
.map((segment) => {
try {
return decodeURIComponent(segment)
return decodeURIComponent(segment);
} catch {
throw new Error('Invalid URL')
throw new Error("Invalid URL");
}
})
});
const owner = segments[0] ?? ''
const repo = (segments[1] ?? '').replace(/\.git$/, '')
if (!owner || !repo) throw new Error('GitHub URL must be /<owner>/<repo>')
const owner = segments[0] ?? "";
const repo = (segments[1] ?? "").replace(/\.git$/, "");
if (!owner || !repo) throw new Error("GitHub URL must be /<owner>/<repo>");
const kind = segments[2] ?? ''
if (!kind) return { owner, repo, originalUrl }
if (kind !== 'tree' && kind !== 'blob') {
return { owner, repo, originalUrl }
const kind = segments[2] ?? "";
if (!kind) return { owner, repo, originalUrl };
if (kind !== "tree" && kind !== "blob") {
return { owner, repo, originalUrl };
}
const ref = segments[3] ?? ''
if (!ref) throw new Error('Missing ref in GitHub URL')
const ref = segments[3] ?? "";
if (!ref) throw new Error("Missing ref in GitHub URL");
const rest = segments.slice(4).join('/')
const normalizedRest = normalizeRepoPath(rest)
const rest = segments.slice(4).join("/");
const normalizedRest = normalizeRepoPath(rest);
if (kind === 'blob') {
if (!rest) throw new Error('Missing path in GitHub URL')
if (!normalizedRest) throw new Error('Invalid path in GitHub URL')
const dir = normalizedRest.split('/').slice(0, -1).join('/')
return { owner, repo, ref, path: dir || undefined, originalUrl }
if (kind === "blob") {
if (!rest) throw new Error("Missing path in GitHub URL");
if (!normalizedRest) throw new Error("Invalid path in GitHub URL");
const dir = normalizedRest.split("/").slice(0, -1).join("/");
return { owner, repo, ref, path: dir || undefined, originalUrl };
}
if (rest && !normalizedRest) throw new Error('Invalid path in GitHub URL')
return { owner, repo, ref, path: normalizedRest || undefined, originalUrl }
if (rest && !normalizedRest) throw new Error("Invalid path in GitHub URL");
return { owner, repo, ref, path: normalizedRest || undefined, originalUrl };
}
export async function resolveGitHubCommit(
parsed: GitHubImportUrl,
fetcher: typeof fetch,
): Promise<GitHubImportResolved> {
const repoUrl = `https://${GITHUB_HOST}/${parsed.owner}/${parsed.repo}`
const ref = parsed.ref?.trim() || 'HEAD'
const path = normalizeRepoPath(parsed.path ?? '')
const repoUrl = `https://${GITHUB_HOST}/${parsed.owner}/${parsed.repo}`;
const ref = parsed.ref?.trim() || "HEAD";
const path = normalizeRepoPath(parsed.path ?? "");
const commit =
ref === 'HEAD'
ref === "HEAD"
? await resolveHeadCommit(parsed, fetcher)
: await resolveRefCommit(parsed, ref, fetcher)
: await resolveRefCommit(parsed, ref, fetcher);
return {
owner: parsed.owner,
@@ -110,42 +110,42 @@ export async function resolveGitHubCommit(
path,
repoUrl,
originalUrl: parsed.originalUrl,
}
};
}
async function resolveRefCommit(parsed: GitHubImportUrl, ref: string, fetcher: typeof fetch) {
const apiUrl = `https://api.github.com/repos/${parsed.owner}/${parsed.repo}/commits/${encodeURIComponent(ref)}`
const apiUrl = `https://api.github.com/repos/${parsed.owner}/${parsed.repo}/commits/${encodeURIComponent(ref)}`;
const response = await fetcher(apiUrl, {
headers: {
Accept: 'application/vnd.github+json',
'User-Agent': 'clawdhub/github-import',
Accept: "application/vnd.github+json",
"User-Agent": "clawhub/github-import",
},
})
if (!response.ok) throw new Error('GitHub ref not found')
const body = (await response.json()) as { sha?: unknown }
const sha = typeof body.sha === 'string' ? body.sha : ''
if (!/^[a-f0-9]{40}$/i.test(sha)) throw new Error('GitHub commit sha missing')
return sha.toLowerCase()
});
if (!response.ok) throw new Error("GitHub ref not found");
const body = (await response.json()) as { sha?: unknown };
const sha = typeof body.sha === "string" ? body.sha : "";
if (!/^[a-f0-9]{40}$/i.test(sha)) throw new Error("GitHub commit sha missing");
return sha.toLowerCase();
}
async function resolveHeadCommit(parsed: GitHubImportUrl, fetcher: typeof fetch) {
let url = `https://${GITHUB_HOST}/${parsed.owner}/${parsed.repo}/archive/HEAD.zip`
let url = `https://${GITHUB_HOST}/${parsed.owner}/${parsed.repo}/archive/HEAD.zip`;
for (let i = 0; i < MAX_REDIRECTS; i += 1) {
const response = await fetcher(url, { redirect: 'manual' })
const location = response.headers.get('location')
if (!location) break
const next = new URL(location, url)
const response = await fetcher(url, { redirect: "manual" });
const location = response.headers.get("location");
if (!location) break;
const next = new URL(location, url);
if (next.hostname !== GITHUB_HOST && next.hostname !== CODELOAD_HOST) {
throw new Error('Unexpected redirect host')
throw new Error("Unexpected redirect host");
}
url = next.toString()
url = next.toString();
}
const maybe = url.split('/').at(-1) ?? ''
const maybe = url.split("/").at(-1) ?? "";
if (!/^[a-f0-9]{40}$/i.test(maybe)) {
throw new Error('Could not resolve commit for HEAD')
throw new Error("Could not resolve commit for HEAD");
}
return maybe.toLowerCase()
return maybe.toLowerCase();
}
export async function fetchGitHubZipBytes(
@@ -153,273 +153,273 @@ export async function fetchGitHubZipBytes(
fetcher: typeof fetch,
limits?: { maxZipBytes?: number },
): Promise<Uint8Array> {
const maxZipBytes = limits?.maxZipBytes ?? 25 * 1024 * 1024
const url = `https://${CODELOAD_HOST}/${resolved.owner}/${resolved.repo}/zip/${resolved.commit}`
const maxZipBytes = limits?.maxZipBytes ?? 25 * 1024 * 1024;
const url = `https://${CODELOAD_HOST}/${resolved.owner}/${resolved.repo}/zip/${resolved.commit}`;
const response = await fetcher(url, {
headers: { 'User-Agent': 'clawdhub/github-import' },
})
if (!response.ok) throw new Error('GitHub archive download failed')
headers: { "User-Agent": "clawhub/github-import" },
});
if (!response.ok) throw new Error("GitHub archive download failed");
const lengthHeader = response.headers.get('content-length')
const lengthHeader = response.headers.get("content-length");
if (lengthHeader) {
const contentLength = Number.parseInt(lengthHeader, 10)
const contentLength = Number.parseInt(lengthHeader, 10);
if (Number.isFinite(contentLength) && contentLength > maxZipBytes) {
throw new Error('GitHub archive too large')
throw new Error("GitHub archive too large");
}
}
const reader = response.body?.getReader()
const reader = response.body?.getReader();
if (!reader) {
const buffer = new Uint8Array(await response.arrayBuffer())
if (buffer.byteLength > maxZipBytes) throw new Error('GitHub archive too large')
return buffer
const buffer = new Uint8Array(await response.arrayBuffer());
if (buffer.byteLength > maxZipBytes) throw new Error("GitHub archive too large");
return buffer;
}
const chunks: Uint8Array[] = []
let total = 0
const chunks: Uint8Array[] = [];
let total = 0;
while (true) {
const { done, value } = await reader.read()
if (done) break
if (!value) continue
total += value.byteLength
if (total > maxZipBytes) throw new Error('GitHub archive too large')
chunks.push(value)
const { done, value } = await reader.read();
if (done) break;
if (!value) continue;
total += value.byteLength;
if (total > maxZipBytes) throw new Error("GitHub archive too large");
chunks.push(value);
}
const out = new Uint8Array(total)
let offset = 0
const out = new Uint8Array(total);
let offset = 0;
for (const chunk of chunks) {
out.set(chunk, offset)
offset += chunk.byteLength
out.set(chunk, offset);
offset += chunk.byteLength;
}
return out
return out;
}
export type ZipEntryMap = Record<string, Uint8Array>
export type ZipEntryMap = Record<string, Uint8Array>;
export function buildGitHubZipForTests(entries: Record<string, string>) {
const asBytes = Object.fromEntries(
Object.entries(entries).map(([path, text]) => [path, new TextEncoder().encode(text)]),
)
return Uint8Array.from(zipSync(asBytes, { level: 1 }))
);
return Uint8Array.from(zipSync(asBytes, { level: 1 }));
}
export function stripGitHubZipRoot(entries: ZipEntryMap): ZipEntryMap {
const paths = Object.keys(entries)
if (paths.length === 0) return {}
const first = paths[0] ?? ''
const firstRoot = first.split('/')[0] ?? ''
if (!firstRoot) return entries
const prefix = `${firstRoot}/`
if (!paths.every((path) => path.startsWith(prefix))) return entries
const out: ZipEntryMap = {}
const paths = Object.keys(entries);
if (paths.length === 0) return {};
const first = paths[0] ?? "";
const firstRoot = first.split("/")[0] ?? "";
if (!firstRoot) return entries;
const prefix = `${firstRoot}/`;
if (!paths.every((path) => path.startsWith(prefix))) return entries;
const out: ZipEntryMap = {};
for (const [path, data] of Object.entries(entries)) {
const stripped = path.slice(prefix.length)
if (!stripped) continue
out[stripped] = data
const stripped = path.slice(prefix.length);
if (!stripped) continue;
out[stripped] = data;
}
return out
return out;
}
export function detectGitHubImportCandidates(entries: ZipEntryMap): GitHubImportCandidate[] {
const candidates: GitHubImportCandidate[] = []
const candidates: GitHubImportCandidate[] = [];
for (const path of Object.keys(entries)) {
const normalized = normalizeRepoPath(path)
const lower = normalized.toLowerCase()
const isSkill = SKILL_FILENAMES.some((name) => lower === name || lower.endsWith(`/${name}`))
if (!isSkill) continue
const dir = normalized.split('/').slice(0, -1).join('/')
const readmePath = normalized
const raw = new TextDecoder().decode(entries[path] ?? new Uint8Array())
const frontmatter = parseFrontmatter(raw)
const name = typeof frontmatter.name === 'string' ? frontmatter.name : undefined
const normalized = normalizeRepoPath(path);
const lower = normalized.toLowerCase();
const isSkill = SKILL_FILENAMES.some((name) => lower === name || lower.endsWith(`/${name}`));
if (!isSkill) continue;
const dir = normalized.split("/").slice(0, -1).join("/");
const readmePath = normalized;
const raw = new TextDecoder().decode(entries[path] ?? new Uint8Array());
const frontmatter = parseFrontmatter(raw);
const name = typeof frontmatter.name === "string" ? frontmatter.name : undefined;
const description =
typeof frontmatter.description === 'string' ? frontmatter.description : undefined
typeof frontmatter.description === "string" ? frontmatter.description : undefined;
candidates.push({
path: normalizeRepoPath(dir),
readmePath,
name: name?.trim() || undefined,
description: description?.trim() || undefined,
})
});
}
return uniqCandidates(candidates)
return uniqCandidates(candidates);
}
function uniqCandidates(candidates: GitHubImportCandidate[]) {
const seen = new Set<string>()
const out: GitHubImportCandidate[] = []
const seen = new Set<string>();
const out: GitHubImportCandidate[] = [];
for (const candidate of candidates) {
const key = `${candidate.path}::${candidate.readmePath}`
if (seen.has(key)) continue
seen.add(key)
out.push(candidate)
const key = `${candidate.path}::${candidate.readmePath}`;
if (seen.has(key)) continue;
seen.add(key);
out.push(candidate);
}
return out.sort((a, b) => a.path.localeCompare(b.path))
return out.sort((a, b) => a.path.localeCompare(b.path));
}
export function listTextFilesUnderCandidate(
entries: ZipEntryMap,
candidatePath: string,
): Array<{ path: string; bytes: Uint8Array }> {
const root = normalizeCandidateRoot(candidatePath)
const out: Array<{ path: string; bytes: Uint8Array }> = []
const root = normalizeCandidateRoot(candidatePath);
const out: Array<{ path: string; bytes: Uint8Array }> = [];
for (const [path, bytes] of Object.entries(entries)) {
const normalized = normalizeRepoPath(path)
if (!isUnderRoot(normalized, root)) continue
if (!isTextPath(normalized)) continue
out.push({ path: normalized, bytes })
const normalized = normalizeRepoPath(path);
if (!isUnderRoot(normalized, root)) continue;
if (!isTextPath(normalized)) continue;
out.push({ path: normalized, bytes });
}
return out.sort((a, b) => a.path.localeCompare(b.path))
return out.sort((a, b) => a.path.localeCompare(b.path));
}
export function computeDefaultSelectedPaths(params: {
candidate: GitHubImportCandidate
files: Array<{ path: string; bytes: Uint8Array }>
maxDepth?: number
maxAdds?: number
candidate: GitHubImportCandidate;
files: Array<{ path: string; bytes: Uint8Array }>;
maxDepth?: number;
maxAdds?: number;
}) {
const maxDepth = params.maxDepth ?? 4
const maxAdds = params.maxAdds ?? 200
const byPath = new Map(params.files.map((file) => [file.path, file.bytes]))
const candidateRoot = normalizeCandidateRoot(params.candidate.path)
const selected = new Set<string>()
let added = 0
const maxDepth = params.maxDepth ?? 4;
const maxAdds = params.maxAdds ?? 200;
const byPath = new Map(params.files.map((file) => [file.path, file.bytes]));
const candidateRoot = normalizeCandidateRoot(params.candidate.path);
const selected = new Set<string>();
let added = 0;
const add = (path: string) => {
const normalized = normalizeRepoPath(path)
if (!isUnderRoot(normalized, candidateRoot)) return
if (!byPath.has(normalized)) return
const normalized = normalizeRepoPath(path);
if (!isUnderRoot(normalized, candidateRoot)) return;
if (!byPath.has(normalized)) return;
if (!selected.has(normalized)) {
selected.add(normalized)
added += 1
selected.add(normalized);
added += 1;
}
}
};
add(params.candidate.readmePath)
add(params.candidate.readmePath);
const visited = new Set<string>()
const visited = new Set<string>();
const queue: Array<{ path: string; depth: number }> = [
{ path: params.candidate.readmePath, depth: 0 },
]
];
while (queue.length > 0) {
const item = queue.shift()
if (!item) break
if (item.depth >= maxDepth) continue
if (visited.has(item.path)) continue
visited.add(item.path)
const item = queue.shift();
if (!item) break;
if (item.depth >= maxDepth) continue;
if (visited.has(item.path)) continue;
visited.add(item.path);
const bytes = byPath.get(item.path)
if (!bytes) continue
if (!item.path.toLowerCase().endsWith('.md')) continue
const bytes = byPath.get(item.path);
if (!bytes) continue;
if (!item.path.toLowerCase().endsWith(".md")) continue;
const text = new TextDecoder().decode(bytes)
const refs = extractMarkdownRelativeTargets(text)
const text = new TextDecoder().decode(bytes);
const refs = extractMarkdownRelativeTargets(text);
for (const ref of refs) {
if (added >= maxAdds) break
const resolved = resolveMarkdownTarget(item.path, ref)
if (!resolved) continue
add(resolved)
if (resolved.toLowerCase().endsWith('.md') && byPath.has(resolved)) {
queue.push({ path: resolved, depth: item.depth + 1 })
if (added >= maxAdds) break;
const resolved = resolveMarkdownTarget(item.path, ref);
if (!resolved) continue;
add(resolved);
if (resolved.toLowerCase().endsWith(".md") && byPath.has(resolved)) {
queue.push({ path: resolved, depth: item.depth + 1 });
}
}
if (added >= maxAdds) break
if (added >= maxAdds) break;
}
return Array.from(selected).sort()
return Array.from(selected).sort();
}
export function buildGitHubImportFileList(params: {
candidate: GitHubImportCandidate
files: Array<{ path: string; bytes: Uint8Array }>
defaultSelectedPaths: string[]
candidate: GitHubImportCandidate;
files: Array<{ path: string; bytes: Uint8Array }>;
defaultSelectedPaths: string[];
}): GitHubImportFileEntry[] {
const selected = new Set(params.defaultSelectedPaths)
const selected = new Set(params.defaultSelectedPaths);
return params.files.map((file) => ({
path: file.path,
size: file.bytes.byteLength,
defaultSelected: selected.has(file.path),
}))
}));
}
export function normalizeRepoPath(path: string) {
const stripped = path.replace(/^\/+/, '').trim()
if (!stripped) return ''
const cleaned = stripped.split('/').filter(Boolean).join('/')
if (!cleaned || cleaned.includes('\\') || cleaned.includes('..')) return ''
return cleaned
const stripped = path.replace(/^\/+/, "").trim();
if (!stripped) return "";
const cleaned = stripped.split("/").filter(Boolean).join("/");
if (!cleaned || cleaned.includes("\\") || cleaned.includes("..")) return "";
return cleaned;
}
export function normalizeCandidateRoot(candidatePath: string) {
const normalized = normalizeRepoPath(candidatePath)
return normalized ? `${normalized}/` : ''
const normalized = normalizeRepoPath(candidatePath);
return normalized ? `${normalized}/` : "";
}
function isUnderRoot(path: string, rootWithSlash: string) {
if (!rootWithSlash) return true
return path === rootWithSlash.slice(0, -1) || path.startsWith(rootWithSlash)
if (!rootWithSlash) return true;
return path === rootWithSlash.slice(0, -1) || path.startsWith(rootWithSlash);
}
function isTextPath(path: string) {
const lower = path.toLowerCase()
const ext = lower.split('.').at(-1) ?? ''
if (!ext) return false
return TEXT_FILE_EXTENSION_SET.has(ext)
const lower = path.toLowerCase();
const ext = lower.split(".").at(-1) ?? "";
if (!ext) return false;
return TEXT_FILE_EXTENSION_SET.has(ext);
}
export function suggestDisplayName(candidate: GitHubImportCandidate, fallbackBase: string) {
const base = candidate.name?.trim() || fallbackBase.trim()
if (!base) return ''
const base = candidate.name?.trim() || fallbackBase.trim();
if (!base) return "";
return base
.replace(/[-_]+/g, ' ')
.replace(/\s+/g, ' ')
.replace(/\b\w/g, (char) => char.toUpperCase())
.replace(/[-_]+/g, " ")
.replace(/\s+/g, " ")
.replace(/\b\w/g, (char) => char.toUpperCase());
}
export function suggestVersion(latestVersion?: string | null) {
const latest = latestVersion?.trim() || ''
const latest = latestVersion?.trim() || "";
if (latest && semver.valid(latest)) {
return semver.inc(latest, 'patch') ?? '0.1.0'
return semver.inc(latest, "patch") ?? "0.1.0";
}
return '0.1.0'
return "0.1.0";
}
export function extractMarkdownRelativeTargets(markdown: string): string[] {
const out: string[] = []
const pattern = /!?\[[^\]]*]\(([^)]+)\)/g
const out: string[] = [];
const pattern = /!?\[[^\]]*]\(([^)]+)\)/g;
for (const match of markdown.matchAll(pattern)) {
const raw = (match[1] ?? '').trim()
if (!raw) continue
const isAngleWrapped = raw.startsWith('<') && raw.endsWith('>')
const cleaned = raw.replace(/^<|>$/g, '').trim()
if (!cleaned) continue
const target = isAngleWrapped ? cleaned : (cleaned.split(/\s+/)[0] ?? '')
if (!target) continue
if (target.startsWith('#')) continue
const lower = target.toLowerCase()
if (lower.startsWith('http:') || lower.startsWith('https:')) continue
if (lower.startsWith('mailto:')) continue
out.push(target)
const raw = (match[1] ?? "").trim();
if (!raw) continue;
const isAngleWrapped = raw.startsWith("<") && raw.endsWith(">");
const cleaned = raw.replace(/^<|>$/g, "").trim();
if (!cleaned) continue;
const target = isAngleWrapped ? cleaned : (cleaned.split(/\s+/)[0] ?? "");
if (!target) continue;
if (target.startsWith("#")) continue;
const lower = target.toLowerCase();
if (lower.startsWith("http:") || lower.startsWith("https:")) continue;
if (lower.startsWith("mailto:")) continue;
out.push(target);
}
return out
return out;
}
export function resolveMarkdownTarget(fromPath: string, target: string) {
const withoutHash = target.split('#')[0] ?? ''
const withoutQuery = (withoutHash.split('?')[0] ?? '').trim()
if (!withoutQuery) return null
if (withoutQuery.startsWith('/')) return null
if (withoutQuery.includes('\\') || withoutQuery.includes('..')) return null
const withoutHash = target.split("#")[0] ?? "";
const withoutQuery = (withoutHash.split("?")[0] ?? "").trim();
if (!withoutQuery) return null;
if (withoutQuery.startsWith("/")) return null;
if (withoutQuery.includes("\\") || withoutQuery.includes("..")) return null;
const fromDirParts = normalizeRepoPath(fromPath).split('/').slice(0, -1)
const targetParts = withoutQuery.split('/').filter(Boolean)
const combined = [...fromDirParts, ...targetParts]
const normalized: string[] = []
const fromDirParts = normalizeRepoPath(fromPath).split("/").slice(0, -1);
const targetParts = withoutQuery.split("/").filter(Boolean);
const combined = [...fromDirParts, ...targetParts];
const normalized: string[] = [];
for (const part of combined) {
if (part === '.') continue
if (part === '..') return null
normalized.push(part)
if (part === ".") continue;
if (part === "..") return null;
normalized.push(part);
}
return normalizeRepoPath(normalized.join('/')) || null
return normalizeRepoPath(normalized.join("/")) || null;
}
+18
View File
@@ -0,0 +1,18 @@
export const GITHUB_PROFILE_SYNC_WINDOW_MS = 6 * 60 * 60 * 1000;
export function shouldScheduleGitHubProfileSync(
user:
| {
deletedAt?: number;
deactivatedAt?: number;
githubProfileSyncedAt?: number;
}
| null
| undefined,
now: number,
) {
if (!user || user.deletedAt || user.deactivatedAt) return false;
const lastSyncedAt = user.githubProfileSyncedAt ?? null;
if (lastSyncedAt && now - lastSyncedAt < GITHUB_PROFILE_SYNC_WINDOW_MS) return false;
return true;
}
+53
View File
@@ -0,0 +1,53 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from "vitest";
import type { GitHubBackupContext } from "./githubBackup";
import { readGitHubBackupFile } from "./githubRestoreHelpers";
function makeContext(): GitHubBackupContext {
return {
token: "token",
repo: "owner/repo",
repoOwner: "owner",
repoName: "repo",
branch: "main",
root: "skills",
};
}
afterEach(() => {
vi.unstubAllGlobals();
vi.restoreAllMocks();
});
describe("githubRestoreHelpers", () => {
it("decodes base64 payloads (including newlines) into bytes", async () => {
const content = "SGVs\n bG8h"; // "Hello!" with whitespace/newline
vi.stubGlobal(
"fetch",
vi.fn(async () => ({
ok: true,
json: async () => ({ content, encoding: "base64" }),
text: async () => "",
})),
);
const bytes = await readGitHubBackupFile(makeContext(), "Owner", "slug", "SKILL.md");
expect(bytes).not.toBeNull();
expect(Buffer.from(bytes!).toString("utf8")).toBe("Hello!");
});
it("throws on unsupported GitHub content encoding", async () => {
vi.stubGlobal(
"fetch",
vi.fn(async () => ({
ok: true,
json: async () => ({ content: "eA==", encoding: "utf-16" }),
text: async () => "",
})),
);
await expect(readGitHubBackupFile(makeContext(), "Owner", "slug", "SKILL.md")).rejects.toThrow(
/Unsupported GitHub content encoding/i,
);
});
});
+159
View File
@@ -0,0 +1,159 @@
"use node";
import type { GitHubBackupContext } from "./githubBackup";
const GITHUB_API = "https://api.github.com";
const META_FILENAME = "_meta.json";
const USER_AGENT = "clawhub/skills-restore";
type GitHubContentsEntry = {
name?: string;
path?: string;
type?: string; // 'file' | 'dir'
size?: number;
};
type GitHubBlobResponse = {
content?: string;
encoding?: string;
size?: number;
};
/**
* List all files in a skill's backup directory (excluding _meta.json).
* Uses the Contents API scoped to the target directory instead of fetching
* the entire repository tree, which is critical for bulk restore performance.
* Returns relative file paths (e.g. "SKILL.md", "lib/helper.ts").
*/
export async function listGitHubBackupFiles(
context: GitHubBackupContext,
ownerHandle: string,
slug: string,
): Promise<string[]> {
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug);
return listFilesRecursive(context, skillRoot, "");
}
/**
* Recursively list files under a directory using the GitHub Contents API.
* Each call is scoped to one directory, avoiding full-repo tree downloads.
*/
async function listFilesRecursive(
context: GitHubBackupContext,
basePath: string,
relativePath: string,
): Promise<string[]> {
const dirPath = relativePath ? `${basePath}/${relativePath}` : basePath;
try {
const entries = await githubGet<GitHubContentsEntry[]>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/contents/${encodePath(dirPath)}?ref=${context.branch}`,
);
if (!Array.isArray(entries)) return [];
const files: string[] = [];
for (const entry of entries) {
if (!entry.name || !entry.type) continue;
const entryRelative = relativePath ? `${relativePath}/${entry.name}` : entry.name;
if (entry.type === "file") {
// Skip the meta file
if (entry.name === META_FILENAME) continue;
files.push(entryRelative);
} else if (entry.type === "dir") {
// Recurse into subdirectories
const subFiles = await listFilesRecursive(context, basePath, entryRelative);
files.push(...subFiles);
}
}
return files;
} catch (error) {
if (isNotFoundError(error)) return [];
throw error;
}
}
/**
* Read a single file from the GitHub backup repository.
* Returns the file content as a Uint8Array, or null if not found.
*/
export async function readGitHubBackupFile(
context: GitHubBackupContext,
ownerHandle: string,
slug: string,
filePath: string,
): Promise<Uint8Array | null> {
const skillRoot = buildSkillRoot(context.root, ownerHandle, slug);
const fullPath = `${skillRoot}/${filePath}`;
try {
const response = await githubGet<GitHubBlobResponse>(
context.token,
`/repos/${context.repoOwner}/${context.repoName}/contents/${encodePath(fullPath)}?ref=${context.branch}`,
);
if (!response.content) return null;
if (response.encoding && response.encoding !== "base64") {
throw new Error(`Unsupported GitHub content encoding: ${response.encoding}`);
}
return fromBase64Bytes(response.content);
} catch (error) {
if (isNotFoundError(error)) return null;
throw error;
}
}
function buildSkillRoot(root: string, ownerHandle: string, slug: string) {
const ownerSegment = normalizeOwner(ownerHandle);
return `${root}/${ownerSegment}/${slug}`;
}
function normalizeOwner(value: string) {
const normalized = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]/g, "-")
.replace(/-+/g, "-")
.replace(/^-+|-+$/g, "");
return normalized || "unknown";
}
function encodePath(path: string) {
return path
.split("/")
.map((segment) => encodeURIComponent(segment))
.join("/");
}
function fromBase64Bytes(value: string) {
// GitHub may include newlines in the base64 payload.
const normalized = value.replace(/\s/g, "");
return new Uint8Array(Buffer.from(normalized, "base64"));
}
async function githubGet<T>(token: string, path: string): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
headers: {
Authorization: `token ${token}`,
Accept: "application/vnd.github+json",
"User-Agent": USER_AGENT,
},
});
if (!response.ok) {
const message = await response.text();
throw new Error(`GitHub GET ${path} failed: ${message}`);
}
return (await response.json()) as T;
}
function isNotFoundError(error: unknown) {
return (
error instanceof Error && (error.message.includes("404") || error.message.includes("Not Found"))
);
}
+194 -189
View File
@@ -1,96 +1,96 @@
'use node'
"use node";
import { createPrivateKey, createSign } from 'node:crypto'
import type { Id } from '../_generated/dataModel'
import type { ActionCtx } from '../_generated/server'
import { createPrivateKey, createSign } from "node:crypto";
import type { Id } from "../_generated/dataModel";
import type { ActionCtx } from "../_generated/server";
const GITHUB_API = 'https://api.github.com'
const DEFAULT_REPO = 'clawdbot/souls'
const DEFAULT_ROOT = 'souls'
const META_FILENAME = '_meta.json'
const USER_AGENT = 'clawdhub/souls-backup'
const GITHUB_API = "https://api.github.com";
const DEFAULT_REPO = "clawdbot/souls";
const DEFAULT_ROOT = "souls";
const META_FILENAME = "_meta.json";
const USER_AGENT = "clawhub/souls-backup";
type BackupFile = {
path: string
size: number
storageId: Id<'_storage'>
sha256: string
contentType?: string
}
path: string;
size: number;
storageId: Id<"_storage">;
sha256: string;
contentType?: string;
};
type BackupParams = {
slug: string
version: string
displayName: string
ownerHandle: string
files: BackupFile[]
publishedAt: number
}
slug: string;
version: string;
displayName: string;
ownerHandle: string;
files: BackupFile[];
publishedAt: number;
};
type RepoInfo = {
default_branch?: string
}
default_branch?: string;
};
type GitRef = {
object: { sha: string }
}
object: { sha: string };
};
type GitCommit = {
sha: string
tree: { sha: string }
}
sha: string;
tree: { sha: string };
};
type GitTreeEntry = {
path?: string
type?: string
}
path?: string;
type?: string;
};
type GitTree = {
tree?: GitTreeEntry[]
}
tree?: GitTreeEntry[];
};
type MetaFile = {
owner: string
slug: string
displayName: string
owner: string;
slug: string;
displayName: string;
latest: {
version: string
publishedAt: number
commit: string | null
}
version: string;
publishedAt: number;
commit: string | null;
};
history: Array<{
version: string
publishedAt: number
commit: string
}>
}
version: string;
publishedAt: number;
commit: string;
}>;
};
export type GitHubBackupContext = {
token: string
repo: string
repoOwner: string
repoName: string
branch: string
root: string
}
token: string;
repo: string;
repoOwner: string;
repoName: string;
branch: string;
root: string;
};
export function isGitHubSoulBackupConfigured() {
return Boolean(
process.env.GITHUB_APP_ID &&
process.env.GITHUB_APP_PRIVATE_KEY &&
process.env.GITHUB_APP_INSTALLATION_ID,
)
process.env.GITHUB_APP_PRIVATE_KEY &&
process.env.GITHUB_APP_INSTALLATION_ID,
);
}
export async function getGitHubSoulBackupContext(): Promise<GitHubBackupContext> {
const repo = process.env.GITHUB_SOULS_REPO ?? DEFAULT_REPO
const root = process.env.GITHUB_SOULS_ROOT ?? DEFAULT_ROOT
const [repoOwner, repoName] = parseRepo(repo)
const token = await createInstallationToken()
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`)
const branch = repoInfo.default_branch ?? 'main'
const repo = process.env.GITHUB_SOULS_REPO ?? DEFAULT_REPO;
const root = process.env.GITHUB_SOULS_ROOT ?? DEFAULT_ROOT;
const [repoOwner, repoName] = parseRepo(repo);
const token = await createInstallationToken();
const repoInfo = await githubGet<RepoInfo>(token, `/repos/${repoOwner}/${repoName}`);
const branch = repoInfo.default_branch ?? "main";
return { token, repo, repoOwner, repoName, branch, root }
return { token, repo, repoOwner, repoName, branch, root };
}
export async function fetchGitHubSoulMeta(
@@ -98,14 +98,14 @@ export async function fetchGitHubSoulMeta(
ownerHandle: string,
slug: string,
): Promise<MetaFile | null> {
const soulRoot = buildSoulRoot(context.root, ownerHandle, slug)
const soulRoot = buildSoulRoot(context.root, ownerHandle, slug);
return fetchMetaFile(
context.token,
context.repoOwner,
context.repoName,
`${soulRoot}/${META_FILENAME}`,
context.branch,
)
);
}
export async function backupSoulToGitHub(
@@ -113,46 +113,51 @@ export async function backupSoulToGitHub(
params: BackupParams,
context?: GitHubBackupContext,
) {
if (!isGitHubSoulBackupConfigured()) return
if (!isGitHubSoulBackupConfigured()) return;
const resolved = context ?? (await getGitHubSoulBackupContext())
const soulRoot = buildSoulRoot(resolved.root, params.ownerHandle, params.slug)
const resolved = context ?? (await getGitHubSoulBackupContext());
const soulRoot = buildSoulRoot(resolved.root, params.ownerHandle, params.slug);
const ref = await githubGet<GitRef>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/ref/heads/${resolved.branch}`,
)
const baseCommitSha = ref.object.sha
);
const baseCommitSha = ref.object.sha;
const baseCommit = await githubGet<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits/${baseCommitSha}`,
)
const baseTreeSha = baseCommit.tree.sha
);
const baseTreeSha = baseCommit.tree.sha;
const existingTree = await githubGet<GitTree>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees/${baseTreeSha}?recursive=1`,
)
);
const prefix = `${soulRoot}/`
const prefix = `${soulRoot}/`;
const existingPaths = new Set(
(existingTree.tree ?? [])
.filter((entry) => entry.type === 'blob' && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? ''),
)
.filter((entry) => entry.type === "blob" && entry.path?.startsWith(prefix))
.map((entry) => entry.path ?? ""),
);
const newPaths = new Set<string>()
const newPaths = new Set<string>();
const treeEntries: Array<{
path: string
mode: '100644'
type: 'blob'
sha: string | null
}> = []
path: string;
mode: "100644";
type: "blob";
sha: string | null;
}> = [];
for (const file of params.files) {
const content = await fetchStorageBase64(ctx, file.storageId)
const blobSha = await createBlob(resolved.token, resolved.repoOwner, resolved.repoName, content)
const path = `${soulRoot}/${file.path}`
newPaths.add(path)
treeEntries.push({ path, mode: '100644', type: 'blob', sha: blobSha })
const content = await fetchStorageBase64(ctx, file.storageId);
const blobSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
content,
);
const path = `${soulRoot}/${file.path}`;
newPaths.add(path);
treeEntries.push({ path, mode: "100644", type: "blob", sha: blobSha });
}
const existingMeta = await fetchMetaFile(
@@ -161,22 +166,22 @@ export async function backupSoulToGitHub(
resolved.repoName,
`${soulRoot}/${META_FILENAME}`,
resolved.branch,
)
const metaPath = `${soulRoot}/${META_FILENAME}`
const metaDraft = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, null)
const metaDraftContent = `${JSON.stringify(metaDraft, null, 2)}\n`
);
const metaPath = `${soulRoot}/${META_FILENAME}`;
const metaDraft = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, null);
const metaDraftContent = `${JSON.stringify(metaDraft, null, 2)}\n`;
const metaDraftSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaDraftContent),
)
newPaths.add(metaPath)
treeEntries.push({ path: metaPath, mode: '100644', type: 'blob', sha: metaDraftSha })
);
newPaths.add(metaPath);
treeEntries.push({ path: metaPath, mode: "100644", type: "blob", sha: metaDraftSha });
for (const path of existingPaths) {
if (newPaths.has(path)) continue
treeEntries.push({ path, mode: '100644', type: 'blob', sha: null })
if (newPaths.has(path)) continue;
treeEntries.push({ path, mode: "100644", type: "blob", sha: null });
}
const newTree = await githubPost<{ sha: string }>(
@@ -186,7 +191,7 @@ export async function backupSoulToGitHub(
base_tree: baseTreeSha,
tree: treeEntries,
},
)
);
const commit = await githubPost<GitCommit>(
resolved.token,
@@ -196,24 +201,24 @@ export async function backupSoulToGitHub(
tree: newTree.sha,
parents: [baseCommitSha],
},
)
);
const metaFinal = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, commit.sha)
const metaFinalContent = `${JSON.stringify(metaFinal, null, 2)}\n`
const metaFinal = buildMetaFile(params, existingMeta, resolved.repo, baseCommitSha, commit.sha);
const metaFinalContent = `${JSON.stringify(metaFinal, null, 2)}\n`;
const metaFinalSha = await createBlob(
resolved.token,
resolved.repoOwner,
resolved.repoName,
toBase64(metaFinalContent),
)
);
const metaTree = await githubPost<{ sha: string }>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/trees`,
{
base_tree: commit.tree.sha,
tree: [{ path: metaPath, mode: '100644', type: 'blob', sha: metaFinalSha }],
tree: [{ path: metaPath, mode: "100644", type: "blob", sha: metaFinalSha }],
},
)
);
const metaCommit = await githubPost<GitCommit>(
resolved.token,
`/repos/${resolved.repoOwner}/${resolved.repoName}/git/commits`,
@@ -222,7 +227,7 @@ export async function backupSoulToGitHub(
tree: metaTree.sha,
parents: [commit.sha],
},
)
);
await githubPatch(
resolved.token,
@@ -230,7 +235,7 @@ export async function backupSoulToGitHub(
{
sha: metaCommit.sha,
},
)
);
}
function buildMetaFile(
@@ -240,15 +245,15 @@ function buildMetaFile(
baseCommitSha: string,
latestCommitSha: string | null,
): MetaFile {
let history = [...(existing?.history ?? [])]
let history = [...(existing?.history ?? [])];
if (existing?.latest?.version) {
const previousCommit = existing.latest.commit ?? commitUrl(repo, baseCommitSha)
const previousCommit = existing.latest.commit ?? commitUrl(repo, baseCommitSha);
const previous = {
version: existing.latest.version,
publishedAt: existing.latest.publishedAt,
commit: previousCommit,
}
history = [previous, ...history.filter((entry) => entry.version !== previous.version)]
};
history = [previous, ...history.filter((entry) => entry.version !== previous.version)];
}
return {
@@ -261,7 +266,7 @@ function buildMetaFile(
commit: latestCommitSha ? commitUrl(repo, latestCommitSha) : null,
},
history: history.slice(0, 200),
}
};
}
async function fetchMetaFile(
@@ -275,63 +280,63 @@ async function fetchMetaFile(
const response = await githubGet<{ content?: string }>(
token,
`/repos/${repoOwner}/${repoName}/contents/${encodePath(path)}?ref=${branch}`,
)
if (!response.content) return null
const raw = fromBase64(response.content)
return JSON.parse(raw) as MetaFile
);
if (!response.content) return null;
const raw = fromBase64(response.content);
return JSON.parse(raw) as MetaFile;
} catch (error) {
if (isNotFoundError(error)) return null
throw error
if (isNotFoundError(error)) return null;
throw error;
}
}
async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<'_storage'>) {
const blob = await ctx.storage.get(storageId)
if (!blob) throw new Error('File missing in storage')
const buffer = Buffer.from(await blob.arrayBuffer())
return buffer.toString('base64')
async function fetchStorageBase64(ctx: ActionCtx, storageId: Id<"_storage">) {
const blob = await ctx.storage.get(storageId);
if (!blob) throw new Error("File missing in storage");
const buffer = Buffer.from(await blob.arrayBuffer());
return buffer.toString("base64");
}
async function createInstallationToken() {
const appId = process.env.GITHUB_APP_ID
const installationId = process.env.GITHUB_APP_INSTALLATION_ID
const appId = process.env.GITHUB_APP_ID;
const installationId = process.env.GITHUB_APP_INSTALLATION_ID;
if (!appId || !installationId) {
throw new Error('GitHub App credentials missing')
throw new Error("GitHub App credentials missing");
}
const jwt = createAppJwt(appId)
const jwt = createAppJwt(appId);
const response = await fetch(`${GITHUB_API}/app/installations/${installationId}/access_tokens`, {
method: 'POST',
method: "POST",
headers: buildHeaders(jwt, true),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub App token failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub App token failed: ${message}`);
}
const payload = (await response.json()) as { token?: string }
if (!payload.token) throw new Error('GitHub App token missing')
return payload.token
const payload = (await response.json()) as { token?: string };
if (!payload.token) throw new Error("GitHub App token missing");
return payload.token;
}
function createAppJwt(appId: string) {
const privateKey = loadPrivateKey()
const now = Math.floor(Date.now() / 1000)
const header = { alg: 'RS256', typ: 'JWT' }
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId }
const encodedHeader = base64Url(JSON.stringify(header))
const encodedPayload = base64Url(JSON.stringify(payload))
const signingInput = `${encodedHeader}.${encodedPayload}`
const sign = createSign('RSA-SHA256')
sign.update(signingInput)
sign.end()
const signature = sign.sign(privateKey)
return `${signingInput}.${base64Url(signature)}`
const privateKey = loadPrivateKey();
const now = Math.floor(Date.now() / 1000);
const header = { alg: "RS256", typ: "JWT" };
const payload = { iat: now - 60, exp: now + 9 * 60, iss: appId };
const encodedHeader = base64Url(JSON.stringify(header));
const encodedPayload = base64Url(JSON.stringify(payload));
const signingInput = `${encodedHeader}.${encodedPayload}`;
const sign = createSign("RSA-SHA256");
sign.update(signingInput);
sign.end();
const signature = sign.sign(privateKey);
return `${signingInput}.${base64Url(signature)}`;
}
function loadPrivateKey() {
const raw = process.env.GITHUB_APP_PRIVATE_KEY
if (!raw) throw new Error('GITHUB_APP_PRIVATE_KEY is not configured')
const normalized = raw.replace(/\\n/g, '\n')
return createPrivateKey(normalized)
const raw = process.env.GITHUB_APP_PRIVATE_KEY;
if (!raw) throw new Error("GITHUB_APP_PRIVATE_KEY is not configured");
const normalized = raw.replace(/\\n/g, "\n");
return createPrivateKey(normalized);
}
async function createBlob(token: string, repoOwner: string, repoName: string, content: string) {
@@ -340,104 +345,104 @@ async function createBlob(token: string, repoOwner: string, repoName: string, co
`/repos/${repoOwner}/${repoName}/git/blobs`,
{
content,
encoding: 'base64',
encoding: "base64",
},
)
if (!result.sha) throw new Error('GitHub blob missing sha')
return result.sha
);
if (!result.sha) throw new Error("GitHub blob missing sha");
return result.sha;
}
async function githubGet<T>(token: string, path: string): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
headers: buildHeaders(token),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub GET ${path} failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub GET ${path} failed: ${message}`);
}
return (await response.json()) as T
return (await response.json()) as T;
}
async function githubPost<T>(token: string, path: string, body: unknown): Promise<T> {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'POST',
method: "POST",
headers: buildHeaders(token),
body: JSON.stringify(body),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub POST ${path} failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub POST ${path} failed: ${message}`);
}
return (await response.json()) as T
return (await response.json()) as T;
}
async function githubPatch(token: string, path: string, body: unknown) {
const response = await fetch(`${GITHUB_API}${path}`, {
method: 'PATCH',
method: "PATCH",
headers: buildHeaders(token),
body: JSON.stringify(body),
})
});
if (!response.ok) {
const message = await response.text()
throw new Error(`GitHub PATCH ${path} failed: ${message}`)
const message = await response.text();
throw new Error(`GitHub PATCH ${path} failed: ${message}`);
}
}
function buildHeaders(token: string, isAppJwt = false) {
return {
Authorization: `${isAppJwt ? 'Bearer' : 'token'} ${token}`,
Accept: 'application/vnd.github+json',
'User-Agent': USER_AGENT,
}
Authorization: `${isAppJwt ? "Bearer" : "token"} ${token}`,
Accept: "application/vnd.github+json",
"User-Agent": USER_AGENT,
};
}
function parseRepo(repo: string) {
const [owner, name] = repo.split('/')
if (!owner || !name) throw new Error('GITHUB_SOULS_REPO must be owner/repo')
return [owner, name] as const
const [owner, name] = repo.split("/");
if (!owner || !name) throw new Error("GITHUB_SOULS_REPO must be owner/repo");
return [owner, name] as const;
}
function normalizeOwner(value: string) {
const normalized = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9-]/g, '-')
.replace(/-+/g, '-')
.replace(/^-+|-+$/g, '')
return normalized || 'unknown'
.replace(/[^a-z0-9-]/g, "-")
.replace(/-+/g, "-")
.replace(/^-+|-+$/g, "");
return normalized || "unknown";
}
function commitUrl(repo: string, sha: string) {
return `https://github.com/${repo}/commit/${sha}`
return `https://github.com/${repo}/commit/${sha}`;
}
function buildSoulRoot(root: string, ownerHandle: string, slug: string) {
const ownerSegment = normalizeOwner(ownerHandle)
return `${root}/${ownerSegment}/${slug}`
const ownerSegment = normalizeOwner(ownerHandle);
return `${root}/${ownerSegment}/${slug}`;
}
function encodePath(path: string) {
return path
.split('/')
.split("/")
.map((segment) => encodeURIComponent(segment))
.join('/')
.join("/");
}
function base64Url(value: string | Buffer) {
const buffer = typeof value === 'string' ? Buffer.from(value) : value
return buffer.toString('base64').replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/g, '')
function base64Url(value: string | Uint8Array) {
const buffer = typeof value === "string" ? Buffer.from(value) : Buffer.from(value);
return buffer.toString("base64").replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/g, "");
}
function toBase64(value: string) {
return Buffer.from(value).toString('base64')
return Buffer.from(value).toString("base64");
}
function fromBase64(value: string) {
return Buffer.from(value, 'base64').toString('utf8')
return Buffer.from(value, "base64").toString("utf8");
}
function isNotFoundError(error: unknown) {
return (
error instanceof Error && (error.message.includes('404') || error.message.includes('Not Found'))
)
error instanceof Error && (error.message.includes("404") || error.message.includes("Not Found"))
);
}
+141
View File
@@ -0,0 +1,141 @@
import type { Doc } from "../_generated/dataModel";
import type { MutationCtx, QueryCtx } from "../_generated/server";
export const GLOBAL_STATS_KEY = "default";
type SkillVisibilityFields = Pick<
Doc<"skills">,
"softDeletedAt" | "moderationStatus" | "moderationFlags"
>;
type GlobalStatsReadCtx = Pick<MutationCtx | QueryCtx, "db">;
type GlobalStatsWriteCtx = Pick<MutationCtx, "db">;
export function isPublicSkillDoc(skill: SkillVisibilityFields | null | undefined) {
if (!skill || skill.softDeletedAt) return false;
if (skill.moderationStatus && skill.moderationStatus !== "active") return false;
if (skill.moderationFlags?.includes("blocked.malware")) return false;
return true;
}
export function getPublicSkillVisibilityDelta(
before: SkillVisibilityFields | null | undefined,
after: SkillVisibilityFields | null | undefined,
) {
const beforePublic = isPublicSkillDoc(before);
const afterPublic = isPublicSkillDoc(after);
if (beforePublic === afterPublic) return 0;
return afterPublic ? 1 : -1;
}
function getErrorMessage(error: unknown) {
if (typeof error === "string") return error;
if (error && typeof error === "object" && "message" in error) {
const message = (error as { message?: unknown }).message;
if (typeof message === "string") return message;
}
return "";
}
export function isGlobalStatsStorageNotReadyError(error: unknown) {
const message = getErrorMessage(error).toLowerCase();
if (!message) return false;
const referencesGlobalStats = message.includes("globalstats") || message.includes("by_key");
if (!referencesGlobalStats) return false;
return (
message.includes("table") ||
message.includes("index") ||
message.includes("schema") ||
message.includes("not found") ||
message.includes("does not exist") ||
message.includes("unknown")
);
}
export async function countPublicSkillsForGlobalStats(ctx: GlobalStatsReadCtx) {
const digests = await ctx.db
.query("skillSearchDigest")
.withIndex("by_active_updated", (q) => q.eq("softDeletedAt", undefined))
.collect();
let count = 0;
for (const digest of digests) {
if (isPublicSkillDoc(digest)) count += 1;
}
return count;
}
export async function setGlobalPublicSkillsCount(
ctx: GlobalStatsWriteCtx,
count: number,
now = Date.now(),
) {
const normalizedCount = Math.max(0, Math.trunc(Number.isFinite(count) ? count : 0));
try {
const existing = await ctx.db
.query("globalStats")
.withIndex("by_key", (q) => q.eq("key", GLOBAL_STATS_KEY))
.unique();
if (existing) {
await ctx.db.patch(existing._id, { activeSkillsCount: normalizedCount, updatedAt: now });
} else {
await ctx.db.insert("globalStats", {
key: GLOBAL_STATS_KEY,
activeSkillsCount: normalizedCount,
updatedAt: now,
});
}
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return;
throw error;
}
}
export async function adjustGlobalPublicSkillsCount(
ctx: GlobalStatsWriteCtx,
delta: number,
now = Date.now(),
) {
const normalizedDelta = Math.trunc(Number.isFinite(delta) ? delta : 0);
if (normalizedDelta === 0) return;
let existing:
| {
_id: Doc<"globalStats">["_id"];
activeSkillsCount: number;
}
| null
| undefined;
try {
existing = await ctx.db
.query("globalStats")
.withIndex("by_key", (q) => q.eq("key", GLOBAL_STATS_KEY))
.unique();
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return;
throw error;
}
if (!existing) {
// No baseline yet (e.g. fresh deploy). Initialize via full recount once.
const count = await countPublicSkillsForGlobalStats(ctx);
await setGlobalPublicSkillsCount(ctx, count, now);
return;
}
const nextCount = Math.max(0, existing.activeSkillsCount + normalizedDelta);
await ctx.db.patch(existing._id, { activeSkillsCount: nextCount, updatedAt: now });
}
export async function readGlobalPublicSkillsCount(ctx: GlobalStatsReadCtx) {
try {
const stats = await ctx.db
.query("globalStats")
.withIndex("by_key", (q) => q.eq("key", GLOBAL_STATS_KEY))
.unique();
return stats?.activeSkillsCount ?? null;
} catch (error) {
if (isGlobalStatsStorageNotReadyError(error)) return null;
throw error;
}
}
+18
View File
@@ -0,0 +1,18 @@
function toHeaderRecord(init?: HeadersInit): Record<string, string> {
if (!init) return {};
if (init instanceof Headers) return Object.fromEntries(init.entries());
if (Array.isArray(init)) return Object.fromEntries(init);
return { ...(init as Record<string, string>) };
}
export function mergeHeaders(...inits: Array<HeadersInit | undefined>): Record<string, string> {
const out: Record<string, string> = {};
for (const init of inits) {
Object.assign(out, toHeaderRecord(init));
}
return out;
}
export function corsHeaders(origin: string = "*"): Record<string, string> {
return { "Access-Control-Allow-Origin": origin };
}
+296
View File
@@ -0,0 +1,296 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { applyRateLimit, getClientIp } from "./httpRateLimit";
type MockRateLimitStatus = {
allowed: boolean;
remaining: number;
limit: number;
resetAt: number;
};
type MockRateLimitPlan = {
ip: MockRateLimitStatus;
user?: MockRateLimitStatus;
tokenValid?: boolean;
userActive?: boolean;
};
function makeRateLimitCtx(plan: MockRateLimitPlan) {
const runQuery = vi.fn(async (_fn: unknown, args: Record<string, unknown>) => {
if ("tokenHash" in args) {
if (plan.tokenValid === false) return null;
return { _id: "token_1", revokedAt: undefined };
}
if ("tokenId" in args) {
if (plan.userActive === false) return null;
return { _id: "users_123", deletedAt: undefined, deactivatedAt: undefined };
}
if ("key" in args && "limit" in args && "windowMs" in args) {
const key = String(args.key);
if (key.startsWith("ip:")) return plan.ip;
if (key.startsWith("user:")) return plan.user;
}
throw new Error(`Unexpected runQuery args: ${JSON.stringify(args)}`);
});
const runMutation = vi.fn(async (_fn: unknown, args: Record<string, unknown>) => {
const key = String(args.key);
const source = key.startsWith("user:") ? plan.user : plan.ip;
if (!source) throw new Error(`Missing rate limit source for ${key}`);
return { allowed: source.allowed, remaining: source.remaining };
});
return {
runQuery,
runMutation,
} as unknown as Parameters<typeof applyRateLimit>[0];
}
describe("getClientIp", () => {
let prev: string | undefined;
beforeEach(() => {
prev = process.env.TRUST_FORWARDED_IPS;
});
afterEach(() => {
if (prev === undefined) {
delete process.env.TRUST_FORWARDED_IPS;
} else {
process.env.TRUST_FORWARDED_IPS = prev;
}
});
it("returns null when cf-connecting-ip is missing (CF-only default)", () => {
const request = new Request("https://example.com", {
headers: {
"x-forwarded-for": "203.0.113.9",
},
});
delete process.env.TRUST_FORWARDED_IPS;
expect(getClientIp(request)).toBeNull();
});
it("keeps forwarded headers disabled when TRUST_FORWARDED_IPS=false", () => {
const request = new Request("https://example.com", {
headers: {
"x-forwarded-for": "203.0.113.9",
},
});
process.env.TRUST_FORWARDED_IPS = "false";
expect(getClientIp(request)).toBeNull();
});
it("returns first ip from cf-connecting-ip", () => {
const request = new Request("https://example.com", {
headers: {
"cf-connecting-ip": "203.0.113.1, 198.51.100.2",
},
});
expect(getClientIp(request)).toBe("203.0.113.1");
});
it("uses forwarded headers when opt-in enabled", () => {
const request = new Request("https://example.com", {
headers: {
"x-forwarded-for": "203.0.113.9, 198.51.100.2",
},
});
process.env.TRUST_FORWARDED_IPS = "true";
expect(getClientIp(request)).toBe("203.0.113.9");
});
it("prefers x-forwarded-for over x-real-ip when trusted mode is enabled", () => {
const request = new Request("https://example.com", {
headers: {
"x-forwarded-for": "203.0.113.9, 198.51.100.2",
"x-real-ip": "198.51.100.77",
},
});
process.env.TRUST_FORWARDED_IPS = "true";
expect(getClientIp(request)).toBe("203.0.113.9");
});
});
describe("applyRateLimit headers", () => {
afterEach(() => {
vi.restoreAllMocks();
});
it("returns delay-seconds Retry-After on 429 (not epoch)", async () => {
vi.spyOn(Date, "now").mockReturnValue(1_000_000);
const runMutation = vi.fn();
const ctx = {
runQuery: vi.fn().mockResolvedValue({
allowed: false,
remaining: 0,
limit: 20,
resetAt: 1_030_500,
}),
runMutation,
} as unknown as Parameters<typeof applyRateLimit>[0];
const request = new Request("https://example.com", {
headers: { "cf-connecting-ip": "203.0.113.1" },
});
const result = await applyRateLimit(ctx, request, "download");
expect(result.ok).toBe(false);
if (result.ok) return;
expect(result.response.status).toBe(429);
expect(result.response.headers.get("Retry-After")).toBe("31");
expect(result.response.headers.get("X-RateLimit-Reset")).toBe("1031");
expect(result.response.headers.get("RateLimit-Reset")).toBe("31");
expect(runMutation).not.toHaveBeenCalled();
});
it("includes rate-limit headers without Retry-After when allowed", async () => {
vi.spyOn(Date, "now").mockReturnValue(2_000_000);
const ctx = {
runQuery: vi.fn().mockResolvedValue({
allowed: true,
remaining: 19,
limit: 20,
resetAt: 2_015_000,
}),
runMutation: vi.fn().mockResolvedValue({
allowed: true,
remaining: 18,
}),
} as unknown as Parameters<typeof applyRateLimit>[0];
const request = new Request("https://example.com", {
headers: { "cf-connecting-ip": "203.0.113.1" },
});
const result = await applyRateLimit(ctx, request, "download");
expect(result.ok).toBe(true);
if (!result.ok) return;
const headers = new Headers(result.headers);
expect(headers.get("X-RateLimit-Limit")).toBe("20");
expect(headers.get("X-RateLimit-Remaining")).toBe("18");
expect(headers.get("X-RateLimit-Reset")).toBe("2015");
expect(headers.get("RateLimit-Limit")).toBe("20");
expect(headers.get("RateLimit-Remaining")).toBe("18");
expect(headers.get("RateLimit-Reset")).toBe("15");
expect(headers.get("Retry-After")).toBeNull();
});
it("allows authenticated users when user bucket is healthy and shared ip bucket is exhausted", async () => {
vi.spyOn(Date, "now").mockReturnValue(3_000_000);
const ctx = makeRateLimitCtx({
ip: {
allowed: false,
remaining: 0,
limit: 20,
resetAt: 3_040_000,
},
user: {
allowed: true,
remaining: 42,
limit: 120,
resetAt: 3_010_000,
},
});
const request = new Request("https://example.com", {
headers: {
authorization: "Bearer clh_token",
"cf-connecting-ip": "203.0.113.1",
},
});
const result = await applyRateLimit(ctx, request, "download");
expect(result.ok).toBe(true);
if (!result.ok) return;
const headers = new Headers(result.headers);
expect(headers.get("X-RateLimit-Limit")).toBe("120");
expect(headers.get("X-RateLimit-Remaining")).toBe("42");
expect(headers.get("Retry-After")).toBeNull();
});
it("does not consume ip bucket for authenticated requests", async () => {
vi.spyOn(Date, "now").mockReturnValue(3_100_000);
const ctx = makeRateLimitCtx({
ip: {
allowed: true,
remaining: 19,
limit: 20,
resetAt: 3_140_000,
},
user: {
allowed: true,
remaining: 41,
limit: 120,
resetAt: 3_110_000,
},
});
const request = new Request("https://example.com", {
headers: {
authorization: "Bearer clh_token",
"cf-connecting-ip": "203.0.113.1",
},
});
const result = await applyRateLimit(ctx, request, "download");
expect(result.ok).toBe(true);
const runMutation = (ctx as unknown as { runMutation: ReturnType<typeof vi.fn> }).runMutation;
const consumedKeys = runMutation.mock.calls.map(([, args]) => String(args.key));
expect(consumedKeys.some((key) => key.startsWith("user:"))).toBe(true);
expect(consumedKeys.some((key) => key.startsWith("ip:"))).toBe(false);
});
it("denies authenticated users when user bucket is exhausted even if ip bucket is healthy", async () => {
vi.spyOn(Date, "now").mockReturnValue(4_000_000);
const ctx = makeRateLimitCtx({
ip: {
allowed: true,
remaining: 19,
limit: 20,
resetAt: 4_020_000,
},
user: {
allowed: false,
remaining: 0,
limit: 120,
resetAt: 4_030_000,
},
});
const request = new Request("https://example.com", {
headers: {
authorization: "Bearer clh_token",
"cf-connecting-ip": "203.0.113.1",
},
});
const result = await applyRateLimit(ctx, request, "download");
expect(result.ok).toBe(false);
if (result.ok) return;
expect(result.response.status).toBe(429);
expect(result.response.headers.get("X-RateLimit-Limit")).toBe("120");
expect(result.response.headers.get("X-RateLimit-Remaining")).toBe("0");
expect(result.response.headers.get("Retry-After")).toBe("30");
});
it("falls back to ip enforcement when bearer token is invalid", async () => {
vi.spyOn(Date, "now").mockReturnValue(5_000_000);
const ctx = makeRateLimitCtx({
tokenValid: false,
ip: {
allowed: false,
remaining: 0,
limit: 20,
resetAt: 5_030_000,
},
});
const request = new Request("https://example.com", {
headers: {
authorization: "Bearer invalid",
"cf-connecting-ip": "203.0.113.1",
},
});
const result = await applyRateLimit(ctx, request, "download");
expect(result.ok).toBe(false);
if (result.ok) return;
expect(result.response.status).toBe(429);
expect(result.response.headers.get("X-RateLimit-Limit")).toBe("20");
expect(result.response.headers.get("Retry-After")).toBe("30");
});
});
+209
View File
@@ -0,0 +1,209 @@
import { internal } from "../_generated/api";
import type { ActionCtx } from "../_generated/server";
import { getOptionalApiTokenUserId } from "./apiTokenAuth";
import { corsHeaders, mergeHeaders } from "./httpHeaders";
const RATE_LIMIT_WINDOW_MS = 60_000;
export const RATE_LIMITS = {
read: { ip: 180, key: 900 },
write: { ip: 45, key: 180 },
download: { ip: 30, key: 180 },
} as const;
type RateLimitResult = {
allowed: boolean;
remaining: number;
limit: number;
resetAt: number;
};
export async function applyRateLimit(
ctx: ActionCtx,
request: Request,
kind: keyof typeof RATE_LIMITS,
): Promise<{ ok: true; headers: HeadersInit } | { ok: false; response: Response }> {
const userId = await getOptionalApiTokenUserId(ctx, request);
const ip = getClientIp(request) ?? "unknown";
const ipSource = getClientIpSource(request);
const hasClientIp = ip !== "unknown";
// Authenticated requests are enforced and consumed by user bucket only to
// avoid draining shared IP quota.
if (userId) {
const userResult = await checkRateLimit(ctx, `user:${userId}`, RATE_LIMITS[kind].key);
const headers = rateHeaders(userResult);
if (!userResult.allowed) {
console.info("rate_limit_denied", {
kind,
auth: true,
userAllowed: false,
ipAllowed: null,
ipSource,
hasClientIp,
});
return {
ok: false,
response: new Response("Rate limit exceeded", {
status: 429,
headers: mergeHeaders(
{
"Content-Type": "text/plain; charset=utf-8",
"Cache-Control": "no-store",
},
headers,
corsHeaders(),
),
}),
};
}
return { ok: true, headers };
}
// Anonymous requests remain IP-enforced.
const ipResult = await checkRateLimit(ctx, `ip:${ip}`, RATE_LIMITS[kind].ip);
const headers = rateHeaders(ipResult);
if (!ipResult.allowed) {
console.info("rate_limit_denied", {
kind,
auth: false,
userAllowed: null,
ipAllowed: ipResult.allowed,
ipSource,
hasClientIp,
});
return {
ok: false,
response: new Response("Rate limit exceeded", {
status: 429,
headers: mergeHeaders(
{
"Content-Type": "text/plain; charset=utf-8",
"Cache-Control": "no-store",
},
headers,
corsHeaders(),
),
}),
};
}
return { ok: true, headers };
}
export function getClientIp(request: Request) {
const cfHeader = request.headers.get("cf-connecting-ip");
if (cfHeader) return splitFirstIp(cfHeader);
if (!shouldTrustForwardedIps()) return null;
const forwarded =
request.headers.get("x-forwarded-for") ??
request.headers.get("x-real-ip") ??
request.headers.get("fly-client-ip");
return splitFirstIp(forwarded);
}
function getClientIpSource(request: Request) {
if (request.headers.get("cf-connecting-ip")) return "cf-connecting-ip";
if (!shouldTrustForwardedIps()) return "none";
if (request.headers.get("x-forwarded-for")) return "x-forwarded-for";
if (request.headers.get("x-real-ip")) return "x-real-ip";
if (request.headers.get("fly-client-ip")) return "fly-client-ip";
return "none";
}
async function checkRateLimit(
ctx: ActionCtx,
key: string,
limit: number,
): Promise<RateLimitResult> {
// Step 1: Read-only check to avoid write conflicts on denied requests.
const status = (await ctx.runQuery(internal.rateLimits.getRateLimitStatusInternal, {
key,
limit,
windowMs: RATE_LIMIT_WINDOW_MS,
})) as RateLimitResult;
if (!status.allowed) {
return status;
}
// Step 2: Consume with a mutation only when still allowed.
let result: { allowed: boolean; remaining: number };
try {
result = (await ctx.runMutation(internal.rateLimits.consumeRateLimitInternal, {
key,
limit,
windowMs: RATE_LIMIT_WINDOW_MS,
})) as { allowed: boolean; remaining: number };
} catch (error) {
if (isRateLimitWriteConflict(error)) {
return {
allowed: false,
remaining: 0,
limit: status.limit,
resetAt: status.resetAt,
};
}
throw error;
}
return {
allowed: result.allowed,
remaining: result.remaining,
limit: status.limit,
resetAt: status.resetAt,
};
}
function rateHeaders(result: RateLimitResult): HeadersInit {
const nowMs = Date.now();
const resetSeconds = Math.ceil(result.resetAt / 1000);
const resetDelaySeconds = Math.max(1, Math.ceil((result.resetAt - nowMs) / 1000));
return {
"X-RateLimit-Limit": String(result.limit),
"X-RateLimit-Remaining": String(result.remaining),
"X-RateLimit-Reset": String(resetSeconds),
"RateLimit-Limit": String(result.limit),
"RateLimit-Remaining": String(result.remaining),
"RateLimit-Reset": String(resetDelaySeconds),
...(result.allowed ? {} : { "Retry-After": String(resetDelaySeconds) }),
};
}
export function parseBearerToken(request: Request) {
const header = request.headers.get("authorization") ?? request.headers.get("Authorization");
if (!header) return null;
const trimmed = header.trim();
if (!trimmed.toLowerCase().startsWith("bearer ")) return null;
const token = trimmed.slice(7).trim();
return token || null;
}
function splitFirstIp(header: string | null) {
if (!header) return null;
if (header.includes(",")) return header.split(",")[0]?.trim() || null;
const trimmed = header.trim();
return trimmed || null;
}
function shouldTrustForwardedIps() {
const value = String(process.env.TRUST_FORWARDED_IPS ?? "")
.trim()
.toLowerCase();
// Hardening default: CF-only. Forwarded headers are trivial to spoof unless you
// control the trusted proxy layer.
if (!value) return false;
if (value === "1" || value === "true" || value === "yes") return true;
return false;
}
function isRateLimitWriteConflict(error: unknown) {
if (!(error instanceof Error)) return false;
return (
error.message.includes("rateLimits") &&
error.message.includes("changed while this mutation was being run")
);
}
+35
View File
@@ -0,0 +1,35 @@
import { describe, expect, it } from "vitest";
import {
parseBooleanQueryParam,
parseBooleanQueryParamOptional,
resolveBooleanQueryParam,
} from "./httpUtils";
describe("parseBooleanQueryParam", () => {
it("returns true for true-like values", () => {
expect(parseBooleanQueryParam("true")).toBe(true);
expect(parseBooleanQueryParam("1")).toBe(true);
expect(parseBooleanQueryParam(" TRUE ")).toBe(true);
});
it("returns false for missing and false-like values", () => {
expect(parseBooleanQueryParam(null)).toBe(false);
expect(parseBooleanQueryParam("")).toBe(false);
expect(parseBooleanQueryParam("false")).toBe(false);
expect(parseBooleanQueryParam("0")).toBe(false);
expect(parseBooleanQueryParam("yes")).toBe(false);
});
it("supports optional parsing for precedence-sensitive callers", () => {
expect(parseBooleanQueryParamOptional(null)).toBeUndefined();
expect(parseBooleanQueryParamOptional("false")).toBe(false);
expect(parseBooleanQueryParamOptional("1")).toBe(true);
});
it("prefers the primary param over the legacy alias when both are present", () => {
expect(resolveBooleanQueryParam("false", "1")).toBe(false);
expect(resolveBooleanQueryParam("true", "0")).toBe(true);
expect(resolveBooleanQueryParam(null, "1")).toBe(true);
expect(resolveBooleanQueryParam(null, null)).toBeUndefined();
});
});
+16
View File
@@ -0,0 +1,16 @@
export function parseBooleanQueryParam(value: string | null) {
if (!value) return false;
const normalized = value.trim().toLowerCase();
return normalized === "true" || normalized === "1";
}
export function parseBooleanQueryParamOptional(value: string | null) {
if (value == null) return undefined;
return parseBooleanQueryParam(value);
}
export function resolveBooleanQueryParam(primaryValue: string | null, legacyValue: string | null) {
return (
parseBooleanQueryParamOptional(primaryValue) ?? parseBooleanQueryParamOptional(legacyValue)
);
}
+42
View File
@@ -0,0 +1,42 @@
import { describe, expect, it, vi } from "vitest";
/* @vitest-environment node */
import type { Id } from "../_generated/dataModel";
import { takeTopNonSuspiciousTrendingEntries, type LeaderboardEntry } from "./leaderboards";
describe("takeTopNonSuspiciousTrendingEntries", () => {
it("keeps scanning past suspicious entries until it finds enough clean skills", async () => {
const skillId = (value: string) => value as Id<"skills">;
const entries: LeaderboardEntry[] = [
{ skillId: skillId("skills:suspicious-1"), score: 300, installs: 300, downloads: 10 },
{ skillId: skillId("skills:suspicious-2"), score: 200, installs: 200, downloads: 9 },
{ skillId: skillId("skills:clean"), score: 100, installs: 100, downloads: 8 },
];
const ctx = {
db: {
get: vi.fn(async (id: Id<"skills">) => {
if (id === skillId("skills:clean")) {
return {
_id: id,
softDeletedAt: undefined,
moderationFlags: [],
moderationReason: undefined,
};
}
return {
_id: id,
softDeletedAt: undefined,
moderationFlags: ["flagged.suspicious"],
moderationReason: undefined,
};
}),
},
};
const items = await takeTopNonSuspiciousTrendingEntries(ctx as never, entries, 1);
expect(items).toEqual([
{ skillId: skillId("skills:clean"), score: 100, installs: 100, downloads: 8 },
]);
});
});

Some files were not shown because too many files have changed in this diff Show More